Hello all, it’s time for the weekly migration subsystem meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.

Core migration issues

Next video meeting

2021-07-01 - The hope is that most or all of the maintainers will attend. We will try to focus on longer-term goals than in the weekly meeting.

Agenda Items

Add your items here.

0️⃣ Who is here today? What is the most confusing part of the Migrate API?

quietone Vicki, what popped into my head was handle_multiples
benjifisher Benji here. Yes, that is confusing. And the process plugins in migrate_plus that deal with that property are not any better: ssingle_value and multiple_values.
dinarcon :wave: Hello. That some source/process/destination plugins do not document its configuration configuration options in a centralized way. There is an issue ability implementing an interface for some. There are other parts like idmap plugins and some configuration options that might be daunting to grok.
alison Alison here :pikachu_wave: "stubs" are v confusingThe terminology took a while for me to grasp (source/process/destination + extract/transform/load) (P.S. I wasted time early on trying to wrap my head around both trios of words -- I was SO CONFUSED, even though somehow now I look at them and they make total sense to me??)I mention the terminology because understanding key terms is SO crucial for getting help (from Google and/or other people).(EDIT: I'm not sure my answers have to do with the API...)EDIT 2: Ok another answer that night be more appropriate:I get confused when I see D6/D7-specific plugins and non-version-specific plugins that seem to be related, ish? -- I can't even explain exactly what I mean, I hope it makes sense. (edited)
anmolgoyal74 Hi..  catching up late..
gaurav mahlawat Catching up. :wave::skin-tone-2:
jibran Jibran, Sub process plugin, migration look up.
benjifisher We added some documentation recently for migration_lookup: https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins/ch...
benjifisher We should add some simple examples for sub_process. (It does have a _, right?) It can get complicated, but we should start with the easy stuff.

1️⃣ What should we talk about today? Suggest topics here and I will add threads. I will also check for comments on the issue for today's meeting.

quietone What is the next step for this, #2976098: MigrateExecutable should add details for the migration & destination property to exceptions that cause a row failure
dinarcon I have not been following the progress, but it would be good to talk about Drush 10.4+ and Migrate Tools working together.

2️⃣ Action items

quietone Ask in #documentation about documenting source plugin configuration. @benjifisher
quietone Consider converting  @see in plugin configuration to @link  @quietone
quietone Report back how the source_id hash works. @quiteone

3️⃣ Statistics

benjifisher Fixed since last week's meeting: 2 (not counting the issue for the meeting).
benjifisher RTBC: 10, 2 of which are Major.
benjifisher NR: 29, including 3 Major and 2 that has not been updated in more than three months.
benjifisher Nice graphs: https://docs.google.com/spreadsheets/d/1o0Rjlc1vnnLP5bM5P-SMMyGzqn7258hi...
quietone Nice to see the effort in doing reviews is paying off.
quietone On the other hand, I have worked on the patches for many of those and someone else needs to review.

4️⃣ How can I help? Comment in this thread if you are looking for ways to help. Give us some idea of what you would like to do: documentation, code review, testing, project management, ...

6️⃣ Is it me or is the testbot flaky?

benjifisher OK, I know that HEAD was broken for a bit on May 3, so tests were failing all over the place. I am not asking about that.
benjifisher I see issue labeled NR with merge requests, and the testbot is not running. Is something broken, or is there a step I am forgetting?
quietone I haven't noticed. Link?
benjifisher Now that 9.2.0-alpha1 is released, all the issues that were targeting 9.2.x now target 9.3.x. (There was a glitch there, too.) But the testbot still runs those patches against 9.2.x.
benjifisher #3211895: ThemeInstaller::[un]install() fails if files/css directory does not exist
quietone I had a similar problem with a patch that was RTBC. I did a retest and it still went to 9.2, I did another retest and then it was on 9.3. https://drupal.slack.com/archives/C1BMUQ9U6/p1620268924052900
benjifisher Before the meeting, I looked at all the RTBC migration issues. They have all been updated to 9.3.x, and they have all been tested in the last two days. I think all of them were tested against 9.2.x.
benjifisher Actually, one of them had not been tested. I had to trigger a test for #2974128: Missing parameter in process plugin DefaultValue example .
benjifisher And see the thread right before this meeting: @Nick Dickinson Wilde says that the spell checker complained about a line that was not touched by the patch.
Nick Dickinson Wilde yeah - touched file but not a touched line

7️⃣ MigrateExecutable should add details for the migration & destination property to exceptions that cause a row failure

benjifisher #2976098: MigrateExecutable should add details for the migration & destination property to exceptions that cause a row failure
quietone I've been working on the patch, I am not sure of the next step so that Migrate Tools is not broken. (edited)
benjifisher I guess we need to review the issue for migrate_tools and get it committed.
benjifisher What about the suggestion from @wimleers (he/him) in #42 that we save the extra data in separate columns?
benjifisher It'd also open the door for a single migrate_messages table, to allow searching all migration messages with a single query, rather than dozens (or even hundreds) of migration_message_* tables.That would be a major change. I was thinking about similar questions, for other reasons. How does the sourceid_hash work? If I have a File migration and a Media migration with the same source, will the hashes be the same? If so, then we will need to use the hash and the plugin ID as a compound database key.

8️⃣ Drush 10.4+ and Migrate Tools

benjifisher @marvil07, any updates since last week? Is there anything blocking you?
benjifisher https://gitlab.com/drupalspoons/migrate_tools/-/issues/118
benjifisher Marco's latest comment is from April 23.

9️⃣ How should plugins document their configuration?

benjifisher @dinarcon, I did not quite follow what you wrote in 1️⃣. Maybe there was one typo too many.
quietone Which plugins?
benjifisher One of my favorite examples is the batch_size option for the SqlBase class. That is an abstract class, so I cannot call it a source plugin.batch_size is inherited by all descendant classes, but it is not easily discoverable.
benjifisher API docs for SqlBase: https://api.drupal.org/api/drupal/core%21modules%21migrate%21src%21Plugi...
quietone The docblock for mostly all source plugins have an @see to SqlBase, It was recently added in #3189481: [Meta] Add source plugin documentation to the codebase
benjifisher API docs for the d7_user source plugin do not mention any configuration: https://api.drupal.org/api/drupal/core%21modules%21user%21src%21Plugin%2...
dinarcon The issue I referred to is the one quietone shared above. The idea is that it would be easy to know what configuration options are available for all migration plugins.
dinarcon I have seen people asking what options are available for a source plugin for example. And I have found some that required digging to understand how to use them.
quietone That is odd. There are 2 @see in the file.
benjifisher So this point of confusion is well on the way to being fixed?
dinarcon Right, some configuration options come from base classes and others depend on the plugin class being used. I should have looked for concrete examples.
dinarcon For context, I created https://understanddrupal.com/articles/drupal-migrations-reference-list-c... and https://understanddrupal.com/articles/drupal-migrations-reference-list-c... pretty much manually. It would be great to have a way to programmatically generate similar reference documentation, ideally hosted on Drupal .org (edited)
quietone Ignore previous comment, somehow I was looking at 9.1.x.The link above does have the two @see statements, for User.php
dinarcon I think new configuration options have been added since I wrote those articles. So, something that can be generated automatically as configuration options are added would be nice. (edited)
benjifisher Something like a {@inheritconfg} PHPDoc tag? Of course, we have to allow inheritance and adding nee keys.
quietone Oh, nice idea. That would encourage people to look at the parent classes.
benjifisher I was thinking that we could extend PHPDoc so that it would have complete documentation in the API docs.
benjifisher I will post a question in the #documentation channel.
quietone Oh course, there is PHP8 annotations ...

1️⃣0️⃣ Terminology: ETL vs. source/process/destination

benjifisher @alison, I think this counts as an answer to my question.
benjifisher I think the history is that Drupal developed its own terminology (source, process, destination) before anyone pointed out that there are more standard terms for these.
alison I had a moment just now where I thought we remotely jinxed or something, I didn't catch on that you were adding topics for each point of confusion people listed haha
benjifisher What can we do to relieve the confusion?Replace source, process, destination with extract, transform, load.Use both sets of terms together in all of our documentation.The overview page uses the ETL terms: https://www.drupal.org/docs/drupal-apis/migrate-api/migrate-api-overviewBut I do not think we use them anywhere else.
alison That history makes sense to me.And, it took me a while to understand the terminology even without there being "duplicate" terms -- I can't explain why it was confusing, which is a bummer, I wish I could -- it's one of those things that totally makes sense to me now, and I don't remember what I didn't understand :confounded:
alison (so to be clear, I was confused about the terminology anyway, it wasn't just the overlapping words)
alison I think the use of both sets of terms on the overview is useful. I wouldn't want to double-up everywhere. I'll think about it, tho.
alison But also, I think it's just, tricky stuff, when it's new to you/ til you get it. I'm glad I was able to be confused and figure it out before actually doing migration work. If I were helping someone get started, I would encourage them to spend time wrapping their head around the concepts beforehand or early on -- not everyone learns like that, of course, and I think I imagine myself as someone who learns by doing, but I'm glad I got a handle on the words first.

1️⃣1️⃣ Wrap up

benjifisher Thanks for participating! I will update 2️⃣. Please continue to add comments in the threads. In 1-7 days, we will post a transcript for today's meeting.
quietone @benjifisher thanks for facilitating!

Participants:

quietone, benjifisher, dinarcon, alison, anmolgoyal74, gaurav mahlawat, jibran, Nick Dickinson Wilde

Comments

quietone created an issue. See original summary.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

benjifisher’s picture

benjifisher
he/him or they/them
Primary language English
Location Boston area
commented

We might discuss whether to use the label "Upgrade messages" or "Migration messages" and whether it belongs in the Migrate module or Migrate Drupal UI: #3063856: Add ability to view migrate_message table data.

quietone credited Gauravmahlawat.

quietone credited NickDickinsonWilde.

quietone credited alisonjo315.

quietone credited anmolgoyal74.

quietone credited dinarcon.

quietone credited jibran.

quietone’s picture

quietone commented
Issue summary: View changes
quietone’s picture

quietone commented
Issue summary: View changes
quietone’s picture

quietone commented
Issue summary: View changes
Status: Active » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.