Hello all, it’s time for the weekly migration subsystem meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.

Core migration issues:

Next video meeting

2021-04-08

Agenda Items

Add your items here.

0️⃣ Who is here today? Are you planning anything for DrupalCon Global next month? (Are you at least registered?)

dinarcon :wave: Mauricio here. No DrupalCon plans yet.
benjifisher Benji, one of the maintainers of the migration subsystem. I am registered, but have not made any plans.
Matroskeen :wave: Ivan from Ukraine. No plans yet
anmolgoyal74 Hi:wave:, Anmol from India, No plan as of now.
gaurav mahlawat Hi, Gauravmahlawat from india. No plans yet.
quietone Vicki, No plans

1️⃣ What should we talk about today? Suggest topics here and I will add threads. I will also check for comments on the issue for today's meeting.

Matroskeen #3199727: Create plugin for recursive YAML discovery

2️⃣ Action items. To be added later.

3️⃣ Statistics

benjifisher Fixed since last week's meeting: 3 (not counting the issue for the meeting).
benjifisher RTBC: 9, 1 of which is Minor, the rest Normal. All have been updated in the last 3 weeks.
benjifisher NR: 39, including 1 Critical, 5 Major, and 1 that has not been updated in more than two months.
benjifisher It was going to be 4 Fixed, but I set one of them back to "Patch to be ported" and triggered a test for 8.9.x.

4️⃣ Is it OK to raise funds for documentation for the code written by others?

benjifisher Short answer: yes. @dinarcon
dinarcon For context, I brought this up because it would be good to update the 31 days of migration series. One of the major motivations is Drush including migration runners since 10.4 which means 2/3 of series includes instructions that is no longer correct. But many things have changed since August 2019 in the Migrate ecosystem. There are many improvements and fixes to do to the examples. I want to translate the series to Spanish and French. And produce a video series at least in English. So, lots of work ahead.
benjifisher Just be clear in your fundraising about how much will be public and how much you will use in paid training classes.
benjifisher There are companies that base their business models on training for Drupal (and presumably other companies for other OS software). If people expect a benefit, and are willing to pay for it, I do not see a problem.The benefit could be direct use of public documentation or advertising as a sponsor of it. :shrug:
benjifisher You are not competing for funds with any developers, as far as I know.
dinarcon I am planning to make all content related to the 31 days of migration series available for free on the website. That being said, I do sell eBook copies of the series. If the videos were created, they would also be free to watch online and I would sell the downloads for offline viewing. To date, I might have sold about 15 copies of the book and gifted it at many camps. These sales are not a significant of source income. I think they are mostly from people who want to support my writing efforts. (edited)

5️⃣ Issues related to highwater marks.

benjifisher @wimleers (he/him) has been wworking on #2859314: Highwater condition with unjoined maps skips unprocessed and NEEDS_UPDATE rows. @quietone suggests reviewing other related issues.

6️⃣ ContentEntity migration source has option to include revision ID as a source key

benjifisher The issue was fixed in 9.1.x and 9.2.x. The CR needs to be updated asnd published: https://www.drupal.org/node/3191344
benjifisher This is the CR for the issue I mentioned in 3️⃣. It may also be fixed for 8.9.x.

7️⃣ Add back rollbacks through the UI

benjifisher #2687849: Add back rollbacks on migrate_drupal_ui
benjifisher This issue was RTBC. The latest patch failed automated tests. @quietone
quietone I would prefer to not do this. It would be better for people to do a fresh minimal install. For one, rollback does not rollback config and cause confusion.

8️⃣ Migrate Plus, Migrate Run, and Migrate Tools

benjifisher Copied from the issue for today's meeting:Now that Drush 10.4.0 has been released, we can run migrations from the CLI without any additional modules. I think we have to make some changes to Migrate Plus and/or Migrate Tools.At the same time, I think each of those modules does too much:Migrate Plus: several migrate plugins AND support migrations as config entitiesMigrate Tools: Drush integration AND a UI for migrations defined as config entitiesIf I want the extra plugins but not support for config entities, then I am stuck. If I want the UI for config entities, then I cannot install the latest version of Drush.Proposal to discuss:Migrate Run: mark it as "minimally maintained" or "bug fixes only" or whatever the appropriate status is.Migrate Plus: Create a 6.0 branch where one or both of the feature sets is in a submodule.Migrate Tools: Either create a 6.0 branch without the Drush support or move the UI to another submodule of Migrate Plus.Alternatively, we could keep just the plugins in Migrate Plus and move the support for config entities to Migrate Tools. I think that option would be more disruptive.
benjifisher @heddn
quietone Be aware that I can't use --limit or --idlist with Drush 10. https://github.com/drush-ops/drush/issues/4679
benjifisher That seems like a bug. I assume it will be fixed, sooner or later. Is there any reason to think otherwise?
heddn @benjifisher that wasn't in migrate run, so it didn't make it into drush core
quietone It is in the drush docs, https://www.drush.org/latest/commands/migrate_import/

9️⃣ Site upgrades for good causes.

benjifisher I think an interesting project would be to update a D6 or D7 site to D9 as part of a contribution event.
benjifisher I am currently doing some prep work for that. See the discussion in the #boston channel. I need a candidate site. I prefer something local to Boston, but at this point I am willing to consider anything.
benjifisher I do not expect to do a site upgrade in a day the first time around. But once I have some experience with the process, and documentation and other structure, I think it is something that could be done with a group of volunteers at a contribution event.
benjifisher The goal is to help the good cause and give participants some experience in migration. Also experience in site building, theming, whatever.
dinarcon This is a great initiative! Thanks for leading the efforts!
quietone Awesome idea!
benjifisher Thanks. If anyone has a candidate site, please DM me.

1️⃣0️⃣ How can I help? Comment in this thread if you are looking for ways to help. Give us some idea of what you would like to do: documentation, code review, testing, project management, ...

benjifisher @anmolgoyal74: Last week you said you would try to get set up to test migration issues. Any progress? If you asked any questions in this channel, then I missed them.
benjifisher If you are following my suggestion to use Lando, then I have been thinking of other ways to manage things, related to 9️⃣.
anmolgoyal74 I have started.. But didn’t able to progress much. I am not using lando.
ijf8090 I'd be interested in helping out w. documentation and testing. I have a positive gift for mis-interpretation and breaking stuff.  I want to use this super-power for good not evil :). If I can make something work - then pretty much anybody will be able to use it.
ijf8090 Right now I am looking to migrate 2,000 static pages from a 10 year old web site.

1️⃣1️⃣ Extended callback process plugin to call functions with multiple parameters

benjifisher #2882276: Extended callback process plugin to call functions with multiple parameters
benjifisher This issue proposes a new process plugin for the migrate_plus module. But I recently had an idea of a really simple way to extend the core callback plugin instead. See my comments on the issue several hours ago.
benjifisher The usual criterion for adding  something to the core migrate module is that it has to support Drupal upgrades. Are we willing to make an exception here? @heddn, @quietone, @mikelutz (he/him)
benjifisher We will need examples in the doc block and some more tests, but the only code change is to replace return call_user_func($this->configuration['callable'], $value);with either if (!empty($this->configuration['splat'])) { return call_user_func($this->configuration['callable'], ...$value); } return call_user_func($this->configuration['callable'], $value);or $args = empty($this->configuration['splat']) ? [$value] : $value; return call_user_func($this->configuration['callable'], ...$args);
dinarcon Some time ago I created a custom process plugin for this with this code in transform:$params = $this->configuration['params'] ?? []; return call_user_func($this->configuration['callable'], $value, ...$params);(edited)
dinarcon Well, it also uses an extra configuration option so it is not really better than the other two proposals.
benjifisher I like the simplicity of my idea. Or maybe I just like it because it is my idea. It is good to get opinions from others.
mikelutz (he/him) I think we should do this in core.
mikelutz (he/him) If you are looking for an excuse, If we have this in core, we could probably deprecate the substr plugin.
mikelutz (he/him) Maybe even explode, although we would need a way to declare the output as multiple.
mikelutz (he/him) That might be a separate ticket, but we probably should let the migration declare whether the output of callback should be considered multiple or not, for things like explode.
benjifisher The migrate_plus module has two process plugins, multiple_values and single_value (something like that) to do just that. I would have made it one plugin.
benjifisher The migrate_plus module also has a plugin for formatting dates, which could also be deprecated.
mikelutz (he/him) I'm aware of that, but we can't use those plugins to deprecate explode in core using callback...
benjifisher ... unless we add something similar to core, in another issue.
quietone I agree this would be useful in core as well as contrib.

1️⃣2️⃣ Create plugin for recursive YAML discovery

benjifisher #3199727: Create plugin for recursive YAML discovery
benjifisher I commented here the last time we brought this up. Let me add a comment to the issue.
Matroskeen Thanks! So the main question is:Should it be part of the Drupal core or contrib module (existing or a new one)?My intention was to add it to core, but I didn’t consider that there is no real use-case for it in core APIs. (edited)
Matroskeen Currently, I don’t have a strong opinion about this and will accept to the consensus of core maintainers. (edited)
benjifisher My comment:There is an issue somewhere to make it easier to reuse process pipelines. I cannot find the issue right now: maybe it is for the Migrate Plus module, not core.One option is to create the subdirectory `migrations/process/` in your module and have YAML files in there. Then write a plugin manager that creates process plugins from those YAML files. These plugins would work by creating pipelines from existing plugins.This is just a half-baked idea: I have not done any work to implement it. But I would like to keep this option open. If this issue is implemented in a way that migrations/process/foo.yml is used to define a migration plugin instead of a process plugin, then it makes my half-baked idea harder to implement. (edited)
benjifisher It can be done in contrib. So can my half-baked idea. There is not a strong argument for having it in core.Or if there is an argument for having it in core, which comment(s) on the issue should I read?For now, my vote is not to do it in core.
Matroskeen In my initial prototype, we can exclude certain directories from discovery by supplying a regular expression. See original issue: [#3195568]It was already already done to exclude migrations/state directory and can be extended to exclude migrations/process as well. (edited)
Matroskeen I have just one argument: if it would be outside of the core, then less people would know about it and even less will use it :man-shrugging:
benjifisher So #3199727 would create a configurable discovery plugin but not use it aywhere? And then #3195568 (whether in core or contrib) would use that to search migrations/** for migrations?
Matroskeen Yep. It was done in the same way for current discovery: #2671034: Create plugin per file YAML discovery (edited)
benjifisher OK, so it does not conflict with my half-baked idea. And they could even work well together if my idea catches on.
benjifisher If you start a sandbox project with your recursive discovery plugin, then I will try to make some time to add my "reusable process plugin" feature to it. If we gain some traction and people start to use it, then maybe we can bring it into core or promote it to a full module.
Matroskeen Ok, sounds good!
benjifisher We might also add it to Migrate Plus. See 8️⃣.
Matroskeen I would prefer add it to the existing module, I think it would make more sense.Has any decision been made about moving the development of migrate_plus back to drupal.org?

1️⃣3️⃣ Wrap up (edited) 

benjifisher Thanks for participating! I will update 2️⃣. Please continue to add comments in the threads. In 1-7 days, we will post a transcript for today's meeting.

Participants:

dinarcon, benjifisher, Matroskeen, anmolgoyal74, gaurav mahlawat, quietone, heddn, ijf8090, mikelutz (he/him)

Comments

quietone created an issue. See original summary.

quietone’s picture

quietone commented
Issue summary: View changes
benjifisher’s picture

benjifisher
he/him or they/them
Primary language English
Location Boston area
commented

Some issues to discuss:

Review open issues related to highwater marks, including #2859314: Highwater condition with unjoined maps skips unprocessed and NEEDS_UPDATE rows.

Now that Drush 10.4.0 has been released, we can run migrations from the CLI without any additional modules. I think we have to make some changes to Migrate Plus and/or Migrate Tools.

At the same time, I think each of those modules does too much:

  • Migrate Plus: several migrate plugins AND support migrations as config entities
  • Migrate Tools: Drush integration AND a UI for migrations defined as config entities

If I want the extra plugins but not support for config entities, then I am stuck. If I want the UI for config entities, then I cannot install the latest version of Drush.

Proposal to discuss:

  • Migrate Run: mark it as "minimally maintained" or "bug fixes only" or whatever the appropriate status is.
  • Migrate Plus: Create a 6.0 branch where one or both of the feature sets is in a submodule.
  • Migrate Tools: Either create a 6.0 branch without the Drush support or move the UI to another submodule of Migrate Plus.

Alternatively, we could keep just the plugins in Migrate Plus and move the support for config entities to Migrate Tools. I think that option would be more disruptive.

quietone’s picture

quietone commented

Check this CR and publish, https://www.drupal.org/node/3191344

quietone credited Gauravmahlawat.

quietone credited Matroskeen.

quietone credited anmolgoyal74.

quietone credited dinarcon.

quietone credited heddn.

quietone credited mikelutz.

quietone’s picture

quietone commented
Issue summary: View changes

quietone credited ijf8090.

quietone’s picture

quietone commented
Status: Active » Needs review

Add credit

quietone’s picture

quietone commented
Status: Needs review » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.