Come together with the global Drupal community in Rotterdam, 28 Sept – 1 Oct 2026. Sessions, contribution, connection, and Early Bird savings until 8 June.Hello all, it’s time for the weekly migration subsystem meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.
Core migration issues
Next video meeting 2022-07-07 2100Z (tentative)
The hope is that most or all of the maintainers will attend. We will try to focus on longer-term goals than in the weekly meeting.
0️⃣ Who is here today? Did you brush your teeth last night?
| mikelutz (he/him) | Hi all. I definitely brushed my teeth yesterday.. |
| benjifisher | :wave: Me, too. I also flossed. |
| danflanagan8 | :tooth:<- brushed ’em. Haven’t been to the dentist since before covid though. |
| efpapado | Hello, I wash brush my teeth twice every day :smile: (edited) |
| Matroskeen | Hello :wave: Of course I did!I also finished my dentists appointments just before the :ru: invasion. |
| quietone | Hi. I sure did. |
| benjifisher | FYI, I am working on a presentation about Drupal security. One point I want to make is that the important practices are things you already know (but may or may not practice).I thought of tooth brushing as an analogy. Then it occurred to me that this question is a good proxy for "are you taking care of yourself or are you totally stressed out?" (edited) |
1️⃣ What should we talk about today? Suggest topics here and I will add threads. I will also check for comments on the issue for today's meeting.
| danflanagan8 | I’ve been using (and enjoying) @benjifisher’s snippet plugin from this issue. Anyone have thoughts on the future of this issue? #3123534: Process plugin: snippet to re-use YAML config |
| efpapado | Hi! I have uploaded 2 small modules that offer one process plugin each, and today I thought that it could be a good idea to have a page in documentation that would list all the contrib modules of the migrate ecosystem. I created one: https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins/li... (edited) |
2️⃣ Action items. To be added later.
| benjifisher | Review and test the latest patch on #2953111: Only migrate role permissions that exist on the destination. @benjifisher.We did not discuss this issue at today's meeting, but it is definitely active. (edited) |
| benjifisher | Review the latest version of #3262395: $migration_dependencies has inconsistent structure. This is another issue we did not discuss. @quietone, @heddn, can you take a look? |
| benjifisher | Decide whether to convert https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins/li... to a sub-guide. @benjifisher |
| quietone | from mikelutz in 8️⃣ Add a regular item to the meetings to look at wiki documentation, maybe once or twice per year. |
3️⃣ Statistics
| benjifisher | Fixed since last week's meeting: 0. |
| benjifisher | RTBC: 5, all Normal priority. |
| benjifisher | NR: 37, including 2 Critical, 3 Major, and 9 that have not been updated in more than four months. |
| benjifisher | Google sheet for recording stats: https://docs.google.com/spreadsheets/d/1o0Rjlc1vnnLP5bM5P-SMMyGzqn7258hi... |
4️⃣ Comment in this thread if you are looking for ways to help. Give us some idea of what you would like to do: documentation, code review, testing, project management, ...
5️⃣ Previous minutes.
| benjifisher |
|
| benjifisher | Is there anything to follow up? Anything need to be changed in those minutes? |
6️⃣ Announcements
| mikelutz (he/him) | No meeting next week due to DrupalCon Portland. Come join us at the migration contribution table! |
7️⃣ Create a "pipeline" process plugin to re-use YAML config
| benjifisher | #3123534: Process plugin: snippet to re-use YAML config |
| danflanagan8 | I’ve been using @benjifisher’s snippet plugin with success. |
| danflanagan8 | The issue was just tagged as being related to #3250040: Shared configuration for migrations |
| danflanagan8 | Are they just “related” or is the shared configuration a complete solution to the problem of a shared pipeline? |
| mikelutz (he/him) | While I appreciate the need for this, I’m still not a huge fan of this implementation. I’ve been trying to figure out why it doesn’t sit well with me, and I’m convinced it comes down to further divergence of the yml between core migrations and those using migrate_plus/tools. I’m not a huge fan of the second issue either, but I get it. I don’t think the second issue is the same thing. The second issue looks to be a way to do shared configuration without using migrate groups, which the new core drush commands don’t directly support. |
| mikelutz (he/him) | I really think the long term solution to both is to bring migration groups into core (admittedly a lot of work) |
| mikelutz (he/him) | And then pipeline reuse could be implemented in core via the migration groups. |
| mikelutz (he/him) | So I guess the question is do we settle on going the ugly way of doing this in contrib or do we have someone that would put the extra work into doing this in a better way in core. |
| danflanagan8 |
I’m convinced it comes down to further divergence of the yml between core migrations and those using migrate_plus/tools. I don’t see the snippet process plugin as causing any fundamental divergence. |
| benjifisher | I think the snippet or pipeline idea lets you do things that you cannot do with shared configuration.
Shared config lets you use the same pipeline on the same field in different migrations. With snippet, you can use the same pipeline on different fields. Or you can insert the same 4 steps into any pipeline. |
| benjifisher | I actually have not used it. @danflanagan8 has more practical experience with it than I do. :wink: |
| danflanagan8 | Try it! You’ll like it! |
| mikelutz (he/him) | @danflanagan8 Well, there’s already divergence with both the concept of migration groups, and the lack of support in migration configs for derived migration. This adds another yml configuration supported by migrate plus and not core. I’m not saying it’s the end of the world, All I’m saying is that given unlimited development resources, I would prefer to align them by bringing groups into core than diverge them by adding yet another migrate_plus only configuration. But we don’t have unlimited development resources, and there’s every chance that an effort to bring groups into core and build reusable pipelines on that would stall. |
| danflanagan8 |
This adds another yml configuration supported by migrate plus and not core. |
| danflanagan8 | Ah, I was looking at it as yml supported by the snippet process plugin, not directly supported by migrate_plus. Of course indirectly it’s supported by migrate_plus if that’s where snippet lives. (edited) |
| mikelutz (he/him) | In my (admittedly idealized) world, tools would provide UI and extended drush support and plus would be primarily support for migrate as config and a suite of specialized process plugins that don’t make sense in core, but ideally any underlying structural changes to the yml files or number or types of yml files would be in core whether they live as plugins or config. |
| mikelutz (he/him) | Again, this is all super idealized stuff, and I know as well as anyone that migrate is not idealized. |
| benjifisher | I kind of think that we should split up migrate_plus: one module for migrations-as-config and one module for process plugins that are widely used but do not make it into core.
But that is not likely to happen. The snippet plugin is squarely in the "more plugins" part. You seem to be talking about it as if it affects how the YAML is interpreted. |
| danflanagan8 | The pipeline plugin, that also has a patch or two in that issue, does indeed create a new config entity type and comes with more overhead. But the snippet plugin doesn’t really do anything “new” with yml. It just lets you put the yml in a new place, more or less. |
| benjifisher | Right. The pipeline and snippet plugins have the same goal: re-use pipelines. But they have very different implementations.
Maybe I caused some confusion at the top of this thread, suggesting that they were interchangeable. |
| danflanagan8 | The issue also has something like 4 distinct proposed resolutions. That can lead to confusion obviously! |
| benjifisher | My comment #12 on the issue, from 2021-06-02, tried to summarize the different approaches. |
| danflanagan8 | And I guess I brought the issue up here because at one time it seemed like there was a lot of interest and I’m finding one of the proposed solutions very useful. |
8️⃣ List of contrib modules that offer process plugins
| benjifisher | New documentation page: https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins/li... |
| benjifisher | I love process plugins. When I come up with a good, reusable one, I usually contribute it to the Migrate Plus module. Did you try that before creating your own modules? |
| efpapado | Hmm no, maybe I should have done that. |
| benjifisher | The entity_lookup plugin in migrate_plus is already based on entity queries. How is entity_field_lookup different? |
| efpapado | entity_field_lookup can query against any field, not just the entity ID |
| damienmckenna | Are you only considering modules that have generic plugins, or would Metatag's custom plugins for Nodewords and Metatag-D7 count? |
| efpapado | If you're asking me, I was considering all plugins of type MigrateProcessPlugin |
| efpapado | So if the "core" metatag module has also a process plugin, it could also be listed. In these cases, maybe a separate section for modules that are not migrate-specific |
| benjifisher | Let's add to the list. Once we have more than a handful of items, we can think about organization.
I guess there is no need to document them unless they are likely to be used in custom migrations. If not, then we can just add a list of "other modules" with an un-annotated list of links. |
| Matroskeen | entity_lookup in migrate_plus is limited and complex in the same time.It tries to guess many things, that could be just listed in the yaml. In the same time, it doesn't allow you to put multiple conditions and refine the lookup result.
I think we can consider adding a new variation of it, which is a straigtforward wrapper for entity query. (edited) |
| benjifisher | I see that the "Migrate process plugins" guide already has child pages for 3 contributed process plugins. Do we want to move those to the new page? Should we make the new page a sub-guide? |
| benjifisher | Good point: all that guessing makes entity_lookup complex. On the other hand, you have the option of supplying all the parameters in the YAML, and then it bypasses the guessing. |
| benjifisher | @efpapado, I am pretty sure that you can use entity_lookup to query on a field, not just the entity ID. As I said, it uses entity queries. |
| mikelutz (he/him) | While I’m all for documenting this, it raises the question of maintenance of this page, as migration plugins are often written by someone for a specific purpose and then left unmaintained, so there would need to be a way to regularly review the list and purge old and add new. About the only thing I find more frustrating than no documentation is outdated/incorrect documentation. |
| efpapado | @benjifisher it's possible that when we wrote the entity_field_lookup, the entity_lookup was on an earlier version that could not use any field. It's been 2 years but I remember we concluded that it did not fit us. Also we thought that we should support also additional conditions for the query through the yml configuration. It seems that now the biggest part of the functionality is indeed covered by entity_lookup. |
| damienmckenna | How about using the new search system that was added to gitlab? |
| damienmckenna | https://git.drupalcode.org/search?group_id=2&repository_ref=9.4.x&scope=... |
| danflanagan8 |
Should we make the new page a sub-guide? I like this idea. That would be a good place for the Migrate Conditions guide, which is kind of in limbo currently. |
| benjifisher | One of my first, and biggest, migrations was for Pega Systems, about 5 years ago, with @marvil07. We wrote the first DOM process plugins for that. And Marco pointed out to me that entity_lookup was based on entity queries, which made it very flexible.It took me a while to get the point. |
| benjifisher | :+1: for adding a link to GitLab search to the page.It is the nature of community-maintained documentation that it is sometimes out of date. Maybe always. :wink: |
| benjifisher | I am a maintainer of the guide, so I can make the changes. I added an item to 2️⃣ for this. |
| benjifisher | I just created a sub-guide: https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins/pr...
I moved the page at the top of this thread to the new sub-guide. (That changes the page's URL, but the link here should redirect to the new one.) If there are other process plugins that deserve more documentation, then add a page to the new guide. I can add your new page to the navigation menu. |
Comments
Comment #2
quietone commentedComment #7
quietone commentedComment #8
quietone commentedComment #9
quietone commentedThere were no request for changes at Migrate Meeting 2022-05-19 1400Z.
Comment #11
quietone commentedAccidentally made several migrate minutes issue on the wrong version. Moving to 9.4.x