Core migration issues

Next video meeting 2025-04-24

Hello all, it’s time for the biweekly migration subsystem meeting. The meeting will take place in slack in various threads. This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every second Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to. #3456078: [meeting] Migrate Meeting 2024-07-18 2100Z)See the parent issue for an idea of the typical agenda.
➤Transcript will be exported and posted to the agenda issue. For anonymous comments, start with a 👤 emoji. To take a comment or thread off the record, start with a 🚫 emoji.

0️⃣ Who is here today?

1️⃣ What should we talk about today? Suggest topics here and I will add threads. I will also check for comments on the issue for today's meeting.

benjifisher Change the filter in the details page for migration messages to a condition object
benjifisher migrate mapping & messages table names are truncated, can lead to incorrect mapping lookups
benjifisher Move the d8_config source plugin to the migrate module
benjifisher Migrations fail due to missing dependency when dependency has skipped rows by the source plugin
benjifisher Should migration plugins declare their configuration?
smustgrave Could use review[#3171980]
smustgrave Don’t think I’m the best to do so

2️⃣ Action items. To be added later.

3️⃣ Statistics

Migrate Initiative Meeting TBA
Migrate Initiative Meeting Google sheet for recording stats: https://docs.google.com/spreadsheets/d/1o0Rjlc1vnnLP5bM5P-SMMyGzqn7258hi...
benjifisher Fixed in the last two weeks: 1
benjifisher RTBC: 3, all Normal priority.
benjifisher NR: 7, including 2 Major and 1 that has not been updated in more than a month.

4️⃣ Comment in this thread if you are looking for ways to contribute. Give us some idea of what you would like to do: documentation, code review, testing, project management, ...

5️⃣ Previous minutes.

Migrate Initiative Meeting https://www.drupal.org/project/drupal/issues/3506466 (edited)
benjifisher We are behind on tending to the meeting issues.#3497391: [meeting] Migrate Meeting 2025-01-30 2100Z (2025-01-30) is NR, thanks to @ddavis. The two meetings from February still need transcripts.

6️⃣ Announcements

benjifisher There is a BoF session scheduled to discuss the new Modeler API module at DrupalCon Atlanta (Monday, March 24): https://events.drupal.org/atlanta2025/bofs/all. This new module will provide the basis for editing migrations in the admin UI, with the same interface as the ECA module.
benjifisher Right after that, in the same room, there is another BoF: "Is there anything the Migrate API cannot do?"This will be a live version of the ever-helpful #migration Slack channel.

7️⃣ Change the filter in the details page for migration messages to a condition object

Migrate Initiative Meeting https://www.drupal.org/project/drupal/issues/3495936 (edited)
Migrate Initiative Meeting (Original Request) https://drupal.slack.com/archives/C226VLXBP/p1741875272152899?thread_ts=... (edited)
benjifisher I have already reviewed this issue, and I need to look at the recent updates.

8️⃣ migrate mapping & messages table names are truncated, can lead to incorrect mapping lookups

Migrate Initiative Meeting https://www.drupal.org/project/drupal/issues/2845340 (edited)
Migrate Initiative Meeting (Original Request) https://drupal.slack.com/archives/C226VLXBP/p1741875457243239?thread_ts=... (edited)
benjifisher I am very nervous about changing names of existing database tables, but we have to do something because it is a pretty serious bug.
benjifisher @mikelutz (he/him), do you have capacity to review this issue?
mikelutz (he/him) I have more capacity than I would like at the moment. :joy:. Yeah, I can move this along.

9️⃣ Move the d8_config source plugin to the migrate module

Migrate Initiative Meeting https://www.drupal.org/project/drupal/issues/3506605 (edited)
Migrate Initiative Meeting (Original Request) https://drupal.slack.com/archives/C226VLXBP/p1741875653779289?thread_ts=... (edited)
benjifisher I worked on this issue, so we need someone else to review it.
benjifisher This is one of the things we want to get done before we remove the migrate_drupaal module in D12.
mikelutz (he/him) I looked at it the other day. seems fine except we don’t need the source module annotation in the new plugin.
mikelutz (he/him) Think I typed that up and never hit enter.
benjifisher It looks to me as if it uses attributes, not annotations. Even the first commit in the MR uses attributes.

🔟 Migrations fail due to missing dependency when dependency has skipped rows by the source plugin

Migrate Initiative Meeting https://www.drupal.org/project/drupal/issues/2797505 (edited)
Migrate Initiative Meeting (Original Request) https://drupal.slack.com/archives/C226VLXBP/p1741875781767059?thread_ts=... (edited)
benjifisher I have not looked at this issue, but from the title it seems like a big DX improvement.

1️⃣1️⃣ Should migration plugins declare their configuration?

Migrate Initiative Meeting No issue yet. This is the first time I have brought this up.
Migrate Initiative Meeting (Original Request) https://drupal.slack.com/archives/C226VLXBP/p1741876203861139?thread_ts=... (edited)
benjifisher In the last meeting, we talked about editing migrations in the admin UI. If we want to do that, then we should provide a way to validate the choices made in the UI. First step: what are the configuration keys. Next step: how do you validate the config values?
benjifisher The source plugins already have the fields() method, which could be used in this way.
mikelutz (he/him) I’ve long wanted to have process plugins declare expected config and expected input and output
benjifisher But all plugins (source, process, destination, field) have config. All we currently do is document it, including "see the parent class for additional configuration options".
benjifisher Maybe this is a question for the larger plugin system, not just migration.
benjifisher

... expected input and output

A long time ago, @marvil07 suggested using the TypedData system to declare that. I am not sure how that would work.
marvil07 It sounds handy to have validation on configuration for migration plugins.

AFAIR I tend to add validation on the constructor of the plugin, which is not ideal.

Having it as part of normal validation would be great.
marvil07 On the expected input/output, that may be a separate issue, and is specially useful for migrate process plugins.

Right now, AFAIK, the assumption is the same as unix programs, text is the interface passed along the pipes.

That is quite nice, but there may be useful cases to pass other things, like the dom plugin set we added with @benjifisher a while ago.

Not sure about the actual solution for that.

IIRC I did ask Berdir about using typed data there, but I totally forgot now what was the outpus of that conversation :sweat_smile: , my memory suggest that it was an idea that was discarded.
dinarcon Somewhat related to a previous conversation https://drupal.slack.com/archives/C226VLXBP/p1637271127038900
benjifisher I am thinking about how to implement the validation. If you pass an array of configuration to the plugin's constructor, then could the constructor create a config entity (without saving any config to the database) and validate that, using the config validation system? Would the plugin declare/decide which schema to use, or would the schema declare that it applies to the plugin?
dinarcon I was originally thinking of just listing configuration options, not validating them. In some cases, configuration options are exclusive. Not sure the config validation system can handle that unless we use custom constraints.
marvil07 @benjifisher that's a good point, the how may be not yet clear, but happy to see interest about this. It may be worth moving this conversation to an actual issue. I may get to implement a draft over dcon if someone does not beat me (that is totally OK!)
marvil07 Ah sorry for the channel echo, unintended :see_no_evil:

1️⃣2️⃣ Menu migration should happen before block migration

Migrate Initiative Meeting https://www.drupal.org/project/drupal/issues/3171980
Migrate Initiative Meeting (Original Request) https://drupal.slack.com/archives/C226VLXBP/p1741876445861569?thread_ts=... (edited)
benjifisher All of that will be removed along with the migrate_drupal module. OTOH, it is a very small change: +7 -5, with most of the changes in tests.
benjifisher Back in 2020, @mikelutz (he/him) set it to NW for tests. Are the test changes just fixing tests broken by the change or are they actually testing the intention of the change?  A quick look at the MR is not enough to decide.
benjifisher @smustgrave: There are 7 issues in the NR status for the migration system, and I already started threads for a few of them. Why do you call attention to this one in particular?
smustgrave Just happened to be the one I had open :) certainly can wait till next meeting. But the code it’s changing I definitely dont think I can review it
mikelutz (he/him) Well, at the time this was a bug and the policy was all bug fixes required a test showing the bug.
mikelutz (he/him) So if a warning was being emitted during the migration, I wanted it captured in a test.
mikelutz (he/him) I think I still do, but I can add this one to my list.

1️⃣3️⃣ Wrap up

benjifisher Thanks for the discussion! We will do this again in two weeks. Or maybe we will skip the next meeting since it will be the last day of DrupalCon Atlanta.
benjifisher I have to get moving. I have to catch a train for DrupalCamp NJ, which starts tomorrow.

Comments

benjifisher created an issue. See original summary.

benjifisher credited dinarcon.

benjifisher credited marvil07.

benjifisher credited mikelutz.

benjifisher credited smustgrave.

benjifisher’s picture

benjifisher
he/him or they/them
Primary language English
Location Boston area
commented
Issue summary: View changes
Status: Active » Needs review
smustgrave’s picture

smustgrave commented
Status: Needs review » Fixed

Appears all threads have been captured.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.