Come together with the global Drupal community in Rotterdam, 28 Sept – 1 Oct 2026. Sessions, contribution, connection, and Early Bird savings until 8 June.Hello all, it’s time for the weekly migration initiative meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to here: https://www.drupal.org/project/drupal/issues/3158024. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.
Core migration issues: RTBC, NR, all open.
0️⃣ Who is here today? Did you sleep well last night?
| mikelutz | Michael Lutz - Fair, according to Fitbit, I got a sleep score of 78/100, 6hr, 29 minutes asleep, 59 minutes awake, 12% restless. (not sure if you wanted that much detail) :stuck_out_tongue: (edited) |
| benjifisher | I had insomnia. I also have a Fitbit: 63/100, 6:23 asleep, 1:41 awake |
| heddn | Lucas, I think I slept well. I was asleep so I'm not 100% sure if I tossed and turned the whole time or not. I don't think I did, because my wife would have told me if I had. |
| benjifisher | My wife had insomnia, too. |
| Bisonbleu | Will you forgive a stranger who overheard your conversation for barging in…I was awake at 4h with langcode: fr statements and paragraphs_item_field_data and migrate_map_ tables floating in my head… oh and an unknown girlfriend…Migrations will do that to a newbie I guess :wink: |
| benjifisher | Will you forgive a cultural stereotype? The phrase "unknown girlfriend" seems very French. Do you attribute that to the migrations? :wink: |
| Bisonbleu | Hm… interesting, in what way does it appear very French? Disclaimer: I don’t have a girlfriend at the present time, and the lady in my dreams I have never met before :thinking_face: |
| damienmckenna | Catching up after the fact. |
| mikelutz | I've been doing migrations for a long time, they still give me nightmares.. |
| mikelutz | Just like I finished college 20 years ago and still have nightmares about a surprise final that I haven't studied for. |
| dinarcon | Hi :wave: Catching up with the meeting. |
| quietone | Hi, also catching up. |
1️⃣ What should we talk about today? Suggest topics here and I will add threads.
| quietone | A big thank you to daffie for reviewing and RTBC'ing #3110669: Migrate d7 menu language content settings |
2️⃣ Action items. To be added later.
| benjifisher | #3006750: Remove memory management from MigrateExecutable: @quietone, can you move this issue forward? |
| benjifisher | #3134470: Switch to entity owner in EntityContentBase during validation: @heddn |
| benjifisher | #3047328: Allow logging for non-strings values: @benjifisher |
| benjifisher | #3063856: Add ability to view migrate_message table data: @benjifisher |
3️⃣ Statistics
| benjifisher | There was a push by the core committers to close out RTBC issues. At one point, there were only two RTBC issues in the core migration system.We skipped last week's meeting, so it has been two weeks since I last recorded this. |
| benjifisher | Fixed (in the last two weeks): 12 |
| benjifisher | RTBC: 7, including 1 Major and 1 more than a month old (edited) |
| benjifisher | NR: 42, including 3 Major, 3 that are nearly 2 months old, and 20 more that are more than 1 month old. (edited) |
| benjifisher | I did not count meeting issues in the Fixed count. In all cases, "old" means since last comment. |
| benjifisher | The NR list is long. The total number has been fairly steady, but there has been movement (new issues added, others resolved). |
4️⃣ Move memory management from MigrateExecutable to an event subscriber
| benjifisher | #3006750: Remove memory management from MigrateExecutable |
| benjifisher | I suggested major changes to make the new class more testable. I was outvoted: that can be done in a follow-up. Current status is NW, to address a few points in Comment #62. |
5️⃣ Switch to entity owner in EntityContentBase during validation
| benjifisher | #3134470: Switch to entity owner in EntityContentBase during validation |
| benjifisher | Feedback requested at the last meeting. There was feedback on Slack and a review posted on the issue. Current status is NW. |
| heddn | I need to circle back and post a new patch |
6️⃣ Allow logging for non-strings values
| benjifisher | #3047328: Allow logging for non-strings values |
| benjifisher | This is the oldest of the NR issues. I will review it before the next meeting. |
| benjifisher | Good news: I reviewed it.Bad news: I set it back to NW. |
7️⃣ Why not make messages into entities? (edited)
| benjifisher | It would not be hard, and then they would be available to views. We could use views or entity queries instead of direct database queries when listing them. |
| heddn | Sure, great idea. We could probably even wrap the existing message tables (maybe) or do something. |
| heddn | BC is always a concern |
| benjifisher | Yes, BC is a great motivator to get things right the first time around. ;) |
| benjifisher | If we add a MigrateMessage entity type, and the bundles correspond to migrations, then maybe we can use the same table names. The annotations on the entity class can be used to map the required keys to the existing database fields (columns).If we have to add new columns, then there is a BC issue: someone may have custom queries that join with other tables. If they use a field name from another table and do not specify which source table it comes from ... |
| benjifisher | We would have to rewrite the code that manages the map and message tables. They are created dynamically, and I think they are truncated at some point. Are they ever explicitly deleted? |
| heddn | there is an API for deleting them. truncating is done manually. |
| steinmb | Late, I know. Are there performance regression by turning them into entities? Except that concern from me, it is a good idea. |
| heddn | probably not as all properties would be columns on a single database table. i.e. the fields would be base fields |
8️⃣ Expose full set of debugging data in migrate_message table (filterable/searchable)
| benjifisher | #3063856: Add ability to view migrate_message table data (edited) |
| benjifisher | There has been some progress on this issue, currently NR. I owe a review of the latest patch, worked on by @Fredy and @quietone. |
| benjifisher | One thing worth discussing: the original proposal was to have a single page, and one of the filters would select the migration. The current patch, modeled on Migrate Tools (?) has a summary page with links to separate pages, one for each migration. Those pages allow you to sort, search, and page through results. |
| benjifisher | I think I like the idea of a summary page, showing you which migrations actually have messages (and how many). I certainly do not want to select one migration at a time and reload the page to find out that there are not any messages!With the original proposal, I guess we could disable (or even remove) the options corresponding to the migrations with 0 messages. |
| benjifisher | We need screenshots! |
| benjifisher | I did some manual testing and added screenshots. Back to NW. Thanks to @heddn for adding a link to an issue related to one of the bugs I noticed.@quietone or @Fredy, can you update the patch? |
'9️⃣ Add a migration source plugin for JSON:API
| benjifisher | #3150949: Add a migration source plugin for JSON:API |
| benjifisher | @mglaman mentioned that someone might have started this already with a Commerce application in mind. I have not seen any follow-up. AFAIK no one made any progress here, although it was a suggested issue for DrupalCon contribution. |
| mglaman | I forgot to follow up. Posted message https://drupal.slack.com/archives/C1TLCCF9B/p1595514236160200 |
| benjifisher | Thanks! There are no new comments on the issue since then, so maybe we are not so lucky. |
| benjifisher | On the issue, I already mentioned this 2016 blog post by @KarenS: https://www.lullabot.com/articles/pull-content-from-a-remote-drupal-8-si... |
| benjifisher | Here is part of the configuration from that post:source: plugin: url data_fetcher_plugin: http data_parser_plugin: json urls: http://sourcesite.com/jsonapi/node/article ids: nid: type: integer item_selector: data/ fields: - name: nid label: 'Nid' selector: /attributes/nidOne place we might start is with a failing test, and this gives us an idea of what the test migration would look like. We would use the new jsonapi source instead of the url source from Migrate Plus (?) with its data_* options.For testing purposes, we would want to support getting the JSON from a file as well as from http.IMO it is important that the source plugin manage paging. That is part of what it means to be a JSON:API source and not a JSON source. I am not sure how to test that. I guess the primary source file has a "link" to the "next page" of results. I have not looked at the JSON:API spec: does this break it? That is, does the spec allow file:// for links as well as https://? |
| heddn | yes, paging is required (I think) |
| benjifisher | Does the spec give us enough information that we can leave off the fields: section of the source config? That would be sweet. |
| heddn | probably. we could dynamically surface it most likely |
| benjifisher | I looked at Links in the spec: https://jsonapi.org/format/#document-links. A link object has to contain a URL, but it does not say anything about the protocol of that URL. The examples are all http:// , but I do not see any reason not to use file:// for testing purposes. |
| mglaman | It could be any protocol AFAIK |
1️⃣ 0️⃣ Wrap up
| benjifisher | Thanks for participating! I already updated 2️⃣ . Please continue to add comments. In 1-7 days, I will post a transcript for today's meeting. |
Participants:
mikelutz, benjifisher, heddn, Bisonbleu, dinarcon, quietone, steinmb, mglaman, damienmckenna
Comments
Comment #2
benjifisherWe will not have a meeting on 2020-07-16, since that will be during DrupalCon Global. I will update the title of this issue to 2020-07-23, but not until the original date has passed.
We will have a BoF at DrupalCon Global: see the BoF schedule.
We will also have a "virtual table" during contribution day: please join the Migrate API contribution group.
Comment #3
benjifisherWe skipped the 2020-07-16 meeting because of DrupalCon Global. I am repurposing this issue (i.e., changing the title) for 2020-07-23.
Comment #4
benjifisherComment #5
benjifisherIn case anyone is curious, the first few agenda items are
Anyone can contribute to (1) and we will add a thread for it.
We fill in (2) after the scheduled meeting, and add threads for any open items from the previous meeting's action items.
Comment #13
benjifisherComment #15
benjifisherComment #16
benjifisher