Hello all, it’s time for the weekly migration initiative meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to here: https://www.drupal.org/project/drupal/issues/3178489. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.

Core migration issues:

0️⃣ Who is here today? What is the next Drupal camp you plan to attend? (Hint: I am leading a migration BoF at NEDCamp on 2020-11-06.)

irinaz Hi, this is Irina Zaks, working on Feeds-like UI for Migrate
quietone Hi. I've never been to a Drupal camp. There are all too far away.
benjifisher This year, they are all on Zoom or Hopin. But there are still timezone issues.
irinaz @benjifisher could you send link to that BoF?
benjifisher Here is the whole schedule. Migrations are at 10:00 ET. https://nedcamp.org/days/2020/bof-schedule-2020
dinarcon :wave: NEDCamp and DrupalCamp Costa Rica
damienmckenna Hey there, won't be able to attend NEDCamp this year, will be selling my house :disappointed:
irinaz @benjifisher,  7am pacific might be a bit too early for me... https://nedcamp.org/bof/drupal-migrations but I will try :slightly_smiling_face:
irinaz @dinarcon when is DrupalCamp Costa Rica?
quietone Yes, I should t adjust to video conferencing.
dinarcon https://www.drupalcamp.cr/ November 5-7
benjifisher @damienmckenna Are you moving locally or long distance?
Nick Dickinson Wilde For a change vaguely here. Don't have any camp plans for the rest of the year. Way too many deadlines right now in too many things and I'm tired and wishing for in person interactions. (edited)
damienmckenna @benjifisher: Ireland :slightly_smiling_face:

1️⃣ What should we talk about today? Suggest topics here and I will add threads.

irinaz Hi @benjifisher, it was great to see you and talk to you at BADCamp.  I would like to talk about what feeds_migrate module can do and how migrate team can benefit from this module (edited)
quietone Would like to get reviews on all the issues in this meta, #2456259: [META] Drupal 7 to Drupal 8 Migration path
quietone Congratulate ourselves that #2208401: [META] Remaining multilingual migration paths is fixed!
benjifisher :tada:
damienmckenna Would appreciate some discussion on the possibility of either media_migration not working with migrate_upgrade, or wondering if something else might be adding conflicts? In short, #3170639: Provide workaround for Migrate Tools' broken migration dependency discovery was committed but it didn't resolve the problem for me.
Nick Dickinson Wilde Got a bit of a weird thing...Using migrate_tools/haven't tested this with migrate_run (actually never used it so far).running drush ms [error] Message: Failed to connect to your database server. The server reports the followingmessage: /No database connection configured for source plugin variable/. * Is the database server running? * Does the database exist, and have you entered the correct database name? * Have you entered the correct username and password? * Have you entered the correct database hostname?None of my migrations use the variable plugin.My migration key is d7 - but it can be anything except migrate and it will throw this.If instead the database key is set to migrate then I get all the Drupal 6 and 7 core migrations trying to report status (and some failing because it is a D7 database so structure is wrong for D7 ones).I can add a custom tag to all my migrations and then run drush ms --Tag mycustomtag But... that isn't great to have to remember to do. Is there any way we could have a skip-core-migrations options*? (or any other recommendations?)*I mean if there was a PR for that would it be likely accepted?

2️⃣ Action items. To be added later.

benjifisher Most of last week's items were done. :slightly_smiling_face: One exception: #2925899: MigrateUpgradeImportBatch does not use source_private_file_path & source_base_path correctly, making it impossible to have public & private files in separate locations (NR, @benjifisher)
benjifisher Update https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins/ch... as described in 4️⃣. Needs a volunteer
benjifisher #2852463: Create a trait and base class to implement \Drupal\Component\Plugin\ConfigurableInterface:Update issue summary. @mikelutz (he/him) (and then un-assign yourself)Review.
benjifisher NR: #3000050: Replace \Drupal\Component\Utility\Environment::checkMemoryLimit with a PHP memory checking helper class and service. @benjifisher
benjifisher NR: child issues of #2456259: [META] Drupal 7 to Drupal 8 Migration path
benjifisher #3150949: Add a migration source plugin for JSON:API: Create fixtures (valid file-based JSON:API sources for tests). @quietone
benjifisher NR: #3095237: Migrate Drupal 7 date field "todate" value
benjifisher Issue summary update for #3080256: Add alter event to alter generated configuration and/or related issues (see 1️⃣0️⃣). @damienmckenna
benjifisher Test and, if appropriate, issue summary update: #3152789: Add required variables config to Variable migrate source plugin, and if one of those are missing, return zero rows. @Nick Dickinson Wilde (edited)

3️⃣ Statistics

benjifisher Fixed since last week' meeting:#2208401: [META] Remaining multilingual migration paths (woot!)
benjifisher RTBC: 7, all Normal priority, two have been waiting more than a month.
benjifisher NR: 27, including 5 Major; one issue (Major) has been waiting more than a month (first item under 2️⃣).

4️⃣ Documentation

benjifisher One of the fixed issues is for updating this documentation page: https://www.drupal.org/node/2350651. It now mentions the migrate_run module.
benjifisher One of last week's action items (done) was to add this new page to the guide's menu: https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins/ch...
benjifisher I think it would be good to expand it:Give more-or-less complete YAML for both the User migration and the Node migration that depends on it.Give some example database queries to illustrate how you can troubleshoot.Mention what happens for more complicated migrations: multiple source and/or destination IDs.

5️⃣ Feeds Migrate

benjifisher https://www.drupal.org/project/feeds_migrate
benjifisher @irinaz, is there any recent progress to report? Are there any core issues blocking progress?
irinaz @benjifisher, there has been lots of updates during last year, and there is most recent patch for plugin processors #3002362: Foundation: Feeds migrate Processors/Tamper UI #comment-13873376
irinaz We are looking for community help to test this patch
benjifisher In the past, I have had trouble working on that module. Development has been split between d.o and GitHub. If I apply the current patch and follow the instructions in the issue summary, can I actually test it?
irinaz Yes, all development has been merged and now everything is on drupal.org.
irinaz Let me know if there is more info about this module that I can post in this channel :)
megachriz @benjifisher Yes, there are core issues that are partly blocking progress. These are #3093407: Add functional test for creating/editing migrations
benjifisher I remember looking at those issues at MidCamp 2019. My main contribution was to nudge Mike Lutz on #2852463: "Hey, can you reply to Lucas's question in #77?" I did not notice that the fix was reverted, more than a month later.
megachriz @benjifisher In Feeds Migrate UI errors are now displayed because some Migrate plugins make assumptions about what config is being passed to the constructor. So we need them to implement ConfigurableInterface.

6️⃣ Move memory management from MigrateExecutable to an event subscriber

benjifisher #3006750: Remove memory management from MigrateExecutable
benjifisher This is now postponed on #3000050: Replace \Drupal\Component\Utility\Environment::checkMemoryLimit with a PHP memory checking helper class and service. Both @quietone and I have posted reviews there, and @quietone made an updated patch based on my review. We need to decide whether it needs to be a service, ...

7️⃣ [META] Drupal 7 to Drupal 8 Migration path

benjifisher #2456259: [META] Drupal 7 to Drupal 8 Migration path
benjifisher It looks as though three of hte child issues are marked NR.
benjifisher @quietone, is the issue summary up to date? Is there still important stuff in the sandbox waiting to be ported? Or should we just be looking at the child isues?
quietone Issue summary is probably out of date. Two of the three child issues are waiting for a review after a reroll, they were both already RTBC.
quietone Assigned the meta to myself to update the IS
quietone :checkbox: IS updated.

8️⃣ Add a migration source plugin for JSON:API

benjifisher #3150949: Add a migration source plugin for JSON:API
quietone I was thinking of trying this client, https://github.com/swisnl/json-api-client
benjifisher Corect me if I am wrong, but I think the latest patch is mostly an update of hte previous one, porting the JSON source from Migrate Plus.
quietone Yes, that is correct.

9️⃣ Migrate Drupal 7 date field "todate" value

benjifisher #3095237: Migrate Drupal 7 date field "todate" value
benjifisher After the discussion last week, @quietone posted a new patch. Back to NR.

1️⃣0️⃣ Migrate Tools and Media Migration

quietone This is the missing dependency problem? That is referred at #3157708: Missing dependencies when using Migrate Upgrade#comment-13843516
benjifisher Fly-by guess: media_migration is using a deriver, and this does not work with migrations defined by configuration.If that is anywhere close to what is going on, then I think the problem is in Migrate Plus, not Migrate Tools. Have you considered using migrate_run instead? (Cf. 4️⃣)
benjifisher I am not sure about the specific commentin that link, but Damien mentioned #3170639: Provide workaround for Migrate Tools' broken migration dependency discovery in 1️⃣, and that refers to the same issue.
quietone I agree with huzooka that it should be fixed in Tools (or Plus) not have many workarounds. (edited)
damienmckenna For now I'm going to continue using the patch that wasn't committed, hopefully that'll be enough (haven't looked at it on my one project for a few weeks).
quietone @benjifisher I don't use migrate plus or migrate tools but I would have expected the problem to be in tools. Why do you think it is migrate plus?
benjifisher I thing that Plus creates the plugins from the configuration, and Tools is responsible for running whatever migrations (or migration plugins) it finds (filtered by tags or groups provided).As I said, it was a fly-by guess. I only read the issue description. (edited)
benjifisher If it is a question of dependencies, can you work around it by exporting config, editing the dependencies, and then re-importing? Or is one of these modules modifying the dependencies declared in the config?
quietone I would like someone to try building the migration config with an empty prefix (--prefix='') and not use the default of 'upgrade'. (edited)
benjifisher I see this linked issue: #3080256: Add alter event to alter generated configuration (created by @heddn, patches and comments from @damienmckenna). I guess the goal of that is to provide a hook in migrate_upgrade that other modules can implement.That is getting closer to solving the problem once instead of having work-arounds.That approach is doomed to fail if the dependencies declared in config are munged.
damienmckenna Yeah, some feedback on ^ would be really appreciated.
benjifisher Can we start with an issue summary update, for at least one of these issues? Fully describe the status quo:From the user's point of view. Include the error message for searchability. (Not a word?)Your best understanding of what is going wrong. Where are dependencies originally declared, and where are they being modified?If you think that the alter hook is the right approach, then add this to the summary of that issue.Even better (but not necessarily the right first step) would be a failing test that demonstrates the description of what goes wrong.

1️⃣1️⃣[error] Message: Failed to connect to your database server. The server reports the followingmessage: /No database connection configured for source plugin variable/.

benjifisher I think there is a recent issue that is related to this.
benjifisher Maybe this one? #3152789: Add required variables config to Variable migrate source plugin, and if one of those are missing, return zero rows
dinarcon I think this might be related as well #3123950: 'Field discovery failed for Drupal core version 7' errors if migration source database key is not 'migrate'
benjifisher If it is either of those issues, then someone should paste the actual error message into the issue description so that it shows up in search results.
benjifisher @Nick Dickinson Wilde: this thread's for you.
Nick Dickinson Wilde Thanks the first one looks like it is it
Nick Dickinson Wilde I'll try the patch/confirm before updating the issue though

1️⃣2️⃣ Wrap up

benjifisher Thanks for participating! I will update 2️⃣. Please continue to add comments in the threads. In 1-7 days, we will post a transcript for today's meeting.
dinarcon Thanks for facilitating Benji
damienmckenna Yes, thanks again Benji, it's much appreciated.

Participants:

irinaz, quietone, benjifisher, dinarcon, damienmckenna, Nick Dickinson Wilde, megachriz

Comments

benjifisher created an issue. See original summary.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

benjifisher’s picture

benjifisher
he/him or they/them
Primary language English
Location Boston area
commented

With my two roles as migrate co-maintainer and member of the Usability team, I should take responsibility for #2687849: Add back rollbacks on migrate_drupal_ui (currently NR and tagged for usability review).

benjifisher’s picture

benjifisher
he/him or they/them
Primary language English
Location Boston area
commented
Issue summary: View changes

quietone credited DamienMcKenna.

quietone credited NickDickinsonWilde.

quietone credited dinarcon.

quietone credited irinaz.

quietone’s picture

quietone commented
Issue summary: View changes

quietone credited MegaChriz.

quietone’s picture

quietone commented
quietone’s picture

quietone commented
Status: Active » Fixed
benjifisher’s picture

benjifisher
he/him or they/them
Primary language English
Location Boston area
commented

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.