[meeting] Migrate Meeting 2021-05-20 [#3214258]

Hello all, it’s time for the weekly migration subsystem meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.

Core migration issues

Next video meeting

2021-07-01 - The hope is that most or all of the maintainers will attend. We will try to focus on longer-term goals than in the weekly meeting.

Agenda Items

Add your items here.

0️⃣ Who is here today? Say hello and tell us something you like to do.

dinarcon	Hello :wave:
marvil07	Hi!I like to sit quietly and see landscapes. :evergreen_tree: :mountain:
benjifisher	:wave: Benji Fisher, celebrating one year of being a co-maintainer. I like to square dance, but that has been no-no during the pandemic.
marvil07	Congrats @benjifisher!
mikelutz (he/him)	Hello all!
irinaz	hello, all
quietone	Vicki, I like to walk in Wilton's Bush and look for eels.
benjifisher	What do you do when you find an eel? Just look at it?
quietone	Yes, they are protected there.
jibran	Jibran
anmolgoyal74	Hi, I like to go for long drives…
gaurav mahlawat	Hello, Catching up late.
Neslee	Hey, i like cycling and photography
Arthur Deryckere (KenowaX)	Hi :slightly_smiling_face:

1️⃣ What should we talk about today? Suggest topics here and I will add threads. I will also check for comments on the issue for today's meeting.

marvil07	I have an update on drush/migrate_tools! :raised_hands:
dinarcon	Add an option to migrate_lookup to not do return perform a reset on $destination_ids if the array holds a single value. This can help when migrating entity reference revisions (paragraphs) using sub_process.
quietone	Ways to encourage reviews. Many at NR are patches by me so I can't do review. And there is an unfortunate side effect - I am reluctant to make patches for other issues because of the low chance of a review.
huzooka	I would very much appreciate your suggestions and concerns: #3213636: Migrate text fields with conflicting text_processing setting as formatter text fields

2️⃣ Action items. To be added later.

quietone	NR #3189876: Add documentation for file source plugins @quietone
quietone	NR #3213012: [meeting] Migrate Meeting 2021-05-13 @quietone

3️⃣ Statistics

benjifisher	Fixed since last week's meeting: 3, 1 of which is Major (counting 1 fixed during last week's meeting, not counting the issue for the meeting).
benjifisher	RTBC: 9, 1 of which is Major.
benjifisher	NR: 30, including 2 Major and 5 that has not been updated in more than three months.
benjifisher	Google sheet for recording stats:https://docs.google.com/spreadsheets/d/1o0Rjlc1vnnLP5bM5P-SMMyGzqn7258hi...

4️⃣ How can I help? Comment in this thread if you are looking for ways to help. Give us some idea of what you would like to do: documentation, code review, testing, project management, ...

5️⃣ JSON parsing

benjifisher	#3007709: Add XPath-style filtering ability in JSON data parser plugin
benjifisher	I did not realize how poorly supported JSON was until someone asked for help on this channel a few days ago. With XML sources, you have true XPath support (using simple_xml data parser from the migrate_plus module). With JSON, you have very limited options for your fields and item selectors.
benjifisher	I would love to get some testing and get that issue ^^ fixed in migrate_plus. Does anyone have some JSON migrations where they can test it?
irinaz	@benjifisher we have some json migrations in one of our older projects, I can try to find those if that will be helpful
irinaz	there is also json data here https://understanddrupal.com/articles/migrating-json-files-drupal - will that work for testing?
benjifisher	That is a possibility.
benjifisher	Once we do that, maybe we can think about JSON:API in the migrate_plus module instead of core. We do not have to jump through hoops to allow external libraries.
benjifisher	#3150949: Add a migration source plugin for JSON:API
benjifisher	@Thomas Wagner, how is your migration going? If the issue at the top of this thread is working for you, then please add a comment there. (edited)

6️⃣ Create a trait and base class to implement \Drupal\Component\Plugin\ConfigurableInterface

benjifisher	Create a trait and base class to implement \Drupal\Component\Plugin\ConfigurableInterface
benjifisher	@irinaz, @mikelutz (he/him)
mikelutz (he/him)	We should totally do that.
irinaz	I cannot do code review because I do not have enough qualifications, but I will be happy to test #2852463: Create a trait and base class to implement \Drupal\Component\Plugin\ConfigurableInterface
benjifisher	It is such a good idea that the issue was fixed until some mean person decided to revert it. :wink:
irinaz	@mikelutz (he/him) what needs to happen for this issue to be fixed?
benjifisher	Thanks, I meant to paste in the link.
mikelutz (he/him)	I need to love it and nurture it and spend time with it like I’ve been promising @benjifisher for months.. I kinda feel like a deadbeat dad right now, promising everything, but not showing up to the kids baseball game.
irinaz	@mikelutz (he/him) I hear you. How about not showing for kids baseball game but going to a bar for some beer instead? What would be your incentive to prioritize this issue?
mikelutz (he/him)	It’s not a matter of incentive, I’m just deep in a project at work, and I haven’t had time to do any work on Drupal in some time. I added an @todo in my work task tracking system, hopefully that will force me to set aside an hour to catch up and work on it next week.
benjifisher	Feel free to ping me on Slack when you find time for it. It might help to have some synchronous discussion.
irinaz	Let me know if I can help in any way.

7️⃣ Drush and Migrate Tools

benjifisher	@marvil07, what is new?
benjifisher	https://gitlab.com/drupalspoons/migrate_tools/-/issues/118
marvil07	Now that migrate_tools is back on drupal.org, I have created a new issue there to track the progress on the migrate_tools side of the changes.See #3213947: Drush core migrate commands integration refactor I have started a new branch on an issue fork that handles the migrate-status command and also a compatibility layer for an option on migrate_tools but not on drush core.
marvil07	No MR opened yet, I will wait till I port other commands.Relevant current diff for web UI peak at https://git.drupalcode.org/issue/migrate_tools-3213947/-/compare/9d9c02f...
benjifisher	I notice that you can now use migrate_tools:dev-5.x with the latest drush/drush.
marvil07	Yes, commands are overriden by default, i.e. you get migrate_tools commands and lose access to drush core equivalents.
dinarcon	Is that a temporary measure? Or migrate_tools will always override Core commands?
dinarcon	Has there been conversations to remove the composed conflict of Drush 10.4+ on migrate_tools if it is possible to install now?
benjifisher	I think the conflict is for migrate_tools:<5.0, which is why dev-5.x works.
marvil07	Exactly, but I am guessing that once 5.x is released it drush core may up the version con the composer conflict line.
benjifisher	Why would they do that? (they = Drush or the Drush maintainers)
marvil07	They added the conflict, but it actually works, so it was probably about suggesting the command set was taken, so it is just a candid guess :nerd_face:

8️⃣ Add documentation for remaining source plugins

benjifisher	#3199741: Add documentation for remaining source plugins
benjifisher	This issue is RTBC, with a MR instead of a patch. So it re-tests every 2 days. The latest test (yesterday) failed, but the testbot did not set it back to NW.
quietone	Ah, that needs to go back to NW. I will comment now.
benjifisher	It looks like an unrelated failure: field_layout\FunctionalJavascript\FieldLayoutTest.
benjifisher	Because of the failed test or something else you just remembered?
quietone	I remembered two days ago - we should not have used @see where we did.
Matroskeen	@quietone I had another suggestion for @see section: #3189876: Add documentation for file source plugins#mr185-note25593It would be great if you can review, so I can address this in remaining tasks.

9️⃣ Scan for snippets of process plugins

benjifisher	#3214186: Scan for snippets of process plugins
benjifisher	I owe @Matroskeen a review of the related issue #3214464: Let other modules modify the scanning rules
quietone	Will migrate scanner also act as a repository for snippets?
benjifisher	First we have to decide whether that is really the right place for the snippet plugin.
benjifisher	Currently it is implemented as a configurable process plugin. Part of the configuration is the path to the file (relative to migrations/process/).
benjifisher	I originally thought of creating a process plugin for each snippet. If I do it that way, then it belongs in migrate_scanner.
benjifisher	In the current form, I think it is more appropriate for migrate_plus.
benjifisher	Does anyone want to convince me that a plugin per snippet is a better idea than a configurable plugin? @Matroskeen? @mikelutz (he/him)? @marvil07
marvil07	I think tried wrongly an ended up suggesting the opposite at https://drupal.slack.com/archives/C226VLXBP/p1621102949204500?thread_ts=... :sweat_smile:If derivers are used (I have not read the code yet) then the one process plugin with deriver can cover the case of configuration plus unique name together.
mikelutz (he/him)	A big part of me wants to support this use case in core, though in a different way..
benjifisher	How would you handle it?
mikelutz (he/him)	Do we have a core ticket for reusable process pipelines? I thought we made one at some point.
mikelutz (he/him)	Untitled id: my_migration source: plugin: sql process: name: Click to expand inline (20 lines)
mikelutz (he/him)	Support that in core for a single migration and then support reuse through migrate_plus migration groups.
mikelutz (he/him)	I see #3123534: Process plugin: snippet to re-use YAML config / https://gitlab.com/drupalspoons/migrate_plus/-/issues/133. for migrate_plus.
mikelutz (he/him)	The trick to doing it my way is to find some migrations in core that could be simplified in that format.
mikelutz (he/him)	I don’t know if there are.
benjifisher	I just found those.You can replace null_coalesce with a little pipeline including array_filter.
mikelutz (he/him)	Eh, null_coalesce is fine, I think. It’s more efficient anyway, it stops at the first non-null value. array filter has to check all of them just to pull the first one later.
benjifisher	In my use cases, that is micro-efficiency.
benjifisher	I am still curious what your approach is.
benjifisher	There are a lot of process plugins added by core modules other than migrate. I compiled a list: https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins/li...…]-other-core-modules
benjifisher	I think some effort went into making the migrate module's process plugins orthogonal. Maybe some of these could be replaced with a pipeline.
mikelutz (he/him)	My approach is the snippet I posted above.
benjifisher	Oh, I missed that.That would be reusable within one migration, but not between migrations.Maybe, with migrate_plus, it could be used within a migration group.
mikelutz (he/him)	Right. I said that above too.
benjifisher	As I said, ... :wink:

1️⃣0️⃣ How to tame the NR queue?

quietone	Anyone wanting to learn about reviewing an issue is welcome to ask me any question they have. There is more to review than code review.
benjifisher	@quietone, I spend much more time on review these days than on writing my own patches. If I review an issue and see something that needs to be fixed, I hold myself back so that I can keep the reviewer role.
benjifisher	For the last few weeks, I have been spending a lot of time on #2571235: [regression] Roles should depend on objects that are building the granted permissions, which is getting very close. I should have more time soon to work on the NR queue for migration.
quietone	Yes, I have been keeping an eye on that, you and alexpott have a nice flow going with that one.
marvil07	@quietone I truly appreciate the amazing and prolific work you do on the migration queue :thankful:I have felt like that in the past, i.e. seeing patches added and waiting for review for a long time, years in some cases, so I totally get it.I guess more hands are needed on the review queue, and I also guess that the reviewer role is a bit intimidating for many people.I have no answers on how to promote more code reviews now, but taking about it is the first step, so thanks for bringing it up!
marvil07	:s/taking/talking/
quietone	thank you.Yes, review can sound intimidating.A little while ago I forced myself to do reviews and more reviews. I then found it was difficult to keep up/remember to follow up on the new work done.I am trying to say that getting a timely flow between the reviewer(s) and others working on the issue dramatically reduces the time the issue will get to RTBC or otherwise resolved.
benjifisher	When I review an issue, I make an effort to keep working on it.I know, there are several such issues where I owe you a re-review. :shrug: I try.
quietone	I think benjifisher is right - that a reviewer needs to keep working on the issue. Something I missed when I did reviews on more issues that I could follow up on in a reasonable timeframe.

1️⃣1️⃣ Expose full set of debugging data in migrate_message table (filterable/searchable)

benjifisher	#3063856: Add ability to view migrate_message table data
quietone	The test needs work so this work can move from migrate_drupal_ui to migrate. I hope to do that this week.
quietone	I played with it late last night and found that there is no filtering on the messages. Didn't we have that working?
benjifisher	@quietone, I think that filtering did work. Your implementation of 86.6 did not work. See my comment #94.
benjifisher	Specifically, 94.4.
quietone	OK. thanks for the nudge.

1️⃣2️⃣ Wrap up

benjifisher	Thanks for participating! I will update 2️⃣. Please continue to add comments in the threads. In 1-7 days, we will post a transcript for today's meeting.
marvil07	:thankful:
dinarcon	Thanks for facilitating Benji! :raised_hands:

Participants:

dinarcon, marvil07, benjifisher, mikelutz (he/him), irinaz, quietone, jibran, anmolgoyal74, gaurav mahlawat, Neslee, Arthur Deryckere (KenowaX), huzooka, Matroskeen