[meeting] Migrate Meeting 2021-06-10 [#3217504]

Hello all, it’s time for the weekly migration subsystem meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.

Core migration issues

Next video meeting

2021-07-01 - The hope is that most or all of the maintainers will attend. We will try to focus on longer-term goals than in the weekly meeting.

0️⃣ Who is here today? Did you see the annular eclipse a few hours ago?

marvil07	o/ I did not know about that eclipse :sweat_smile:
mikelutz (he/him)	No, couldn't really see it here.
Neslee	Hey, didn't see here too
Matroskeen	Ivan, maintainer of Migrate Scanner module :yum:It was rain here, so no luck. (edited)
benjifisher	My wife woke up at 5:30 AM. The sun was technically above the horizon, but there are too many hills, houses, trees to see it. :disappointed:
benjifisher	I know @mikelutz (he/him) is West of me and would not see it, but maybe some of our European friends had more luck. Too bad about the :cloud: and rain in Ukraine.
benjifisher	@Neslee, is this your first time here? What part of the world is home for you?
anmolgoyal74	Hi.. I think it is not visible in India
huzooka	Hi - EU, it was partial (8% coverage)
Arthur Deryckere (KenowaX)	Yop ! Didn't see it. =(
Neslee	@benjifisher its my second time, and i am from Goa
benjifisher	Welcome back! Is there something you would like to talk about?
Neslee	Ya i have added my comment under point 4
gaurav mahlawat	Hello :wave:
alison	Hi! -- didn't see the eclipse, too early for me :parrotsleep:
quietone	Vicki

1️⃣ What should we talk about today? Suggest topics here and I will add threads. I will also check for comments on the issue for today's meeting.

Matroskeen I have a small announcement - Migrate Scanner got a RC recently - https://www.drupal.org/project/migrate_scannerManual testing and feedback is welcome :slightly_smiling_face: Matroskeen Also, I wanted to ask about the following case:I have to migrate multiple nodes into one block. This is 2-steps migration:1) nodes -> paragraphs;2) nodes -> block;In the 2nd step I need to have my nodes in a single row.I ended up writing a custom source plugin, but I’m wondering if there is a better option. (edited)

2️⃣ Action items. To be added later.

3️⃣ Statistics

benjifisher	Fixed since last week's meeting: 0 (not counting the issue for the meeting).
benjifisher	RTBC: 11, 2 of which are Major.
benjifisher	NR: 29, including 2 Major and 11 that have not been updated in more than three months.
benjifisher	Google sheet for recording stats: https://docs.google.com/spreadsheets/d/1o0Rjlc1vnnLP5bM5P-SMMyGzqn7258hi...
benjifisher	I guess the core committers are busy with security releases and the upcoming 9.2.0.
Matroskeen	larowlan mentioned this recently in #bugsmash channel:there has been a lot of freezes lately, between security releases and 9.2 beta/rcs(edited)

4️⃣ How can I help? Comment in this thread if you are looking for ways to help. Give us some idea of what you would like to do: documentation, code review, testing, project management, ...

Neslee	I started to get into migration tagged issue queue from last week, and will be working on testing/code review
benjifisher	Are you set up for testing? There are a few extra steps compared to typical Drupal core issues, at least if you are testing migrate_drupal.
Neslee	Yes, i believe. If you have anything else extra, that you want me to look into, docs link?
benjifisher	There are lots of issues that need review (as reported in 3️⃣) https://www.drupal.org/project/issues/drupal?status=8&component=migratio...
Neslee	Ya i had when over this filtered list, planning to start :+1:
benjifisher	There are also issues for Migrate Plus: https://www.drupal.org/project/issues/migrate_plus?status=8
Neslee	Ah this initiative is also working on even contrib modules? didnt knew that
benjifisher	Technically, we are not an initiative. But maybe we should be. If you want to create a page for the migration community initiative, that would be another way to help!

5️⃣ Previous minutes.

benjifisher	Is there anything to follow up? Anything need to be changes in those minutes?
benjifisher	#3216355: [meeting] Migrate Meeting 2021-06-03

6️⃣ Migrate Scanner release candidate

benjifisher	https://www.drupal.org/project/migrate_scanner
benjifisher	@Matroskeen, can you give a short description to save others the trouble of following the link? Maybe a little copy/paste.
Matroskeen	Yes, sure.This module provides a recursive directory scanner for migrations files. So you can place migration files into sub-directories like:migrations - foo - foo_bar.yml - foo_foo.yml - bar - baz - bar_baz.yml
Matroskeen	As a bonus, it allows to alter discovered migration files by implementing hook_migrate_scanner_patterns_alter hook.In this way, I was able to get rid of unnecessary migrations when using Migrate Source UI module.Kudos to @benjifisher for the nice feature request and code review.
Arthur Deryckere (KenowaX)	I started using it this week. It's really helpful. I was worried migration tags wouldn't be compatible as it's from migrate plus but it works fine.
benjifisher	@Matroskeen, where is a good place to provide feedback? Other than this thread?
benjifisher	@Arthur Deryckere (KenowaX), how many migrations are you managing?
Matroskeen	Issues/feature requests - in the issues queue.Other feedback can be either in this thread or general Slack channel. (edited)
Arthur Deryckere (KenowaX)	@benjifisher about 20 in 3 different languages. Making the total 60. And it keeps on growing. We're planning to reduce that number in the futur but used the migrate scanner as a way to keep a structured file system in the mean time.

7️⃣ Add documentation for file source plugins

benjifisher	#3189876: Add documentation for file source plugins
benjifisher	It is 2:12 AM for @quietone now, but she asked,What examples, if any, do you think should be in the docs for the file source plugins?
benjifisher	We already got one reply in this channel from @huzooka :I would simply look into the tests and repeat the plugin configs being tested in a human-friedly way :slightly_smiling_face:
huzooka	:slightly_smiling_face:
benjifisher	I have never used the D6 upload plugins (d6_upload, d6_upload_instance) so I have no suggestions there.
benjifisher	I see this in the current patch, for the d6_file source plugin: * Available configuration keys: * - site_path: (optional) The path to the site directory relative to Drupal * root. Defaults to 'sites/default'.That is a good start, but the question is about examples, and I think good examples will help a lot.
benjifisher	This is really confusing. People do not realize what the right directory structure is. When I first started doing migrations, I figured it out by looking at the "file not found" error messages.So I think it is worth adding two full examples.
benjifisher	OMG, I did not realize we did this in prepareRow() (D7 version): `$path = preg_replace('#' . preg_quote($this->configuration['constants']['source_base_path']) . '#', '', $path,` 1);We should make source_base_path a top-level configuration item if we are going to use it like this. But that is an issue for another day.
benjifisher	What happens if $this->configuration['constants']['source_base_path'] is not set? :thinking_face:
benjifisher	It is kind of crazy. We strip out source_base_path in prepareRow() and then we add it back with the concat plugin in the process phase. :scream:
benjifisher	Let's start with an example based on d7_file.yml, but skip the step where we concatenate with the empty string: `source: plugin: d7_file scheme: public constants: source_base_path: '' process: uri: plugin: file_copy source: - filepath - uri` All of that inside @code tags. Then explain what the path to the files directory has to be to make this work. And test it.Then we need a second example where source_base_path is not empty ...
alison	Glad this is moving along! I commented a month ago that the info I want in documentation is included with the patch that's been written so far, so I'm a happy camper.

8️⃣ Proposed new process plugin: service

benjifisher	Last week, @mikelutz (he/him) wrote,Also just the difficulties with dependency injection. You either have to store the whole container in the plugin, or break dependency injection.and I asked,I do not understand DI well enough. The container is an object, and storing an object just means storing its handle, right? I do not see the problem with that, unless the plugin gets serialized.
huzooka	I will repeat myself - we can use a deriver
benjifisher	If breaking DI means we cannot test it properly, maybe that is a blocker for core but not migrate_plus. It would be pretty easy to extend the callable plugin and use \Drupal::service() to load the requested service. @heddn, would that meet the standards of migrate_plus?
benjifisher	If we use a deriver, then the usage is something likesome_field: plugin: service:pathauto.alias_cleaner method: cleanString source: whateverright?
huzooka	yepp
benjifisher	Wouldn't that mean a lot of work during plugin discovery, loading all services? And caching all of them?
huzooka	No,I don't think so (edited)
huzooka	BTW if we don't inject services, then we can still test them it (edited)
benjifisher	I already admittedI do not understand DI well enough.What is the problem with breaking DI for this pluign?
heddn	I don't have concerns with breaking DI. But I'm also not sure why we would need to break it. We get configuration passed into a plugin creation. public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration = NULL)
benjifisher	Right, so we have the container and the configuration available in the create() method. This might be even easier to implement than I thought.
quietone	Or have a destination pipeline?
benjifisher	I do not know what you mean by a destination pipeline. Did you mean to comment on 9️⃣?
benjifisher	It is so easy to implement that it feels like cheating. #3218356: Process plugin: service

9️⃣ Many-to-one migrations

benjifisher	Quoting @Matroskeen from 1️⃣:I have to migrate multiple nodes into one block. This is 2-steps migration:1) nodes -> paragraphs;2) nodes -> block;In the 2nd step I need to have my nodes in a single row.I ended up writing a custom source plugin, but I’m wondering if there is a better option.
benjifisher	I think that is the only way to do it currently.
benjifisher	I think it would be useful to have a destination plugin that merges multi-valued fields instead of skipping them or overwriting them. Would that work for this use case?
Matroskeen	Hmm… not sure to be honest.I was thinking about the opposite direction - changing the source rows.It can be done by adding option to SourcePluginBaseand convert existing rows into one with items key. (edited)
benjifisher	When you say "opposite direction", you mean making changes in the source plugin instead of destination plugin, right?I think we agree this is a many-to-one problem, not one-to-many.
benjifisher	Probably neither of our suggestions will be added to core, but they might be added to Migrate Plus.
benjifisher	Can you give an example of what your source plugin configuration would look like?
Matroskeen	There is no configuration yet. This is how it looks in my custom plugin:protected function initializeIterator() { $single_row = ['items' => iterator_to_array(parent::initializeIterator())]; return new \ArrayIterator($single_row);}
benjifisher	How does that work? Does it simply create a source plugin that always returns a single row?
benjifisher	Clarification: my previous question meant if we were to modify SourcePluginBase (your suggestion) then what would the YAML look like in a migration?
Matroskeen	I didn’t try it yet, but I’m already facing issues related to missing ID property :sweat_smile:Chances are my approach is not gonna work at all
Matroskeen	When I figure it out, I’ll make a feature request to migrate_plus module.But the idea is the following - create a single row and place original rows as a single row property.
benjifisher	Think about my suggestion. If we have a row for each paragraph, and we can figure out which node to attach the paragraph to, then all we need is some way to add items to an existing field.I think a new destination plugin is the way to do that, but maybe there is some other way.
mikelutz (he/him)	How are you identifying which block your nodes are going into?
mikelutz (he/him)	Or is it literally every node in the migration becomes a paragraph, all of which are added to a single paragraph field in a custom block?
Matroskeen	Yep, that’s right.Actually, when I have a custom block, I would need one more step - place that block into layout builder of the homepage.My approach with making one row worked, but now I have to think more about @benjifisher’s suggestion. (edited)
Matroskeen	To give more context:This is gonna be a slideshow, where each slide is a node in D7, but paragraph in D8/D9. (edited)

1️⃣0️⃣ Compatibility with Drush 10.4+

benjifisher	@marvil07, any recent progress? There was a question in this channel earlier today.
benjifisher	Related: with issues like https://gitlab.com/drupalspoons/migrate_tools/-/issues/92 should we reopen the related issue on d.o, #2701121: New batch started because of not enough reclaimed memory seems to forget which migrations have already run leading to missing migration dependencies ?Maybe this is something where we could encourage new contributors to help with the issue management. What do you think, @heddn?
marvil07	@benjifisher No progress over last few weeks on my side, I have been focusing on migrate code reviews :slightly_smiling_face:
benjifisher	Are you the one responsible for so many RTBC issues?
radelson	Still curious about this question ! => We're having memory issues during one of our migration and if we end up using BatchMigrateExecutable (this issues #2701121: New batch started because of not enough reclaimed memory seems to forget which migrations have already run leading to missing migration dependencies ), I'd like to contribute our work on it but I'd like to have some pointers on what's the best way to have it merged somewhere now that migrate:import command is not in migrate_tools anymore

1️⃣1️⃣ Wrap up

benjifisher	Thanks for participating! I will update 2️⃣. Please continue to add comments in the threads. In 1-7 days, we will post a transcript for today's meeting.
Matroskeen	Thanks!

Participants:

marvil07, mikelutz (he/him), Neslee, Matroskeen, benjifisher, anmolgoyal74, huzooka, Arthur Deryckere (KenowaX), gaurav mahlawat, alison, quietone, heddn, radelson