Hello all, it’s time for the weekly migration subsystem meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.
2021-07-01 - The hope is that most or all of the maintainers will attend. We will try to focus on longer-term goals than in the weekly meeting.
Add your items here.
| benjifisher |
Hi, I am Benji, one of the maintainers of the migration subsystem.I should get myself some speakers, because I do not like wearing headphones all day ... |
| mikelutz (he/him) |
Definitely not. I wouldn’t be able to concentrate on code if the music had any lyrics, and while I don’t hate instrumentals, I’m just not in the mood for them while working. |
| Joshua Turton (srjosh) |
I'm here! And constantly. I have many different playlists, depending on mood. |
| webchick |
I'm webchick, a Drupal product manager. I sadly also can't listen to music while working, but I love listening to music while driving. :smile: Mostly punk/indie stuff, esp. stuff heavy on screaming chicks. :smile: (edited) |
| quietone |
Vicki, sometimes and if I do it is Baroque, with headphones My partner plays music nearly all day and sometimes I wear cancellation headphones so I can work. |
| heddn |
Lucas here. I sometimes listen to music if I need to zone out all background noise around me. Usually in spanish (my second language) so it isn't too hard to tune the music out if I need to. |
| alison |
Alison here! Happy Pride Month to all who celebrate on the outside or inside :heart_hands: :progress-prideflag-rounded:I'm with Mike, I can't concentrate with music, especially if I like it. But, I do have The Office running in the background as often as possible. I can't explain why it works for me, I'm very easily distracted, it makes no sense. My only guess is that I've seen the first 6-ish seasons soooo many times, I know almost every line, maybe my brain can pop in and out at any moment???? (edited) |
| gaurav mahlawat |
Hello :wave: |
| anmolgoyal74 |
Hello! |
| Matroskeen |
Hi, I’m Ivan and I’m reading meeting discussions in the morning :wave:I usually listen to music, although I turn it off when I need to focus, for example learning something new. (edited) |
| benjifisher |
#3123534: Process plugin: snippet to re-use YAML config |
| benjifisher |
I reopened this issue, now that Migrate Plus issues are back on d.o. I added a comment listing several options, including the ones we discussed ... umm ... 2 weeks ago. (maybe 3 weeks) |
| Joshua Turton (srjosh) |
This... seems really useful. |
| benjifisher |
Three of the options involve a new process plugin, and two of those are already implemented in patches. (One is mine.) |
| Joshua Turton (srjosh) |
Of the options in that thread, I favor the one that's simplest for devs to learn - which is IMO the pipeline method. It's yaml, which is already known. |
| Joshua Turton (srjosh) |
doesn't require any new install of a preprocessor, or weird syntax in the header of the yaml file. |
| benjifisher |
@mikelutz (he/him), I was thinking of your proposal. I think it is complicated to have the process plugin "peek" at the current migration to get its configuration. |
| benjifisher |
My snippet plugin is also YAML. |
| benjifisher |
IMO, the main difference is that pipeline gets its pipelines from config entities, and snippet gets its pipelines from mymodule/migrations/process/*.yml. |
| Joshua Turton (srjosh) |
(I'm also on the record as favoring the config entity approach for migrations due to its centralization, i.e. groups and migrations in the same folder) |
| mikelutz (he/him) |
You think my proposal is complicated and needs to peek at the migration? |
| mikelutz (he/him) |
I don’t care what route you go, but mine would not require a process plugin to peek at the migration. The migration would replace the pipeline with the actual plugins somewhere around Drupal\migrate\Plugin\Migration::getProcessNormalized() (edited) |
| benjifisher |
I mis-remembered. You suggestedprocess: name: pipeline: trim_and_capitalize source: nameI was thinking of something likeprocess: name: plugin: pipeline pipeline: trim_and_capitalize source: nameSo your proposal is not a new process plugin at all. |
| benjifisher |
I still think that implementation is more complicated than the options that create a new process plugin. You are adding code to something already complicated instead of yet another process plugin (YAPP). |
| mikelutz (he/him) |
Right, it would make the pipelines first class citizens within the migration. |
| mikelutz (he/him) |
I definitely disagree on the complexity of implementation. I think that could be coded up in a handful of lines in core. The only complicated part is the test suite for it. |
| benjifisher |
Are you going to allowprocess: name: - pipeline: trim_and_capitalize source: name - plugin: some_other_process |
| alison |
Read this thread but haven't read the issue yet, so if my question is irrelevant, lmk and I'll believe you :grin:In the past, I've put "shared" process stuff in migration group YML files, i.e. we had a bunch of things that were going to be the same for every node migration, so we put those processes into the migration group YML file that all our node migrations were part of...To be clear, I don't mean "you don't need any of this b/c of what I'm saying about groups", I'm wondering about the use case / motivation / end result. I hope I'm making sense. |
| Joshua Turton (srjosh) |
If you read the thread from the first link on d.o, there's a really good explanation of that, I think (edited) |
| benjifisher |
In short: what if you want to apply the same pipeline to two different text fields? I do not think that can be done with migration groups. |
| mikelutz (he/him) |
You certainly could, Benji. |
| alison |
In short: what if you want to apply the same pipeline to two different text fields? I do not think that can be done with migration groups.I think you could, BUT, it would have to be the same fields for all migrations that use that group, whereas (I think?) with the plugin you're proposing, you'd have more flexibility / the content types don't need to be quite so similar in order to reuse your config YML.(That said, y'all's responses have helped me understand the issue/goal -- and it does seem to be similar to how we used migration groups, at least in my head. Thank you!) |
| mikelutz (he/him) |
You can put a process in for a field in a migration group and apply that process to that field on multiple migrations. Migration group configuration is simply merged into all Migrations in the group. This is about having 40 different fields in one migration (or potentially across multiple migrations) that all need to be piped through skip_on_empty, trim, lowercase, and string replace for each one. How can we make that easier/cleaner than copy pasting those 4 plug-ins over and over |
| Joshua Turton (srjosh) |
true. However, the benefit to a pipeline process plugin (whatever shape that takes) would be that you could designate a specific set of baseline stuff for a type of field, then also add in additional special-case stuff for a specific field of that type. Just adding fields to a group makes that difficult. |
| mikelutz (he/him) |
I suspect I'm going to want this on my current project, so there's a good chance I'm going to code up my solution as a proof of concept core patch. I think you will be surprised as to how small the patch will actually be, sans tests. |
| Joshua Turton (srjosh) |
@mikelutz (he/him) let me know if you need eyes on that. |
| benjifisher |
The other two options are already available as patches. It will be great to have actual code to compare. |
| benjifisher |
why [did] various migration_* modules [make] the decision to move away from DrupalSpoons and instead back to Drupal.org? |
| heddn |
because the community kept asking to move back to d.o. plain and simple. |
| benjifisher |
@heddn is the one who moved to DrupalSpoons. He really liked some of the features, including MRs (before they were avaiable on d.o and maybe still easier to use) and flexible tags. |
| benjifisher |
Right, and people did not like the fragmentation of having to look in two places, depending on whether they were working on Drupal core or one module or another. |
| heddn |
I really liked its issues. pasting images inline into the body of a field is very easy. on drupal.org, that just isn't possible right now |
| benjifisher |
Another advantage of d.o is being able to reference other issues with [#1234567] syntax.Another is the ability to move issues from one queue to another, such as Drupal core to Migrate Plus and back. |
| heddn |
the reason to move back was against my own personal desire. i was (am) trying to listen to the voice of the community. |
| webchick |
Any insight into why users were demanding it? Having to look in two places makes sense. Is that because some issues were in the old place and some were in the new place? Or because issues were on D.o, MRs were on GitLab? etc. |
| mikelutz (he/him) |
My feelings are about the same. The move back wasn’t that there was something wrong with spoons, just that d.o. picked up some of the important features like MRs and it was nice to not be fractured from the community. |
| heddn |
having the ability to control my own pipelines via gitlab.yml was nice |
| webchick |
This is all SUPER helpful btw, thank you. |
| webchick |
Oh I see. So because the migration system spans both core AND Migrate Plus (or whatever) that created headaches. |
| heddn |
I wrote the parts of the "migration" of issues into gitlab for spoons. We had everything in spoons. It was just that google had indexed d.o and folks had a hard time finding drupalspoons (edited) |
| webchick |
Ahhh. Makes sense. |
| webchick |
Also is that migration code available anywhere? |
| webchick |
And really? You can't cross-reference issues in Gitlab?! |
| heddn |
the UX needed improvement, but about the time moshe and I turned to look into that, d.o suddenly released MRs for core and it took a lot of the wind out of our sails |
| heddn |
you can cross reference inside of gitlab. you can't easily cross reference between gitlab and d.o issues. where d.o issues know about gitlab references |
| webchick |
Gottttcha |
| webchick |
Ok, so if ALL Drupal projects, including core, were to move to GitLab, you'd have a lot fewer complaints it sounds like. (edited) |
| heddn |
its because drupal wasn't all in on gitlab, much of everything was still on d.o. and even simple things like moving an issue from d.o into drupalspoon was complicated because we were in both worlds |
| benjifisher |
The other day, I wrote a wrapper function so that I could use it with the callback process plugin:function mymodule_pathauto_alias ($title) { return \Drupal::service('pathauto.alias_cleaner') ->cleanString($title);} |
| benjifisher |
I used it in a migration that creates Redirect entities like so: redirect_source/path: - plugin: callback callable: mymodule_pathauto_alias source: title_from_csv |
| benjifisher |
Wouldn't it be nice if we could do this instead? redirect_source/path: - plugin: service service: pathauto.alias_cleaner method: cleanString source: title_from_csv |
| benjifisher |
Or maybe, instead of a new process plugin, add yet another option to the callback plugin: redirect_source/path: - plugin: callback service: pathauto.alias_cleaner callable: cleanString source: title_from_csv |
| benjifisher |
Basically, the callback plugin works with global things (functions, static methods). Services are global. |
| benjifisher |
Just recently, we added a way to use callback with more than one argument. It would be nice to have the same option with services. #2882276: Extended callback process plugin to call functions with multiple parameters |
| mikelutz (he/him) |
I don’t like it, but I haven’t come up with a good reason why yet.. |
| benjifisher |
Maybe it is because you respect the ETL paradigm more than I do. Accessing the service container during the transform/process phase breaks that. |
| jibran |
Do we have an issue for this yet? |
| mikelutz (he/him) |
Yes, that's basically it, I think. Also just the difficulties with dependency injection. You either have to store the whole container in the plugin, or break dependency injection. I would rather identify services that might be useful and have a new plugin for each (not in core, obviously) than to have a catch all. |
| huzooka |
@mikelutz (he/him) process plugins also can be derived - so what if we create a clever deriver for a new process plugin? |
| benjifisher |
I do not understand DI well enough. The container is an object, and storing an object just means storing its handle, right? I do not see the problem with that, unless the plugin gets serialized. |
benjifisher, mikelutz (he/him), Joshua Turton (srjosh), webchick, quietone, heddn, alison, gaurav mahlawat, anmolgoyal74, Matroskeen, jibran, huzooka
Comments
Comment #13
quietone commentedComment #14
quietone commentedComment #15
quietone commentedAre there any action items? I don't see any.
Comment #16
quietone commentedNo changes required from following meeting. Closing.