[meeting] Migrate Meeting 2025-02-27 2100Z

Core migration issues

• RTBC
• NR
• all open
• Needs subsystem maintainer review
• Statistics, Google spreadsheet

Next video meeting 2025-04-24

Hello all, it’s time for the biweekly migration subsystem meeting. The meeting will take place in slack in various threads. This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every second Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to. #3456078: [meeting] Migrate Meeting 2024-07-18 2100Z)See the parent issue for an idea of the typical agenda.
➤Transcript will be exported and posted to the agenda issue. For anonymous comments, start with a 👤 emoji. To take a comment or thread off the record, start with a 🚫 emoji.

0️⃣ Who is here today?

1️⃣ What should we talk about today? Suggest topics here and I will add threads. I will also check for comments on the issue for today's meeting.

2️⃣ Action items. To be added later.

3️⃣ Statistics

4️⃣ Comment in this thread if you are looking for ways to contribute. Give us some idea of what you would like to do: documentation, code review, testing, project management, ...

5️⃣ Previous minutes.

6️⃣ Announcements

7️⃣ Convert MigrateSource plugin discovery to attributes

8️⃣ DrupalCon

9️⃣ Entity lookup documentation

🔟 Call for volunteers

:one::one: Editing migrations with eca

benjifisher	Vaporware: let's leverage ECA to get a UI for editing migrations (as migrate_plus.migration.* config entities).
benjifisher	We obviously need this, and it fits with the "ambitious site builder" target audience. The question is how hard it will be to do.
benjifisher	At #florida Drupal Camp, @cosmicdreams brought up an overlapping idea. Clearly, the time is right.
benjifisher	We agree that Step 1 is to generate flowcharts in the style of ECA to visualize migrations.
benjifisher	We disagree about Step 2. I want it to be editing migrations through that UI. @cosmicdreams wants it to be visualizing data flow through the flowchart.
benjifisher	We agree on the end goals. The only question is what to do second.
benjifisher	Once we have a flowchart, maybe we can integrate it with migrate_sandbox for data flow. What do you think, @danflanagan8 ?
benjifisher	How hard is it? At the moment, harder than I had hoped. I started a thread in the #eca channel to discuss it. @jurgenhaas seems interested in refactoring things to make it easier.
benjifisher
benjifisher	But, as I warned at the top of the thread, this is still vaporware.
danflanagan8	I always assumed this must be really hard because I know that there was a lot of work on Migrate UI that stalled real bad. As for Migrate Sandbox, I think it's fine for what it does. But the code's kind of a mess and my gut feeling is that starting from scratch would be smarter than trying to repurpose Migrate Sandbox into something it was never really intended to be.
benjifisher	The ECA module has already solved a lot of the hard problems. It translates an ECS config entity into a flowchart. I should really say that it translates it into a "model" and that the recommended modeller is BPMN.io (as in the bpmn_io module, recommended for eca).
benjifisher	You can use the modeller to inspect and edit the ECA config entity.
benjifisher	How hard can it be to replace "eca config entity" in all that with migrate_plus config entity?
benjifisher	Well, for now the answer is "harder than I had hoped".
cosmicdreams	I still think. that's where we should start.  Let's not compound the complexity of directly using ECA.  Let's use BPMN.IO directly to build out a graph that looks like a migration.We anticipate that we might need to create new components for the Source, Process, and Destination that make up the ETL responsibilities of the migraiton.
jurgenhaas	This indeed sounds like a good starting point. I just wonder what you mean by "create new components for the Source, Process, and Destination". The components in a BPMN based UI are the circle, arrows and rectangles. Are you suggesting changing them? Or is it that you want to map e.g. source plugins to circles, process and destination to rectangles, etc.?
jurgenhaas	When it comes to executing that plan, at least the first part to extract all modeller related functionality from ECA and pushing that into a separate modeller API module, I'd like to build the architecture for that such that we can ensure seamless ECA migration and also get a second "consumer" like migrate to leverage it so that we can prove that it really is generic and functional. I want to be hands-on in that process. But resources and funding are a bit of an issue. Just throwing that in so that you know where we're coming from.
cosmicdreams	Also, @jurgenhaas: I'll try to distill my pitch for how I am envisioning this module for you this weekend.  I can't explain it without visual aids
jcandan	As noted here, I have the start of this ready to push to a project.However, I want to get a better idea from the team here about why a BPMN diagram would be helpful.
jcandan	Diagraming what I’d put together earlier…
jcandan	I realized that there’s no flow to diagram. We may as well just go with forms. (edited)
jcandan	But, if we change the items charted to be the fields…
jcandan	Then a flow chart begins to make more sense. We can even run into conditional or sub-process loop flows.
jcandan	Anybody have any thoughts on this second approach?
benjifisher	I think of the source, process, and destination sections of the flowchart as separate. The interesting "flow" is when we have a process pipeline. It gets most interesting when we use the sub_process plugin. There is also some "flow" (maybe) for nested arrays, like source -> constants -> path -> 'sites/all/files'.
benjifisher	If you look at the ECA module, it supports alternatives to BPMN. One of the options is form-based. My impression is that it is perfectly functional, but no one likes it.
cosmicdreams	@jcandan You can create a sandbox project and push your code there right?  Somewhere somebody suggested a better name for this project that we came up with.  I can't find that now.
cosmicdreams	Ha! it was you.  "Migrate Tools Visualizer"
jurgenhaas	X-posting https://drupal.slack.com/archives/C0287U62CSG/p1741426001023289 here, as I really would like us to consider the benefits of a generic approach with all of the migrate maintainers. I think we have a huge opportunity to build something together which still serves application-specific needs, but will help us to save tons of hours while not having to maintain redundant technologies.
benjifisher	@jurgenhaas: I agree with you :100: , but other people have other ideas and I do not want to discourage them from exploring those ideas. Maybe we can work in parallel tracks, and when the Modeller API is in alpha the other track can decide to start using it. I will also follow up on the thread in #eca
cosmicdreams	I also like this idea.  After spending a little bit of time researching different modeling apps, the complexity isn't in defining a model that the modeler's rendering engine knows how to display, but with doing anything of significance with the result. We want a Modeler API that can: render the kind of diagrams we need A UX paradigm for interacting with all the bits/states/nodes/components/whatever Can allow developers to be in control over these interactions for custom implementations Simple graphing / flowchart making doesn't appear to go far enough
cosmicdreams	If I could wave a magic wand and get whatever I want, I would want a canvas like what XB built that we can create our own set of components for.  The Modeler API would provide: An initial set of components Methods for parsing data into metadata (BPMN uses XML) Methods for parsing the metadata in a diagram Implement admin interactions for each component
benjifisher	At FL DC, @mrconnerton pointed out that BPMN.io is more than a flowchart editor. I think it is possible to design the Modeler API so that it is extensible. It defines a plugin type and a plugin manager. OOTB it gives you some of what you need: displaying and editing. The bmnn_io module implements the plugin type. Then you can extend the classes from bpmn_io and add additional functionality.
benjifisher	I think you are right: one "l" in Modeler.
cosmicdreams	We definitely need to go down the road of using BPMN to its fullest.  I've been exploring using its libraries directly, without Drupal in the mix. So far I have seen a way to create additional components beyond the set provided by the library.  I do recall us talking about that and maybe we won't need to extend.  If it's not an option then we can see if not being able to extend it is really a problem
jcandan	There is no need to reinvent the wheel with novel BPMN.iO components. The real work will be in determining the interaction user experience and what the flow diagram should consist of.

1️⃣2️⃣ Wrap-Up