Problem/Motivation

This is a spin-off from #3195568: [PP-1] Support recursive discovery for migration plugins.

Sometimes (especially for large custom migrations) it might be useful to organize plugin files into some group/sub-directories.
There is no such plugin discovery yet, that allows scanning directory recursively.

My particular use-case: organize a bunch of yml files for the custom migration.

Proposed resolution

Introduce a new YamlRecursiveDirectoryDiscovery plugin that can scan directories for YAML files recursively.

Note: The plugin should have the ability to ignore some directories.
For example, migrations define the migration state in migrations/state directory, which should be omitted.

Remaining tasks

Prepare a merge request
Review
Add a change record
Commit

API changes

New discovery plugin to scan yml files recursively.

Issue fork drupal-3199727

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Matroskeen created an issue. See original summary.

Matroskeen’s picture

The basic implementation is ready, but we need the ability to ignore some directories during the discovery.

For example, migrations define the migration state in migrations/state directory, which should be omitted.

Matroskeen’s picture

Issue summary: View changes
Status: Active » Needs review

I think it's ready for review.

Matroskeen’s picture

Issue summary: View changes
Matroskeen’s picture

Issue summary: View changes
andypost’s picture

This discovery have no usage in core so no reason to extend api surface, I suggest to use it in contrib and close the issue as Won't fix

Matroskeen’s picture

But it might be used for migration plugins, and we can organize some core migrations into d6/d7 directories as was done here: https://git.drupalcode.org/project/drupal/-/merge_requests/281/diffs?com...

Anyway, I see your point. Let it stay here for a while and let me think about it 😊

quietone’s picture

I can hear mikelutz agreeing with #7.

For migrations, if the option for recursive discovery was available migrate would have split migrations into d6 and d7 sub directories. But the time for that has passed.

For contrib, speaking for Commerce Migrate, I don't think there are enough migrations in any of the sub modules to warrant moving them. But that doesn't mean that other contrib or custom work would not find this feature useful. On the other hand, many implementations use migrations as config, so they are in one directory anyway.

Are there any other core systems where this would be useful?

In summary, I think recursive discovery is, in general, a good idea. I am just looking for a way to justify it being in core.

benjifisher’s picture

There is an issue somewhere to make it easier to reuse process pipelines. I cannot find the issue right now: maybe it is for the Migrate Plus module, not core.

One option is to create the subdirectory migrations/process/ in your module and have YAML files in there. Then write a plugin manager that creates process plugins from those YAML files. These plugins would work by creating pipelines from existing plugins.

This is just a half-baked idea: I have not done any work to implement it. But I would like to keep this option open. If this issue is implemented in a way that migrations/process/foo.yml is used to define a migration plugin instead of a process plugin, then it makes my half-baked idea harder to implement.

benjifisher’s picture

From discussion at #3200725: [meeting] Migrate Meeting 2021-03-04 I see that this would not conflict with the idea I brought up in #10. In fact, I see that this issue just provides for the discovery plugin, and #3195568: [PP-1] Support recursive discovery for migration plugins could use it and be configured not to search the process/ subdirectory. So I have no objection there.

Still, I tend to agree with #7. Let's create this in a contrib module. If people find it useful, then we can think about adding it to core.

The issue I was thinking of is #3123534: Create a "pipeline" process plugin to re-use YAML config, which has been migrated to Drupal Spoons.

andypost’s picture

heddn’s picture

I like yaml anchors.

benjifisher’s picture

Unless I am missing something, YAML anchors make it possible to reuse a bit of YAML within one file. #3123534: Create a "pipeline" process plugin to re-use YAML config is about reusing process plugins in many files.

Again, I may be missing something: #12 and #13 are really about #3123534, not this issue, right?

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Matroskeen’s picture

Version: 9.3.x-dev » 10.0.x-dev
Status: Needs review » Closed (won't fix)

The plugin for recursive discovery was moved into a brand-new contrib module: https://www.drupal.org/project/migrate_scanner (code review is appreciated 😋)
If there will be a use-case for Drupal core, I'll be happy to help supply this into core codebase.

Thanks, everyone!