Problem/Motivation
This is a spin-off from #3195568: [PP-1] Support recursive discovery for migration plugins.
Sometimes (especially for large custom migrations) it might be useful to organize plugin files into some group/sub-directories.
There is no such plugin discovery yet, that allows scanning directory recursively.
My particular use-case: organize a bunch of yml files for the custom migration.
Proposed resolution
Introduce a new YamlRecursiveDirectoryDiscovery
plugin that can scan directories for YAML files recursively.
Note: The plugin should have the ability to ignore some directories.
For example, migrations define the migration state in migrations/state
directory, which should be omitted.
Remaining tasks
Prepare a merge request
Review
Add a change record
Commit
API changes
New discovery plugin to scan yml files recursively.
Issue fork drupal-3199727
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3199727-recursive-yml-discovery changes, plain diff MR !335
Comments
Comment #3
MatroskeenThe basic implementation is ready, but we need the ability to ignore some directories during the discovery.
For example, migrations define the migration state in
migrations/state
directory, which should be omitted.Comment #4
MatroskeenI think it's ready for review.
Comment #5
MatroskeenComment #6
MatroskeenComment #7
andypostThis discovery have no usage in core so no reason to extend api surface, I suggest to use it in contrib and close the issue as Won't fix
Comment #8
MatroskeenBut it might be used for migration plugins, and we can organize some core migrations into d6/d7 directories as was done here: https://git.drupalcode.org/project/drupal/-/merge_requests/281/diffs?com...
Anyway, I see your point. Let it stay here for a while and let me think about it 😊
Comment #9
quietone CreditAttribution: quietone as a volunteer commentedI can hear mikelutz agreeing with #7.
For migrations, if the option for recursive discovery was available migrate would have split migrations into d6 and d7 sub directories. But the time for that has passed.
For contrib, speaking for Commerce Migrate, I don't think there are enough migrations in any of the sub modules to warrant moving them. But that doesn't mean that other contrib or custom work would not find this feature useful. On the other hand, many implementations use migrations as config, so they are in one directory anyway.
Are there any other core systems where this would be useful?
In summary, I think recursive discovery is, in general, a good idea. I am just looking for a way to justify it being in core.
Comment #10
benjifisherThere is an issue somewhere to make it easier to reuse process pipelines. I cannot find the issue right now: maybe it is for the Migrate Plus module, not core.
One option is to create the subdirectory
migrations/process/
in your module and have YAML files in there. Then write a plugin manager that creates process plugins from those YAML files. These plugins would work by creating pipelines from existing plugins.This is just a half-baked idea: I have not done any work to implement it. But I would like to keep this option open. If this issue is implemented in a way that
migrations/process/foo.yml
is used to define a migration plugin instead of a process plugin, then it makes my half-baked idea harder to implement.Comment #11
benjifisherFrom discussion at #3200725: [meeting] Migrate Meeting 2021-03-04 I see that this would not conflict with the idea I brought up in #10. In fact, I see that this issue just provides for the discovery plugin, and #3195568: [PP-1] Support recursive discovery for migration plugins could use it and be configured not to search the
process/
subdirectory. So I have no objection there.Still, I tend to agree with #7. Let's create this in a contrib module. If people find it useful, then we can think about adding it to core.
The issue I was thinking of is #3123534: Create a "pipeline" process plugin to re-use YAML config, which has been migrated to Drupal Spoons.
Comment #12
andypostAnother option could be using yaml 1.2 anchors https://yaml.org/spec/1.2/spec.html#id2765878 to group reusable process plugins
edit example usage https://docs.ansible.com/ansible/latest/user_guide/playbooks_advanced_sy...
Comment #13
heddnI like yaml anchors.
Comment #14
benjifisherUnless I am missing something, YAML anchors make it possible to reuse a bit of YAML within one file. #3123534: Create a "pipeline" process plugin to re-use YAML config is about reusing process plugins in many files.
Again, I may be missing something: #12 and #13 are really about #3123534, not this issue, right?
Comment #16
MatroskeenThe plugin for recursive discovery was moved into a brand-new contrib module: https://www.drupal.org/project/migrate_scanner (code review is appreciated 😋)
If there will be a use-case for Drupal core, I'll be happy to help supply this into core codebase.
Thanks, everyone!