Problem/Motivation
When the source data structure is somehow controlled, such as might be the case with migrations such as Migrate Google Sheets, the ability to dynamically derive fields based on the source data instead of hand-crafting all the fields can be an incredibly useful bit of automation.
This is another issue inspired from the same use case that brought about #2780965: Add support for original data values with JSON data_parser plugin
Proposed resolution
Create a new trait that can be included in source plugins and leveraged to find a usable intersection between source and destination data models, and automatically configure the row & migration to pull that data in.
Remaining tasks
Generalize the field gathering mechanism to be entity neutral. It is currently tied to nodes.
User interface changes
None.
API changes
None.
Data model changes
None.
Comment | File | Size | Author |
---|---|---|---|
#8 | dynamically_determine-2789699-8.patch | 5.14 KB | tjh |
#7 | dynamically_determine-2789699-7.patch | 5.11 KB | tjh |
#4 | dynamically_determine-2789699-4.patch | 5.04 KB | tjh |
#2 | dynamically_determine-2789699-2.patch | 4.79 KB | Grayside |
Comments
Comment #2
Grayside CreditAttribution: Grayside at Phase2 for Norwegian Cruise Line commentedThis is the version that I mention is a bit entity specific to nodes.
Comment #3
Grayside CreditAttribution: Grayside at Phase2 for Norwegian Cruise Line commentedSomething like this functionality is also implemented as part of Migrate Default Content, further illustrating the usefulness of a common facility.
Comment #4
tjh CreditAttribution: tjh at Phase2 commentedUpdated DynamicFieldTrait to be open to entities that aren't nodes.
Comment #5
mikeryanI... am not a fan of this at all.
First off, there needs to be a clearer issue summary - I thought this had something to do with identifying source fields, but what it's actually trying to do is map source fields to destination fields, which is not the least bit clear.
Secondly, this involves the source plugin examining the destination plugin in order to edit the process pipeline of the migration dynamically - not only that, it's doing it over and over for every row. And it only generates direct mappings - more complex fields, or references to other migrations, may need more processing than a simple get.
I think the best way to do automatic mapping of fields is statically - a one-time operation altering/generating a migration configuration entity with the automatic mappings. If/when #2470920: Implement the field mapping editor gets done, we could add a "Generate default mappings" button or somesuch to align matching field names.
Comment #6
Grayside CreditAttribution: Grayside at Phase2 for Norwegian Cruise Line commentedThe use case for this is most common in content staging use cases where your source data is either from a replicate of the destination or is at least modeled as a near replica. We are successfully using this to create bundle-neutral migrations. Generated mapping configurations sounds really useful, but it doesn't address this problem in the same way.
https://www.drupal.org/project/migrate_google_sheets is another project using something to dynamically derive the available fields.
Tying this to the destination plugin is mostly done in the name of further stability, making sure there are no extra fields in the source data we should simply ignore. That part could be removed if we want to trust the data.
Comment #7
tjh CreditAttribution: tjh at Phase2 commentedNeeded to determine the entity's bundle as well inside of DynamicFieldTrait::filterToDestinationProperties(), rather than hardcode to type. 'type' is the bundle key for nodes, vid is the bundle key for taxonomy term entities.
Comment #8
tjh CreditAttribution: tjh at Phase2 commentedUpdated patch actually calls the row's bundle correctly.
Changes:
To:
Comment #9
jhchnc CreditAttribution: jhchnc as a volunteer commentedWell, this is probably sloppy, but it took minimal time to implement, and might be what a future Google searcher is seeking:
In a nutshell: read your YAML file in PHP, and do something like this: