Problem/Motivation
We need to expand the capabilities of our importer to entity imports. Not sure it's actually simpler, but it feels simpler to start with taxonomy.
Specifically, for this issue, we need to be able to:
- Import content with existing simple tags
- Import content with new simple tags
By "simple tags," I mean something like free tags, where content is tagged with specific terms, but the terms do not have their own data (besides the term title itself).
Start by adding this to the existing simple_content_migration we already have. If it is necessary, later spin it off into a new migration.
Issue fork ai_migration-3549329
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
majorrobot commentedI think I've finally figured out how to expand entity relationships in our bundle schemas. I've overridden
web/modules/contrib/schemata/schemata_json_schema/src/Normalizer/jsonapi/RelationshipFieldDefinitionNormalizer.phpfor entity relationships that start with "field_".A potential issue is that it is recursive, so it could lead to a very large schema -- or perhaps an infinite loop if there is a circular reference.
I also still need to fix up
normalizeResponse()in AiMigrator so that it doesn't normalize out the relationships in the AI response.We should also build in some sort of filter in the migration yml so that we only add fields that are relevant to the schema.
Here is a sample of the schema Drupal is building right now.
Comment #4
majorrobot commentedFinally have an MR for this: https://git.drupalcode.org/project/ai_migration/-/merge_requests/27.
QA Steps:
ddev poserddev drush crddev drush migrate:rollback complex_content_migrationddev drush migrate:import complex_content_migration[warning] Referenced entity not found: taxonomy_term--subjects.Comment #6
dmundraFYI here is the command to import the new config changes from the example module.
ddev drush config-import --partial --source=modules/custom/ai_migration/modules/ai_migration_example/config/optionalComment #7
dmundraTests worked (with gemini) and pipeline is passing. Couple of thoughts:
* I did get an error 'Failed to decode cleaned AI JSON response:' with OpenAPI when I forgot to add the taxonomy terms.
* When fixing the tests, we didn't have test to catch this change was needed https://git.drupalcode.org/project/ai_migration/-/merge_requests/27/diff... when I made the changes in https://git.drupalcode.org/project/ai_migration/-/merge_requests/27/diff...
* Maybe @kducharm wants to do more code review, I might not be the best for deep code analysis
Comment #8
dmundraLooks like error still persisted when I re-tried OpenAI with gpt-4.1-mini with a lot of output in the error. It does import one the 'The Martian Chronicles' one but fails on the other.
Comment #9
dmundraCreated #3561784: Error handling when failing to decode JSON for the error handling.
Comment #10
dmundraComment #11
majorrobot commented@dmundra -- thx for testing and fixing up the tests. I had forgotten to mention adding the taxonomy terms. Do you think we should add an install hook to add those? We don't have instructions dedicated to the ai_migration_example module. We could add directions there, but my guess is that they will be missed. Thoughts?
Comment #12
dmundraYa automating the creation of taxonomy terms is a good idea just to make it easier to test. Want me to do that?
Comment #13
dmundraDone. Here are updated QA steps if updating from existing installation:
QA Steps:
ddev poserddev drush crddev drush config-import --partial --source=modules/custom/ai_migration/modules/ai_migration_example/config/installddev drush config-import --partial --source=modules/custom/ai_migration/modules/ai_migration_example/config/optionalddev drush updbddev drush migrate:rollback complex_content_migrationddev drush migrate:import complex_content_migration[warning] Referenced entity not found: taxonomy_term--subjects.Comment #14
dmundraI realized it would be good to have the subject terms created on install as well so refactored the code run both on install and post update hook. I tested install hook using tugboat site and tested the update hook locally.
Comment #15
majorrobot commented@dmundra thx for adding in the taxonomy import!
It works for me when I install the ai_migration_example module from scratch.
With the module already enabled (and taxonomy deleted), I can't get it to work correctly.
Steps:
1. $ ddev drush cr
2. $ ddev drush updb
[success] No pending updates.3. $ ddev drush config-import --partial --source=modules/custom/ai_migration/modules/ai_migration_example/config/optional
It looks like updb does not running the post_update hook for some reason. I tried to see if it was already run, and it looks like it wasn't:
This isn't a use case I'm too concerned about, but there may be a couple people out there who will need to update their locals (including other devs on this project).
So, maybe give it a look?
Something else I had a question about -- why move the subject taxonomy to config/install? If it's in config/optional, there's no error thrown if you already have the taxonomy (which I sometimes do, for instance).
Comment #16
dmundraRight so the issue is you have run config import on the config/install folder first and then run it on the config/optional one for the import to complete. Maybe there is way to run both folders at the same time.
I had to move the subject taxonomy to config/install so support the hook_install adding the taxonomy terms. It failed when it was in config/optional. So might be a order of operations, config/install -> hook_install -> config/optional? Not sure.
Ya not sure why updb didn't run the post update hook as it is supposed to.
Comment #17
dmundraDiscussed the taxonomy work. Merging and closing the ticket.
Comment #20
dmundra