Problem/Motivation

We need to expand the capabilities of our importer to entity imports. Not sure it's actually simpler, but it feels simpler to start with taxonomy.

Specifically, for this issue, we need to be able to:

  • Import content with existing simple tags
  • Import content with new simple tags

By "simple tags," I mean something like free tags, where content is tagged with specific terms, but the terms do not have their own data (besides the term title itself).

Start by adding this to the existing simple_content_migration we already have. If it is necessary, later spin it off into a new migration.

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

majorrobot created an issue. See original summary.

majorrobot’s picture

I think I've finally figured out how to expand entity relationships in our bundle schemas. I've overridden web/modules/contrib/schemata/schemata_json_schema/src/Normalizer/jsonapi/RelationshipFieldDefinitionNormalizer.php for entity relationships that start with "field_".

A potential issue is that it is recursive, so it could lead to a very large schema -- or perhaps an infinite loop if there is a circular reference.

I also still need to fix up normalizeResponse() in AiMigrator so that it doesn't normalize out the relationships in the AI response.

We should also build in some sort of filter in the migration yml so that we only add fields that are relevant to the schema.

Here is a sample of the schema Drupal is building right now.

majorrobot’s picture

Priority: Normal » Major
Status: Active » Needs review

Finally have an MR for this: https://git.drupalcode.org/project/ai_migration/-/merge_requests/27.

QA Steps:

  1. ddev poser
  2. ddev drush cr
  3. Roll back any previous migrations of the complex_content_migration type.
  4. ddev drush migrate:rollback complex_content_migration
  5. If you do not have any terms in the Subjects taxonomy, add these (capitalization matters!):
    • Fiction
    • Science fiction
  6. In your terminal, run ddev drush migrate:import complex_content_migration
  7. Confirm there are not any errors (crossing fingers). However, you will likely see a warning, which will be dealt with in another ticket: [warning] Referenced entity not found: taxonomy_term--subjects.
  8. Go to the content list and confirm that you have 2 new entries:
    • The Martian Chronicles
    • The Time Machine
  9. Click on Edit next to each entry.
  10. Confirm that each entry has 2 entries for Subjects:
    • Fiction
    • Science fiction

dmundra made their first commit to this issue’s fork.

dmundra’s picture

Assigned: majorrobot » dmundra

FYI here is the command to import the new config changes from the example module.

ddev drush config-import --partial --source=modules/custom/ai_migration/modules/ai_migration_example/config/optional

dmundra’s picture

Tests worked (with gemini) and pipeline is passing. Couple of thoughts:

* I did get an error 'Failed to decode cleaned AI JSON response:' with OpenAPI when I forgot to add the taxonomy terms.
* When fixing the tests, we didn't have test to catch this change was needed https://git.drupalcode.org/project/ai_migration/-/merge_requests/27/diff... when I made the changes in https://git.drupalcode.org/project/ai_migration/-/merge_requests/27/diff...
* Maybe @kducharm wants to do more code review, I might not be the best for deep code analysis

dmundra’s picture

Looks like error still persisted when I re-tried OpenAI with gpt-4.1-mini with a lot of output in the error. It does import one the 'The Martian Chronicles' one but fails on the other.

dmundra’s picture

dmundra’s picture

Assigned: dmundra » kducharm
majorrobot’s picture

@dmundra -- thx for testing and fixing up the tests. I had forgotten to mention adding the taxonomy terms. Do you think we should add an install hook to add those? We don't have instructions dedicated to the ai_migration_example module. We could add directions there, but my guess is that they will be missed. Thoughts?

dmundra’s picture

Assigned: kducharm » dmundra
Status: Needs review » Needs work

Ya automating the creation of taxonomy terms is a good idea just to make it easier to test. Want me to do that?

dmundra’s picture

Assigned: dmundra » Unassigned
Status: Needs work » Needs review

Done. Here are updated QA steps if updating from existing installation:

QA Steps:

  1. ddev poser
  2. ddev drush cr
  3. Run config import to import new content-types we are using to test this migration (install both the install folder and optional folder)
  4. ddev drush config-import --partial --source=modules/custom/ai_migration/modules/ai_migration_example/config/install
  5. ddev drush config-import --partial --source=modules/custom/ai_migration/modules/ai_migration_example/config/optional
  6. Run update database command to run post update hooks for the ai_migration_example module.
  7. ddev drush updb
  8. Roll back any previous migrations of the complex_content_migration type.
  9. ddev drush migrate:rollback complex_content_migration
  10. In your terminal, run ddev drush migrate:import complex_content_migration
  11. Confirm there are not any errors (crossing fingers). However, you will likely see a warning, which will be dealt with in another ticket: [warning] Referenced entity not found: taxonomy_term--subjects.
  12. Go to the content list and confirm that you have 2 new entries:
    • The Martian Chronicles
    • The Time Machine
  13. Click on Edit next to each entry.
  14. Confirm that each entry has 2 entries for Subjects:
    • Fiction
    • Science fiction
dmundra’s picture

I realized it would be good to have the subject terms created on install as well so refactored the code run both on install and post update hook. I tested install hook using tugboat site and tested the update hook locally.

majorrobot’s picture

Assigned: Unassigned » dmundra

@dmundra thx for adding in the taxonomy import!

It works for me when I install the ai_migration_example module from scratch.

With the module already enabled (and taxonomy deleted), I can't get it to work correctly.

Steps:
1. $ ddev drush cr
2. $ ddev drush updb
[success] No pending updates.
3. $ ddev drush config-import --partial --source=modules/custom/ai_migration/modules/ai_migration_example/config/optional

+------------+-----------------------------------------------------------+-----------+
| Collection | Config                                                    | Operation |
+------------+-----------------------------------------------------------+-----------+
|            | node.type.complex_content_migration                       | Update    |
|            | field.field.node.complex_content_migration.field_subjects | Update    |
+------------+-----------------------------------------------------------+-----------+

 ┌ Import the listed configuration changes? ────────────────────┐
 │ Yes                                                          │
 └──────────────────────────────────────────────────────────────┘


In ConfigImportCommands.php line 291:
                                                                                                                                                                                                                                 
  The import failed due to the following reasons:                                                                                                                                                                                
  Configuration <em class="placeholder">field.field.node.complex_content_migration.field_subjects</em> depends on the <em class="placeholder">taxonomy.vocabulary.subjects</em> configuration that will not exist after import.  

It looks like updb does not running the post_update hook for some reason. I tried to see if it was already run, and it looks like it wasn't:

$ ddev drush state:get system.post_update

$

This isn't a use case I'm too concerned about, but there may be a couple people out there who will need to update their locals (including other devs on this project).

So, maybe give it a look?

Something else I had a question about -- why move the subject taxonomy to config/install? If it's in config/optional, there's no error thrown if you already have the taxonomy (which I sometimes do, for instance).

dmundra’s picture

Assigned: dmundra » majorrobot

Right so the issue is you have run config import on the config/install folder first and then run it on the config/optional one for the import to complete. Maybe there is way to run both folders at the same time.

I had to move the subject taxonomy to config/install so support the hook_install adding the taxonomy terms. It failed when it was in config/optional. So might be a order of operations, config/install -> hook_install -> config/optional? Not sure.

Ya not sure why updb didn't run the post update hook as it is supposed to.

dmundra’s picture

Status: Needs review » Fixed

Discussed the taxonomy work. Merging and closing the ticket.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

  • dmundra committed 83796ac1 on 1.0.x authored by majorrobot
    Issue #3549329: Add custom normalizers and denormalizers to shape and...
dmundra’s picture

Status: Fixed » Closed (fixed)