Taxonomy terms not displayed after migration

@Benia, please add the steps to reproduce this to the issue summary.

It sounds that this is rather a migrate problem than a fields problem no ?

@Benia: Regarding this:

If I go to the taxonomy vocabulary page of every vocabulary whatsoever, and Just hit "Save" --- All of the terms in the vocabulary are deleted.

Is this yet another issue? This would be critical priority.

@Benia: This issue needs a new title and a summary update, with a focus on the most serious aspect, because it will be difficult for people to understand. Have you seen the template and "How to create a good issue"? They help.

The component should be "taxonomy.module".

@Benia with an issue like this, it's most likely to be diagnosed if you're able to provide a copy of the Drupal 6 database and/or the migrated Drupal 8 database. Obviously would need to remove any sensitive information.

Moving to taxonomy module and tagging for migrate for now.

@benia see #22 - a copy of the pre- and/or post-migration databases would be most useful here. mysqldump would be best.

If I go to the taxonomy vocabulary page of every vocabulary whatsoever, and Just hit "Save" --- All of the terms in the vocabulary are deleted.

It isn't clear to me if the D8 taxonomy tables have been checked to see if the data is there. Or, what is missing.

Of course, there is data in these tables

The issue summary says the data is deleted. I want to be sure I understand that point. So, to summarize, if you look in say, taxonomy_term_field_data you see the vocabulary and terms that no longer display since pressing save on a taxonomy vocabulary page. That it, the data is there but not displaying.

The MetaTag patch touches the taxonomy. Perhaps it incorrectly migrated something? Is it possible to rerun the migration with out the MetaTag patch? And then see if the problem persists? Just a thought.

Reviewed in weekly Migrate standup. See https://www.drupal.org/node/2735059 for a schedule of these meetings. Marking postponed for now.

@Benia, it's not possible to 'reset' the storage. Also we can't provide a fix for this without being able to see the data or having clear steps to reproduce the bug.

Can you provide a dump of your database? drush sql-dump --sanitize will remove e-mail addresses etc. Potentially just the term tables might be enough (but note the storage uses multiple tables, not just one).

Also could you check dblog and look for PHP errors on the site, that might also help.

@Benia If you only can share the term tables, make sure you include them all, but, yes.

@Benia a configuration export from your site would probably help too.

Retitling - the terms are not actually deleted (they are present in the database), but they are not displayed in places where the hierarchy table is joined.

Your taxonomy_term_hierarchy table is empty - this strongly suggests this is an instance of #2744639: Taxonomy term hierarchy migration incomplete, which was fixed in 8.1.8. You say:

If I remember correctly I first migrated from Drupal 7.44 to Drupal 8.1.3 (I now use 8.1.8). That's all there is to it, since then the problem started.

Just to be clear - you have not rerun your migration under 8.1.8, your migration was actually last run under 8.1.3? Then that's it, what you have is the remnants of a bug which has been fixed since your migration.

I believe you should be able to fix your database with the following query (be sure to backup your database before trying this!):

Edit: Actually, don't try this query, try the patch in the following comment.

insert into taxonomy_term_hierarchy (tid, parent)
select tid, 0 from taxonomy_term_data
where tid not in (select tid from taxonomy_term_hierarchy)

Actually, forget about the manual query, please apply the attached patch and run db updates on your D8 site - see if that fixes it for you.

Patch reviewers - should the update go into the taxonomy module instead?

Linking to the issue preventing a cleaner update function: #2599450: Drupal\Core\Entity\Query\QueryException: 'parent' not found in Drupal\Core\Entity\Query\Sql\Tables->ensureEntityTable

Which appears to be the focus on this more current issue #2543726: Make $term->parent behave like any other entity reference field, to fix REST and Migrate support and de-customize its Views integration. Looks like solve that issue may solve the one you've linked.

Bumping down to Major - migrate is experimental and not required to do database updates, and we do have the "Migrate critical" tag on this.

Updating the issue summary to reflect the details of what's going on here.

You need to run database updates - what was added was an update function.

drush -y dbupdate

Having trouble finding the D8 administrator docs, with the major doc re-org going on...

Can you verify that the patch actually took? Look at core/modules/migrate/migrate.install and make sure that "function migrate_update_8002()" is in there. If it is... I don't understand why it wouldn't be recognized.

Hmm, I don't understand why drush -y updatedb didn't pick it up - perhaps if you tried through the UI (http://example.com/update.php, where "example.com" is your web address)?

@Benia do you actually have migrate module installed on the site or did you uninstall after the migration?

@catch: Good point - that's an excellent reason to put the update function into the taxonomy module...

@Benia: Please give this patch a try - it puts the update function in the taxonomy module. Be sure to run db updates (drush -y updatedb, or go to /update.php on your server).

Cleaner version of the patch, please skip the previous one.

Shall we move this over to taxonomy's queue then?

Also, this should really use the batch API. Entity save's are expensive and if there are a lot of terms to re-save, this is going to timeout.

Shamelessly stealing from system_update_8200()...

*Sigh* - had temporarily set the limit to 10 because my test db had only 33 terms, restoring the 50...

@Benia - please use the last patch here. Don't forget to run the db updates after applying the patch.

patch -p1 < filename

On a single line. That is the correct command. what you are doing doesn't actually do anything.

Thanks for testing it Benia!

We leave this in "Needs review" now for technical review of the code, so we can get this patch into Drupal core and fix this situation for anyone else who runs into it...

Updated target version of the update function, obviously it's not going into 8.1.x.

@catch: Does this need a test? Note we'd have to munge the db (taxonomy_term_hierarchy table) to create the situation it's fixing.

Thanks @catch for pointing me here. I have previously run migrations prior to #2744639: Taxonomy term hierarchy migration incomplete being implemented so was missing taxonomy_term_hierarchy data and could still not see the terms in the ui. I have applied the patch from #76, run the DB update and confirm it worked successfully. I can now see all my imported terms in the ui under /admin/structure/taxonomy - thank you!

I now just need to figure out how to get the nodes to be associated with taxonomy terms as per the source site - however that is a separate issue.

)-: I've just rolled back all the migrations and done a migrate-import --all and the taxonomy_term_hierarchy has reverted back. I no cannot see the taxonomy terms again.

1. +++ b/core/modules/taxonomy/taxonomy.install
   @@ -0,0 +1,45 @@
   +function taxonomy_update_8200(&$sandbox) {
   

   $sandbox should be type hinted.

2. +++ b/core/modules/taxonomy/taxonomy.install
   @@ -0,0 +1,45 @@
   +  if (!array_key_exists('tids', $sandbox)) {
   

   Should be !isset() && is_array()<code>, because <code>array_key_exists() will return TRUE if the key exists and the value is NULL or some other invalid cruft.

3. +++ b/core/modules/taxonomy/taxonomy.install
   @@ -0,0 +1,45 @@
   +  $tids_to_process = array_slice($sandbox['tids'], 0, 50);
   +  $terms = Term::loadMultiple($tids_to_process);
   +  /** @var Drupal\taxonomy\Entity\Term $term */
   +  foreach ($terms as $term) {
   +    $term->set('parent', 0);
   +    $term->save();
   +  }
   

   If any of the terms throw an exception or otherwise fail when $term->save() is called, $sandbox['tids'] might end up in an uncertain state. A while loop that continually pops term IDs out of the array would make me more comfortable.

4. +++ b/core/modules/taxonomy/taxonomy.install
   @@ -0,0 +1,45 @@
   +  $sandbox['#finished'] = empty($sandbox['tids']) ? 1 : ($sandbox['max'] - count($sandbox['tids'])) / $sandbox['max'];
   

   How is $sandbox['#finished'] used?

Should address 1, 2, 3 in #79.

I'm sure the calculation of #finished can be clarified better though.

Adding interdiff

I've tested this with 1000 taxonomy terms nested three deep by

1) creating them in D7.44
2) running migrate_drupal through the ui to Drupal 8.1.3 and observing the initial problem where terms aren't displayed
3) updating to the code to 8.2.x HEAD plus the latest patch here and running update.php.

The terms are again visible and correctly nested.

Hence putting back to needs review.

+++ b/core/modules/taxonomy/taxonomy.install
@@ -0,0 +1,45 @@
+    array_shift($sandbox['tids']);
+    $term->set('parent', 0);
+    $term->save();

The array_shift() call should be the last thing in the loop, so that it will only be called if the term saves without blowing up.

Other than that, I'm OK with this.

New patch reflecting comments made in #83 and interdiff.

Preemptively RTBC assuming Drupal CI passes everything.

Nice fix and sleuthing to find out the problem. Unfortunately this is exactly the type of upgrade path that needs testing. I think the best way will be to use the bare update and use database methods to insert terms without hierarchy and then run the updates.

Add test for update hook.

Self-assigning for re-review.

1. +++ b/core/modules/taxonomy/src/Tests/Update/TaxonomyUpdateTest.php
   @@ -0,0 +1,76 @@
   +  protected function setUp() {
   

   Missing @inheritdoc comment.

2. +++ b/core/modules/taxonomy/src/Tests/Update/TaxonomyUpdateTest.php
   @@ -0,0 +1,76 @@
   +    $connection->insert('taxonomy_term_data')
   +      ->fields(['tid', 'vid', 'uuid', 'langcode'])
   +      ->values([1, 'test_vocabulary', 'uuid-abc', 'en'])
   +      ->values([2, 'test_vocabulary', 'uuid-xyz', 'en'])
   +      ->execute();
   +    $connection->insert('taxonomy_term_field_data')
   +      ->fields(['tid', 'vid', 'langcode', 'name', 'weight', 'default_langcode'])
   +      ->values([1, 'test_vocabulary', 'en', 'Term1', 0, 1])
   +      ->values([2, 'test_vocabulary', 'en', 'Term2', 0, 1])
   +      ->execute();
   +    $connection->insert('taxonomy_term_hierarchy')
   +      ->fields(['tid', 'parent'])
   +      ->values([1, 0])
   +      ->execute();
   

   I'd like to see a few comments explaining what is being set up here.

3. +++ b/core/modules/taxonomy/src/Tests/Update/TaxonomyUpdateTest.php
   @@ -0,0 +1,76 @@
   +      ->execute()->fetchCol();
   

   Nit: The fetchCol() calls should be on their own line.

4. +++ b/core/modules/taxonomy/src/Tests/Update/TaxonomyUpdateTest.php
   @@ -0,0 +1,76 @@
   +    $this->assertIdentical($term_tids, $term_parent_tids);
   

   Use assertSame(), not assertIdentical().

1. Added @inheritdoc comment.
2. Added comments to explain test set up.
3. Moved fetchCol() onto its own line.
4. No change - it seems there is no assertSame() in Drupal\simpletest\TestBase

No change - it seems there is no assertSame() in Drupal\simpletest\TestBase

Good catch. My bad.

Otherwise, I think this looks fine and dandy. Let us proceed with it. Thanks, @Jo Fitzgerald!

1. +++ b/core/modules/taxonomy/taxonomy.install
   @@ -0,0 +1,45 @@
   +/**
   + * @addtogroup updates-8.2.x
   + * @{
   + */
   +
   +/**
   + * Fix taxonomy terms previously migrated with no hierarchy.
   + */
   +function taxonomy_update_8200(array &$sandbox) {
   

   This at best will make it into 8.3.x

2. +++ b/core/modules/taxonomy/taxonomy.install
   @@ -0,0 +1,45 @@
   +  // We need to use a database query due to https://www.drupal.org/node/2599450.
   

   This should point to https://www.drupal.org/node/2543726

3. +++ b/core/modules/taxonomy/taxonomy.install
   @@ -0,0 +1,45 @@
   +    $query = Database::getConnection()
   +      ->select('taxonomy_term_data', 'td')
   +      ->fields('td', ['tid']);
   +    $query->leftJoin('taxonomy_term_hierarchy', 'th', 'td.tid = th.tid');
   +    $query->isNull('th.parent');
   +    $sandbox['tids'] = $query->execute()->fetchCol();
   +    $sandbox['max'] = count($sandbox['tids']);
   

   It's a shame we have to use a database query and not an entity query.

   Also should we be concerned about the number of tids here potentially? I guess it could be very very long. Maybe we should get a new 50 every time until there is none?

4. +++ b/core/modules/taxonomy/taxonomy.install
   @@ -0,0 +1,45 @@
   +  $tids_to_process = array_slice($sandbox['tids'], 0, 50);
   

   We're debating adding a setting for 50 in ...[issue]

5. Did we consider just running a single query on the taxonomy_term_hierarchy table to update NULL to 0? An advantage of this is that I'm not sure about the Term::loadMultiple() in a hook_update_N()

Doesn't really seem critical at this point - it's been a long time since the bug causing the situation was fixed.

looks like i am having the same issue

Greetings
I am migrating taxonomy from a tab file, The script and everything runs well and all the 279 taxonomies show in the migration status as beeing completed.
But when i go to view my taxonomies in drupal, only 180 of the taxonomies show up in the taxonomy vocabulary. I even tried creating a view to see all the taxonomies but it only shows 180 and i cant find the rest.
Here is a my migration script

id: migrate_taxonomy
label: 'Migrate the Main Taxonomy terms from the csv file'
migration_group: migrate_group
source:
  plugin: csv
  # Full path to the file.
  path: modules/custom/migration_content/source/REF_Taxonomy.TAB
  delimiter: "\t"
  header_row_count: 0
  ids:
    - Taxonomy_ID
process:
  tid: tid
  vid:
    plugin: default_value
    default_value: vocab_taxonomy # vacabulary to populate
  name: FullName # field in cvs to get the actual term names to become the term name
  parent_id:
    - plugin: skip_on_empty
      method: process
      source: ParentID
    - plugin: migration_lookup
      migration: migrate_taxonomy
  parent:
    plugin: default_value
    default_value: 0
    source: '@parent_id'
destination:
  plugin: entity:taxonomy_term

And here is the Tab file sample
Taxonomy_ID		FullName			ParentID
1				Fruit				0
2				mango			1
3				apple			1

I know this is a really old thread, but came across it while looking at some old patches during an update. And it reminded me of another issue that sounds related that I documented here: #2985762: Terms created when override_selector = TRUE don't appear on Terms List/Overview page

People experiencing missing terms after a migration may want to check that out. It has some thoughts on a workaround and a patch, although it's pretty old and not sure if it still applies.

The bug causing this was fixed in July 2016 but this update function was never committed and it would fix long standing migrations.

Moving to migration system since this fix is a fix for a problem caused by migration and the migrate maintainers will see it.

This issue was discussed as a migrate maintainer meeting, #3199420: [meeting] Migrate Meeting 2021-02-25 Item #10. The conclusion was that this can be closed as outdated. Read the minutes for the full discussion.