Orphan term hierarchy records can cause taxonomy_update_8502 to enter an infinite loop

Would you be able/allowed to share a sanitized/anonymized DB dump? Ideally a pre-update backup.

Can you try the patch at #2997960-9: Missing taxonomy hierarchy items in 8.6.0 after running taxonomy_update_8502?

One issue I can see in the update hook is that we might reuse deltas if two terms from the same parent arrive in different batches:

  $tid = -1;

  foreach ($hierarchy as $row) {
    if ($row->tid !== $tid) {
      $delta = 0;
      $tid = $row->tid;
    }

$tid is reset on every batch, so if a previous batch saw the same term ID, we will try to insert delta 0 again.

I think we can prove this in the existing test.

And the fix.

No guarantee of course that this is the original issue, as that sounds like it might be different, especially if the site has less than 100 terms as they should all be processed in one go?

Fancy!

This definitely looks like an improvement, however I agree this is unlikely to fix the issue reported here, since we are dealing with only 5 terms. It might be useful to fix #2997960: Missing taxonomy hierarchy items in 8.6.0 after running taxonomy_update_8502 though. Worth posting over there IMO.

Back to the original report, "Duplicate entry '1-0-0-nl'" feels like this could be the very first row, and so perhaps the update hook has already (partially?) run somehow? Should we truncate the destination table at the start of the first batch, just in case?

If the update is really running twice, we should find out what's going on and fix the root cause, otherwise we might be just hiding a bug taxonomy_update_8502 is uncovering. AFAICT the only way this could be running twice is that taxonomy_update_8502 failed previously, so I guess we would need to know which the original error was.

It would be also good to know whether this is a multilingual site.

In any case, I think we need several inputs from @samuhe before being able to investigate further :)

Same error here. I tried to upgrade to 8.6.0 from the latest 8.5 and got DB errors during upgrade on a dev box. This is what I get with PostgreSQL and drush. The site has around 40.000 taxonomy tags so the update seems to run in batches.

taxonomy module : 
  Clear entity_bundle_field_definitions cache for new parent field settings.
  Clear caches due to updated taxonomy entity views data.
  Add a 'published' = TRUE filter for all Taxonomy term views and converts  existing ones that were using the 'content_translation_status' field.

Do you wish to run all pending updates? (y/n): y
Performing taxonomy_update_8501                                                  [ok]
Performing taxonomy_update_8502                                                  [ok]
Performing taxonomy_update_8502                                                  [ok]
Performing taxonomy_update_8502                                                  [ok]
Performing taxonomy_update_8502                                                  [ok]
Performing taxonomy_update_8502                                                  [ok]
Performing taxonomy_update_8502                                                  [ok]
Performing taxonomy_update_8502                                                  [ok]
SQLSTATE[23505]: Unique violation: 7 ERROR:  duplicate key value violates unique [error]
constraint "taxonomy_term__parent____pkey"
DETAIL:  Key (entity_id, deleted, delta, langcode)=(644, 0, 0, de) already
exists.: INSERT INTO taxonomy_term__parent (bundle, entity_id, revision_id,
langcode, delta, parent_target_id) VALUES (:db_insert_placeholder_0,
...
<-- endless list of placeholders -->
...
:db_insert_placeholder_76799); Array
(
)

Performing taxonomy_update_8502                                                  [ok]
Performing block_content_update_8600                                             [ok]
Performing menu_link_content_update_8601                                         [ok]
Failed: SQLSTATE[23505]: Unique violation: 7 ERROR:  duplicate key value violates[error]
unique constraint "taxonomy_term__parent____pkey"
DETAIL:  Key (entity_id, deleted, delta, langcode)=(644, 0, 0, de) already
exists.: INSERT INTO taxonomy_term__parent (bundle, entity_id, revision_id,
...

I will rerun and investigate on Monday.

@bzrudi71 That looks like it could be the issue I spotted - can you try with the patch from #7 and see if that fixes it?

I am running into same issue here. I have over 70,000 terms in database. I have had to increase the memory limit to 2048M just to get past a allowed memory issue to run this update. I have had to truncate the taxonomy_term_parent table each time before running the update. Now it is failing with - error: Invalid string length.

I used the patch above but same error.

Since I assume there will be a lot of real world users that have older D8 sites with a lot of content that they too will run into this issue. Seems to me that Core maintainers didn't truly test this before releasing 8.6 to the world.

Please advise how to fix this or when a proper patch or update will be available.

Just back from vacation I spend some minutes reading the 8002 taxonomy update code. My guess is the problem is because of a missing orderBy() clause in the select for the hierarchy relation. When doing a select without an order by condition the order is not guaranteed and may cause this issue. Drupal should really throw an exception in case of a select without an order condition ;)
Anyway, I'm back in Office tomorrow and will check and report...

Update: Just discovered related issue #2997960: Missing taxonomy hierarchy items in 8.6.0 after running taxonomy_update_8502. Looks like the right approach to me and should fix this issue as well. Will report results over there...

@chaz.chumley:

You should probably test the patch available at #2997960: Missing taxonomy hierarchy items in 8.6.0 after running taxonomy_update_8502, but could you provide more details about the error: Invalid string length. error? Is it an exception triggered while running taxonomy_update_8502()? Could you post the full message/stack trace?

Since I assume there will be a lot of real world users that have older D8 sites with a lot of content that they too will run into this issue. Seems to me that Core maintainers didn't truly test this before releasing 8.6 to the world.

This is a bold assumption and something very unfair to say. The issues we identified so far were not reliably reproducible or were edge cases, so it is entirely possible that the people that worked on the original issue manually tested the update code without discovering them. And even if you were right, it's not clear to me what's the goal of such a comment, besides blaming the people you're asking support from. There were a lot of 8.6 development releases before 8.6.0 (alpha, beta, RC) that people could have exploited to test the upgrade and report any issues, which would have benefitted everyone in the community much more than complaining about how broken the stable release is.

Most of the issues we identified so far should be addressed by #2997960: Missing taxonomy hierarchy items in 8.6.0 after running taxonomy_update_8502. I'm going to rescope this to specifically figure out what happened in the originally reported case. @longwave and I currently assume that the update function was run twice. To figure out what happened we definitely need more info, I'll update the IS accordingly.

IS updated

Patch in #2997960: Missing taxonomy hierarchy items in 8.6.0 after running taxonomy_update_8502 works for me. Thanks!

Removed, commented on wrong thread

Thanks, here's another report that's similar to what @samuhe experienced: #2997960-51: Missing taxonomy hierarchy items in 8.6.0 after running taxonomy_update_8502.

I ran into this issue when updating to 8.6.1 from 8.5.3 using composer and drush. The first time the update ran drush kept printing out the same line in a loop until I killed it.

 [notice] Update started: taxonomy_update_8502

After killing it and running the update again it would fail with the following log message:

Drupal\Core\Database\IntegrityConstraintViolationException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '19-0-0-en' for key 'PRIMARY': INSERT INTO {taxonomy_term__parent} (bundle, entity_id, revision_id, langcode, delta, parent_target_id) ...

The only way I could figure to get this to run was to truncate the taxonomy_term__parent table and then run the update again. After that the DB update ran smoothly and my site seems to be back up and running correctly.

@johns996 Why did you kill the update during the first run? The info line is printed as long as the taxonomy batch update is running, so for many terms the line may appear hundred of times and this is the normal behavior. Starting a second run will lead to this error and is also expected. So I see no real problem here. Just to make sure I run an update from 8.5 to 8.6.1 without any issue.

Should we file a bug against drush in that it doesn't report batch status very well? It could at least say "continuing" instead of "started" on the second line onwards.

@bzrudi71 I killed my terminal at 50,000 lines of [notice] Update started: taxonomy_update_8502. It seemed like a fairly safe bet that this was an out-of-control loop and not just a normal update. Especially since my site only has 100 nodes and 5 taxonomy terms.

Full disclosure, when I ran the update the second time I didn't use drush and used the update.php script instead. The script only took a few seconds to complete as opposed to the few minutes I was stuck looking a the looping lines on the terminal during my first attempt.

I can roll everything back and try to reproduce the error again since I have a backup.

Yes, please, steps to reproduce are cool :)

I tried to update my Drupal 8.5.7 to 8.6.1 (from tar.gz package and via update.php admin interface), with 3 simple flat taxonomies (all with less than 10 terms) and I found the following error:

taxonomy module
Update #8502
Failed: Drupal\Core\Database\IntegrityConstraintViolationException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '6-0-0-en' for key 'PRIMARY': INSERT INTO {taxonomy_term__parent} (bundle, entity_id, revision_id, langcode, delta, parent_target_id) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5), (:db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11), (:db_insert_placeholder_12, :db_insert_placeholder_13, :db_insert_placeholder_14, :db_insert_placeholder_15, :db_insert_placeholder_16, :db_insert_placeholder_17), (:db_insert_placeholder_18, :db_insert_placeholder_19, :db_insert_placeholder_20, :db_insert_placeholder_21, :db_insert_placeholder_22, :db_insert_placeholder_23), (:db_insert_placeholder_24, :db_insert_placeholder_25, :db_insert_placeholder_26, :db_insert_placeholder_27, :db_insert_placeholder_28, :db_insert_placeholder_29), (:db_insert_placeholder_30, :db_insert_placeholder_31, :db_insert_placeholder_32, :db_insert_placeholder_33, :db_insert_placeholder_34, :db_insert_placeholder_35), (:db_insert_placeholder_36, :db_insert_placeholder_37, :db_insert_placeholder_38, :db_insert_placeholder_39, :db_insert_placeholder_40, :db_insert_placeholder_41), (:db_insert_placeholder_42, :db_insert_placeholder_43, :db_insert_placeholder_44, :db_insert_placeholder_45, :db_insert_placeholder_46, :db_insert_placeholder_47), (:db_insert_placeholder_48, :db_insert_placeholder_49, :db_insert_placeholder_50, :db_insert_placeholder_51, :db_insert_placeholder_52, :db_insert_placeholder_53), (:db_insert_placeholder_54, :db_insert_placeholder_55, :db_insert_placeholder_56, :db_insert_placeholder_57, :db_insert_placeholder_58, :db_insert_placeholder_59), (:db_insert_placeholder_60, :db_insert_placeholder_61, :db_insert_placeholder_62, :db_insert_placeholder_63, :db_insert_placeholder_64, :db_insert_placeholder_65), (:db_insert_placeholder_66, :db_insert_placeholder_67, :db_insert_placeholder_68, :db_insert_placeholder_69, :db_insert_placeholder_70, :db_insert_placeholder_71), (:db_insert_placeholder_72, :db_insert_placeholder_73, :db_insert_placeholder_74, :db_insert_placeholder_75, :db_insert_placeholder_76, :db_insert_placeholder_77), (:db_insert_placeholder_78, :db_insert_placeholder_79, :db_insert_placeholder_80, :db_insert_placeholder_81, :db_insert_placeholder_82, :db_insert_placeholder_83); Array ( [:db_insert_placeholder_0] => section [:db_insert_placeholder_1] => 6 [:db_insert_placeholder_2] => 6 [:db_insert_placeholder_3] => en [:db_insert_placeholder_4] => 0 [:db_insert_placeholder_5] => 0 [:db_insert_placeholder_6] => section [:db_insert_placeholder_7] => 10 [:db_insert_placeholder_8] => 10 [:db_insert_placeholder_9] => en [:db_insert_placeholder_10] => 0 [:db_insert_placeholder_11] => 0 [:db_insert_placeholder_12] => tags [:db_insert_placeholder_13] => 12 [:db_insert_placeholder_14] => 12 [:db_insert_placeholder_15] => en [:db_insert_placeholder_16] => 0 [:db_insert_placeholder_17] => 0 [:db_insert_placeholder_18] => tags [:db_insert_placeholder_19] => 13 [:db_insert_placeholder_20] => 13 [:db_insert_placeholder_21] => en [:db_insert_placeholder_22] => 0 [:db_insert_placeholder_23] => 0 [:db_insert_placeholder_24] => tags [:db_insert_placeholder_25] => 15 [:db_insert_placeholder_26] => 15 [:db_insert_placeholder_27] => en [:db_insert_placeholder_28] => 0 [:db_insert_placeholder_29] => 0 [:db_insert_placeholder_30] => publication_category [:db_insert_placeholder_31] => 17 [:db_insert_placeholder_32] => 17 [:db_insert_placeholder_33] => en [:db_insert_placeholder_34] => 0 [:db_insert_placeholder_35] => 0 [:db_insert_placeholder_36] => publication_category [:db_insert_placeholder_37] => 18 [:db_insert_placeholder_38] => 18 [:db_insert_placeholder_39] => en [:db_insert_placeholder_40] => 0 [:db_insert_placeholder_41] => 0 [:db_insert_placeholder_42] => tags [:db_insert_placeholder_43] => 19 [:db_insert_placeholder_44] => 19 [:db_insert_placeholder_45] => en [:db_insert_placeholder_46] => 0 [:db_insert_placeholder_47] => 0 [:db_insert_placeholder_48] => publication_category [:db_insert_placeholder_49] => 20 [:db_insert_placeholder_50] => 20 [:db_insert_placeholder_51] => en [:db_insert_placeholder_52] => 0 [:db_insert_placeholder_53] => 0 [:db_insert_placeholder_54] => publication_category [:db_insert_placeholder_55] => 21 [:db_insert_placeholder_56] => 21 [:db_insert_placeholder_57] => en [:db_insert_placeholder_58] => 0 [:db_insert_placeholder_59] => 0 [:db_insert_placeholder_60] => publication_category [:db_insert_placeholder_61] => 22 [:db_insert_placeholder_62] => 22 [:db_insert_placeholder_63] => en [:db_insert_placeholder_64] => 0 [:db_insert_placeholder_65] => 0 [:db_insert_placeholder_66] => publication_category [:db_insert_placeholder_67] => 23 [:db_insert_placeholder_68] => 23 [:db_insert_placeholder_69] => en [:db_insert_placeholder_70] => 0 [:db_insert_placeholder_71] => 0 [:db_insert_placeholder_72] => publication_category [:db_insert_placeholder_73] => 24 [:db_insert_placeholder_74] => 24 [:db_insert_placeholder_75] => en [:db_insert_placeholder_76] => 0 [:db_insert_placeholder_77] => 0 [:db_insert_placeholder_78] => publication_category [:db_insert_placeholder_79] => 25 [:db_insert_placeholder_80] => 25 [:db_insert_placeholder_81] => en [:db_insert_placeholder_82] => 0 [:db_insert_placeholder_83] => 0 ) in Drupal\Core\Database\Connection->handleQueryException() (line 683 of /dati/data/virtual/drupal/cms/drupal-8.6.1/core/lib/Drupal/Core/Database/Connection.php).

Was it the first time you attempted the update? Did you experience any error previously? Do you have a DB backup? Would you be able/allowed to share an export of your taxonomy_* tables?

If you're restoring a backup to run the update, make sure you drop all database tables first. It's possible you're running the update into a table that already exists.

I tried a new update, dropping first all DB tables and then restoring from a backup created with Drupal 8.5.6, just before the 8.5.7 update.
This time the update worked correctly, I believe. Here the final messages:

taxonomy module
Update #8502
Taxonomy term hierarchy has been converted to default entity reference storage.
Update #8601
The publishing status field has been added to taxonomy terms.
menu_link_content module
Update #8601
The publishing status entity key has been added to custom menu links.

I was able to reproduce the loop. He's the updates that Drush was about to run before the loop started:

 ------------------- --------------------------------------------- --------------- -------------------------------------------------------------------------------------------------- 
  Module              Update ID                                     Type            Description                                                                                       
 ------------------- --------------------------------------------- --------------- -------------------------------------------------------------------------------------------------- 
  block_content       8600                                          hook_update_n   Add 'reusable' field to 'block_content' entities.                                                 
  dblog               8600                                          hook_update_n   Change 'No logs message available.' area plugin type.                                             
  media               8600                                          hook_update_n   Updates media.settings to support OEmbed.                                                         
  menu_link_content   8601                                          hook_update_n   Add the publishing status entity key to custom menu links.                                        
  taxonomy            8501                                          hook_update_n   Convert the custom taxonomy term hierarchy storage to a default storage.                          
  taxonomy            8502                                          hook_update_n   Copy hierarchy from {taxonomy_term_hierarchy} to {taxonomy_term__parent}.                         
  taxonomy            8503                                          hook_update_n   Update views to use {taxonomy_term__parent} in relationships.                                     
  taxonomy            8601                                          hook_update_n   Add the publishing status fields to taxonomy terms.                                               
  webform             8137                                          hook_update_n   Issue #2983137: WYSIWYG editor not saving inline images for advanced htmltext editor.             
  webform             8138                                          hook_update_n   Issue #2947303: Location element's geocomplete library is not supported.                          
  webform             8139                                          hook_update_n   Issue #2918669: Elements separated by conditional inside multistep wizard are both saved to       
                                                                                    submission.                                                                                       
  webform             8140                                          hook_update_n   Issue #2994411: Allow help to be hidden.                                                          
  webform             8141                                          hook_update_n   Issue #2993327: Provide Default View to Override Submission Results Tab.                          
  webform             8142                                          hook_update_n   Issue #2995413: Allow email handler theme to be customized.                                       
  webform             8143                                          hook_update_n   Issue #2943879: How to display alternate text when the user is not allowed to create a Webform?   
  block_content       add_views_reusable_filter                     post-update     Adds a 'reusable' filter to all Custom Block views.                                               
  image               scale_and_crop_effect_add_anchor              post-update     Add 'anchor' setting to 'Scale and crop' effects.                                                 
  media               storage_handler                               post-update     Clear caches due to the addition of a Media-specific entity storage handler.                      
  system              change_delete_action_plugins                  post-update     Change plugin IDs of delete actions.                                                              
  system              extra_fields                                  post-update     Update all entity displays that contain extra fields.                                             
  system              language_item_callback                        post-update     Force cache clear for language item callback.   @see https:www.drupal.orgnode2851736              
  taxonomy            clear_entity_bundle_field_definitions_cache   post-update     Clear entity_bundle_field_definitions cache for new parent field settings.                        
  taxonomy            clear_views_data_cache                        post-update     Clear caches due to updated taxonomy entity views data.                                           
  taxonomy            handle_publishing_status_addition_in_views    post-update     Add a 'published' = TRUE filter for all Taxonomy term views and converts  existing ones that      
                                                                                    were using the 'content_translation_status' field.                                                
 ------------------- --------------------------------------------- --------------- --------------------------------------------------------

things eventually died with an out-of-memory error

 [notice] Update started: taxonomy_update_8502
 [notice] Update started: taxonomy_update_8502
 [notice] Update started: taxonomy_update_8502
 [notice] Update started: taxonomy_update_8502
 [notice] Update started: taxonomy_update_8502
[error]  Drush command terminated abnormally due to an unrecoverable error.
Error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 15728672 bytes) in /htdocs/DrupalDev/vendor/drush/drush/includes/backend.inc, line 249 

I backed up my taxonomy tables before I ran the update and put the dump in a gist.
https://gist.github.com/johns996/d76e0605bb28a9d2879e0c786342f2e1

Hopefully this helps.

After testing a bit more with this issue, I discovered two things. After I run the update and have it loop and die, even if I truncate the taxonomy_term__parent table and run the updated again it still does the loop. Looking into my taxonomy_term_hierarchy table I see a row that does not line up with my other taxonomy tables:

INSERT INTO `taxonomy_term_hierarchy` (`tid`, `parent`) VALUES
(4, 0),

Once I pull that row out, truncate the taxonomy_term__parent table and run the update again the loop does not happen and the update finishes.

Well spotted!

So, if for some reason there are orphaned records in taxonomy_term_hierarchy, $sandbox['max'] will count these, but the later query uses an inner join and will disregard the orphans, so the batch will never appear complete and this update will loop forever.

The attached patch fixes this by including the join in the count query. The orphaned records are useless and will be safely deleted at the end of the batch.

@johns996 Can you test this patch against your original database and let us know if it completes successfully?

So this only happens with a bad data set, but since we don't know what's causing that bad data set in the first place or how many sites it affects yet, bumping to critical.

Great catch!

+++ b/core/modules/taxonomy/taxonomy.install
@@ -32,7 +32,9 @@ function taxonomy_update_8502(&$sandbox) {
+    $select->join('taxonomy_term_data', 'd', 'h.tid = d.tid');

Can we add a comment about why we are adding this?

I'm experimenting the same issue, using a clean DB dump from my previous Drupal 8.5.7 install and I'm getting this error on drush updb

Failed: SQLSTATE[23000]: Integrity constraint violation: 1062 : INSERT INTO {taxonomy_term__parent}

The patch at #34 is not working for me.

@idiaz.roncero When you restore the 8.5.7 database, are you sure that the taxonomy_term__parent table is being deleted? It should not exist in the dump, so simply restoring over the top will not work. If this is still failing with the table deleted, can you share your taxonomy_term_data and taxonomy_term_hierarchy tables?

@longwave Your patch worked perfectly against my original database and no loop was encountered.

Cool, we just need some additional test coverage then (and maybe adding a comment to the new join line ;).

Not sure how an infinite loop will fare on the testbot, let's see.

Hm, so that didn't do what I expected!

I don't think we need to reproduce the infinite loop, we just need to assert that the orphan was not migrated

I don't think we need to reproduce the infinite loop

I disagree. I think we need a test-only patch that includes an orphan and fails. And given HEAD's current code, when there's an orphan, the failure is that the batch never finishes. It's interesting that #41's test-only patch passes. I wonder why it does.

The unexpected pass in #41 is a misunderstanding of the data structures in the test fixture, there are actually four terms created. This version seemed to loop forever when I ran it locally.

Spurious fail so I queued a retest

Nice work!

Adding issue credit to people who helped resolve the currently titled issue.

Pushed to 8.7.x and 8.6.x. Thanks!

An issue summary update would still be great for future reference, if anyone feels up to writing it.

@idiaz.roncero: Re #37, I think you were running into #2997960: Missing taxonomy hierarchy items in 8.6.0 after running taxonomy_update_8502 rather than this issue. The patch there was committed and released in 8.6.1 before your comment though. Did you try with 8.6.1?

I believe the issue reported originally and by other people here was caused by the fact that the update was attempted multiple times without truncating the taxonomy_term__parent table in between attempts.

Issue summary updated

Crossposted with #56. Removed the following bit from the proposed solution as we did not implement it, and hopefully it's no longer needed now.

Inform users that they need to truncate or delete the taxonomy_term__parent table before retrying the update.

Are we going to tag a new release?

Not my call, but I don't think so, this is not causing data loss AFAICT.

This will go out no later than the October 3rd window whatever happens.

johns996 is as far as I can tell the only person who has confirmed orphaned records in taxonomy_hierarchy - the other reports here are not necessarily the same issue (some look like attempted repeat updates with the new database table not removed for example).

If we get further confirmed reports of sites hitting this issue with orphaned records, I could see releasing 8.6.2 out of cycle, but otherwise I think this can probably wait for October 3rd.

I just upgraded from 8.5.x to 8.6.1. when I run the update.php, still got this kind of ERROR message. (See Attached)

Maybe the Error is caused by the table name, I found there are two table name: "taxonomy_term_hierarchy" and "term_hierarchy".

the error message showed "taxonomy_term_hierarchy", but in my database, I see the table name is "term_hierarchy".

@hellobank: thanks for reporting that.

in my database, I see the table name is "term_hierarchy"

Did you migrate your site from Drupal 6 directly, skipping Drupal 7, or did you go from 6 -> 7 -> 8? Also, what version of Drupal 8 did you initially migrate into?

yes, migrated from D6 -〉D8 directly.

Not quite sure the initially migrate version, maybe 8.2.x.

Before migrate to 8.5,6 to 8.6.1 I have performed this two query on the database to delete orphans taxonomy_term_hierarchy record.

• DELETE `taxonomy_term_hierarchy` FROM `taxonomy_term_hierarchy` left join `taxonomy_term_data` ON `taxonomy_term_hierarchy`.`tid` = `taxonomy_term_data`.`tid` WHERE `taxonomy_term_data`.`tid` IS NULL
• DELETE `taxonomy_term_hierarchy` FROM `taxonomy_term_hierarchy` left join `taxonomy_term_data` ON `taxonomy_term_hierarchy`.`parent` = `taxonomy_term_data`.`tid` WHERE `taxonomy_term_hierarchy`.`parent` >0 AND `taxonomy_term_data`.`tid` IS NULL

After that I have executed "drush up -y" and after some messageas "Performing taxonomy_update_8502" the update end.
I think thi is the best possible from this update.

What I've done to restore our terms:

1. Apply patch 2997982-45.patch from #45
2. Truncate the taxonomy_term__parent:
   drush php-eval "\Drupal::database()->truncate('taxonomy_term__parent')->execute();"
3. Set the install schema version of the Taxonomy module back to 8500:
   drush php-eval "drupal_set_installed_schema_version('taxonomy', 8500)"
4. Run updates including --entity-updates:
   drush updb --entity-updates

#67 works for restore my terms, thanks.

PS: with drush php-eval "\Drupal::database()->truncate('taxonomy_term__parent')->execute();" (without the ")" at the end. )

Seems like it would be a good idea to have a patch that runs those two commands from #67. Maybe even patches the update that needs to re-run (according to that post). Though this seems a bit like a chicken vs egg conundrum.

I created another issue regarding this update in case anyone is experiencing that as well: #3005162: taxonomy_update_8601 failing

I haven't used drush yet. Isn't it possible to give a fast database hack insteed of "drush". Or refix it in the next version?

In response to #68, thanks, I've updated my original post to prevent problems for future visitors.

Patch from #45 works with instructions from #67, thanks guys.

Included is a patch adds the truncate table line to taxonomy_update_8501() as suggested in #69. I left out the schema version line, since that seems hacky and doesn't belong in an update hook, in my opinion. Add this patch after the one from #45.

NOTE: This is only necessary if the update failed before, since the taxonomy_term__parent did not exist prior to 8.6.1. A drush sql-drop and a 'clean' 8.5.4 DB backup did the trick for me (with patch #45).

Can confirm patch from #73 above works, thanks!

in #73 taxonomy_term__parent table is truncated on every batch step,
so for example, if taxonomy_term_hierarchy table had 1050 rows, and each batch step pick 50 rows, taxonomy_term__parent will have 50 rows at the end of execution.

TL;DR, in short:

1. cd /path/to/docroot/
2. wget https://www.drupal.org/files/issues/2018-09-12/2997982-45.patch
3. wget https://www.drupal.org/files/issues/2018-10-17/2997982_truncate_taxonomy...
4. git apply 2997982-45.patch
5. git apply 2997982_truncate_taxonomy_term__parent_table.patch
6. drush cr
7. drush updb

I'm super confused - did this make it into 8.6.2?
I'm going from 8.5.4 to 8.6.2 and still experiencing this issue.
Am I missing something?
Do I need to go to latest 8.5.x first?

If the issue is marked "fixed", and i don't have a failed update, seems like i should be able to follow the standard update procedure, no?

No, 8.6.2 was a security-only release. You will have to wait for 8.6.3 (assuming it is not another security-only release) or use 8.6.x-dev.

Thanks @longwave

To clarify: if i'm hitting this bug when updating from 8.5.4 (or 8.5.6), is it safe to go to 8.5.8?
Then, can i go from 8.5.8 to 8.6.2 without any core patches?

I'm still having this issue with 8.6.3 even though it is in the change log. I'm guessing that #73 didn't make it in?

Adding the patch in #73 to 8.6.3 does fix the issue for me though.

@aaronbauman I am not quite clear what you are asking here, but you should be able to go to 8.6.3 directly now and solve this issue for good. Please make a database backup first!

@jaarong To be clear when you say "this issue" do you mean the infinite loop or the "Duplicate entry"? The "Duplicate entry" implies that the update hook had already partially run and that you had not deleted the taxonomy_term__parent before reverting to the older database, which should not exist in any 8.5.x database backup.

Sorry, I wasn't aware there were separate issues. In typing my reply, I remembered that I had tried the update on production with 8.6.1. It deleted some of my taxonomy terms or at least removed them from view. I reverted to 8.5.7 and everything appeared to be normal. But let me sum up what I am seeing:

I am running 8.5.8 and all taxonomy terms are present.

While updating to 8.6.3 without #73, I do get the duplicate entry error referenced in this issue and it won't complete the update.

Updating to 8.6.3 (with #73), updb completes successfully, about half of my terms are removed (some in a hierarchy, some not).

I'm not sure what to do at this point, and I'm very glad that security fixes are being offered for 8.5 till May 2019. So let's assume that I somehow ran the update_hook in a previous 8.6 version, is there a way forward for me that doesn't include restoring a database backup that is now 2 months old?

I don't have that many terms and don't have lots of content assigned to the terms deleted, so if I had to do this manually, my preference would be to just recreate the terms and assign them back to the content. That being better than restoring a 2 month old db and losing a lot of content I have entered. But will the db be ok at that point for future updates?

about half of my terms are removed

Removed or unpublished? Note that there's also #2998221: Primary language taxonomy terms unpublished during 8.6.0 upgrade.

The patch in #45 went out with Drupal 8.6.3.

This does not fix the 'duplicate error' because that is caused by re-running an update on a database that's already been updated.

You can run 'TRUNCATE taxonomy_term__parent;' from the MySQL cli to empty that table before attempting to re-run the update (this is what the patch in #73 does), or ideally restore a clean backup of your 8.5.x database and run updates on that.

I don't have multiple languages, but one direction that thread did take me in is making a view to show all of my terms. Oddly enough, all of them are still there. They are published according to views. They simply don't show up in the vocabulary list. They are associated with the proper content, but when you edit that content node, the term isn't there.

Saving the term seems to restore the term to the vocabulary page and restore the term on the node edit form. Hierarchy no longer exists for the restored terms, but in my case, that isn't going to take that long to fix.

Saving all terms through Views Bulk Operations also works to get them to reappear. It also looks like maybe the previous 8.6 try unchecked auto-alias on all terms that existed at the time.

All in all, I'm hoping this is all that it takes to restore this to what it was. Maybe someone could let me know if the above all sounds relatively normal and that saving the terms again is something that should fix the problems above?

Edit: Again, this is after updating with 8.6.3 and patch from #73 above.

This patch worked for me after I truncated the taxonomy_term__parent table that was created during the first attempt at upgrading. The upgrade replaces the taxonomy_hierarchy table with taxonomy_term__parent.

Updating to 8.6.3 (with #73), updb completes successfully, about half of my terms are removed (some in a hierarchy, some not).

#73 can make most of taxonomy hierarchy items lost when processing in multiple batches.

This problem is due to $database->truncate('taxonomy_term__parent')->execute(); runs on each batch.

2019-03-06T12:07:31.838507Z  8 Query  TRUNCATE taxonomy_term__parent
2019-03-06T12:07:31.879301Z  8 Query  INSERT INTO taxonomy_term__parent (bundle, entity_id, revision_id, langcode, delta, parent_target_id) VALUES ...
2019-03-06T12:07:31.888240Z  8 Query  TRUNCATE taxonomy_term__parent
2019-03-06T12:07:31.902783Z  8 Query  INSERT INTO taxonomy_term__parent (bundle, entity_id, revision_id, langcode, delta, parent_target_id) VALUES ...
2019-03-06T12:07:31.908652Z  8 Query  TRUNCATE taxonomy_term__parent
2019-03-06T12:07:31.925778Z  8 Query  INSERT INTO taxonomy_term__parent (bundle, entity_id, revision_id, langcode, delta, parent_target_id) VALUES ...
2019-03-06T12:07:31.985031Z  8 Query  TRUNCATE taxonomy_term__parent
2019-03-06T12:07:32.002145Z  8 Query  INSERT INTO taxonomy_term__parent (bundle, entity_id, revision_id, langcode, delta, parent_target_id) VALUES ...
2019-03-06T12:07:32.010066Z  8 Query  TRUNCATE taxonomy_term__parent
2019-03-06T12:07:32.023794Z  8 Query  INSERT INTO taxonomy_term__parent (bundle, entity_id, revision_id, langcode, delta, parent_target_id) VALUES ...
2019-03-06T12:07:32.032689Z  8 Query  TRUNCATE taxonomy_term__parent
2019-03-06T12:07:32.047611Z  8 Query  INSERT INTO taxonomy_term__parent (bundle, entity_id, revision_id, langcode, delta, parent_target_id) VALUES ...

So use this one instead of #73.