Problem/Motivation

On a site with a large content base, the first usage rebuild (drush entity-usage:recreate, or the batch from the settings form) fails on one chunk and then never recovers. Every following chunk fails too, with a growing "Duplicate entry" error, and each error is logged with the full failed query. On our site one run left a watchdog table of several GB.

Three problems work together:

  1. EntityUsage::bulkInsert() empties $this->inserts only after $query->execute(). The batch worker catches the exception, logs it and moves on to the next chunk, so the rows of the failed
    chunk are still in the queue and are sent again on top of the rows of the next chunk. The query grows at every chunk and keeps hitting the same duplicate primary key, so the failure becomes permanent.
  2. In EntityUsageBatchManager::updateSourcesRevisioned(), $context['sandbox']['current_id'] is set inside the loop over $entity_storage->loadMultipleRevisions(). That method gives no order guarantee, so current_id can end up lower than a revision id already handled. The next query (> current_id) then returns revisions that were already tracked, and their rows are inserted a second time:
    this is what raises the first duplicate key error. It is also set inside the try, so a chunk that throws is replayed for ever.
  3. A failed multi-row insert puts the whole SQL statement and every placeholder in the exception message. One chunk holds thousands of rows, so a single dblog entry can weigh tens of MB.

Steps to reproduce

  1. Install Entity Usage on a site with enough content to need several chunks (the default chunk size is 5000 revisions).
  2. Enable the paragraph and taxonomy_term source types, so the same target is reached by many sources.
  3. Run drush entity-usage:recreate.
  4. The run fails with "Integrity constraint violation: 1062 Duplicate entry ... for key 'PRIMARY'". Every next chunk fails with the same error, and the watchdog table grows very fast.

Proposed resolution

  1. EntityUsage::bulkInsert(): take the rows and reset $this->inserts before running the query, so a failed chunk is not resent with the next one.
  2. EntityUsageBatchManager::updateSourcesRevisioned(): set current_id from the last id of the chunk (the id list is already sorted ASC), outside the try.
  3. Log the exception with the message cut at a fixed length (4096 characters in the patch). The head of the message holds the error itself, the tail is only the query.

Remaining tasks

  • Review.
  • Tests: a kernel test on bulkInsert() that makes the query throw and checks the queue is empty after the call.

User interface changes

None.

API changes

None.

Data model changes

None.

CommentFileSizeAuthor
#4 3622049.png127.78 KBcsakiistvan
Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

julien tekrane created an issue. See original summary.

csakiistvan’s picture

Assigned: Unassigned » csakiistvan
csakiistvan’s picture

Assigned: csakiistvan » Unassigned
Status: Needs review » Reviewed & tested by the community
StatusFileSize
new127.78 KB

✅ Tested and works — replayed the batch worker loop on 8.x-2.x with a script (each chunk: enableBulkInsert(), registerUsage(), bulkInsert(), exception caught), where chunk 2 hits a duplicate key and chunk 3 holds 200 valid rows only. Before the MR chunk 3 failed on chunk 2's row with a 156296-character exception message and left 3 rows in entity_usage; after applying the MR and running ddev drush cr, chunk 2 still fails on its real duplicate but chunk 3 succeeds and the table holds 203 rows, and logBulkException() cuts a 200000-character message to 4134 characters. A full drush entity-usage:recreate gives the same result patched and unpatched.

alexpott made their first commit to this issue’s fork.

alexpott’s picture

Version: 8.x-2.2 » 5.x-dev
Status: Reviewed & tested by the community » Needs work

@julien tekrane - nice find and I agree with the solution.

I'm adding tests and working on a 5.x version too.

alexpott’s picture

Status: Needs work » Needs review
Issue tags: -Needs tests

  • alexpott committed 8705cfe1 on 5.x
    fix: #3622049 Bulk rebuild replays the rows of a failed chunk: duplicate...

  • alexpott committed a21bf73e on 8.x-2.x
    fix: #3622049 Bulk rebuild replays the rows of a failed chunk: duplicate...
alexpott’s picture

Status: Needs review » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.