Problem/Motivation

The generation batch uses a numerical position offset within the result set, which means that if one or more already-processed/hashed entities happen to be deleted while the batch is running, then that number of items could be skipped by the batch process.

Steps to reproduce

Haven't actually run into this, more planning to enable and run the batch in an active site. Hypothetically, should be able to trigger with:

  1. Create two managed file entities
  2. Enable/configure filehash
  3. Introduce a pause at the end of the batch process callback
  4. Execute the hash generation batch
  5. While in the pause, after processing the first entity, delete the first entity

The second entity would not presently get processed.
The actual failure mode appears to be a non-terminating loop: The DB query should just return an empty result set which we will iterate over forever while returning that we are not $context['finished'] because we do not further increment $context['results']['processed'] and $context['sandbox']['count'] is stale (as is the $context['results']['processed'], since it is counting that it processed something that is no-longer in the system).

Proposed resolution

Track and use the last fid to offset in the query. The query is already ordered by fid.
Exit the batch when the query returns an empty result set.

Remaining tasks

User interface changes

API changes

Data model changes

Issue fork filehash-3543131

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

adam-vessey created an issue. See original summary.

adam-vessey’s picture

Status: Active » Needs review

MR together; should address, I think?

mfb’s picture

I think this should also be query a optimization when there are tens of thousands of files - it can jump to the next fid rather than scanning through a large row offset.

  • mfb committed c9baff86 on 3.x authored by adam-vessey
    Issue #3543131 by adam-vessey: Hash generation batch can skip items,...
mfb’s picture

Status: Needs review » Fixed

Now that this issue is closed, please review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, please credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.