Currently users can easily run into HTTP status 429 errors when indexing lots of items with attachments at once since SearchStax imposes a rate limit on its Document Extractor API endpoint.
We should enable users to also set a limit in Drupal to avoid running into these problems and just retry file extraction later.

Issue fork searchstax-3591124

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

drunken monkey created an issue. See original summary.

drunken monkey’s picture

Status: Active » Needs review

  • drunken monkey committed 0ce7dd89 on 1.x
    feat: #3591124 Added rate limiting for attachment extraction.
    
    By:...
drunken monkey’s picture

Merged the MR, will be tested in HEAD.

  • drunken monkey committed 025b6316 on 1.x
    feat: #3591124 Added a warning on the index View tab when files have...
drunken monkey’s picture

Merged a second MR adding a warning message to the index “View” tab if some files are still pending extraction (for already indexed items).

  • drunken monkey committed 6919efff on 1.x
    fix: #3591124 Modified the file extraction warning message to only...
drunken monkey’s picture

Modified the warning message to only contain the total number of queued files since the other number could be incorrect due to #3601364: Queued files in key-value store are not properly deleted.

drunken monkey’s picture

Status: Needs review » Fixed

Released as part of version 1.12.0.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

drunken monkey’s picture

Note that in versions of the Search API Attachments module older than 10.0.3 (without the fix for #3372383: Allow extractor plugin to throw DelayedRequeueException) it was not possible to add a delay when failing extraction of a file. Therefore, when the rate limit is reached the file extraction might be retried immediately. In the worst case scenario, a file extraction might be retried five times almost instantly, always running into the same rate limit, at which point the Search API Attachments module would give up and not try extracting that file anymore.

If possible, it is therefore recommended to update to version 10.0.3 or higher of the Search API Attachments module.

This problem is mitigated by the fact that the same problem would already happen when running into the rate limit of the SearchStax Document Extractor endpoint. Therefore, there would be no additional problem, this new functionality will just not be as efficient in solving the existing problem as would otherwise be the case.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

maskedjellybean’s picture

Thanks for this! Works great as far as I can tell. It cleaned up the logs while indexing.