Problem/Motivation

In issue 3613098, failures on import were surfaced to the user. This issue will address why the failures are happening in the first place.

Steps to reproduce

  • Check out bibcite_crossref from the issue 3613098 merge request
  • Import 12-20 references using a text file and bibcite_import
  • Note the number of imported items reported after, and if less than the amount in the file, the error messages presented (likely 429 or 503 Too Many Requests).

Proposed resolution

Stay under Crossref’s published rate limits to avoid “Too Many Requests” error.

If no email is provided in the module config, then stay under 5 requests per second. If an email is provided and passed along to the request, then stay within 10 requests per second.

Remaining tasks

Code, test, have reviewed.

User interface changes

NA.

API changes

NA.

Data model changes

NA.

Issue fork bibcite_crossref-3613388

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

byronveale created an issue. See original summary.

byronveale opened merge request !7

byronveale’s picture

byronveale commented
Status: Active » Needs review

Opened Merge Request, the 429 came from CrossrefClient::request() firing DOI lookups back-to-back with no rate limiting. This update, authored with assistance from Claude Opus 4.8, makes the client respect Crossref's REST API limits with:

  1. Proactive throttling (prevents 429s): Each response's X-Rate-Limit-Limit / X-Rate-Limit-Interval headers are read to derive a minimum spacing between requests (e.g. 50/1s → 20 ms). throttle() sleeps as needed before each request so the import stays under the advertised rate.
  2. Retry with backoff (recovers from 429/503): On a 429 Too Many Requests or transient 503, the request is retried up to MAX_ATTEMPTS (5), honoring the Retry-After header (seconds or HTTP-date) when present, otherwise exponential backoff capped at 30s. After exhausting retries it rethrows, so the failure still surfaces through the reporting added earlier.

The polite-pool mailto (which raises Crossref's limits) was already sent and is preserved.

Verified via a Guzzle MockHandler harness: retry-after-429 succeeds, sequential requests are correctly spaced (1.51s for 4 requests at 2/s), and persistent 429s throw after 5 attempts. php -l and phpcs (Drupal, DrupalPractice) are clean; the existing kernel test passes.

Note: fetching still happens synchronously during form validation, so throttling adds wall-clock time (small per request). For very large DOI lists, moving the fetch into the import batch would be the next structural improvement.

byronveale’s picture

byronveale commented
Assigned: byronveale » Unassigned

Forgot to note, this depends on a commit from the issue #3613098 Merge Request, so I made the current Merge Request dependent on that one.

bkosborne’s picture

bkosborne
Primary language English
Location New Jersey, USA
commented
Status: Needs review » Needs work

I found that Crossref publishes their rate limits, I think the logic here should be simplified to follow the published limits.

If no email is provided in the module config, then ensure we stay under 5 requests per second. If an email is provided and passed along to the request, then stay within 10 requests per second.

Then we don't need to worry about retry logic.

byronveale’s picture

byronveale commented
Title: Avoid import errors using proactive rate-limiting and retry-with-backoff » Avoid import errors using published Crossref rate limits
Assigned: Unassigned » byronveale
Issue summary: View changes

As per @bkosborne’s comment, simplifying this by staying under Crossref’s published rate limits.

bkosborne’s picture

bkosborne
Primary language English
Location New Jersey, USA
commented
Status: Needs work » Reviewed & tested by the community

Looks good now. I tested with 50 entries in a file and didn't get any rate limit errors.

I did notice a possible issue which the way this is all set up. The import process uses Drupal's batch API and the batches are chunked so it imports 50 entities per operation. The issue we have is that decoding the DOI numbers is not part of that batch process, it happens before that. So if the user tries importing 500 items from one DOI file, it will take a minimum of 100 seconds just to decode all the DOIs because of the rate limiting. That's 100 seconds the form submission will be "pending" by the browser. It's of course half of that if the user provides an email since we can double the rate limit. But still.

In any case, that again can be resolved in some follow up, probably in a refactor of how the Bibcite Import module works to make the decoding process part of the batch operations, too.

byronveale opened merge request !8

byronveale’s picture

byronveale commented

Thanks @bkosborne. Opened new MR to get 3613098-report-import-errors commit in here.

  • byronveale committed 1856eac2 on 2.0.x 
    feat: #3613388: avoiding 429 or 503 Too Many Requests errors with simple...
byronveale’s picture

byronveale commented
Status: Reviewed & tested by the community » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.