Problem / Motivation

We are processing 25k+ items across multiple Brightcove queues using drush queue:run. During long-running executions the queue workers consistently begin to fail after ~300 seconds.

Logs show that once the execution approaches the 5-minute mark, Brightcove API requests start returning 401 UNAUTHORIZED responses: 

[info] Processing item 842778799 from brightcove_video_delete_queue_worker queue. [283.2 sec, 52.62 MB] [error] Invalid status code: expected 200–299, got 401.

Stopping and restarting the drush queue:run process allows the queue to continue for another ~5 minutes, which indicates the Brightcove access token is expiring during the worker execution and only being refreshed on process restart.

Steps to reproduce

  1. Populate any Brightcove-related queue with enough items for execution to exceed 5 minutes.
  2. Run the queue with: drush queue:run
  3. Observe that after ~300s API calls begin returning 401 UNAUTHORIZED and processing fails until the process is restarted.

Proposed resolution

Detect 401 responses during queue processing and refresh the Brightcove access token in-process so long-running queue runs do not require manual restarts.

Implementation notes / considerations:

  • Refresh the token via the existing Brightcove auth service (or add a small token-refresh helper) when a 401 is received or when the token is near expiry.
  • Retry the failed request after a successful token refresh (with a safe retry limit to avoid tight failure loops).

Issue fork brightcove-3568814

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

pablonicolas created an issue. See original summary.

nickolaj made their first commit to this issue’s fork.

nickolaj opened merge request !36

nickolaj’s picture

nickolaj
Primary language Ukrainian
commented
Status: Active » Needs review

When a 401 error occurs during queue processing, the patch now clears the cached API clients and releases the item back to the queue for retry. This allows fresh tokens to be obtained automatically on the next attempt instead of deleting items when tokens expire.

pablonicolas’s picture

pablonicolas
Location Santa Catarina, Brazil
commented
Status: Needs review » Needs work

Thank you for looking at it @nickolaj!

I tested the changes on my end, and unfurtunatelly it is still an issue:

[info] Processing item 794662306 from brightcove_video_delete_queue_worker queue. [301.33 sec, 65.28 MB]
 [info] Processing item 794662309 from brightcove_video_delete_queue_worker queue. [301.83 sec, 65.39 MB]
 [error]  Invalid status code: expected 200-299, got 401.
[{"error_code": "UNAUTHORIZED", "message": "Permission denied."}] [302.23 sec, 65.4 MB]

But one thing that I noticed after raising this issue is that the 5-minute token won't be a problem if the queue run is triggered by Drupal cron. Since the Queue worker plugin annotation sets the cron interval to 30 seconds, it would never expire at runtime.

The solution for my case was to set a cron job at the server level to trigger the queue run every 5 minutes, with a maximum run time of 250 seconds.