Problem/Motivation

The stage_file_proxy:dl Drush command downloads all missing managed files from the origin server in a tight loop, with no delay between requests. When downloading a large number of files, this can trigger rate limiting on the origin server (e.g. a CDN, reverse proxy, or web server configured to throttle rapid successive requests), causing downloads to fail with client errors and leaving files unretrieved.

Steps to reproduce

  1. Configure stage_file_proxy.settings.origin to point to a production site whose web server or CDN enforces rate limiting on file requests.
  2. Ensure there are many managed files missing locally.
  3. Run drush stage-file-proxy-dl.
  4. Observe that after a number of rapid requests, the origin server starts returning errors (e.g. HTTP 429/503) and files fail to download, logged as errors by the command.

Proposed resolution

Add a new --interval option to the stage_file_proxy:dl command that lets the user specify a delay, in milliseconds, to wait between each file download. When set to a value greater than 0, the command pauses for the given interval after each download attempt, allowing site owners to throttle the request rate and stay within the origin server's rate limits. When left at the default of 0, behavior is unchanged and no delay is introduced.

Remaining tasks

  • Add the interval option to StageFileProxyCommands::dl().
  • Sleep for the configured interval between download attempts.
  • Document the new option in the command's help text.
  • Add/update tests covering the new option, if feasible.

User interface changes

None. This only affects the Drush command-line interface, adding a new optional --interval flag to stage_file_proxy:dl (aliases stage-file-proxy-dl, sfdl).

API changes

No changes to public APIs or services. The change is limited to the Drush command class and does not affect DownloadManagerInterface or other module APIs.

Data model changes

None.

AI was used to draft this issue and write the code for the merge request, but everything was reviewed before submitting.

Issue fork stage_file_proxy-3625234

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

dieterholvoet created an issue. See original summary.

dieterholvoet opened merge request !120

dieterholvoet’s picture

dieterholvoet commented
Status: Active » Needs review
smustgrave’s picture

smustgrave commented
Status: Needs review » Fixed

Since this is just touching the drush command I'm fine with it.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.