Problem/Motivation

The batch from LocaleFetch::buildUpdateBatch() is used in the installer, when installing modules and theme, when adding a new language, cron automatic translation updates (only when configured, not enabled by default) and when we have stale data about translation updates but need to update translations. So most of the time with locale :)

This adds 4 operations per project and language: version check, status check, download and import. The status check does a HEAD request to the translation server, and if the file was newer the download operation right after that GETs the same file. This separation is for operation reuse I believe, as there is a report page which checks status without download and the update form fetches without re-checking when the status is fresh — but in buildUpdateBatch() (used by the installer, optionally cron and the stale-status update path) the two always happen right after each other which is unfortunate. Network latency could make this really slow.

A Drupal CMS 2.x-dev install in German takes 47 projects, results in 188 batch operations out of which 98 do requests to ftp.drupal.org (50 HEAD + 48 GET). A GET with If-Modified-Since would do both in one request: ftp.drupal.org supports the conditional request.

While looking at this, it rudned out that English on the default server pattern is only flagged as unusable for the GET request but but the HEAD is still sent — that request should also be skipped. Included here for simplicity for now.

Steps to reproduce

Install Drupal CMS in German, check how many HTTP requests it takes.

Proposed resolution

  • Refactor existing duplicated try/catch and redirect handling in checkRemoteFileStatus() and downloadTranslationSource() to a shared private helper.
  • Add a new LocaleFileManager::fetchTranslationSource() which does a conditional GET returning a RemoteFileInfo. Added a new NotModified = 304 enum case, plus included the saved file on success.
  • New LocaleFetch::batchConditionalFetch() replacing the status check (HEAD) + download (GET) pair in buildUpdateBatch()./li>
  • The status-only and fetch-only flows keep their existing operations, unchanged.
  • English + default server pattern no longer makes any request, in both the new operation and batchStatusCheck(). Custom server patterns for English are still checked.

Changes to batch and HTTP requests (Drupal CMS 2.x, German install, 47 projects)

before after
Translation batch operations 188 141
Requests to ftp.drupal.org 98 52

Note that the amount of data transferred does not change much, since the eliminated HEAD exchanges carry only headers, plus the same file bodies are downloaded either way. How much wall time this saves depends entirely on network latency: each removed request is a full round trip plus a TLS connection setup (every batch operation opens a fresh connection!). The halved request and connection load on the translation server applies everywhere, for every use of translation updates though.

Remaining tasks

Review.

User interface changes

None.

Introduced terminology

None.

API changes

Existing methods keep their signatures and behavior.

New public method on two services, one new enum case, one new property on RemoteFileInfo.

Data model changes

None.

Release notes snippet

Localization translation downloads in installation and updates do not run two separate requests to localize.drupal.org anymore (one HEAD, one GET). The new combined GET request makes translated installs and later translation updates faster in case of noticable network latency.

LLM disclosure

LLM was used to diagnose this problem, iterate on solutions and develop results. The results were reviewed and adjusted as a human though.

CommentFileSizeAuthor
#6 3619842-nr-bot_2yqc1leo.txt715 bytesneeds-review-queue-bot

Issue fork drupal-3619842

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

gábor hojtsy created an issue. See original summary.

gábor hojtsy’s picture

Status: Active » Needs review
gábor hojtsy’s picture

Issue summary: View changes
drumm’s picture

Everything that would be good for updates.drupal.org in the related issues would also be good for translation downloads.

It would be good to add 'decode_content' => 'br' to the HTTP request while we are here.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new715 bytes

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.