Problem/Motivation

With many languages and projects, checking for translation updates can get extremely slow, resulting in thousands of separate requests to drupal.org.

There are separate batch operations, each doing one thing for source (project + langcode). One does a HEAD request to get the current timestamp, then another does a download if the remote timestamp is larger.

Steps to reproduce

Proposed resolution

Different ideas were discussed in this slack thread: https://drupal.slack.com/archives/C51GNJG91/p1781688203931809

  • new data structures on drupal.org, for example a single JSON file per project that contains the timestamps for each language. Requires changes to localize.drupal.org, could significantly reduce those HEAD requests from projects * langcodes to just count(projects). Single-language sites or operations like adding a new language would need to fetch more data and not just the headers from the HEAD request. But the underlying files could possibly be cached for a certain time so that adding two languages in short order would be able to reuse a local cache. It was also pointed out that a new remote resource for this would be incompatible with different translation services, which can be set globally or per-project, so this would need to check if the current request is for drupal.org.
  • using async/concurrent guzzle requests to group the requests together. See https://stackoverflow.com/questions/60603792/is-it-possible-to-setup-guz.... Requires completely refactoring the batch operations to do them in groups. concurrency needs to be set, recommendation is to keep this low by @drumm. The API to download/check is also on a different service and would require new methods there.
  • #3619842: Translation install and update batch does a HEAD and then a GET request for every project unnecessarily Instead of a HEAD followed by an optional GET, do a GET with an If-Modified-Since header. Problem: check (remote timestamp) and download are separate batch operations that are only sometimes executed in the same batch, you can also run only the check.

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Comments

berdir created an issue. See original summary.

drumm’s picture

gábor hojtsy’s picture

Title: Optimize locale check/download batch operations » [META] Optimize locale check/download batch operations
Category: Task » Plan
Issue summary: View changes