Problem/Motivation

When importing existing configuration during site install, importing translated strings is very slow

Table of xhprof results

Drupal\local\Gettest::fileToDatabase generates a call to Drupal\locale\PoDatabaseWriter::importString for each string which is then individually written to the database - in my case, for 6 languages, this caused an additional 51,000+ database writes on install and the installer takes around 10m to complete.

Proposed resolution

Support adding more than 1 translated string at a time to the database

Remaining tasks

tbd

API changes

tbd

Release notes snippet

tbd

CommentFileSizeAuthor
xhprof-locale.png604.85 KBjustafish

Comments

justafish created an issue. See original summary.

justafish’s picture

justafish
she/her
Primary language English
Location London, UK
commented
Issue summary: View changes

Version: 9.0.x-dev » 9.1.x-dev

Drupal 9.0.10 was released on December 3, 2020 and is the final full bugfix release for the Drupal 9.0.x series. Drupal 9.0.x will not receive any further development aside from security fixes. Sites should update to Drupal 9.1.0 to continue receiving regular bugfixes.

Drupal-9-only bug reports should be targeted for the 9.1.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.2.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.1.x-dev » 9.3.x-dev

Drupal 9.1.10 (June 4, 2021) and Drupal 9.2.10 (November 24, 2021) were the last bugfix releases of those minor version series. Drupal 9 bug reports should be targeted for the 9.3.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

heddn’s picture

heddn
Primary language English
Location Illinois
commented
Related issues: +#3037156: Modernize locale history functions

Link to #3037156: Modernize locale history functions.

heddn’s picture

heddn
Primary language English
Location Illinois
commented

When running things using mysql backend, the xhprof seen above is what I see on 9.3. When running with a sqlite backend, I see the trace in #3037156-25: Modernize locale history functions.

The install time really took a hit in 9.3 vs 9.2. I saw install time go from ~18 minutes with 40+ languages in 9.2 grow to 2.5-3hrs in 9.3. Something really make it take a lot longer after the upgrade.

heddn’s picture

heddn
Primary language English
Location Illinois
commented
Related issues: +#3125440: Impossible to skip interface translations download on Drupal install

Also linking in #3125440: Impossible to skip interface translations download on Drupal install.

joao.ramos.costa’s picture

joao.ramos.costa commented

Thanks @heddn .
I can report the same issue. An install time from config with 20+ languages in 9.2 took ~8 min and grow to 40 min in 9.3 .

heddn’s picture

heddn
Primary language English
Location Illinois
commented
Related issues: +#2925203: LocaleConfigSubscriber can result in data loss during install

See #2925203: LocaleConfigSubscriber can result in data loss during install, that probably exacerbated the problem. I think someone could remove that from the install by implementing hook_install_tasks_alterand removing _install_config_locale_overrides from the list of install tasks.

But importing strings shouldn't be so painfully slow. We should cache the info data better (without clearing it every time on a partial miss) and we should bulk insert the strings instead of one by one.

dxvargas’s picture

dxvargas commented
Related issues: +#2871357: Installer tasks using multiple batch sets in non-interactive mode do not get executed

I'm adding this related issue that seems to have uncovered this problem.
When I undo the change (use $batch instead of $batch_definition), the installation is much faster, like before.

Seems like translation batches were being skipped until now. After the merge of the related issue, they started to be processed.

We should improve the translation import process for sure.

But this case also shows that translations may not be relevant when installing the site, since the related issue went unnoticed by many during so much time. Shouldn't we have a option to opt out?

dxvargas’s picture

dxvargas commented
Related issues: +#3074765: Performance improvement for importing of project translations

The patch #3 in this related issue helps, I have an improvement in performance when installing it.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

dxvargas’s picture

dxvargas commented

Issue #3074765 is now merged, it would be nice to check the issue here again.
I believe it is the same problem and this issue may be closed.

heddn’s picture

heddn
Primary language English
Location Illinois
commented
Status: Active » Closed (duplicate)

In our case, this was the solution. I'd say this is duplicate. If that isn't the case for someone else, please re-open and list more specifics about the differences.