Problem/Motivation

This loads each item one by one, with the database storage that can be hundreds/thousands of queries.

We can load all at once or chunk instead. Saves 200+ms from drush cim -y.

Steps to reproduce

Proposed resolution

Multiple load the config items in chunks of 500. Sites tend to have between 1500-4000 config objects so this will result in the loading happening in 3-8 queries.

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Issue fork drupal-3610503

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

catch created an issue. See original summary.

catch’s picture

Status: Active » Needs review
StatusFileSize
new449.58 KB
new447.7 KB

Saves a couple of hundred milliseconds on a drush cim -y call with Umami, will be more with more config on a site.

Goes with #3591680: Use the YAML parsing cache collector for config file storage for the file parsing.

catch’s picture

Issue summary: View changes
smustgrave’s picture

Status: Needs review » Needs work

Sorry could we update the issue summary with the proposed change. How did you determine 500?

fwiw I like the idea of chunks after that fun bug with the entity field queries and 60+ fields so +1 to chunking it!

catch’s picture

Issue summary: View changes
Status: Needs work » Needs review

Updated the issue summary.

500 is a bit arbitrary. Most sites have 1500-4000 config objects from what I've seen, so this means 3-8 database queries. Wouldn't want to go over 500, but we could go down to 200 and it would still be 1/200th the database queries. Don't think it's worth making configurable.

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Thanks for humoring me. Think the change makes sense. Lets give it a shot.

godotislate’s picture

1 Q on the MR.

godotislate’s picture

Status: Reviewed & tested by the community » Needs review
catch’s picture

Status: Needs review » Reviewed & tested by the community

Replied on the MR - the memory storage is only used in tests, and that method never actually got called before. Now that it gets called, it fails.

catch’s picture

I pushed a commit to revert that change and then a second commit to fix the one-liner itself. For me I prefer the foreach than the array acrobatics but it makes it a bit more obvious how the current logic is broken. If we want to go back to the foreach, it'd be reverting the last two commits on the MR.

godotislate’s picture

For me I prefer the foreach than the array acrobatics but it makes it a bit more obvious how the current logic is broken.

I don't mind the foreach, and thanks for explaining the change.

Actually, if we're doing the foreach, we can simplify. Adding an MR suggestion.

godotislate’s picture

Status: Reviewed & tested by the community » Needs review
catch’s picture

Status: Needs review » Reviewed & tested by the community

Yes that's even better! Applied the suggestion.

  • godotislate committed bad194b8 on main
    task: #3610503 Optimize StorageCopyTrait::replaceStorageContents()
    
    By:...

  • godotislate committed 72303793 on 11.x
    task: #3610503 Optimize StorageCopyTrait::replaceStorageContents()
    
    By:...
godotislate’s picture

Version: main » 11.x-dev
Status: Reviewed & tested by the community » Fixed

Committed and pushed bad194b to main and 7230379 to 11.x. Thanks!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

catch’s picture

Tentatively tagging for 11.5.0 release highlights.

In combination with #3591680: Use the YAML parsing cache collector for config file storage this cuts more than half the time off a drush config import.

A lot of the remaining time in drush config import is in building a dependency container, but I think the

dr</code cli dumps its container so it's cached.

Between all of these a <code>dr

config import (that doesn't do much actual config changes or is a no-op) should be a couple of hundred milliseconds vs. 2-3 seconds or more with 11.4 and drush. I'm adding up the individual pieces here, haven't actually done a side by side comparison and not even sure dr has a config import command yet.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.