Test scenario:

  1. Ensure #2430219: Implement a key value store to optimise config entity lookups is applying otherwise creating the configuration entities will take ages
  2. Install minimal
  3. Enable contact module
  4. Create 10 field storages on the contact message content entity
  5. Create 50 contact message bundles and add 10 field instances to each
  6. Delete all the field storages.

Doing the deletes...

Deleted field storage contact_message.field_map_storage_0 40.41s
Deleted field storage contact_message.field_map_storage_1 36.68s
Deleted field storage contact_message.field_map_storage_2 34.33s
Deleted field storage contact_message.field_map_storage_3 31.12s
Deleted field storage contact_message.field_map_storage_4 26.82s
Deleted field storage contact_message.field_map_storage_5 24.08s
Deleted field storage contact_message.field_map_storage_6 20.48s
Deleted field storage contact_message.field_map_storage_7 17.34s
Deleted field storage contact_message.field_map_storage_8 14.44s
Deleted field storage contact_message.field_map_storage_9 11.38s

Ouch.

This looks like it is due to #2473983: [meta] Evaluate Entity Field API Scalability.

My scripts to reproduce this are in attached.

Files: 
CommentFileSizeAuthor
#5 3-5-interdiff.txt486 bytesalexpott
#5 2482231.5.patch1.78 KBalexpott
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 95,278 pass(es). View
#3 2482231.3.patch2.46 KBalexpott
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 95,170 pass(es). View
alex_create.php.txt1.04 KBalexpott
alex_delete.php.txt465 bytesalexpott

Comments

alexpott’s picture

xhprof profiling the first field storage delete shows:

Overall Summary	
Total Incl. Wall Time (microsec): 62,766,459 microsecs
Total Incl. MemUse (bytes): 5,340,216 bytes
Total Incl. PeakMemUse (bytes): 19,446,752 bytes
Number of Function Calls: 24,511,495

But the good news is this looks like it is Drupal\Core\Entity\EntityManager::getFieldMap since that takes 90% of the time! So #2473983: [meta] Evaluate Entity Field API Scalability looks the ticket to fixing this. Very happy it is not dependencies. Postponing to check again after that issue is resolved.

Wim Leers’s picture

Status: Postponed » Active
Related issues: +#2482295: Rebuilding field map with many bundles/fields is very slow

The meta got more child issues. Notably #2482295: Rebuilding field map with many bundles/fields is very slow, which I think is the one that this was actually postponed on?

Reopening.

alexpott’s picture

Status: Active » Needs review
FileSize
2.46 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 95,170 pass(es). View

With the patch attached...

Deleted field storage contact_message.field_map_storage_0 1.18s
Deleted field storage contact_message.field_map_storage_1 1.23s
Deleted field storage contact_message.field_map_storage_2 1.36s
Deleted field storage contact_message.field_map_storage_3 1.28s
Deleted field storage contact_message.field_map_storage_4 1.31s
Deleted field storage contact_message.field_map_storage_5 1.34s
Deleted field storage contact_message.field_map_storage_6 1.38s
Deleted field storage contact_message.field_map_storage_7 1.43s
Deleted field storage contact_message.field_map_storage_8 1.45s
Deleted field storage contact_message.field_map_storage_9 1.49s

Without the patch attached...

Deleted field storage contact_message.field_map_storage_0 6.03s
Deleted field storage contact_message.field_map_storage_1 5.89s
Deleted field storage contact_message.field_map_storage_2 5.63s
Deleted field storage contact_message.field_map_storage_3 5.36s
Deleted field storage contact_message.field_map_storage_4 5.1s
Deleted field storage contact_message.field_map_storage_5 4.91s
Deleted field storage contact_message.field_map_storage_6 4.68s
Deleted field storage contact_message.field_map_storage_7 4.46s
Deleted field storage contact_message.field_map_storage_8 4.22s
Deleted field storage contact_message.field_map_storage_9 4.11s

So significant improvements and things become more predictable and quicker.

Berdir’s picture

Nice.

The clearCachedDefinitions() removal overlaps a bit with #2487287: Optimize/clean up cache clears when saving/deleting FieldConfig entities but that's not ready yet, so we can always reroll it. I'm not sure I see how it can be just removed, though?

alexpott’s picture

Issue tags: -revisit before release candidate
FileSize
1.78 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 95,278 pass(es). View
486 bytes

Let's handle the entity manager cache clearing part in #2487287: Optimize/clean up cache clears when saving/deleting FieldConfig entities. The reason I just removed the cache clear was because

    // Clear the cache upfront, to refresh the results of getBundles().

The above line hinted that the only reason we were doing this was to refresh the results of getBundles - however this was made unnecessary by #2482295: Rebuilding field map with many bundles/fields is very slow

With the patch attached:

Deleted field storage contact_message.field_map_storage_0 2.69s
Deleted field storage contact_message.field_map_storage_1 2.72s
Deleted field storage contact_message.field_map_storage_2 2.8s
Deleted field storage contact_message.field_map_storage_3 2.82s
Deleted field storage contact_message.field_map_storage_4 2.77s
Deleted field storage contact_message.field_map_storage_5 2.75s
Deleted field storage contact_message.field_map_storage_6 2.8s
Deleted field storage contact_message.field_map_storage_7 2.79s
Deleted field storage contact_message.field_map_storage_8 2.82s
Deleted field storage contact_message.field_map_storage_9 2.81s
Peak memory used: 38.75Mb

Without patch

d scr alex_delete.php
Deleted field storage contact_message.field_map_storage_0 4.11s
Deleted field storage contact_message.field_map_storage_1 4s
Deleted field storage contact_message.field_map_storage_2 3.94s
Deleted field storage contact_message.field_map_storage_3 3.72s
Deleted field storage contact_message.field_map_storage_4 3.58s
Deleted field storage contact_message.field_map_storage_5 3.46s
Deleted field storage contact_message.field_map_storage_6 3.39s
Deleted field storage contact_message.field_map_storage_7 3.26s
Deleted field storage contact_message.field_map_storage_8 3.18s
Deleted field storage contact_message.field_map_storage_9 3.04s
Peak memory used: 39.50Mb

So as we can see still a very significant speed up and peak memory is largely the same (well actually a little bit better - so the whole static cache worry in the original comment has proved to be unfounded).

alexpott’s picture

The numbers in #5 are with minimal - here are the numbers with standard (more config entities exist)

With patch

d scr alex_delete.php
Deleted field storage contact_message.field_map_storage_0 3.6s
Deleted field storage contact_message.field_map_storage_1 3.57s
Deleted field storage contact_message.field_map_storage_2 3.53s
Deleted field storage contact_message.field_map_storage_3 3.5s
Deleted field storage contact_message.field_map_storage_4 3.53s
Deleted field storage contact_message.field_map_storage_5 3.46s
Deleted field storage contact_message.field_map_storage_6 3.6s
Deleted field storage contact_message.field_map_storage_7 3.64s
Deleted field storage contact_message.field_map_storage_8 3.64s
Deleted field storage contact_message.field_map_storage_9 3.71s
Peak memory used: 45.50Mb

Without patch

Deleted field storage contact_message.field_map_storage_0 5.33s
Deleted field storage contact_message.field_map_storage_1 5.22s
Deleted field storage contact_message.field_map_storage_2 5.13s
Deleted field storage contact_message.field_map_storage_3 4.96s
Deleted field storage contact_message.field_map_storage_4 4.82s
Deleted field storage contact_message.field_map_storage_5 4.71s
Deleted field storage contact_message.field_map_storage_6 4.69s
Deleted field storage contact_message.field_map_storage_7 4.5s
Deleted field storage contact_message.field_map_storage_8 4.39s
Deleted field storage contact_message.field_map_storage_9 4.24s
Peak memory used: 45.75Mb
Berdir’s picture

Status: Needs review » Reviewed & tested by the community

This seems like a good improvement. We'll continue testing and improving this in #2487287: Optimize/clean up cache clears when saving/deleting FieldConfig entities.

  • webchick committed e96d427 on 8.0.x
    Issue #2482231 by alexpott: Deleting configuration entities is super...
webchick’s picture

Status: Reviewed & tested by the community » Fixed
Issue tags: +Performance

Committed and pushed to 8.0.x. Thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.