Problem/Motivation

Found via profiling drush script that loads and serializes about 60,000 entities.

That drush command ends up calling this method over a million times - via field item creation and TypedDataInternalPropertiesHelper::getNonInternalProperties() which is used via the serializer.

This may or not be applicable to 'normal' requests - will profile Umami too once there's a green MR.

The issue is that when you have a lot of entities with the same fields, the field definition information is stored as a property of each instance of each field item. So we always have to ask the field item for the typed data, and can't get it once per field instance.

Only solution I have so far is a static property on that class, but using a WeakMap - this cuts out all the time from the function calls without any noticeable change in memory usage. xhprof overhead might be involved but I'm not sure that accounts for everything.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Issue fork drupal-3606068

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

catch created an issue. See original summary.

catch’s picture

Status: Active » Needs review

catch’s picture

Priority: Normal » Minor

Profiled the front page of Umami. There are 100 calls to this method, which is not enough to make a measurable difference.

catch’s picture

I had another look at the call stack around here and found more similar cases.

With a couple of extra WeakMap caches, I was able to remove 5 million function calls (72m down to 66m) from the drush command I'm testing against.

The trick is that instead of for every field on every entity getting the typed data definitions and then manipulating the field, trying to get the definitions once per field first, and then re-use that with each entity.

A good thing about this is that there appears to be pretty much zero cpu or memory cost from these WeakMap caches so even with a much smaller number of entities, it should hopefully be neutral at worst.

catch’s picture

Looks like half a millisecond saving on the front page of Umami which is still not much but a bit more visible now.

I think there might be a more visible improvement in core on a JSON:API listing endpoint since that will serialize/normalize multiple entities so call into the internal properties helper.

catch’s picture

StatusFileSize
new373.26 KB

The 'after' screenshot was the wrong one.

Also this shows a consistent 10kb memory reduction, which is tiny but it does stack up on the drush command into a bigger saving.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new1.24 KB

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

catch’s picture

Status: Needs work » Needs review
Related issues: +#3599844: EntityNormalizer should allow normalizing a subset of fields

MR should hopefully be green. Quite a lot of serializer unit test expectations to update.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new1.55 KB

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

catch’s picture

Status: Needs work » Needs review