Problem/Motivation

Follow-up from #2503999-32: Large volume entity migrations run out of memory

This issue was to followup on the consequences of invalidating entity caches when reclaiming static memory. Context from catch's comment:

Opened #2558857: Migrations invalidate entity caches when reclaiming memory to track the persistent cache clearing.

There are two cases that can be tricky:

- migrations that need to load entities as part of the migration may slow down a lot - some sites take 10 or 20 hours to migrate so a 10% regression can mean an extra hour or two.
- partial migrations on a live site
And on the subject of the original intent of this issue - the first point isn't really an issue, the cache is only cleared when memory has nearly run out, it will be very infrequent so migrations that load entities will benefit from the cache their entire run except shortly after clearing the cache.

Migrations on a live site (as using migration in a feeds-like way to do regular import) would be the greater concern (although, unless the periodic imports are very large they shouldn't hit the cache clearing code).

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Files: 
CommentFileSizeAuthor
#3 migrations_invalidate-2558857-3.patch732 bytesandriyun
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 107,559 pass(es), 1 fail(s), and 0 exception(s). View

Comments

catch created an issue. See original summary.

andypost’s picture

Issue summary: View changes
andriyun’s picture

Assigned: Unassigned » andriyun
Status: Active » Needs review
FileSize
732 bytes
FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 107,559 pass(es), 1 fail(s), and 0 exception(s). View
andypost’s picture

Assigned: andriyun » Unassigned
Status: Needs review » Reviewed & tested by the community
Issue tags: +Quickfix

Great!

Berdir’s picture

Status: Reviewed & tested by the community » Needs work

That fixes #32 but I don't think that's why @catch opened this issue? This is about finding a way to only invalidate static caches but not the persistent ones.

mikeryan’s picture

Umm... This issue was not about the little entityManager() cleanup, that should have been a new, separate minor issue.
This issue was to followup on the consequences of invalidating entity caches when reclaiming static memory. Context from catch's comment:

Opened #2558857: Migrations invalidate entity caches when reclaiming memory to track the persistent cache clearing.

There are two cases that can be tricky:

- migrations that need to load entities as part of the migration may slow down a lot - some sites take 10 or 20 hours to migrate so a 10% regression can mean an extra hour or two.
- partial migrations on a live site

And on the subject of the original intent of this issue - the first point isn't really an issue, the cache is only cleared when memory has nearly run out, it will be very infrequent so migrations that load entities will benefit from the cache their entire run except shortly after clearing the cache.

Migrations on a live site (as using migration in a feeds-like way to do regular import) would be the greater concern (although, unless the periodic imports are very large they shouldn't hit the cache clearing code).

The last submitted patch, 3: migrations_invalidate-2558857-3.patch, failed testing.

andriyun’s picture

Issue tags: -Quickfix

Patch for quick fix cleanup moved to issue #2559191: Clean-up migrate executable

andypost’s picture

Status: Needs work » Reviewed & tested by the community
Issue tags: +rc eligible, +Quick fix

Let's keep the issue as clean-up as #8 said, and continue in #2545632: [PP1] Move memory reclamation out of migrate executable

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 3: migrations_invalidate-2558857-3.patch, failed testing.

damiankloip’s picture

Status: Needs work » Active

Afaict, that issue won't resolve this one.

Everyone is trying to find workarounds for the simple fact that the ContentEntityStorageBase::resetCache method completely and utterly abuses the API.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

heddn’s picture

Issue summary: View changes

updated IS

catch’s picture

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.