Problem/Motivation

The Drupal\crop\Entity\Crop::postSave() calls the image_path_flush() which flushes all cached images (image styles instances). If the site has tens of image styles and tens of crop types it leads to performance issues.

At the moment the Drupal\crop\Entity\Crop::postSave() is called for each updated/created Crop entity instance - so during a new media file creation the image_path_flush() is called as many times as many crop styles are attached to the media file type.

For example, a site has 20 image styles + 5 crop types attached to a media file entity. So during a file creation the image_path_flush() is called 5 times, the file system is scanned ~100 times (5 * 20) to flush cached image styles.

Here's a screenshot from New Relic that reflects the described issue:

New Relic

Steps to reproduce

  1. Install modules: crop, image_widget_crop, media.
  2. Add several crop types
  3. Add several image styles that are using the crop types
  4. On the "Manage form display" (admin/structure/media/manage/image/form-display) page select the "ImageWidget crop" widget add set several Crop Types
  5. Set a breakpoint at the Drupal\crop\Entity\Crop::postSave() in your IDE
  6. Add a new Media image and see how many times the image_path_flush() is called

Proposed resolution

  • Don't call an image styles flush during a saving of a new Crop entity. As I understand it doesn't make sense since there's no cached image styles at this moment.
  • Flush only image styles that are related to the current Crop type.

Remaining tasks

  1. Prepare a patch.

User interface changes

Nothing.

API changes

Nothing.

Data model changes

Nothing.

Issue fork crop-3200311

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

Chalk created an issue. See original summary.

chalk’s picture

Status: Active » Needs review
StatusFileSize
new1.22 KB

Here's the patch.

generalredneck’s picture

This is a simple and creative fix.

The only situation where this "might" fail is removing the image_path_flush($this->uri->value); on instantiation. Looking through the commit history, there is no mention of why they did it on entity creation as well, but it may happen if a crop is added to a field where an existing file entity is stored. This may happen in the case where someone is using a module to update a file with Media Entity File Replace or IMCE, or if the file entity is programmatically changed somehow (migration?). In these cases a particular style may already have a derivative, but I would guess it would be up to that module to clean them up? I'm grasping at straws honestly, but I want us to test those usecases.

As it is, using with focal point and none of these situations, this works amazingly well and is a vast improvment performance wise, ESPECIALLY on S3 and Pantheon Filesystems.

generalredneck’s picture

FYI I would have post a new relic stack trace, except that the improvements were so fast, new relic didn't keep any of the trace data. We went from 30+ seconds down to ~800ms with this fix on media creation with around 150 image styles and focal point installed.

jedihe’s picture

I can confirm patch #2 improves the time to update a media entity (incl. manual crops, with image_widget_crop) vs. not using the patch.

jedihe’s picture

StatusFileSize
new1.1 KB

I found that triggering the flush only for crop entity updates allows for an outdated image derivative to be served for this scenario:

  1. Site using Crop API + Image Widget Crop.
  2. An image style ImgSt configured to use only one effect (Manual Crop), using crop type CropA.
  3. The entity with the image field is saved, but no crop CropA is defined (widget set to not require it)
  4. The derivative for ImgSt is viewed/loaded (saves file)
  5. The entity is saved again, this time applying a crop for CropA. This creates a crop entity for CropA + this image.
  6. Loading the image derivative does not reflect the just-applied crop for CropA.
  7. Re-save the entity (make sure to do any kind of change to crop CropA). This updates the crop entity for CropA + this image.
  8. Loading the image derivative finally reflects the just-applied crop for CropA.

This effectively means, requiring $update == TRUE forces the user/editor to save a new crop twice (first creates it, second updates it) in order to see already-generated derivatives properly updated.

I'm attaching a patch that triggers the flush both for create and update scenarios of the crop entity. I just removed " && $update".

Note: I tried to reproduce the above scenario in a simplytest.me site, but couldn't. Since I don't have a way to verify that simplytest.me actually applied patch #2 (I definitely added it to the build), I decided to trust what I see in the site I'm working on which is consistent with what I read in the code.

pingevt made their first commit to this issue’s fork.

pingevt’s picture

I was running down a performance issue which this solves for the most part. But found another check that could be added in. Flushing image styles I'm pretty sure makes the assumption we are working with a uri that has a streamwrapper Drupal can work with. In a custom module, I have been making crops for images from a DAMS and have set the uri to an external URL (which i'm pretty sure is still valid as a uri) since I'm not actually working with a file in Drupal. When this get flushed it creates some weird paths to check and getting some errors from Drupal to to unlink, etc. So, I would propose checking for a valid stream wrapper, before flushing as well. It seemed appropriate to piggy back off of these patches, but if this should be a separate issue, let me know.

loze made their first commit to this issue’s fork.

phenaproxima’s picture

Status: Needs review » Needs work

This is certainly an improvement worth making, but two things stand in its way:

  1. Merge conflicts.
  2. A performance fix like this will definitely need some kind of test coverage so that we don't accidentally regress it. It seems to me that a kernel test might be the best fit here?
loze’s picture

Status: Needs work » Needs review

Resolved merge conflicts and added a new kernel test CropImageStyleFlushTest

phenaproxima’s picture

Status: Needs review » Needs work

Wow, what a test! That's really thorough coverage and much appreciated. I have a couple of minor stylistic nits (feel free to ignore those if you don't agree with them), but my bigger concern is the tests being willing to bail out if the CI environment is misconfigured. That means the tests will not be reliable -- they might just be skipped even if something is broken. If the CI environment is misconfigured in such a way that it breaks these tests -- which is unlikely -- then that means we have to fix something about the CI environment, not the test.

loze’s picture

Status: Needs work » Needs review

Yea, those CI checks were left of from me debugging the test. I have removed them and applied your other suggestions.
Should be good now. Thanks!

phenaproxima’s picture

Status: Needs review » Fixed

No complaints here. Started a merge train for 8.x-2.x. Thanks!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.