Problem/Motivation

For sites that serve their images from example Amazon S3 or some other cloud based file storage, the image style flush operation can get expensive, so I'm proposing a change that could reduce unnecessary image style flushes.

When creating a new File Entity with an image, it's a multi-step process:

  1. Upload a new file
  2. Determine destination (public, etc -- if there's a choice)
  3. Add any info to the fields on the entity (if present)
  4. Save

When the file is uploaded, this triggers FileEntity::postSave(). This is the initial save. When you click the "Save" button, this this triggers FileEntity::postSave() again, but in this case, it is an update.

Taking a look at this method in detail:

/**
 * Implements hook_file_insert().
 */
public function postSave(EntityStorageInterface $storage, $update = TRUE) {
  parent::postSave($storage, $update);
  // Save file metadata.
  if (!empty($this->metadata)) {
  if ($update) {
    db_delete('file_metadata')->condition('fid', $this->id())->execute();
  }
    $query = db_insert('file_metadata')->fields(array('fid', 'name', 'value'));
    foreach ($this->getAllMetadata() as $name => $value) {
      $query->values(array(
        'fid' => $this->id(),
        'name' => $name,
        'value' => serialize($value),
      ));
    }
    $query->execute();
  }

  if ($update) {
    if (\Drupal::moduleHandler()->moduleExists('image') && $this->getMimeTypeType() == 'image' && $this->getSize()) {
      // If the image dimensions have changed, update any image field references
      // to this file and flush image style derivatives.
      if ($this->original->getMetadata('width') && ($this->getMetadata('width') != $this->original->getMetadata('width') || $this->getMetadata('height') != $this->original->getMetadata('height'))) {
        $this->updateImageFieldDimensions();
      }

      // Flush image style derivatives whenever an image is updated.
      image_path_flush($this->getFileUri());
    }
  }
}

We see that whenever the File Entity is updated, the image styles are flushed. Currently, image styles would be flushed even if the only change made was an update to a field on the entity (so a change that would not affect the image style derivative).

There appears to be a few scenarios that would require flushing image styles:

  • the width or height changed
  • the file URI changed
  • the file was replaced

Also, I did take a look at the D7 version of this module and it is the same setup as in D8: flush the image styles whenever the entity is updated.

Proposed resolution

I propose that the image_path_flush() call moves up into the width / height change check:

if ($update) {
    if (\Drupal::moduleHandler()->moduleExists('image') && $this->getMimeTypeType() == 'image' && $this->getSize()) {
      // If the image dimensions have changed, update any image field references
      // to this file and flush image style derivatives.
      if ($this->original->getMetadata('width') && ($this->getMetadata('width') != $this->original->getMetadata('width') || $this->getMetadata('height') != $this->original->getMetadata('height'))) {
        $this->updateImageFieldDimensions();
        
        // Flush image style derivatives.
        image_path_flush($this->getFileUri());
      }
    }
  }

I tried some testing to test if the image styles would flush appropriately. For this, I made sure that the default view mode for the File Entity used an image style rather than original image.

Replaced Image with new image that has different dimensions:

  1. Create a new File Entity and save it will my initial image (filename = "NB1_1025_mai_smaller_delete.jpg")
  2. Edited this entity and used the Replace File button and uploaded a new image with different dimensions (filename = "GE2_8890_smaller_mai_delete.jpg")
  3. Saved the entity

The image style is flushed as expected and shows as the updated image. I noticed that the file uri did not change.

Replaced Image with an image that is exactly the same except for the filename:

  1. Create a new File Entity and save it will my initial image (filename = "NB1_1025_mai_smaller_delete.jpg")
  2. Edited this entity and used the Replace File button and uploaded a new image with different dimensions (filename = "NB1_1025_mai_smaller_delete copy.jpg")
  3. Saved the entity

The image style is flushed as expected (confirmed via xdebug) and shows as the updated image (which is exactly the same). Again, I noticed that the file uri did not change. I also tried these same steps when a slightly altered version of NB1_1025_mai_smaller_delete copy.jpg (The modification = added a red rectangle to the image -- so height/width remains the same).

I must be missing the case when the image file uri changes...when the file is replaced, it looks like Drupal\file_entity\Form\FileEditForm::submitForm() takes care of deleting the old file:

Snippet from Drupal\file_entity\Form\FileEditForm::submitForm():

if (file_unmanaged_copy($entity_replacement->getFileUri(), $this->entity->getFileUri(), FILE_EXISTS_REPLACE)) {
  $entity_replacement->delete();
  \Drupal::logger('file_entity')->info('File @old was replaced by @new', $log_args);
}

It also looks like by the time we get to FileEntity::postSave(), the original file uri and the current file uri are the same.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

mirie created an issue. See original summary.

mirie’s picture

Added patch with proposed change for review.

Please let me know if I've missed workflow(s) which would skip image style flushing with this proposed change.

firewaller’s picture

This issue was causing me alot of headache since I have my files hosted on S3 and only generate the image styles on image upload, then when the node saves it deletes the image styles. This person seemed to have the same issue: https://drupal.stackexchange.com/questions/240151/image-styles-deleted-o...

The above patch works as described above. However, it seems like the origin of the issue is a legacy of Drupal 7's file_entity_file_update(), which did not fire on insert. I feel like leaving it as a catchall for updates is necessary since the file might be updated programmatically rather than with a form submit. Instead we should look into changing the $update variable to FALSE on insert, which should solve all scenarios.

joseph.olstad’s picture

OK so to expedite this, maybe commit the above patch as is and a new issue for the insert? And link it to here? Or if you want roll up a new patch with above.

firewaller’s picture

After some debugging, its seems like both the insert and update hooks are fired when a file is inserted (I am using both the Crop API and S3 File System modules, which may contribute to that).

Does the above patch does account for updated images with the same dimensions? I feel like there are likely to be other workflows missed because of this patch.

Alternatively, I would suggest we follow the Crop API module's usage of image_path_flush() in Drupal\crop\Entity\Crop::postSave(), where it has the following comment and config toggle to directly address the S3 performance/UX issue:

/**
 * {@inheritdoc}
 */
public function postSave(EntityStorageInterface $storage, $update = TRUE) {
  parent::postSave($storage, $update);

  // If you are manually generating your image derivatives instead of waiting
  // for them to be generated on the fly, because you are using a cloud
  // storage service (like S3), then you may not want your image derivatives
  // to be flushed. If they are you could end up serving 404s during the time
  // between the crop entity being saved and the image derivative being
  // manually generated and pushed to your cloud storage service. In that
  // case, set this configuration variable to false.
  $flush_derivative_images = \Drupal::config('crop.settings')->get('flush_derivative_images');
  if ($flush_derivative_images) {
    image_path_flush($this->uri->value);
  }
}
firewaller’s picture

Attached is a patch with the proposed approach seen in Crop API module.

pawel_r’s picture

Patch #6 needs rewrite - "file_entity_update_8005" already exists.

It's not the best approach - in many cases you wish to flush those derivatives, only few should be preserved.
It should be done in more flexible manner, configuration per field, per module, etc.

Berdir’s picture

Status: Needs review » Needs work

An update function must not use installDefaultConfig, it should only set the specific new key.

Not sure about this, I get that it's a performance problem, but just not doing it seems like a problem too.