On one of our projects we've met an issue with no free disk space. After an investigation it was found that /tmp directory was full of imagemagick_* files. The project uses https://www.drupal.org/project/s3fs

It was found that imagemagick_imagemagick_pre_parse_file_alter() and imagemagick_imagemagick_arguments_alter() create local copies of remote files, but never delete them.
(Temporary copies are deleted by imagemagick_imagemagick_post_save_alter(), but this one is only called on file save, which is not always the case.)

UPD: The latest 8.x-2.x branch uses an event subscriber instead of hooks, but the issue is still the same.

UPD2: Just tested on our project: five temporary files were left in the temp directory after creating a new image media.

Comments

Leksat created an issue. See original summary.

leksat’s picture

Status: Active » Needs review
StatusFileSize
new1.07 KB
leksat’s picture

StatusFileSize
new1.46 KB

Status: Needs review » Needs work

The last submitted patch, 3: 3112424-3-test-only.patch, failed testing. View results

leksat’s picture

Status: Needs work » Needs review
StatusFileSize
new3.43 KB

Status: Needs review » Needs work

The last submitted patch, 5: 3112424-5.patch, failed testing. View results

leksat’s picture

Status: Needs work » Needs review
StatusFileSize
new2.01 KB
new3.67 KB

The last submitted patch, 7: 3112424-7-test-only.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

leksat’s picture

Issue summary: View changes

The above patch uses drupal_register_shutdown_function() because there is no way to detect when a temporary file can be removed. Or I could not find one 🤷‍♂️

leksat’s picture

Issue summary: View changes
mondrake’s picture

Version: 8.x-2.x-dev » 8.x-3.x-dev
Status: Needs review » Needs work
Issue tags: +needs backport to 8.x-2.x

Excellent catch, thank you! We need to fix 8.x-3.x first, then backport.

+++ b/src/EventSubscriber/ImagemagickEventSubscriber.php
@@ -295,4 +303,18 @@ class ImagemagickEventSubscriber implements EventSubscriberInterface {
+  /**
+   * Removes a temporary file created during operations on a remote file.
+   *
+   * Used with drupal_register_shutdown_function().
+   *
+   * @param string $path
+   *   The temporary file realpath.
+   */
+  public function removeTemporaryRemoteCopy($path) {
+    if (file_exists($path)) {
+      $this->fileSystem->unlink($path);
+    }
+  }
+

I would prefer to use a static function here. See for example how this was done for Image Effects in #3046522: Remove imagemagick texttowrapper temporary files on shutdown, see commit https://git.drupalcode.org/project/image_effects/commit/3893f53.

+++ b/tests/src/Functional/ToolkitImagemagickTest.php
@@ -65,6 +65,44 @@ class ToolkitImagemagickTest extends BrowserTestBase {
+    $this->assertEqual(count(file_scan_directory('temporary://', '/ima.*/')), 0, 'There are no leftovers in the temporary directory after getting metadata from a remote file.');

We could use assertCount here instead, and be more PHPUnit compliant...

leksat’s picture

@mondrake, since file_unmanaged_delete() is deprecated, we will have to use the file_system service, which is already injected into ImagemagickEventSubscriber. That's why I did not use a static method - we have to call a service anyway.

Could you please confirm your wish for a static function one more time :)

leksat’s picture

Assigned: leksat » Unassigned
Status: Needs work » Needs review
StatusFileSize
new1.96 KB
new3.74 KB

For now: used assertCount and re-rolled on 8.x-3.x

The last submitted patch, 13: 3112424-13-test-only.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Status: Needs review » Needs work

The last submitted patch, 13: 3112424-13.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

leksat’s picture

Status: Needs work » Needs review
StatusFileSize
new1.98 KB
new3.75 KB

Whoops, did not notice that 3.x requires Drupal 8.8+

The last submitted patch, 16: 3112424-16-test-only.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

mondrake’s picture

#12 well that was changed in Image Effects 8.x-3.x likewise, see https://git.drupalcode.org/project/image_effects/blob/8.x-3.x/src/Plugin...

Also, see the getSupportedExtensions method in the ImagemagickToolkit class, which is a static - we have an example of using the service from the static method, too.

I do not know, honestly. What concerns me is that keeping it with the instantiated service in the container may have side effects when running the callbacks in drupal_register_shutdown_function. Now it works, but in the future who knows... just for an example, if you were to run the callbacks in register_shutdown_function (i.e. normal PHP, not Drupal), at the stage of executing the callbacks the container is already destructed (reason why we made a change in #3046522-5: Remove imagemagick texttowrapper temporary files on shutdown). OTOH, I see in drupal core usages also with instantiated methods in services.

leksat’s picture

StatusFileSize
new1.79 KB
new3.59 KB

Okay, let's keep it consistent with image_effects 👍

The last submitted patch, 19: 3112424-19-test-only.patch, failed testing. View results

mondrake’s picture

Status: Needs review » Needs work

Thank you @Leksat.

Just a couple minor things, then it's ready:

  1. +++ b/tests/src/Functional/ToolkitImagemagickTest.php
    @@ -61,6 +61,38 @@ class ToolkitImagemagickTest extends BrowserTestBase {
    +    // Get metadata from a remote file.
    +    $image = $this->imageFactory->get('dummy-remote://image-test.png');
    +    $image->getToolkit()->getExifOrientation();
    +
    

    Let's add an assertCount for the files in tmp here too, to show that the file exists before being deleted.

  2. +++ b/tests/src/Functional/ToolkitImagemagickTest.php
    @@ -61,6 +61,38 @@ class ToolkitImagemagickTest extends BrowserTestBase {
    +      if ($callback['callback'] === ['Drupal\imagemagick\EventSubscriber\ImagemagickEventSubscriber', 'removeTemporaryRemoteCopy']) {
    

    Let's use the ::class notation here, so ImagemagickEventSubscriber::class instead of the FQCN string.

leksat’s picture

Status: Needs work » Needs review
StatusFileSize
new2.2 KB
new4 KB

Thank you for hints!

The last submitted patch, 22: 3112424-22-test-only.patch, failed testing. View results

mondrake’s picture

Crediting

  • mondrake committed 243b3a5 on 8.x-3.x authored by Leksat
    Issue #3112424 by Leksat, mondrake: Temporary copies of remote files are...
mondrake’s picture

Version: 8.x-3.x-dev » 8.x-2.x-dev
Status: Needs review » Patch (to be ported)

Committed to 8.x-3.x. Leaving open for backport to 8.x-2.x, if necessary. Thanks!

leksat’s picture

Status: Patch (to be ported) » Needs review
StatusFileSize
new1.33 KB
new1.33 KB
leksat’s picture

StatusFileSize
new2.21 KB
new3.91 KB

The last submitted patch, 28: 3112424-28-8.x-2.x--FAIL.patch, failed testing. View results

mondrake’s picture

Status: Needs review » Needs work
Issue tags: -needs backport to 8.x-2.x
+++ b/src/EventSubscriber/ImagemagickEventSubscriber.php
@@ -295,4 +303,18 @@ class ImagemagickEventSubscriber implements EventSubscriberInterface {
+      \Drupal::service('file_system')->delete($path);

I know, I know... but we need to use the old file_unmanaged_delete here since 8.x-2.x is still supporting Drupal core before 8.7.

leksat’s picture

Status: Needs work » Needs review
StatusFileSize
new2.21 KB
new3.89 KB

🤞

The last submitted patch, 31: 3112424-31-8.x-2.x--FAIL.patch, failed testing. View results

  • mondrake committed fe6280d on 8.x-2.x authored by Leksat
    Issue #3112424 by Leksat, mondrake: Temporary copies of remote files are...
mondrake’s picture

Status: Needs review » Fixed

Committed #31 to 8.x-2.x. Thanks!

leksat’s picture

🎉🎉🎉

Thank you for your patience!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.