Hi,
When I try to edit an image style I get the errorUser warning: Subfolder is not empty in Aws\S3\StreamWrapper->triggerError()

This is my backtrace info:

Aws\S3\StreamWrapper->triggerError('Subfolder is not empty') (Line: 849)
Aws\S3\StreamWrapper->deleteSubfolder('s3://{bucket}/styles/thumbnail_crop/s3/media/images/2018-09', Array) (Line: 364)
Aws\S3\StreamWrapper->Aws\S3\{closure}() (Line: 910)
Aws\S3\StreamWrapper->boolCall(Object) (Line: 365)
Aws\S3\StreamWrapper->rmdir('s3://{bucket}/styles/thumbnail_crop/s3/media/images/2018-09', 8) (Line: 811)
Drupal\s3fs\StreamWrapper\S3fsStream->rmdir('s3://styles/thumbnail_crop/s3/media/images/2018-09', 8)
rmdir('s3://styles/thumbnail_crop/s3/media/images/2018-09') (Line: 256)
Drupal\Core\File\FileSystem->rmdir('s3://styles/thumbnail_crop/s3/media/images/2018-09', NULL) (Line: 1198)
drupal_rmdir('s3://styles/thumbnail_crop/s3/media/images/2018-09') (Line: 924)
file_unmanaged_delete_recursive('s3://styles/thumbnail_crop/s3/media/images/2018-09', NULL) (Line: 920)
file_unmanaged_delete_recursive('s3://styles/thumbnail_crop/s3/media/images', NULL) (Line: 920)
file_unmanaged_delete_recursive('s3://styles/thumbnail_crop/s3/media', NULL) (Line: 920)
file_unmanaged_delete_recursive('s3://styles/thumbnail_crop/s3', NULL) (Line: 920)
file_unmanaged_delete_recursive('s3://styles/thumbnail_crop') (Line: 259)
Drupal\image\Entity\ImageStyle->flush() (Line: 111)
Drupal\image\Entity\ImageStyle->postSave(Object, 1) (Line: 469)
Drupal\Core\Entity\EntityStorageBase->doPostSave(Object, 1) (Line: 395)
Drupal\Core\Entity\EntityStorageBase->save(Object) (Line: 259)
Drupal\Core\Config\Entity\ConfigEntityStorage->save(Object) (Line: 391)
Drupal\Core\Entity\Entity->save() (Line: 637)

Note: I have replaced my bucket name with {bucket{ in the backtrace.

Comments

James Byng created an issue. See original summary.

noemi’s picture

noemi commented

I stumbled upon the same warning, me too while editing an image style.
It seems images are generated correctly, but I receive the warning Subfolder is not empty StreamWrapper.php:746 when flushing images with drush.

malcolm_p’s picture

malcolm_p commented

I have also run across this but haven't seen it cause any issues yet.

raczernecki’s picture

raczernecki commented

Same thing happened to me, cleaning these folders manually helps to get rid of warning but it shouldn't be done like that. Is there any solution yet?

naveenvalecha’s picture

naveenvalecha
He/Him
commented
Version: 8.x-3.0-alpha13 » 8.x-3.x-dev

Moving to 8.x-3.x branch. Can you confirm are you still able to replicate the issue with the latest dev branch?

darvanen’s picture

darvanen
he/him
Primary language English
Location Sydney, Australia
commented

Try the patch from #3161279: Use of validScheme method which doesn't exist from 9.0.0 and see if that helps.

cmlara’s picture

cmlara
he/him
commented

I haven't seen this in my lab and I'm having a bit of trouble mocking this up as test code at the moment to confirm.

I speculate we have one of two scenarios going on

  1. That we are hitting a race condition a race condition with the files having been deleted and in the time the call to rmdir() is being made after the the local database check for an empty directory that a new image style generates populates the folder
  2. That there are files in the folder we do not know about.

Scenario 1 could very well be possible. The AWS SDK rmdir is really intended to get rid of empty objects that are placed that end in '/' (such as 'styles/') to indicate a directory (calling the AWS SDK mkdir()). Currently we don't utilize the mdkir function inside of S3FS, though in #3151913: Can't upload file to S3 bucket on version 8.x-3.0-alpha15 (i didn't know about mkdir when I wrote the comments) the possibility of changing over S3FS to create empty directory keys was considered as a solution.

Possible Scenario 1 Solutions:

  • Don't call AWS rmdir()
  • Suppress the triggerError on the call to AWS rmdir() using '@' and just act on the true/false return.

Scenario 2 is more concerning, I would want to know why files exist outside of the S3FS file table cache knowledge to be sure it is not an issue in our code. If its a system outside of the Drupal instance creating files (shared bucket) than we can ignore the issue, if its S3FS missing records I'm concerned as to why. Wonder if its possible if the rmdir function is having the same race condition but in reverse (we are told by AWS the directory is deleted, a new file is created, we delete the directory from the database) -- I will need to look into that more.

Would any of you be able to provide any information that may support or refute either of the scenarios?

  • cmlara committed a1773d6 on 8.x-3.x 
    Issue #3004034 by James Byng, cmlara: When editing an image style I get...
cmlara’s picture

cmlara
he/him
commented
Status: Active » Fixed

We don't know for sure there are not files in S3 that are not in the database, however the tasks being done for #3201248: Disallow realtime unkown file lookup to S3 -- Reverts #2915322 should make that more obvious in the long run. ImageStyle->flush() expects there to possibly be errors removing the directory and chooses to ignore them which makes sense where we are seeing this.

The patch will still cause rmdir() to evaluate as FALSE so functions can still track errors it just prevents the triggerError() from causing issues.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.