Problem with : S3 File System cache

Huh, interesting. I've never seen that happen before, so let's dig in and try to find out why.

The fact that ignoring the metadata cache makes the 404 go away means that the file isn't being cached properly for some reason. The cache is located in the s3fs_file table in your database. After doing a cache refresh, take a look at that table and try to find the entry that should exist for that file. It may be either missing entirely, or mangled in some way (wrong scheme, missing a root folder, etc.). Let me know what you find in there.

Looking at your URL from 1), I'm seeing something I've never seen before: that "fr" bit before "s3/files/styles/". I'm guessing that's related to your site's language? "fr" looks like the abbreviation for "French". I've never actually tested s3fs with a multi-language site before, so it's entirely possible that this is a bug in s3fs relating to that. If I'm right about that "fr" meaning that you've got a multi-language site set up, could you please tell me how you did that? I'll need to set up my testing site to be multi-lang to be able to exercise, and debug, this problem.

Yes, is_file() does work with the s3:// scheme. The S3StreamWrapper class defines the s3:// scheme, and it works just like public:// and private://, which is what allows core PHP file operations like is_file() to work properly.

I've never heard of that s3fs_migrate module before, but it looks pretty cool! It would actually solve a problem that another user of s3fs is having, so I'm going to suggest that he try it out.

Hmmm, this is a bit concerning: s3://styles/juicebox_square_thumbnail/s3/s3fs-public/node_gallery/20151023_131036.jpg
If I'm remembering correctly, no URI should be formatted that way, as it combines s3:// and the "s3fs-public", which I'm pretty sure should never happen. "s3fs-public" implies that it should have the public:// scheme. This may be a clue about what's wrong.

> In the s3fs_file table I can find the reference of the original image but with the public:// scheme

Ahh, now this seems important. When you used s3fs_migrate, where within your bucket did it put the files that were originally in the public:// file system? If it put them into the "s3fs-public" root folder, that means something different, internally, than if it put them into a different folder. I'll go install and use s3fs_migrate, and see how it works. That might shed some light on this.

Did you find a solution to this situation, I am having a similar problem and am a bit stumped.

I recently discovered that the issue of getting "page not found" when accessing the derivative generating url went away if I commented out some lines

  if (strpos($key, "$public_folder_name/") === 0) {
      // Much like the root folder, the public folder name must be removed from URIs.
      $key = str_replace("$public_folder_name/", '', $key);
      $uri = "public://$key";
    }
    else if (strpos($key, "$private_folder_name/") === 0) {
      $key = str_replace("$private_folder_name/", '', $key);
      $uri = "private://$key";
    }
    else {
      // No special prefix means it's an s3:// file.
      $uri = "s3://$key";
    }

Into

 /* if (strpos($key, "$public_folder_name/") === 0) {
      // Much like the root folder, the public folder name must be removed from URIs.
      $key = str_replace("$public_folder_name/", '', $key);
      $uri = "public://$key";
    }
    else if (strpos($key, "$private_folder_name/") === 0) {
      $key = str_replace("$private_folder_name/", '', $key);
      $uri = "private://$key";
    }
    else {*/
      // No special prefix means it's an s3:// file.
      $uri = "s3://$key";
    //}

I do not fully understand this code; if it is reading files on s3, why would the code determine that the scheme is public:// when clearly the file is on s3://? Forcing it to return $uri = "s3://$key" resolved the issue of thumbnails not generating.

I will note here that I suspect the modified migrate process could be responsible for this problem but I'm really not sure.

Ultimately, when your derivate-generating url gives you page not found, it is usually because there is no metadata for that file in s3fs_file table. Using the "Refresh File Meta Data" button does not work due to above code in my case unless I force the return of s3://

That code block is there to handle the public and private takeover features that you can optionally enable in S3. Those options make S3FS handle URIs which start with "public://" and "private://", instead of Drupal's default filesystem handling them. The feature is useful for when you want every file generated by Drupal, including those with hard-coded public:// URIs (like optimized CSS) to be stored in your S3 bucket.

Internally, S3FS differentiates public:// files from s3:// files by putting them into the subfolder "s3fs-public/" in your S3 bucket. That code you edited converts the real S3 key of the file into an appropriate URI. For example, "s3fs-public/path/to/file.jpg" becomes "public://path/to/file.jpg", while "path/to/other.jpg" becomes "s3://path/to/other.jpg".

I'm not sure why you'd be getting a 404 error from a derivative generation URL, though. Could you describe your issue in more detail?

Thanks for your help. Basically whenever I get this error

The requested page "/s3/files/styles/thumbnail/public/public/profileImage_1.png?itok=AUrjIHfb" could not be found.

, it is because the $metadata variable in the "_s3fs_get_object" (circa line 1218) is empty. If I disable metadata cache, it works but if uncheck the box to enable the file metadata cache, it does not work.

I compared this same variable when running an image that did work properly from the same url path with different image name, $metadata was not empty.

This is how I traced the problem to the earlier code I published; because the metadata record for that file was nonexistent, yet I had refreshed metadata cache.

> "/s3/files/styles/thumbnail/public/public/profileImage_1.png?itok=AUrjIHfb"

Yeah, that's definitely wrong. There almost certainly shouldn't be two "/public"s in there. That's breaking the argument parsing in _s3fs_image_style_deliver(), causing it to assume that the file's URI is public://public/profileimage_1.png". That URI corresponds to the following path in S3: "/s3fs-public/public/profileimage_1.png", which I'm assuming is not the correct path. What *is* the correct path to that file in your bucket?

I'm not exactly when or how the public/public/ was introduced but the problem was consistent with other files whether or not there were nested /public folders.

/s3fs-public/styles/thumbnail/public/public/profileImage_1.png does indeed contain this image on the S3 bucket

but because the metadata is not there /s3/files/styles/thumbnail/public/public/profileImage_1.png? is not redirecting to it.

If I comment out the code as shown in my post #8, then refresh metadata cache and then visit some other pages,

the thumbnails in some cases are in this location on the s3 bucket:

styles/thumbnail/s3/s3fs-public/public/profileImage_36597.jpg

It is interesting to note that with the code as it was, refreshing the cache and refreshing the page breaks the images that were migrated to s3 but the ones that were uploaded afterwards work fine. When I edit the code by forcing s3:// when the cache is rebuilt, the images that were migrated generate thumbnails properly, but the ones uploaded since then, then break.

We are hosting on pantheon and aren't sure if that is a factor or not, or if the migration process had some problems that may be at fault.

Ahhh, that difference between migrated and unmigrated images is a huge clue. It sounds to me like the migration was probably done without knowledge of the s3fs-public/ prefix for public:// files, which would explain why you're getting that behavior.

So lets work that out. How were your files set up on your site before you performed the migration? Were they all in public://? What kind of migration did you actually perform, e.g. did you use drush s3fs-copy-local, the "Copy local public files to s3" button in the Druapl admin UI, or something else, like using the s3fs-migrate module? What's an example of an image file path in your bucket that works correctly with the patch you posted in #8, and one that works without the patch?

With that information, I can work out what may have gone wrong, and hopefully how to fix it.

Thanks for your response. We were unable to use drush to migrate because we could not get it to work on pantheon. I used the s3fs-migrate module, and like the original poster, I modified it to just perform the database reference updates and we moved the files manually to S3. So basically commenting out the lines that actually moved the file. This works, in that the file is referenced to the right location, it is the thumbnails that don't generate.

PATHS:

Without #8 patch
Migrated file with failed thumbnail generation:
example.com/s3/files/styles/thumbnail/s3/s3fs-public/public/profileImage_36597.jpg?itok=mz52SToC is returning "The requested page "/s3/files/styles/thumbnail/s3/s3fs-public/public/profileImage_36597.jpg?itok=mz52SToC" could not be found."

post-migration file with successful thumbnail generation:
example.com/s3/files/styles/thumbnail/public/public/profileImage_1.png?itok=AUrjIHfb the first time, after the first page view they redirect to the s3 image, for example s3.amazonaws.com/s3fs-public/styles/thumbnail/public/public/profileImage_1.png

With #8 patch:

migrated file with successful thumbnail generation:
example.com/s3/files/styles/thumbnail/s3/s3fs-public/public/profileImage_36597.jpg?itok=mz52SToC and redirects to s3.amazonaws.com/styles/thumbnail/s3/s3fs-public/public/profileImage_36597.jpg after first page view.

post-migration file with failed thumbnail generation:
example.com/s3/files/styles/thumbnail/public/public/profileImage_1.png?itok=AUrjIHfb

All the files were in public:// before migration.

This URL seems very suspect:
/s3/files/styles/thumbnail/s3/s3fs-public/public/profileImage_36597.jpg

The relevant part being the "s3/s3fs-public/". The existence of the "s3fs-public" folder is supposed to be an internal implementation detail, so it shouldn't appear in the thumbnail generation URL. I would expect that URL to instead be "/s3/files/styles/thumbnail/public/public/profileImage_36597.jpg", because anything in your bucket within the "s3fs-public" folder should have a URI of public://path/to/file.jpg, rather than s3://s3fs-public/path/to/file.jpg.

This makes me think that either s3fs-migrate, or your own manual copying, did not go as s3fs expects it to go. So I'm going to ask some very particular questions about the original layout of the files in your site, and how you copied them into your bucket.

1. What was the relative path within your Drupal installation's file folders for the "profileImage_36597.jpg" file? Based on the URLs you mentioned, I would expect it to have been sites/default/files/public/profileImage_36597.jpg. If that's not true (especially in regards to that "public" folder), then something most likely went wrong with your migration.

2. When you copied the local files into your bucket, what folder did you put them into? The drush command would have put them in the bucket's s3fs-public/ folder, which would have left their URIs the same: e.g. public://folder/file.jpg (assuming "Use S3 for public:// files" is enabled). Since their URIs would have remained the same, their thumbnail generation URLs would have remained "/s3/files/styles/thumbnail/public/folder/file.jpg".

3. Just to confirm, you still have the "Use S3 for public:// files" setting enabled, right?

After some investigation it appears that an aberration during an import process caused this problem and that it isn't an issue that would be experienced by most users.

Hi I am facing the same problem. Have anyone found the solution?

I'm experiencing a similar issue. In my case, the initial url contains the presigned url parameters as well as an extra parameter &null which causes S3 to refuse the signature as invalid. manually removing the &null from the url results in the file loading perfectly as expected. Refreshing the file metadata cache removes the &null parameter from the presigned url and it works again...

Lukey, what was the aberration you had with your import process? I am also importing preexisting s3 items into the site and am curious if you found a solution to my problem.

I found the source of my issue. The $meta['version'] for the file was set to "null". I'm looking through the S3fsService.php and line 396 appears to attempt to handle exactly the case I have. For some reason, it's still getting through and saved in the cache when I create the cache the first time.

I was able to address the issue temporarily by changing line 476 of src/StreamWrapper/S3fsStream.php from
if (!empty($meta['version'])) {
to
if (!empty($meta['version']) && $meta['version'] != "null") {

Not sure how this is sneaking past the code in S3fsService.php. If I find out more, I'll circle back and update here.

RedEight, I had a hard time determining the exact cause of the problem but it was something to do w/ meta data being null as you said. It has been a while and this piece got migrated to some other platform for us so I wasn't able to go in and confirm. What you pointed out looks familiar though.

I think it might be better to set the check for a "null" VersionId to the S3fsSystem::convertMetadata method itself... That way S3fsStream::writeUriToCache() always receives consistent results.

I think it might be better to set the check for a "null" VersionId to the S3fsSystem::convertMetadata method itself

You mean like this?

Exactly like that. It appears to have fixed my problem perfectly.

Oh good, I'm glad I understood the problem correctly.

I'll go ahead and integrate that patch into the dev version. A new dev build should show up in a few minutes.

I just realized that the patch targets the D7 version, but the line change worked perfectly in the corresponding D8 method.

Oh, yeah, this issue is set to the D7 version, which is why I was tracking it. I'm not a developer for the D8 version. Lucky you, I guess! :)

Yeah, I lucked out. I'll open an issue specifically for D8 and try to roll my own patch.

This is exactly what i facing in D8

@redEight any solution you found pls share.

Thanks
-Jatin

mail2jatingarg, D8 patch here.
#2922049: VersionId can be null

This patch was never committed but is, in fact, necessary. +1 for reopening it and accepting it as RTBC.

I'm not a developer for the D8 version of this module, so someone else is going to have to apply the patch.

Any of my D8 maintainers still around to do this?

I'll review soon as posible for drupal 8. greetings.

The patch in #22 (https://cgit.drupalcode.org/s3fs/commit/?id=b3ce9ff65523f3623885d6a30463...) can be cherry-picked onto 7.x-3.x.

Missing from 7.x-3.x