Broken image links for files uploaded to S3

I see that your files are appearing in the s3fs-public folder of your bucket. Most of the time, you don't want uploaded files to appear in that folder, because it's really only intended for files that Drupal creates itself.

To avoid that, set your File field's "Upload destination" option to "S3 File System" instead of "Public files". That will cause the normal code paths in s3fs to be used, rather than the "public:// takeover" code paths, which may help with your image derivative problem. You'll need to delete (from within Drupal) any files which have already been uploaded into that field, in order to make Drupal allow you to change the setting.

The URL is different because by default, images are served from Drupal as image derivatives. Those URLs are exactly what I expect s3fs to provide for an image derivative of the "large" image style: "/s3/files/styles/" in the base URL, "large" is the image style's name, "s3" is the scheme of the file's URI, and "field/image/Tulips.jpg" is the full path to the file in your bucket.

The question now is why that URL is giving you a 404 error instead of creating the image derivative.

Have you tried clearing your site's cache? Go to /admin/config/development/performance and clicking the "Clear all caches" button. Then reload the article page. Do you still get a 404 from the image derivative URL?

Hmmm, I'm pretty stumped here. I don't think any "/s3/files/styles/..." URL should ever actually be sending 404s in the first place.

Could you check your bucket for a file called "styles/large/s3/field/image/Tulips.jpg", please? If that file is there, the derivative is being created... though I have no idea why you'd be getting a 404. If the file isn't there, the derivative is not being created when you visit the "/s3/files/styles/..." URL, so we'll have to figure out why that's happening.

It's barely possible that the "field/image" folder that you're using for the "File directory" setting could be at fault here. Try blanking that setting, so that s3fs will place the files in the root of your bucket. If that works, that's a bug in s3fs that I'll have to track down.

MY final idea is that this problem might be related to you using Nginx. Some s3fs users in the past have had strange issues with that server, and I've only ever tested s3fs with Apache.

Oh, hmm... that "itok" HTTP GET argument is a clue. For some reason it's missing on the http://mysite/s3/files/styles/... URLs you're seeing. Although that should cause a 403 (access denied), rather than a 404 (not found).

Note that once a derivative has been created the first time, the URL served on the page will change to the actual S3 url (http://YOUR_BUCKET.s3-us-west-2.amazonaws.com/...), instead of the derivative creation URL (http://mysite/s3/files/styles/...). That might be why you're seeing different behavior between Clean and non-Clean URLs.

At this point, I think it's gotta be something related to another module you have installed. The code path for /s3/files/styles/... URLs cannot throw 404s under normal circumstances. So I think something you have installed on your site may accidentally be hijacking that URL and redirecting the execution path to somewhere besides s3fs's code.

I'm currenty running a completely stateless Drupal-installation (as in nothing on disk) ready for auto-scaling using Docker.

s3fs is a crucial part of this, and my only problem at the moment is generation of image styles when using public filesystem (this aspect of s3fs is essential to a stateless Drupal).

The /s3-path is not a favorite of mine, thus I rewrote that functionality in an earlier patch when seeking the functionality for taking over public filesystem.

As there are code generating this /s3-uris, I think that code should rather generate the image style. And as I've argued before; the current solution is not good for first time caching of pages.

@coredumperror: How about abandoning the /s3-path?

What alternate would you suggest? There needs to be some way to generate image styles the first time they get requested. I recently ran into a situation on some of the sites I manage where I needed to generate every image style all at once to avoid a caching issue, and it takes *forever* on a site with anything more than a half dozen image styles. So doing the generation at image upload time is really not feasible.

So if you have a suggestion that generates the image styles at some time other than upload time or first-request time, I'm all ears.

I was just struggling with this same issue when I realized that it was nginx that was 404ing - the image style request wasn't even making it to Drupal in the first place.

Digging around the nginx config I found this snippet:

         # Fighting with ImageCache? This little gem is amazing.
        location ~ (^/sites/.*/files/imagecache/|^/sites/default/themes/.*/includes/fonts/|^/sites/.*/files/styles/) {
            expires max;
            try_files $uri @rewrite;
        }

I modded it to add the s3 file path:

        # Fighting with ImageCache? This little gem is amazing.
        location ~ (^/s3/files/styles/|^/sites/.*/files/imagecache/|^/sites/default/themes/.*/includes/fonts/|^/sites/.*/files/styles/) {
            expires max;
            try_files $uri @rewrite;
        }

and voilá, image styles being sent to s3 and displayed on the site properly.

Nice! Thank you for sharing that snippet. I am entirely unfamiliar with nginx, so any insight that users can provide here will help future users with nginx-based sites much more than I can.

this is a critically important snippet for anyone running NGINX with this module. It should be included in the README.

Any idea why nginx requires extra setup that isn't needed by apache? Being unfamiliar with nginx, I'm not even sure what this snippet is doing.

Right... so an NGINX config file has a few global directives, like hostname and root directory etc, and then is basically just a set of code blocks for how to handle a given request. Each code block is headed either by a regex for what paths it should handle, or a shortcut name so it's easy to send requests there from other regexes. You can do more complicated logic too, but that's not worth going into right now. :)

So most Drupal installations have a code block like this:

location @rewrite {
  rewrite ^/(.*)$ /index.php?q=$1;
}

(this is from the most common source for a demo config)

This block sets a shortcut for @rewrite. Any request that you send to @rewrite will get rewritten into /index.php?q=whatever/the/path/was . You know, the same thing that .htaccess does for apache users.

The reason that you do that in a shortcut rather than just saying location ^/(.*)$ is because there are some conditions that you want to evaluate first, where you might not need to rewrite the location to index.php - like the contents of sites/default/files, for example. or any request for *.php, which you just want to drop and ignore.

So how does that relate to our code snippet? Well, there's also typically a block like this:

         # Fighting with ImageCache? This little gem is amazing.
        location ~ (^/sites/.*/files/imagecache/|^/sites/default/themes/.*/includes/fonts/|^/sites/.*/files/styles/) {
            expires max;
            try_files $uri @rewrite;
        }

That says that for any request matching the regex, first try the URI directly (ie look for an actual file at that location), and if that fails, use the @rewrite rule (send it to index.php so Drupal can process it). That's perfect for Imagecache. It also sets "expires max" which just sets the cache header so clients keep files cached locally.

Since our Imagecache files use a slightly different path structure, we just modify the regex and apply the same rule.

location ~ (^/s3/files/styles/|^/sites/.*/files/imagecache/|^/sites/default/themes/.*/includes/fonts/|^/sites/.*/files/styles/) {

In fact we could probably send the request to @redirect always, but since most people have something very similar in their configs already there's no harm in trying the actual file.

Make sense?

OK, so if I'm reading this right, I should add something like this snippet to the README:

If you're using NGINX rather than Apache, you probably have a config block like this:
    
location ~ (^/sites/.*/files/imagecache/|^/sites/default/themes/.*/includes/fonts/|^/sites/.*/files/styles/) {
  expires max;
  try_files $uri @rewrite;
}

To make s3fs's custom image derivative mechanism work, you'll need to modify that regex it with an additional path, like so:

location ~ (^/s3/files/styles/|^/sites/.*/files/imagecache/|^/sites/default/themes/.*/includes/fonts/|^/sites/.*/files/styles/) {
  expires max;
  try_files $uri @rewrite;
}

Would that be appropriate?

I've gone ahead and pushed this up to git. If you don't think the snippet I posted in #16 is appropriate, please let me know.

Hi,

I am using Apache server, S3 and CloudFront. The module I am using is Media 2. The issue I am having is quite similar to the originally posted. When I upload new image using node edit page the image src path come as a "/s3/files/styles/". But on node edit page it is correct. I checked all the configuration are correct.

On admin/config/media/s3fs page I have check the following options "Use S3 for public:// files" and "Use S3 for private:// files"
On admin/config/media/file-system I have changed Default download method to "Amazon Simple Storage Service"
On content type field page "admin/structure/types/manage/news/fields/field_newsimage/field-settings" changed Upload destination to "S3 File System"

After all the configurations I am still getting wrong src path.

Ah, CloudFront... the stupid caching system to end all stupid caching systems. I'm not saying it isn't a great service (you get an astounding amount of service for very little money), but it's dumb, so whenever anything complicated comes along, it crashes and burns.

The way image derivatives work in Drupal is a two-step process:
1) The first render of a page with an image on it sets the derivative creation URL as the img src. Requesting this URL will contact the Drupal server and tell it to create the derivative on S3 and then redirect to. The "/s3/files/styles/" URL you're seeing is the derivative creation URL.
2) Subsequent renders of the page use the actual URL of the derivative file on S3.

Unfortunately, CloudFront's caching mechanism caches step 1, and then every request to that page's URL gives you the cached version with the derivative creation URL in it, instead of letting Drupal give you the one with the real URL.

This shouldn't really be a problem, though, because the creation URL will redirect to the real URL if it detects that the derivative file already exists. So I'm not entirely sure why you're having an issue with the "/s3/files/styles/" URLs. Maybe CloudFront is getting in the way again?

Could you please give a more detailed description of exactly what's going wrong?

Hi,

I have figured out what was causing this the issue. ImageMagick was causing the links the issues and in recent logs this message was showing up.

Unable to generate an image derivative at s3://styles/thumbnail/s3/Screen Shot 2015-11-23 at 4.58.30 pm.png.

I think it is not handling the file path for ImageMagick.

Ah, ImageMagick. That module's been problematic in the past. Unfortunately, I've never used it, so I have no idea why it would even have troubles like it does.

i have fix this issue

 function seven_image_widget($variables) {

  $element = $variables['element'];


   //echo"<pre>";print_r($element['#entity']->field_image['und'][0]['uri']);echo"</pre>";
     
      if($element['#entity_type']=="node")
      {
          $uri = $element['#file']->uri;
          $output = '';
          $output .= '<div class="image-widget form-managed-file clearfix">';
          $image_view_src= file_create_url($uri);
          if (isset($element['preview'])) {
            $element['preview']['#markup']='<img width="100"src="' . $image_view_src .'">';
            $output .= '<div class="image-preview">';
              $output .= drupal_render($element['preview'] );
            $output .= '</div>';
          }

          $output .= '<div class="image-widget-data">';
          if ($element['fid']['#value'] != 0) {
            $element['filename']['#markup'] .= ' <span class="file-size">(' . format_size($element['#file']->filesize) . ')</span> ';
          }
          $output .= drupal_render_children($element);
          $output .= '</div>';
          $output .= '</div>';

          return $output;
  }
}

Here we have an API to get multiple images belongs to responsive.
But the API response giving like below...
So is there a way to make this urls to process ?

field_image_1	"/s3/files/styles/1440px/s3/2018-07/profile_0.png?itok=fgdfg4"
field_image_2	"/s3/files/styles/768px/s3/2018-07/profile_0.png?itok=fg4"
field_image_3	"/s3/files/styles/375px/s3/2018-07/profile_0.png?itok=4545dfcg"

If you named the image styles defined in Drupal as "1440px", "768px", and "375px", that should work.

Above mentioned details are not clear need document or clear explanation for broken image of S3.