I would like to continue the discussion from #1291628: Reduce calls to S3, specifically related to image style processing for image fields.

Problem:
When users upload images to an image field the AmazonS3 module will upload them directly to S3 without creating image styles. This is by design since core only creates image styles when they are first requested, usually when an image is embedded in a node. This means for every image style (thumbnail, medium, large) the original image will be downloaded from S3, the image style will get generated and saved back to S3.

The patch in #1291628: Reduce calls to S3 solves the problem with extra calls to S3 to check for existing image styles and directory paths but does not solve the long delays to get each image style generated.
Even when uploading smaller images to a node and then saving the node it can take minutes for the image style to get generated and displayed to the user.

Possible Solutions:
1. One solution would be to just display a background image that says "image is processing" to let the user know about the delay.

2. Another solution was proposed in patch #18 http://drupal.org/node/1291628#comment-5229502 and #49 http://drupal.org/node/1291628#comment-5533206 to create a local file cache that will generate the image styles locally to show an image style to the user right away. Once they are copied to s3 they will use used in node displays. My initial testing showed a dramatic improvement in performance.

3. Another solution would be to use local storage for the initial image upload and image style generation. Then a batch process would check for all local images that have not been saved to S3. It would make sure that all image styles have been generated for that image first. It would then copy the original image and all the image styles to S3 and change the uri and timestamp in the "file_managed" table (or creating a new record, setting the status of the old record to 0 and changing the fid of the field_data_image field)? In combination with the other patch it could also write the image styles to the amazons3_file table.
It would then delete files from the local file system (not sure if Drupal will automatically remove the image styles then the image has a status=0 in the file_managed table).

The advantage of that approach is that we could possibly remove the extra database calls to check with every request if an image style does exist since we now know for sure it exists on S3. Only when image styles are modified or flushed would they have to get regenerated. Ideally that would happen in batch mode as well.

Personally, I'm leaning towards the batch processing approach. I would think we would have an admin menu to select which image fields should be enabled for "S3 batch upload" and how many to process on each cron run. Maybe show how many have been processed already (uri starts with s3://) and how many are waiting to be processed (uri starts with public://).
The processing would follow the steps above. Is there an easy way to check if all images styles for a specific image are available?
I'm unsure about what to do if an image style changes or all image styles are flushed. On my site with 100,000 images I'm more concerned preventing an "accidental" image flush ;-)

CommentFileSizeAuthor
#8 amazons3_mover.tar_.gz3.02 KBaasarava

Comments

mototribe’s picture

mototribe commented

http://drupal.org/project/media_mover would be another potential solution but it doesn't seem ready to do that in D7 yet?

I just finished importing a few thousand of user and user pictures using the migrate module. I had local image storage selected
and then I ran a script to create all image styles that I needed for those pictures:

	$result = mysql_query("SELECT fid, uid, uri  FROM file_managed
						where uri like 'public://picture/%'
						ORDER BY fid asc", $con);
	
	//loop over the results and create the image styles
	while ($row = mysql_fetch_array($result, MYSQL_BOTH)) {
		$uri = $row['uri'];
		
		foreach(image_styles() as $style) {
			if ($style['name'] == 'user_thumb' || $style['name'] == 'user_square'){
				$dest = image_style_path($style['name'], $uri);
				if (!file_exists($dest)) {
					image_style_create_derivative($style, $uri, $dest);
				}
			}
        }

Once all styles had been created (takes about 0.5 sec for each image style using ImageMagick library, not GD) I manually copied the originals and styles to s3 using the same folder structure (I used Cyberduck on the mac)
Then I renamed the uri in file_managed from public:// to s3:// and everything worked like a charm, even after copying the DB to my dev and live site.
Oh, and I disabled the calls to s3 to check if an image style exists before showing it EVERY TIME since I now know that the image styles are already there.

//commented out line 130-135 of AmazonS3StreamWrapper.inc
if ($path[0] == 'styles' && !$this->getS3()->if_object_exists($this->bucket, $this->getLocalPath())) { ...}

I should probably also remove any image flush functionality to ensure that my image styles don't get removed by accident.
I'm trying to figure out how to setup a permission for drupal access on S3 to only allow upload but not delete
http://stackoverflow.com/questions/9398541/how-to-remove-delete-permissi...

mototribe’s picture

mototribe commented
Title: Optimize AmazonS3 for image fields » Get S3 404 errors
Version: 7.x-1.x-dev » 7.x-1.0-beta6

Since I have disabled the "if_object_exists()" for every single image style request I would still like to be able to find out when an image is showing up broken for a user.
I'm trying to get a simple 404 list from S3 but haven't figured out an easy solution: http://stackoverflow.com/questions/9445130/is-there-a-simple-way-to-repo...

This Javascript trick might be another way to do it:

//http://stackoverflow.com/questions/92720/jquery-javascript-to-replace-broken-images
<script type="text/javascript">
function ImgError(source){
    source.src = "/my-missing-file-routine";
    source.onerror = "";
    return true;
}
</script>
<img src="someimage.png" onerror="ImgError(this);"/>

I would just have to pass the image name to the routine.

visualfox’s picture

visualfox commented

I am working on a patch based on solution #2

@mototribe you should change back the title of this thread to: Optimize AmazonS3 for image fields

justafish’s picture

justafish
she/her
Primary language English
Location London, UK
commented
Title: Get S3 404 errors » Local image field cache
rickvug’s picture

rickvug commented

What about a setting per each image style with the following options:

  1. Generate and immediately save image derivative to S3.
  2. Save image derivative to S3 on cron.
  3. Generate image derivative only on page load.

Option 1 could be used for something like a thumbnail image that would immediately be used by the image field widget. What could happen is that the image is created in the same page request that saves the original file. Option 2 would be a cron job that checks for the existence of the image styles in the amazons3_file table. If the image derivative wasn't already there it would be created on cron so that user's wouldn't need to wait on page load. Option 3 would be what we have now.

Any thoughts on how feasible an approach like this would be? It has be advantage of only keeping files in S3 and using the existing amazons3_file table as intended. I'm wondering if option 1 would cause uploads to become too slow, especially if more than one image style is generated on initial upload. Cron generation may not provide a benefit if the derivative image is likely to be required before cron has the chance to run.

infines’s picture

infines commented

Option 3 would cause huge performance problems, and is actually a step backward.

I'm all for a delay in uploading to generate the image cache style right away. Drupal mostly already does this with local file storage.

visualfox’s picture

visualfox commented

By option 3 you mean this one? "Generate image derivative only on page load." or the third option in the original post?

The cache solution proposed in patch #18 http://drupal.org/node/1291628#comment-5229502 is the way to go in my opinion. I did a proof of concept using the /tmp folder but a more controlled cache solution is needed.

So we cache the file locally when just uploaded (maybe a way to mark which field should be cached) and we have a cron job who delete the file after a while. Maybe create a hook to let third party module to veto the deletion... This solution is nice as you don't have any DB call just a check if the file exists in the cache or not (and that only when check for derivative). If it doesn't then we download it from Amazon S3 or we stream it.

If after that someone need to implement some logic to ensure that the derivative image are correctly generated before the cache get delete that should be put in it's own module (as the case are probably not generic enough). I plan to port patch #18. It's actually really straightforward I just didn't get a chance to work on that just yet.

aasarava’s picture

aasarava commented
StatusFileSize
new amazons3_mover.tar_.gz3.02 KB

Regarding the batch processing approach outlined by mototribe in the original post: I'm attaching a module that does just that. It's essentially a working "Media Mover" module for Drupal 7 that uses the Amazon S3 wrapper.

It works like this:

  1. File uploads use the standard, local public:// or private:// URI scheme -- not the s3 scheme.
  2. On each cron run, the module checks Drupal's file_managed table for local files. (Actually it looks for a subset of these files, like only files in the photos/ and videos/original directories, based on some settings that are currently hard-coded in the module.)
  3. Eligible files are added to Drupal's cron queue.
  4. A cron worker goes through the queue and checks whether each file already has corresponding image style derivatives. If so, it moves the file and its derivates to s3. If not, the file can get requeued in several hours for another try (when hopefully the derivatives have been generated.)

Obviously, because of the hard coding of subdirectories in step 2 and image styles in step 4 above, this module isn't ready for prime time. It'd have to manually edited to work for any given site. Also the module logs to wachdog quite heavily for debugging purposes.

But hopefully, if someone has the time to make the settings configurable, this will provide a start for a good companion module to the amazons3 project.

Edit: Sandbox project set up here: http://drupal.org/sandbox/aasarava/1712322

infines’s picture

infines commented

Storage API has implemented this solution and it works wonderfully. However storage api has other problems that lead me back to using this module instead.

How it works:

1. Images are uploaded to local storage. This allows for speedier thumb generation.
2. On cron run, if all the thumbs are generated the files are moved to S3.
3. Celebration.

infines’s picture

infines commented
Priority: Normal » Major

Any thoughts on this justafish?

natbro-1’s picture

natbro-1 commented

I don't have a solution to offer for the performance issues, but please note that most of the ideas above would work against the benefit I am trying to use the AmazonS3 module for: managing content across multiple front-end drupal nodes in a large deployment. Any content deferred on the local file-system of a single node would mean other nodes don't have access to it for some window of time, leading to 404's which may get cached by servers and proxies, or cause issues with drupal's cache. The great thing about AmazonS3 is that the database and the global/S3 file-system are in sync between as many nodes as you like. As currently written, there is a non-destructive race where multiple front-end nodes may try to generate missing image styles simultaneously if they are getting simultaneous requests for them, but that is all I see.

infines’s picture

infines commented

@natbro, this is why this feature should be added in a sub module or through an on and off optional feature.

I'm not saying we need to do everything Storage API does, but it does allow for an optional initial file destination. Otherwise it works as natbro describes.

wedge’s picture

wedge commented

@aasarava there seems to be some changes between the sandbox and the attached amazons3_mover module. Is the attached version the one that is most up to date?

aasarava’s picture

aasarava commented

The sandbox is newer than the attached one. However even the sandbox is in an early state and need work. If you're familiar with module development, it shouldn't be too hard to customize it for your needs.

From the description, Storage API looks like it might do this better than my module does. @gridbitlabs, what are your concerns about Storage API?

infines’s picture

infines commented

@aasarava Storage API has some great ideas and features. However it is a very wonky module that isn't as maintained as this one. Why try to reinvent the wheel? A great solution for a local image field cache would be what Storage API does and upload images files to the local server then allow the local server to generate the image styles. Once that is complete, image files are moved to the server on cron run. This prevents uploading to amazon, then redownloading to drupal for image style generation, and reuploading to amazon.

infines’s picture

infines commented
Version: 7.x-1.0-beta6 » 7.x-1.0-beta7
mototribe’s picture

mototribe commented

we have implemented a similar solution, works like described in #9

infines’s picture

infines commented

@mototribe Is there a patch or submodule?

8ballsteve’s picture

8ballsteve commented

@aasarava - Your sandbox module looks pretty close to what i need to get in place to implement some initial local storage option.

Have created a custom version that follows field data rather than folders and will post a zip here when it's done.

infines’s picture

infines commented

I've done some searching and I've found this module also...all though it's not for D7, it could offer some insight into this situation.

http://drupal.org/project/imageinfo_cache

infines’s picture

infines commented

There is also this module...though I'm not sure if it just delays page load...

http://drupal.org/node/1915996?no_cache=1360768003

infines’s picture

infines commented

EDIT: Double post.

mototribe’s picture

mototribe commented

no, it's a custom code we've written for our specific use case

infines’s picture

infines commented

could you post the code and explain how it works? or start a sandbox module?

8ballsteve’s picture

8ballsteve commented

Have created a patch for @aasarava sandbox module that allows for a field by field implementation - might be of help to someone.

http://drupal.org/node/1917988

infines’s picture

infines commented

Here is module that I'm looking into for solution...I don't know how well it works with Amazon S3, but hopefully further testing can indicate that...

http://drupal.org/project/rules_image_styles

infines’s picture

infines commented
Title: Local image field cache » Option to upload files on cron run
Priority: Major » Critical

It turns out Rules Image Styles doesn't support stream wrappers and has no plans to, so we are back at square one essentially.

I've been thinking about this issue. Rather than creating a special use case for images, why not just have an option per field that tells the module to:

If the option is checked:
1) Upload Files to Amazon S3 on cron run?
2) Add a pluggable system that allows files to be uploaded to S3 on cron run after certain tasks are completed.

If it remains unchecked:
This module functions the same as it does now.

This could allow for further use cases to take advantage of a similar situation (For example, after image styles are generated or after a word document has been converted to PDF). Then other modules can come in and help solve the problem with their necessary tasks. Images would be supported natively because they're apart of Drupal 7 core.

Thoughts?

balashine’s picture

balashine commented
Title: Option to upload files on cron run » How to Upload the files to both Amazon and the public :// File system

1) I Have installed all the Modules related to the amazon ,
Amazon s3
Amazon Crons
Amazon PHP libraries

2) I enabled the Amazon S3 in the file system,

3) If i change the upload location to s3:// means the images are saved in amazon bucked.

4) If change the file system to public and upload the images not storing in the amazon bucket,

can any one suggest how implement uploading both files to public:// and amazon s3://

justafish’s picture

justafish
she/her
Primary language English
Location London, UK
commented
Priority: Critical » Normal
deviantintegral’s picture

deviantintegral commented
Title: How to Upload the files to both Amazon and the public :// File system » Option to upload files on cron run
deviantintegral’s picture

deviantintegral commented
Status: Active » Fixed

I investigated using cron to upload image styles, but was concerned about the possibility for 404s for sites with a large number of image styles. Also, to help keep costs managed we'd need to add in a way to identify what image styles are used with what fields, otherwise for some sites we'd be generating a huge number of unused derivatives.

I ended up solving the perceptible performance problem with a shutdown function that uploads the image after it's been served. I'll be tagging a beta off of the 7.x-2.x branch today, so for anyone where this is an issue I'd suggest checking that branch out.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.