Amazon S3 fails on large video files. [#2045287]

I'm trying to use Zencoder for encoding and AmazonS3 for my storage. Zencoder is in integration mode so the rendered files are small. Zencoder is storing the completed files on S3 with out issue. The only issue comes in when I try to use S3 to store the raw files. I can't get it to store anything over about 100Mb. I'd like to have files limited to 1000Mb. It works fine if I use local storage for the originals but it fails if I try to use S3.
I have found no errors in my logs.
This is my first Drupal bug report so I apologize if I've done it wrong.

--Mike

Comments

Comment #1

Jorrit commented 18 July 2013 at 21:35

It could be that the time it takes to upload the video is too large and PHP or MySQL times out. What is the behavior you receive when you upload a video that is over 100 MB?

Comment #2

zmikez commented 19 July 2013 at 13:26

I get the uploadprogress bar but mine is broken and doesn't actually show progress. (but that's a separate issue) I see local hard drive activity (external drive) so I think the file is being uploaded. After a while, the uploadprogress bar goes away but instead of listing the file as it would if it had uploaded, it goes back to a "browse" button. ...and as I mentioned before, a big fat nothing in the logs. Below is som of the settings from my system that I think are pertinent to this issue:

max_execution_time 200000
max_input_time 200000
memory_limit 2000M
post_max_size 1500M
upload_max_filesize 1500M
I'm running fcgi mode so this also has to be set: FcgidMaxRequestLen 1073741824

Comment #3

Jorrit commented 19 July 2013 at 13:30

There could also be a Fcgi timeout or a MySQL timeout.

Comment #4

zmikez commented 19 July 2013 at 13:47

I just found the issue. The file field doesn't like fcgi. I switched my site to Apache module and it works fine. Uploadprocress also works. I wonder if uploadprogress is getting in the way.

Comment #5

beanjammin commented 13 September 2013 at 20:58

When you say it is working fine, do you mean that the files are now being uploaded and transfered to S3 ok?

I ask because I'm having trouble with larger files (160MB) where the file upload is working as expected and completes, it then appears to begin the transfer to S3, the local (to the server) copy is then deleted from /tmp and... nothing. The file does not make it to the S3 bucket and no errors are logged in WD. With smaller files (at least up to 25MB) everything works fine.

I am also using S3 in conjunction with Zencoder. Video-7.x-2.10, AmazonS3-7.x-1.0-beta7, AWSSDK-7.x-5.4. I am using PHP5 as an Apache module (Debian Wheezy packages).

Any pointers on where to look for why this is failing would be much appreciated.

Comment #6

Jorrit commented 16 September 2013 at 07:40

Check all timeout values that are involved in the upload process.

Comment #7

beanjammin commented 23 September 2013 at 20:40

Thanks for the follow-up Jorrit.

We have actually run into this same issue outside of the video module, so it would seem to be an amazon_s3 module issue. I am posting to the issue queue there and will update back here if I learn anything.

Comment #8

faizan_abid commented 3 March 2014 at 18:00

Hi!

Any updates on this issue? I am facing the same problem. Uploading files < 25 MB is okay most of the times, but might fail as well. Files above 25 MB don't even appear in /tmp or the S3 bucket. I have made sure my environment settings are set to high values for execution time and upload limits.

Similar to zmikez, my uploadprogress doesn't work too, with both S3 and local files. Have been searching around and haven't found any solution yet.

Comment #9

beanjammin commented 13 March 2014 at 19:01

Issue summary:

View changes

faizen_abid, I wasn't able to get past this issue, so ended up working around it by using the Media Derivatives (https://drupal.org/project/media_derivatives) and Simple FFMPEG Engine (https://drupal.org/project/media_ffmpeg_simple) modules to do the storing and encoding locally. No doubt that's not what you want to hear, but I didn't get very far in my investigation of the S3 upload issue and needed to get the project out the door.