I'm using the Zencoder / S3 combination and videoftp module. It worked fine the first few times I tried it, but I tried to attach a larger video last night (120MB) and everything died on me. I've noticed the same behavior with any file that large or larger; while smaller files seem to be working.

Here is my process:

1) upload video via ftp to "ftpvideos" folder
2) create a new piece of video content and select the uploaded file

after saving, the node is created and file is moved to the "videos" folder as it should be. I get green status messages saying that it is queued for processing, and queued for transfer to S3. No error messages in watchdog (just some warnings about not being able to determine the height and width). So far, everything looks good.

3) run /admin/reports/status/run-cron (or cron.php)

Here .. with large files... it spins for a few seconds and then gives me a white screen of death. Nothing at all. No errors or successes in watchdog either. However, with smaller files, this step works fine: I see the message that it was successfully uploaded to S3. ( The "small" files I've tried are 5 - 50MB, the "large" files 120 - 210MB )

4) if I try running cron again now I get messages in watchdog about attempting to re-run cron while it is already running.
5) at this point, I know I'm screwed, but if run video_scheduler.php for good measure, it logs this in watchdog:
We did not find the file id: 61 or it is still queued for ffmpeg processing or S3 push.

eventually cron will realize it's stuck and allow itself to re-run. When it does, it wont do anything. Just runs and quits - logs in watchdog that it successfully ran, but nothing else. It doesn't appear to think there is anything to process.

It seems to be failing when trying to transfer the file up to my S3 bucket. I confirmed that the file never reached S3, and there is no log on Zencoder's side of them ever seeing it.

Any suggestions?

Comments

hypertext200’s picture

hypertext200
he/him
Primary language Singhalese
Location New York City
commented

you must run cron in the CLI interface, go to CLI and run the cron. If you run cron through the web interface it will never work. Can you send me the link to your site.

bbenone’s picture

bbenone commented

Heshan,

Thanks for the quick response.

Yeah, I tried that as well. I have crontab setup to run both cron.php and video_scheduler.php as you recommended in the documentation (invoking via command line "php cron.php"). Same as doing it over the web, it just hangs after running. I thought maybe it was just taking a long time to upload, but I have given it quiet a while. Is there something I'm missing here?

Not surprisingly, this is the offending line. It's not ever returning from what I can tell:

if ($this->s3->putObjectFile($filepath, $this->bucket, $filename, $perm))

I had just started diving into the Amazon documentation to see if I could make any headway on why it would be hanging. Any pointers?

hypertext200’s picture

hypertext200
he/him
Primary language Singhalese
Location New York City
commented
Status: Active » Fixed

It seems PHP time limit exceed, add time limit to 0 on the script run.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

bbenone’s picture

bbenone commented
Status: Closed (fixed) » Active

re-opening.. i'm still working on this... just kinda slow

mgladding’s picture

mgladding commented

Is there any more info available? I'm having the same problem with large files...

mgladding’s picture

mgladding commented

Also, it looks like the large file is making it's way to Amazon just fine, but it's not being recorded in the video_s3 table. It also seems to clog up the process for any smaller files uploaded after the large one (although the smaller files I uploaded after the large one are being written to the video_s3 table.)

I can also confirm that if I uninstall/reinstall the module then I can upload the smaller files (under 100mb) and they are successfully processed.

In the meantime, I'll be telling my client to edit their videos into parts, but I fix for the problem with large files would be great.

hypertext200’s picture

hypertext200
he/him
Primary language Singhalese
Location New York City
commented

That is a limitation of the http file transfer, that's why I added S3 push to the cron job. I tried directly push file to the S3 and it took me a lot of time to push.

I know when cron start running the php_time_limit is set to 0, then we do not want to worry about the PHP timeout. If the files are transferring from your local computer slowness is totally depends on your connection.

Is there any specific error you found when u transfer large files?

kleinq’s picture

kleinq commented

I'm having the same watchdog error, but the file isn't large - only 3.5mb:
Getting a watchdog error: We did not find the file id: 3713 or it is still queued for ffmpeg processing or S3 push.

The file is a 3.5mb video. Its arrived on S3. But no job has been submitted to Zencoder.

I've uninstalled, reinstalled, no luck.

bbenone’s picture

bbenone commented

Yeah, I've been messing with this again this week too.

In my case it has nothing to do with my local connection speed. I am using ftp to upload them to my host. They can't get from there to S3.

Cron is done so fast that I have a hard time believing that it is a PHP timeout. Memory issue maybe, but I don't see logs of that. I tried setting the php time limit to 0, but that didn't help.

I, also, see that the failed files are actually in my S3 bucket. I just noticed that today, kinda surprised me. So, I guess, even though the "s3->putObjectFile" function is never returning, it is getting the file up there.

I skimmed through the docs some, but this is my first exposure to S3 so I'm a bit of a novice here... are there any other options for getting the files moved from the server to S3? Maybe, like the ftp video module/upload field there should be one for S3 as well? That way we could transfer directly to S3 and then attach? (just an idea).

My current working solution, which I don't like, is to lower the video quality to shrink the file size before uploading.

hypertext200’s picture

hypertext200
he/him
Primary language Singhalese
Location New York City
commented

@kleinq , did u enable the Zencoder module and select it from the Transcoder settings? if you select the Zencoder then it should work.

@bbenone Yep, memory limit is for file upload.

kleinq’s picture

kleinq commented

@heshan.lk I did enable the module and set the configuration via the admin pages. I've also re-installed the module several times. Are there any variables/configuration logs I should be looking at?

kleinq’s picture

kleinq commented

Any Update?
Thanks!

Jorrit’s picture

Jorrit commented
Version: 6.x-4.1-rc6 » 6.x-4.2
Component: Code » Amazon S3
Assigned: Unassigned » Jorrit

I will try to add some set time limits to the cron code.

Jorrit’s picture

Jorrit commented
Status: Active » Postponed (maintainer needs more info)

Please try if version 6.x-4.6 beta 1 fixes this problem for you. In this version the Zencoder and S3 parts have been refactored and tested.

You can read about it at http://drupal.org/node/1346434

Jorrit’s picture

Jorrit commented
Status: Postponed (maintainer needs more info) » Closed (cannot reproduce)

Closed because of lack of response.