Add queued image generation support for large sites [#876900]

Hi,

I'm running a large site, high traffic, with loads of images.

Sometimes there's so many new image to be generated, the PHP process crashes before removing the imagecache /tmp/lock file. This leads to a never-ending loop of imagecache trying to generate the image, but is never able to due to the PHP limit. Imagine dozens imagemagick convert actions at once.

See also this comment,
http://drupal.org/node/461014#comment-1670010

A solution would be to implement a "generation queue".

1. User requests image, not yet generated
2. Imagecache adds image to queue, and shows a default image while we wait, this could be a transparent pixel.gif 1x1 (sized by the images' dimensions)
3. A cron process, or other batch process, processes the queue, and generates images 1-by-1, so never reaching a PHP limit
4. After the queue finishes, the next users will see the generated image

Only the first user will see the transparent pixel. All subsequent visits will show the image.

I'll probably work on this myself and come up with a patch (may take a few weeks). Any suggestions are welcome.

Comments

Comment #1

scotjam commented 9 October 2010 at 17:21

Hi morningtime

Image queue generation sounds like a swell idea.

If you could create the presets on the fly, e.g. via cron using Image Cache Batch (http://drupal.org/project/imagecache_batch), might that alleviate the concern of over loading on too many image converts at a time?

Interested to here what you think, or have discovered, as I'm looking for a pre-emptive solution too, though have not experienced the problem myself.

cheers
scotjam

Comment #2

jordanmagnuson commented 14 October 2010 at 18:08

Definitely interested in this... actually, I'm really surprised this functionality hasn't been added to imagecache in all the time it's been live. I've got a fairly large site that's image-heavy with large images, and any time I flush a preset, or add too many images in too short a time, the whole thing comes crashing down. And I'm on a VPS with 512MB of dedicated memory.