image_gd_load breaking on large images

I'm having trouble with a D7 site OCCASIONALLY breaking while handling a large image (e.g, a 12MB jpeg). The log messages look like this:

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 5784 bytes) in /var/www/vhosts/example.com/httpdocs/modules/system/image.gd.inc on line 237

This is inside image_gd_load, at the line:

return (function_exists($function) && $image->resource = $function($image->source));

I understand that the crash is happening because the server is running up against one system constraint or another on the large image such that something breaks; that's not my question/problem. Rather, my question is why GD is running at all -- what I'm doing on the page is presenting a imagecache-generated thumbnail of an already-uploaded image, so I'd think that the GD-based processing of the image has already taken place -- like I said, there are times when the server successfully renders the image, so the thumbnail should have gotten generated. So why am I hitting these compute-intensive operations on some code that (it seems) should just be spitting up a link to an already-generated image. I don't have enough of an understanding of Drupal's image/imagecache architecture to figure this out; does anybody have any ideas? Of course, ideas on how to avoid this extra processing would be especially appreciated...