After experiencing some weird behavior with images that kept loading for a long time and disappeared after loading was complete, I started digging through all relevant code.
Fifteen minutes later I was able to track the bug down to this module.
When defining a CCK field for uploading images, you can specify a maximum resolution. The function filefield_validate_image_resolution checks for this and resizes the image if necessary. After resizing the image it updates the filesize in the passed file object.
The problem: If you have the ImageAPI module enabled, its resizing function gets called but the filesize is not updated.
I've attached a quick fix patch that I'm using currently, though I'm sure a more elegant patch can be written.