In the D5 version (IIRC) it was possible to set the background color to -1 which would be passed through to the rotate function and make the surrounding area transparent. This doesn't work in the current D6 version.
Here is a patch that re-instates that behaviour.
Of course, it's not very effective with just jpegs, but helps a lot with PNGs and the imagecache_actions pipeline.
Here's a patch which seems to do the trick, including a small explanation in the UI.
Without it, I can't do things like: