Disable rewriting in files + have a 404 handler = trouble

The main Drupal .htaccess have:

ErrorDocument 404 /index.php

The .htaccess we create in the files directory does:

<IfModule mod_rewrite.c>
  RewriteEngine off
</IfModule>

That means that when trying to access an URL that doesn't correspond to a file in the files directory, the index.php is triggered without a q= parameter, making Drupal basically display the front page.

Example URL: http://drupal.org/files/tototo.jpg

I don't understand why we are disabling rewriting in the files directory, does that even make sense? Is that a security issue I'm not aware of that makes that required?