Using private download with reverse proxy

I'm using Pressflow 6 with varnish reverse proxy. It's otherwise working fine but I can't get the rewrite working right.

Apache2 listening 127.0.0.1:8000
Varnish listening *:80

settings.php url_base = http://192.168.0.112:80

admin/settings/file-system
File System Path: sites/default/files

admin/settings/private_download
private download directory: private
htaccess content:

RewriteEngine on
RewriteBase /system/files/private
RewriteRule ^(.*)$ $1 [L,R=301]

I'm using CCK with upload path module to move the files into the private directory. This is also working fine.

The problem is, that when I click at the link on the node for the file it gets redirected to http://192.168.0.112:8000/system/files/private/file.pdf and get's file not found error.

If I remove the :8000 from that URL it works great (ACL including) So I need to get rid of the port 8000. How do I do that?
I tried to change the rewriterule to: RewriteRule ^(.*)$ http://192.168.0.112:80/$1 [L,R=301] but it didn't do anything (also restarted the varnish and apache to flush the caches)