robots.txt not found

What you'll want to do is alter the regex that checks file extensions. The default is:

$conf['fast_404_exts'] = '/\.(txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i';

change it to:

$conf['fast_404_exts'] = '[^robots]\.(txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i';

It will then not match to robots.txt

..you mean to:
$conf['fast_404_exts'] = '/[^robots]\.(txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i';

Yes, sorry about that missing /

I'm debating making this a default... Thoughts?

This has been made a default

Actually, this was not set to default...

I've just installed the last version and I still have a problem with robotstxt module. I need to use settings.php to be able to exclude robots..txt path.

Please consider to make default exception for robots.txt

This is wrong:

$conf['fast_404_exts'] = '[^robots]\.(txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i';

That's going to match any file that ends in r, o, b, t, s followed by . and one of the the extensions in the list. I'm getting an apache 404 for nosuchfile.txt instead of the fast404 response. nosuchfiles.txt gets me the fast404 response.

I think this is what you want, and you should update the readme.txt

$conf['fast_404_exts'] = '/(?<!robots)\.(txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i';

unfortunately, this will also match any-prefix-robots.txt, which we may not want.

patch for above

I actually realized this and corrected it as well when I was providing regex for private files.

However, I used (?!robots) and it seems to work A OK