Apache with MultiViews configured: robots.txt is not displayed, instead the frontpage of my site is shown

The error is caused by the wrong declaration of the menu for robots.txt. If you want to make it work sinply change the robotstxt_menu() function to the following code:

function robotstxt_menu() {
  $access_config = array('administer site configuration');
  $access_content = array('access content');

  $items['robots.txt'] = array(
    'page callback' => 'robotstxt_robots',
    'access arguments' => $access_content,
    'type' => MENU_CALLBACK,
  );
  $items['admin/settings/robotstxt'] = array(
    'title' => 'RobotsTxt',
    'description' => 'Manage your robots.txt file.',
    'page callback' => 'drupal_get_form',
    'page arguments' => array('robotstxt_admin_settings'),
    'access arguments' => $access_config,
  );

  return $items;
}

The difference is in 'access callback' changed in 'access arguments'. The first would define the callback used to check if a user has access to the menu (which by default is user_access()), while the second defines the arguments to pass to such function.

Without that correction the module doesn't work, as it uses the value TRUE while Drupal expects a string.

To see it working you need to disable, and re-enable the module.
If you change the declaration of a module menu by manually changing the code, Drupal will not update the definition it saves on its own database tables. To force Drupal to update the list of menu it keeps, you must go to the modules list page, and disable and re-enable the module of which you changed the menu definition; it also works if you go to that page and click on save.

#1 is wrong code. We do not make robots.txt depend on 'access content'. robots.txt must be shown every time - also on sites that have content disabled. We have had there an issue ~1-2 years ago while D6 version was in early dev and I missed this requirement.

See http://drupal.org/node/109157 and jump to

We support 'access callback' => TRUE (and FALSE of course).

This is the current and correct code:

  $items['robots.txt'] = array(
    'page callback' => 'robotstxt_robots',
    'access callback' => TRUE,
    'type' => MENU_CALLBACK,
  );

@magenbrot: Are you using any type of language detection? Moving first as support until we are sure what's going wrong.

All times I used 'access callback' => TRUE, all I got was that the menu was not available.
To make it appears, I changed the line to 'access arguments' => TRUE.

I cannot say if it's a bug in Drupal, or a bug in Drupal documentation, but I can say that 'access callback' => TRUE doesn't work.

Very strange that robotstxt module works for me on my 6.8 box and all previous versions... never seen this problem myself. Could you investigate if this is a menu caching issue?

I checked the core code which handles the menu callbacks, and the code is the following (the code is truncated to the first lines):

function _menu_check_access(&$item, $map) {
  // Determine access callback, which will decide whether or not the current
  // user has access to this path.
  $callback = empty($item['access_callback']) ? 0 : trim($item['access_callback']);
  // Check for a TRUE or FALSE value.
  if (is_numeric($callback)) {
    $item['access'] = (bool)$callback;
  }

Therefore passing TRUE for 'access callback' in the menu callback definition should work like expected (and how declared in the documentation).
I also changed the menu callback definition of a module I use on my test site to use TRUE for 'access callback', and it works.
If there is a problem, then it's not caused by the menu callback definition.

Thank you for investigating this. Good to know code is correct.

I installed the module on my test site, and I am able to see the robots.txt file.
It is surely not a bug of the module, or I should have the same issue magenbrot has.

Cannot repro, no feedback from magenbrot, must be menu path caching or update.php has not executed cleanly in site maintenance mode.

I'm having the same issue.

It seem to come from the dot in the menu item path : it's annoying ! If I change it to robots_txt it work, but if I keep robots.txt nothing happen, and the frontpage is displayed.

Clear your caches... try to disable and re-enable the module... also check your webserver logs... this may be an Apache mod_security issue or something like this. On my Apache 2 and IIS 5.1 it works well for sure.

I cleared the cache, I uninstalled / resinstalled the module, rebuild menu via devel, the default robots.txt file has been renamed, but I steel have the frontpage instead of a robots file.
I checked my log :
"Negotiation: discovered file(s) matching request: //robots.txt (None could be negotiated)."

@magenbrot: No need to go into maintenance mode for install/uninstall. This is only required for upgrades. I checked your server and it gives me a plain 404 error and than the site shows me the front page! So it's save that you may have cluttered your installation - this is not the standard Drupal behaviour!

Some questions:
1. Have you installed any 404 error handler modules like CustomError module?
2. Do you have the robots.txt in the websites root folder deleted?
3. Are you using the standard .htaccess from Drupal or have you altered the file?

I wonder - why http://test.magenbrot.net/?q=robots.txt works well! Please also provide a list of all enabled modules. Try yourself by disabling one by one, until the issues goes away...

There are a few possible reasons:
1. Drupal Menu caching issue (I'm not aware about such a bug)
2. Apache mod_rewrite rules are wrong
3. A bad module is installed (I expect this is your issue)
4. Application firewalls in action

It's soooo time wasting to analyze this issue... If the above gives you no idea, feel free to contact me for a remote session on your server/site.

Please report back your findings.

A quick look to customerror modules source code and a request to your site path customerror showed me you have customerror installed. Disable it as very first action, please.

Moving to customerror issue queue.

I haven't customerror installed on my website and I have the frontpage instead of the robots.txt file. It's not a cutomerror issue.

If you enter http://test.magenbrot.net/robots.txt in Firefox and take a look to Firebug you see the 404 error (see attached). The page you see should come from CustomError! I expect that CustomError is the source of this issue and I will try to test this later on my own box. As you are having a testsite - are you able to install a plain vanilla Drupal 6 with only robotstxt module enabled, please? Than switch on the other modules - step by step and see when it breaks or go the other way and turn all your modules off - step by step. Also please remove your robots.txt at all from website root or rename it to "robots.rem" - not sure if this helps... but this is how I run my sites. Make sure you use only default .htaccess files...

I'm also running many of the modules you have installed... but do not have:

AdSense with:
AdSense Click Tracking
AdSense core
Managed ads
Amazon API
Better Formats
Comment Subscribe
Custom Error
Counter
Forward
IMCE (not sure if active, but on my box)
Lightbox2 (not sure if active, but on my box)
PROG Gallery
CAPTCHA
Image CAPTCHA