This involves following changes in the GUI:
1) Add to '{host}/admin/user/permissions' an option "Allow setting robot.txt directives" with OR-values:
1a) "No" (selected by default)
1b) "Yes" per user role

2) Add to '{host}/admin/content/types/add' an option "robot.txt derictive" with OR-values:
2a) "Automatically listed as Allow"
2b) "Automatically listed as Disallow"
2c) "Automatically unlisted and UNfiltered" (selected by default)
2d) "Automatically unlisted and filtered (removes any listing of the node in the robot.txt)"
2e) "Manually configurable per (!) node by authorized user roles (check permissions)"

3) If 2e is selected: Add to '{host}/node/[nid]/create' and '{host}/node/[nid]/edit' an option "Robot.txt directive" with OR-values 2a up until 2d.

NB: since Drupal installations with clean URLs enabled allow nodes access via '{host}/node/[nid]' AND '{host}/{automated URL alias}', both (!) paths should be handled with this feature.

Comments

hass’s picture

hass commented

If you really need this, please provide a patch. I wouldn't give editing the robots.txt out of the hands of an admin... and you can use nodewords for node type specific robots entries in meta tags. That said - there is already a solution available and I do not think very much that this static robots.txt should be edited by "end-users".

If you are fine with the nodewords abilities or do not plan to provide a patch, feel free to close.

not_Dries_Buytaert’s picture

not_Dries_Buytaert commented

@hass: The core of this feature request is that at least the admin can set the robot directive per content type and node (must-have feature 2). The ability to set such permission per role is of secondary importance (nice-to-have sub-feature 1), though just a general best-practice for configurable settings.
Unfortunately, I do not master OO PHP sufficiently yet. :-( So, I leave this issue open, so others can find it and perhaps even help out.

hass’s picture

hass commented

Why are you not using the nodewords module?

not_Dries_Buytaert’s picture

not_Dries_Buytaert commented

It should be possible to (either automatically or manually) set directives within the robot.txt file. Site-wide files like those should be properly manageable by itself and not rely on search engines to support a workaround, such as with metatags http://www.w3.org/TR/html401/appendix/notes.html#h-B.4.1.2
Subsidiary, as described in topic http://drupal.org/node/532278 , the module http://drupal.org/project/nodewords doesn't support meta tags per content type.

hass’s picture

hass commented
Status: Active » Closed (works as designed)

Cannot find the nodewords issue, maybe kiam renamed it. Node type meta tags are available via an extra module and in 3.x out of the box as i know. Could also be integrated in latest 1.x versions as the link to the module is no longer on project home

For the robots.txt module it's really impossible to add 1million or more urls to the robots.txt. This would result in monster file we and search engines cannot handle.

I say now - it's by design only for this reason. The meta tags are not workrounds... They are the solution for what You are searching for.

hass’s picture

hass commented

http://drupal.org/project/nodewords_nodetype