HowTo: Block search engines from reading or indexing a specific page

While there are many ways of blocking search engines from reading or indexing certain pages on a site, it may be easiest to use the Metatag module.

The tag that controls this is the "Robots" tag. This meta tag can be found in the "Advanced" section of the meta tags form. The Robots tag is a meta tag equivalent of the robots.txt file that provides a means to more easily control individual pages, rather than making a site-wide change.

The following options are provided for the Robots tag:

• Allow search engines to index this page (assumed).
• Allow search engines to follow links on this page (assumed).
• Prevents search engines from indexing this page.
• Prevents search engines from following links on this page.
• Prevents cached copies of this page from appearing in search results.
• Prevents descriptions from appearing in search results, and prevents page caching.
• Blocks the Open Directory Project description from appearing in search results.
• Prevents Yahoo! from listing this page in the Yahoo! Directory.
• Prevent search engines from indexing images on this page.
• Prevent search engines from offering to translate this page in search results.

By default the first two options are the automatic default, regardless of whether they are actually checked, i.e. all search engines will index all pages and then following links provided on the page.

In order to block a search engine from reading or indexing a specific page, some consideration must be taken as to what the specific goals are:

• Should search engines be able to identify that the page exists, but just not show the page's meta data in its summary?
• Should search engines be able to show the page in search results but not store a cached version?

The most common use case is to use the third option, "Prevents search engines from indexing this page".