Drupal generates a page at /filter/tips, and that page is indexed by search engines, and offered up to the public as a destination on your website.

The fix is to list this page in robots.txt.

Files: 
CommentFileSizeAuthor
#32 hide-filter-tips-and-others-from-search-engines-1137848-32.patch1.12 KBGaëlG
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 78,767 pass(es).
[ View ]
#29 drupal-7-disallow-filter-tips-page-1137848-29-do-not-test.patch579 bytesTor Arne Thune
#28 drupal-8-disallow-filter-tips-page-1137848-28.patch628 bytesTor Arne Thune
PASSED: [[SimpleTest]]: [MySQL] 58,853 pass(es).
[ View ]
#19 drupal-disallow_filter_tips_page-1137848-19.patch918 bytespillarsdotnet
PASSED: [[SimpleTest]]: [MySQL] 190 pass(es).
[ View ]
#18 drupal-disallow_filter_tips_page-1137848-18.patch1.4 KBpillarsdotnet
PASSED: [[SimpleTest]]: [MySQL] 190 pass(es).
[ View ]
#16 drupal-disallow_filter_tips_page-1137848-16.patch700 bytespillarsdotnet
PASSED: [[SimpleTest]]: [MySQL] 190 pass(es).
[ View ]
#8 drupal-1137848-8.patch528 bytestim.plunkett
PASSED: [[SimpleTest]]: [MySQL] 29,415 pass(es).
[ View ]
#3 1137848-3.patch302 bytesBrockBoland
PASSED: [[SimpleTest]]: [MySQL] 29,425 pass(es).
[ View ]
#2 1137848-2.patch0 bytesBrockBoland
PASSED: [[SimpleTest]]: [MySQL] 29,422 pass(es).
[ View ]

Comments

jensimmons’s picture

Add this:Disallow: /filter/tips to line 32 of robots.txt

Like this:
a screenshot of the robots.txt file with one line of code added to exclude this page

Does someone want to make a patch?

BrockBoland’s picture

StatusFileSize
new0 bytes
PASSED: [[SimpleTest]]: [MySQL] 29,422 pass(es).
[ View ]

Sure!

BrockBoland’s picture

Status:Active» Needs review
StatusFileSize
new302 bytes
PASSED: [[SimpleTest]]: [MySQL] 29,425 pass(es).
[ View ]

Aw, fer cryin - fixed attached.

Dave Reid’s picture

Version:7.0» 7.x-dev
Category:bug» feature
Status:Needs review» Needs work

We need to add the un-clean URL version of it as well. Note, that page is not a file, so it should go one section lower.

tim.plunkett’s picture

Version:7.x-dev» 8.x-dev
Category:feature» bug
Status:Needs work» Reviewed & tested by the community
Issue tags:+needs backport to D6, +needs backport to D7

Looks good.

Dave Reid’s picture

Category:bug» feature
Status:Reviewed & tested by the community» Needs work
Issue tags:-needs backport to D6, -needs backport to D7

And as always, fix in 8.x first, then backport easily.

tim.plunkett’s picture

tim.plunkett’s picture

Status:Needs work» Needs review
StatusFileSize
new528 bytes
PASSED: [[SimpleTest]]: [MySQL] 29,415 pass(es).
[ View ]

Updated. Crazy cross-posts. It's what happens when Jen asks for Drupal things on twitter.

BrockBoland’s picture

I've read and understand the backport policy (http://drupal.org/node/767608), but what's the actual process for an issue like this? For a simple item like this, it makes sense that a single patch can be applied to D7 and D8, but in more complex cases where the patches differ, should a separate issue be spun off for the D7 version?

Apologies for being a newb - I haven't done any core patches before.

ksenzee’s picture

should a separate issue be spun off for the D7 version?

No, normally it all stays in the same issue. It works fine for simple stuff like this but it's kind of a messy process for complicated issues.

Also, subscribing. I saw drupal.org/filter/tips in some Google results the other day and said huh what?

jensimmons’s picture

Twitter FTW!!

Yeah, the CVS-centric workflow's been to update things in the dev version of Drupal (now D8), then backport to the current version (D7), and then the one-older version (D6). IMO, this workflow could/should/might change now that we have Git, and we can work with branches instead of patches.... but that's not happened yet. So meanwhile, we are following the same rules that were used two years ago when D6 was brand-new and D7 development had just opened. (Or was that three years ago?)

Issues like this one are the test. Super easy to understand. Super easy to write the code. Not much to debate.... now let's see how long it takes to get this into D7, with the crazy D8-first-rule. Especially since we don't have a D8 co-maintainer, and Angie (webchick) doesn't have commit access to D8. Will this be no-biggy? Or will it take months to fix? Our process post-switch-to-git is still evolving.

Meanwhile, welcome BrockBoland to core development! You've been awarded the "My First Drupal Core Patch" badge. :D YAY!

ksenzee’s picture

I don't think the rule about committing to the newest version first is likely to change just because of git. The process is being discussed over at #1050616: Figure out backport workflow from Drupal 8 to Drupal 7.

ksenzee’s picture

Status:Needs review» Reviewed & tested by the community

Oh, and this passed tests since I was last here, so RTBC.

Dave Reid’s picture

+1 from me as well, although now I will no longer be able to google for sites that have their full html input filter on...which is a good thing!

webchick’s picture

Status:Reviewed & tested by the community» Fixed

Makes sense to me.

Committed to 8.x and 7.x. Thanks!

pillarsdotnet’s picture

Version:8.x-dev» 6.x-dev
Status:Fixed» Needs review
Issue tags:-needs backport to D7
StatusFileSize
new700 bytes
PASSED: [[SimpleTest]]: [MySQL] 190 pass(es).
[ View ]

Requested d6 backport:

Dave Reid’s picture

Status:Needs review» Needs work

Don't forget the Disallow: /?q=filter/tips

pillarsdotnet’s picture

Status:Needs work» Needs review
StatusFileSize
new1.4 KB
PASSED: [[SimpleTest]]: [MySQL] 190 pass(es).
[ View ]

Oops.

pillarsdotnet’s picture

StatusFileSize
new918 bytes
PASSED: [[SimpleTest]]: [MySQL] 190 pass(es).
[ View ]

(sigh) Probably better as one patch. Sorry for the noise.

Damien Tournoud’s picture

Status:Needs review» Reviewed & tested by the community
juliangb’s picture

This has been RTBC for 3 months.

I'm using this patch on my live sites and would greatly like for new D6 releases to include this as standard.

Is there anything stopping this from being committed?

Gábor Hojtsy’s picture

Status:Reviewed & tested by the community» Fixed

Committed to 6.x too, thanks!

Status:Fixed» Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

juliangb’s picture

Version:6.x-dev» 8.x-dev
Status:Closed (fixed)» Active

I'm now finding that Google is not blocking filter/tips because the line in robots.txt has a trailing slash.

We need to remove the slash to ensure that Google always knows to block this page.

pillarsdotnet’s picture

The Redirect module has an option to remove trailing slashes.

juliangb’s picture

Actually the redirect module doesn't help in this instance.

The issue is that in the robots.txt the paths all have trailing slashes, which means that Google does not block any paths without the trailing slashes.

To ensure that it catches everything, we should include a version without the trailing slash in robots.txt.

pillarsdotnet’s picture

Ah. That explains the module which *adds* trailing slashes to everything.

Write a patch, please?

Tor Arne Thune’s picture

Category:feature» bug
Status:Active» Needs review
Issue tags:+Quick fix, +Novice, +needs backport to D7
StatusFileSize
new628 bytes
PASSED: [[SimpleTest]]: [MySQL] 58,853 pass(es).
[ View ]

juliangb is right. It should not have a trailing slash. Attaching a patch that corrects it. As for the suggestion to add a non-trailing-slash-version of paths with a trailing slash, I feel that it deserves its own issue.

Tor Arne Thune’s picture

Uploading the D7 backport.

juliangb’s picture

Status:Needs review» Needs work

Thanks for posting the patch, Tor Arne - a good reminder for me seeing this pop up in my issues tracker.

I disagree with fixing the other links in a separate issue though, hence the "needs work" for now. This would leave a slightly "hacked" state until the other issue was fixed.

GaëlG’s picture

Issue summary:View changes
Issue tags:+#amsterdam2014

I'm on it.

GaëlG’s picture

Status:Needs work» Needs review
StatusFileSize
new1.12 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 78,767 pass(es).
[ View ]

Here's a new patch. I checked in the router table to see if the path can have subpaths. If so, we need to list both formats (end slashes and no end slashes).
/search/ needs indeed to be listed to avoid search results indexing, but it seems not bad to me that the search landing page can be indexed. That's why I did not add /search.

oenie’s picture

fixing the amsterdam sprint tag to amsterdam2014

ronaldmulero’s picture