Archive "back" links dangerous for spiders and robots [#43281]

The archive module can be a big problem with some robots/spiders.

Recently, a spider crawled a drupal website for 2 days and generated
a few Gigs of web traffic, which is *very* annoying.

I found out the spider simply followed the "previous month" link
on the calendar, recursively, (trying to get articles back to 1988, for example).

It's not really a bug, but those situations should be avoided, because
bandwidth is not really cheap, and I dont like to see a single web client
generating 90% cpu usage...

As a workaround, perhaps adding a new module option would be enough:
adding a "minimum" date (ex 31/12/1999) so that previous dates wont show
a "previous month" link.

Or perhaps the Drupal team can think about a more elegant / efficient solution?
It's not really easy : some spiders can have good reasons to follow the
"previous month" links, but I think there should be some option to limit
the damages.

Comments

Comment #1

dries commented 4 January 2006 at 13:37

Version:

4.6.5

» x.y.z

Looks like a bug in CVS HEAD too. Bumping.

Comment #2

markus_petrux commented 4 January 2006 at 15:43

Some ideas:

1) Using rel="nofollow" in the previous month links? At least, Google won't follow'em.

2) Generating those links with javascript.

I guess, option 1 is more elegant and compliant with "good" search engines.

Comment #3

dries commented 4 January 2006 at 15:46

Neither (1) or (2) are advised. The solution is to stop generating links that don't go anywhere.

Comment #4

killes@www.drop.org commented 4 January 2006 at 15:51

I've addressed this problem in the event calendar. People may borrow code there. ;)

Comment #5

gábor hojtsy

he/him

Hungarian

Hungary

commented 8 August 2006 at 19:33

Version:

x.y.z

» 4.7.3

Archive module was removed from core. The contrib archive module replacement prints out no links for dates without content. If you have a quick solution for the 4.7 branch, then that could get committed.