By celticess on

Drupal has quite the .htaccess file. I was used to working with the very basic ones before so I don't quite get what some of the comments are inside the file. I have noticed in my stats several spiders. Something is harvesting my email and I'm getting tones of brazillian spam. I want to stop the spiders. So I went hunting at how to squash the creepy crawlies. I do have a robots.txt blocking all but they don't always respect that.(google seems to)

I found this article: http://www.javascriptkit.com/howto/htaccess13.shtml

I went to go paste it into my drupal .htaccess but it's the bottom line I'm unsure on. I did get the hint I need to paste into the spot near the end but there is a regex that I don't understand. I don't want to mess my site up. Also I don't think the 3 latest spiders I've ran into were on that list. According to my stats the latest spiders are:

Microsoft_Internet_Explorer
Rippers
Hurricane Electric
Link Checkers

While these next two aren't bots per say one is a search engine and the other seems to be some form of site casher similar to using wget to fetch.

cfetch/1.0
BlogSearch/1.1 +http://www.icerocket.com/

Comments

styro’s picture

styro commented

Based on my limited understanding of mod_rewrite, you should be able to add those rules to the the Drupal .htaccess file OK.

Paste them in the following place:

<IfModule mod_rewrite.c>
  RewriteEngine on

  # Snipped existing commented out rules for clarity


  # Put bot filters here, but don't add a second "RewriteEngine on" directive
  RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR] 
  ...
  RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Zeus 
  RewriteRule ^.* - [F,L]
  # end of new bot filter rules


  # Rewrite current-style URLs of the form 'index.php?q=x'.
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]
</IfModule>

Note: I think they should go before the existing rules at the bottom, and don't add the second "RewriteEngine on" bit as it already exists.

And test this first - I'm not enough of an expert to guanrantee it will work :)

celticess’s picture

celticess commented

I had figured out that was likely where it should go I was more worried of messing something up by messing with the:

RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]

as the site I linked to listed it as:

RewriteRule ^.* - [F,L]

I really have no idea what the meanings of the line are though I do get it's a wildcard. Later I might consider letting specific sites index but generally I don't want most spiders hitting it. So far though google is persistent it seems to read the file and behave. I think yahoo might do the same because I've never really even noticed it coming at my site.

styro’s picture

styro commented

It should be ok, if you have two RewriteRule directives

see these links for more info:
http://httpd.apache.org/docs/2.0/misc/rewriteguide.html
http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html#rewriterule

It seems that each RewriteRule directive just applies to the previous set of RewriteCond directives, and that you can have multiple RewriteRule directives.

So I think it should work OK just how it was in my example. But as I say, I'm no export on mod_rewrite which is one of the more complex Apache modules.

celticess’s picture

celticess commented

Yes it should be ok by what those links say. As long as I keep the first rule with it's stuff before it before inserting the new things at the end. I'll go drop it in the file now and let you know how it goes later. I should see a decrease in spiders from my stats though once I add this.

Ok I have now added them all and also added agents off of this site:

http://www.jafsoft.com/searchengines/webbots.html#search_engine_robots_a...

If anyone wants my full listing of bots I'll post them so people can cut and paste them to their own .htaccess Too bad that buy default it doesn't list all the robbots allowing people to turn on what they want. I think even if a robot doesn't crawl you can still list your site in it's database.

celticess’s picture

celticess commented

My .htaccess caused a 500 error... Even when I reduced the spiders list to basic it also did it. Only removing the things to block spiders let it be useable again.(btw I was surfing my site using firefox)

Here is basically what my .htaccess says in it with the uncommented stuff removed:

<Files ~ "(\.(inc|module|pl|sh|sql|theme|engine|xtmpl)|Entries|Repositories|Root|scripts|updates)$">
  Order deny,allow
  Deny from all
</Files>

Options -Indexes
Options +FollowSymLinks

ErrorDocument 404 /index.php
DirectoryIndex index.php

<IfModule mod_php4.c>
  php_value magic_quotes_gpc                0
  php_value register_globals                0
  php_value session.auto_start              0
</IfModule>

<IfModule sapi_apache2.c>
  php_value magic_quotes_gpc                0
  php_value register_globals                0
  php_value session.auto_start              0
</IfModule>

<IfModule mod_expires.c>
  ExpiresByType text/html A1
</IfModule>

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]
</IfModule>

I'll paste in a moment what I added to that last section "mod-rewrite.c that made it go 500 error on me. I'm not sure if I got the syntax wrong anywhere. But I don't think so

celticess’s picture

celticess commented 
  RewriteCond %{HTTP_USER_AGENT} ^AbachoBOT [OR]
  RewriteCond %{HTTP_USER_AGENT} ^abcdatos_botlink [OR]
  RewriteCond %{HTTP_USER_AGENT} ^AESOP_com_SpiderMan [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ah-ha.com\ crawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Scooter [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Mercator [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Scooter2_Mercator_3-1.0 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^roach.smo.av.com-1.0 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Tv<nn>_Merc_resh_26_1_D-1.0 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^AltaVista-Intranet [OR]
  RewriteCond %{HTTP_USER_AGENT} ^FAST-WebCrawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Acoon\ Robot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^antibot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Atomz [OR]
  RewriteCond %{HTTP_USER_AGENT} ^AxmoRobot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Buscaplus\ Robi [OR]
  RewriteCond %{HTTP_USER_AGENT} ^CanSeek/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ChristCRAWLER [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Clushbot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Crawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DaAdLe.com\ ROBOT/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^RaBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DeepIndex [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DittoSpyder [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Jack [OR]
  RewriteCond %{HTTP_USER_AGENT} ^EARTHCOM.info [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Speedy\ Spider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ArchitextSpider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ArchitectSpider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^EuripBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Arachnoidea [OR]
  RewriteCond %{HTTP_USER_AGENT} ^EZResult [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Fast\ PartnerSite\ Crawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^FAST\ Data\ Search\ Crawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^FAST\ Data\ Search\ Document\ Retriever [OR]
  RewriteCond %{HTTP_USER_AGENT} ^KIT-Fireball [OR]
  RewriteCond %{HTTP_USER_AGENT} ^france.misesajour.com [OR]
  RewriteCond %{HTTP_USER_AGENT} ^FyberSearch [OR]
  RewriteCond %{HTTP_USER_AGENT} ^GalaxyBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^geckobot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^GenCrawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^GeonaBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^getRAX [OR]
  RewriteCond %{HTTP_USER_AGENT} ^moget/2.0 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Aranha [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Slurp.so/1.0 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Slurp/2.0j [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Slurp/2.0-KiteHourly [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Slurp/2.0-OwlWeekly [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Slurp/3.0-AU [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Toutatis\ 2.5-2 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Hubater [OR]
  RewriteCond %{HTTP_USER_AGENT} ^IlTrovatore-Setaccio [OR]
  RewriteCond %{HTTP_USER_AGENT} ^IncyWincy [OR]
  RewriteCond %{HTTP_USER_AGENT} ^UltraSeek [OR]
  RewriteCond %{HTTP_USER_AGENT} ^InfoSeek\ Sidewinder [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Mole2/1.0 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^MP3Bot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^C-PBWF-ip3000.com-crawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ip3000.com-crawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Knowledge.com/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^kuloko-bot/0.2 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LNSpiderguy [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Linknzbot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^lookbot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^MantraAgent [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NetResearchServer [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Lycos_Spider_(T-Rex) [OR]
  RewriteCond %{HTTP_USER_AGENT} ^JoocerBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^HenryTheMiragoRobot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^MojeekBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^mozDex/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^MSNBOT/0.1 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Navadoo\ Crawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Gulliver [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ObjectsSearch/0.01 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^PicoSearch/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^PJspider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DIIbot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^nttdirectory_robot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^griffon [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Spider/maxbot.com [OR]
  RewriteCond %{HTTP_USER_AGENT} ^gazz/1.0 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NationalDirectory-SuperSpider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^dloader(NaverRobot)/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^dumrobo(NaverRobot)/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Openfind\ piranha,Shark [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Openbot/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^CrawlerBoy\ Pinpoint.com [OR]
  RewriteCond %{HTTP_USER_AGENT} ^user<n>.ip3000.com [OR]
  RewriteCond %{HTTP_USER_AGENT} ^QweeryBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^AlkalineBOT [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SeznamBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Search-10 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Fluffy\ the\ spider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Scrubby/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^asterias [OR]
  RewriteCond %{HTTP_USER_AGENT} ^speedfind\ ramBot\ xtreme [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Kototoi/0.1 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SearchByUsa [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Searchspider/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SightQuestBot/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Spider_Monkey/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Surfnomore\ Spider\ v1.1 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^teoma_agent1 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Teradex_Mapper [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ESISmartSpider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Spider\ TraficDublu [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Tutorial\ Crawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^updated/0.1beta [OR]
  RewriteCond %{HTTP_USER_AGENT} ^UK\ Searcher\ Spider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Vivante\ Link\ Checker [OR]
  RewriteCond %{HTTP_USER_AGENT} ^appie [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Nazilla [OR]
  RewriteCond %{HTTP_USER_AGENT} ^marvin/infoseek [OR]
  RewriteCond %{HTTP_USER_AGENT} ^MuscatFerret [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WhizBang!\ Lab [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ZyBorg [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WSCbot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Yandex [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Yellopet-Spider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Iron33 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ActiveBookmark [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ALink [OR]
  RewriteCond %{HTTP_USER_AGENT} ^AMeta [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ASPSearch\ URL\ Checker [OR]
  RewriteCond %{HTTP_USER_AGENT} ^BlogBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^BMChecker [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Bookmark\ Buddy [OR]
  RewriteCond %{HTTP_USER_AGENT} ^CheckWeb [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DRKSpider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Watchman [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DoctorHTML [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Email\ Extractor [OR]
  RewriteCond %{HTTP_USER_AGENT} ^FavOrg [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Favorites\ Sweeper [OR]
  RewriteCond %{HTTP_USER_AGENT} ^FreshLinks.exe [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Funnel\ Web\ Profiler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Html\ Link\ Validator [OR]
  RewriteCond %{HTTP_USER_AGENT} ^HTMLParser [OR]
  RewriteCond %{HTTP_USER_AGENT} ^The\ Informant [OR]
  RewriteCond %{HTTP_USER_AGENT} ^The\ Intraformant [OR]
  RewriteCond %{HTTP_USER_AGENT} ^InternetLinkAgent [OR]
  RewriteCond %{HTTP_USER_AGENT} ^InternetPeriscope [OR]
  RewriteCond %{HTTP_USER_AGENT} ^javElink [OR]
  RewriteCond %{HTTP_USER_AGENT} ^jdwhatsnew.cgi [OR]
  RewriteCond %{HTTP_USER_AGENT} ^JRTS\ Check\ Favorites\ Utility [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Lambda\ LinkCheck [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LinkLint-checkonly [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LinkAlarm [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Linkbot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Linkman [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LinkProver [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LinkScan\ Server [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LinkSweeper [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Link\ Valet\ Online [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LinkVerify\ Spider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Morning\ Paper [OR]
  RewriteCond %{HTTP_USER_AGENT} ^MoveAnnouncer [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NetLookout [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NetMechanic [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NetMind-Minder [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NetMonitor [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Netprospector\ JavaCrawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^online\ link\ validator [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Rational\ SiteCheck [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Robozilla [OR]
  RewriteCond %{HTTP_USER_AGENT} ^RPT-HTTPClient [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SiteBar [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SpurlBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SurfMaster [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SyncIT [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Watchfire\ WebXM [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WatzNew\ Agent [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebSite-Watcher [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebTrends\ Link\ Analyzer [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Weblink\ Scanner [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Xenu's\ Link\ Sleuth [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Z-Add\ Link\ Checker [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Alligator [OR]
  RewriteCond %{HTTP_USER_AGENT} ^BatchFTP [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DA [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DLExpert [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Download\ Master [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Download\ Ninja [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Download\ Wonder [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Ez\ Auto\ Downloader [OR]
  RewriteCond %{HTTP_USER_AGENT} ^FreshDownload [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
  RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
  RewriteCond %{HTTP_USER_AGENT} ^MyGetRight [OR]
  RewriteCond %{HTTP_USER_AGENT} ^GetSmart [OR]
  RewriteCond %{HTTP_USER_AGENT} ^HiDownload [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Kapere [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Kontiki\ Client [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LeechGet [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LightningDownload [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
  RewriteCond %{HTTP_USER_AGENT} ^MetaProducts\ Download\ Express [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NetZip\ Downloader [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NetButler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NetPumper [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Nitro\ Downloader [OR]
  RewriteCond %{HTTP_USER_AGENT} ^PuxaRapido [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SpeedDownload [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebDownloader\ for\ X\ 1.30 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebPictures\ Downloader [OR]
  RewriteCond %{HTTP_USER_AGENT} ^X-Uploader [OR]
  RewriteCond %{HTTP_USER_AGENT} ^citenikbot/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^CLIPS-index [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Computer_and_Automation_Research_Institute_Crawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^cosmos [OR]
  RewriteCond %{HTTP_USER_AGENT} ^D2KWebCrawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DiaGem/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Digimarc\ WebReader [OR]
  RewriteCond %{HTTP_USER_AGENT} ^EchO!/2.0 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^FinaleRobot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Ideare\ -\ SignSite [OR]
  RewriteCond %{HTTP_USER_AGENT} ^GentleSpider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Gulper\ Web\ Bot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^IRLbot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^KnowItAll [OR]
  RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^MultiText [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NEC\ Research\ Agent [OR]
  RewriteCond %{HTTP_USER_AGENT} ^OntoSpider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^sherlock_spider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^S.T.A.L.K.E.R. [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Steeler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ru-robot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^USyd-NLP-Spider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebGather [OR]
  RewriteCond %{HTTP_USER_AGENT} ^xyro [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Zao/0.2 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Zao-Crawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^JoBo [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NetAttache [OR]
  RewriteCond %{HTTP_USER_AGENT} ^PageDown [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ParaSite [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DISCoFinder [OR]
  RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DigOut4U [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Kenjin\ Spider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Searchworks\ Spider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SiteMapper [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Teleport Pro [OR]
  RewriteCond %{HTTP_USER_AGENT} ^URL2File [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Web2Map [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Webrobot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebSnatcher [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebTwin [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebVCR [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Albert\ Indexer [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ADSAComponent [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ADSARobot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Beamer [OR]
  RewriteCond %{HTTP_USER_AGENT} ^beholder [OR]
  RewriteCond %{HTTP_USER_AGENT} ^e-sense [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Checkbot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^contype [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Convera\ Internet\ Spider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ConveraCrawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^CyberNavi_WebGet [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DaviesBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^deepweb [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Excalibur\ Internet\ Spider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Expired\ Domain\ Sleuth [OR]
  RewriteCond %{HTTP_USER_AGENT} ^GigaBaz [OR]
  RewriteCond %{HTTP_USER_AGENT} ^GigaBazVStheWeb [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Giskard [OR]
  RewriteCond %{HTTP_USER_AGENT} ^grub-client [OR]
  RewriteCond %{HTTP_USER_AGENT} ^htdig [OR]
  RewriteCond %{HTTP_USER_AGENT} ^infoGIST [OR]
  RewriteCond %{HTTP_USER_AGENT} ^InternetArchive [OR]
  RewriteCond %{HTTP_USER_AGENT} ^InternetSeer [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ipiumBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LEIA [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LexiBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^logikabot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Mata\ Hari [OR]
  RewriteCond %{HTTP_USER_AGENT} ^metabot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Mister\ Pix\ II [OR]
  RewriteCond %{HTTP_USER_AGENT} ^MOSES\ 2.0\ Spider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^MonkeyCrawl [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NetCruiser [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NPBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NetZippy [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NZBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Opencola [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ORA_checksite [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Onekit.com\ -\ PAD\ File\ Get. [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Oxxbot1 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Pansophica [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Phoaks [OR]
  RewriteCond %{HTTP_USER_AGENT} ^PICgrabber [OR]
  RewriteCond %{HTTP_USER_AGENT} ^PictureOfInternet [OR]
  RewriteCond %{HTTP_USER_AGENT} ^PicSpider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^PintaSpider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Pita\ (Chub.Stanford.EDU) [OR]
  RewriteCond %{HTTP_USER_AGENT} ^PitSpyder\ Thread<n>0 [OR]
  RewriteCond %{HTTP_USER_AGENT} ^PolyBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Rumours-Agent [OR]
  RewriteCond %{HTTP_USER_AGENT} ^RepoMonkey\ Bait\ &\ Tackle [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Robot2.0(PingSoft) [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SlySearch [OR]
  RewriteCond %{HTTP_USER_AGENT} ^CrawlWave [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SQ\ Webscanner [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Sqworm [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Tagword [OR]
  RewriteCond %{HTTP_USER_AGENT} ^TaWWWantula [OR]
  RewriteCond %{HTTP_USER_AGENT} ^TeraCrawl [OR]
  RewriteCond %{HTTP_USER_AGENT} ^unchaos_crawler [OR]
  RewriteCond %{HTTP_USER_AGENT} ^unlostBot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^URLBlaze [OR]
  RewriteCond %{HTTP_USER_AGENT} ^utopy [OR]
  RewriteCond %{HTTP_USER_AGENT} ^UrlScope [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Vagabondo [OR]
  RewriteCond %{HTTP_USER_AGENT} ^vspider [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Webbandit [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Webclipping.com [OR]
  RewriteCond %{HTTP_USER_AGENT} ^webcollage [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebCompass [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebGenie [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Web Hound [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Web Magnet [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Webpush [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebSymmetrix [OR]
  RewriteCond %{HTTP_USER_AGENT} ^webrank [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WhosTalking [OR]
  RewriteCond %{HTTP_USER_AGENT} ^XupiterToolbar [OR]
  RewriteCond %{HTTP_USER_AGENT} ^YottaShopping_Bot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^BlogSearch/1.1\ +http://www.icerocket.com/ [OR]
  RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^cfetch [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Custo [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^findlinks [OR]
  RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^HMView [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^HTTrack [NC,OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Hurricane\ Electric [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Holmes [NC,OR]
  RewriteCond %{HTTP_USER_AGENT} ^sherlock [NC,OR]
  RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Indy\ Library [NC,OR] 
  RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^larbin [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Link Checkers [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Microsoft_Internet_Explorer [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^psbot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Rippers [OR]
  RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Wget [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Widow [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR] 
  RewriteCond %{HTTP_USER_AGENT} ^Zeus 
  RewriteRule ^.* - [F,L]
cybe’s picture

cybe commented

I just added this nice email-harvester bot trap:-

http://www.kloth.net/internet/bottrap.php

to my site, and it seems to be working well. I had tons of bots hammering my logs with 404's because they didn't look at the BASE HREF for some reason...

Anyone have any comments about this simple script?

cybe’s picture

cybe commented

The trap has now caught near to 40 different IP's going into a folder which is forbidden by robots.txt and has a huge warning to ward of potential humanbeings who get lost there (they shouldn't)

All carrying the Java/1.4.1_04 (or other version) browser identification.

cybe’s picture

cybe commented

I've replaced it with this: http://danielwebb.us/software/bot-trap/ which is almost the same but a bit better, allowing real humanBeings to unban themselves if they for some reason end up in the trap.

I also added:


$br = strtolower($_SERVER['HTTP_USER_AGENT']); // what browser they use.

if(ereg("java", $br)) {

sleep(10);
} else {

}

to index.php since 99.9% of all the bots that get caught in the trap have the user-agent Java....

NGRhodes’s picture

NGRhodes commented

Im using bad behavior. So far it has caught about 10 different bots in 2 weeks, numerous times and no false positives so far.

http://drupal.org/node/30501

cybe’s picture

cybe commented

I keep switching badbehavior on and off. Once I got stopped by badbehavior, and I'm not sure why, it might have been an aggressive fasterfox setting. A friend also had problems accessing my site, and even if one shuts down the aggressive fasterfox tweaks the persons IP-address still remains banned (not sure?).

Whilst looking at the badbehavior log I see so many cases that look legit, and badbehavior isn't telling me what's wrong with it.

Example: Are they really what they say they are or fakes? Too new services for badbehavior to recognize them? 

Header 'Pragma' without 'Cache-Control' prohibited for HTTP/1.1 requests	2006-04-28 15:09:44	69.90.230.126		PubSub-RSS-Reader/1.1 (http://www.pubsub.com/)