Understood this is a dead module, adding this here in case someone picks it up.

http:BL needs to be loaded before most other modules (Captcha specifically), otherwise it doesn't properly deny access from blacklisted IPs.

How to update a module's weight
http://drupal.org/node/110238

Temporary solution is to manually edit the System table, an example:

UPDATE `table-name`.`system` SET `weight` = '-5' WHERE `system`.`filename` = 'sites/all/modules/httpbl/httpbl.module';

Comments

Michael-IDA’s picture

"-15" works better (on some sites), but even then when it's the most negative of all modules in the system table, page not founds seem to get issued before httpbl checks the IP. Could be a caching issue though.

{Just adding the note here for eventual reference}

bryrock’s picture

Status: Active » Postponed (maintainer needs more info)

Not a "dead module." Is this a general recommendation, or do you have evidence that httpbl is not blocking blacklisted IPs?

I have this module installed on a number of my employer's higher trafficked sites, some with and some without captcha, and it consistently blacklists and blocks anywhere from a half-dozen to several dozen IPs a day.

If configured optimally, the blacklisted IPs are added to Drupal's core blocked_ips table. Unless overridden in settings.php, that table is checked on every page request.

Michael-IDA’s picture

Title: Module load weight / » Module load weight

"do you have evidence that httpbl is not blocking blacklisted IPs?"

Yes.

CAPTCHA, being either a module with a lower sort by name than httpbl or having been enabled first and being in the system table first, is doing the “comment_form post blocked by CAPTCHA ...” (dozen times in a row for the same IP) and http:BL isn't getting called. I can guess that CAPTCHA exits out, or something similar, on a form block, hence no other modules run.

Changing the system weight lets http:BL process the IP correctly. CAPTCHA then blocks (most) spam from IPs that are 'clean' according to Honey Pot.

I'm adding this, https://drupal.org/node/354516#comment-6406368 , to the version of http:BL I'm using with clients, see as well: https://drupal.org/node/1248942#comment-6708832 . Although I'm looking for a different hook to use, as I'd only want it run when a module is enabled, not every bootstrap.

Best,
Sam

PS: Didn't see any activity in over a year, sorry for calling it dead.

Michael-IDA’s picture

And even this doesn't seem to help in all cases.

Just had CAPTCHA block 8 comment spams in 3 seconds that http:BL didn't blacklist.

hmmm, I'll guess it's that Honey Pot didn't return valid data, but I'd have to hack in a write to disk log for every request. Which wouldn't be that hard... Call it debug...

I'll get back with you next week and post the results, this is annoying me.

Sam

bryrock’s picture

Do you have the IPs of the commenters that CAPTCHA blocked?

I'm not sure you understand what httpBL is supposed to do. It only greylists or blacklists IPs found @ Project Honeypot, with threat-scores.

If you know the IP of the spammer, you can look it up @ Project Honeypot and see if it is even listed at all, and what the threat-score is if it is. There are still many spammers out there not yet in the PH list.

httpBL does have its own debug log. Just don't leave it on too long because it is a performance expense. It will show what IP's were evaluated and what the results were.

Michael-IDA’s picture

Yes, I looked up the IP @ Project Honeypot, it was a comment spammer and should have triggered, which is why I suspect the data coming back from Honey Pot was bad (as their forums show that happens a reasonably significant amount of the time).

And there is no "debug" flag in the code I'm running (6.x-2.0-rc1), unless you're talking about adding entries to dblog, which doesn't add anything if it didn't grey or black list the IP. (Maybe 7.x has it, but the clients that need this are all on 6.x)

It's trivial to add a disk write at the lookup point (function httpbl_dnslookup) to temporarily capture all request to disk see what actually is being returned for every request. (Which will also show if captcha, or anything else, is not allowing httpbl to run by the lack of entries.)

Obviously I'm not going to run this on a production clients install, yes I know exactly what httpBL is suppose to do (and what it can't).

Sorry if this comes across grumpy, been a long day.

Best,
Sam

bryrock’s picture

Version: 7.x-1.x-dev » 6.x-2.0-rc1

Changing version.

And there is no "debug" flag in the code I'm running (6.x-2.0-rc1), unless you're talking about adding entries to dblog, which doesn't add anything if it didn't grey or black list the IP. (Maybe 7.x has it, but the clients that need this are all on 6.x)

Sorry, but you posted this as a version 7x issue, and that version has exhaustive debug logging (bordering on overkill). When turned on it will log each IP checked and what the response code was from project honeypot. It was never implemented in 6x because logging is very expensive in D6. Logging is what first got me involved in this project, and if you're so inclined you can read the exhaustive debate between praseodym (this project's originator) and myself, #570742: Logging is not working.

When testing with the debug logging while porting this to 7x, I did see instances where IPs were not blocked because the response codes came back clean, yet I knew they were bad IPs because I had preselected them for testing because of their high threat scores on manual lookups. An hour later I would receive the correct response code and the IP would be blocked, as expected. In other words, yes, sometimes, apparently, for whatever reasons, there were invalid responses from p.h., but those misses were intermittent and beyond my control, and the module works when they are valid, so I didn't fret about it. By and large project honeypot stops more bad traffic than it misses. I've seen low (legitimate) traffic sites do an astonishing amount of blocking on a daily basis.

Perhaps for whatever reason you can't use the option to block on all page requests, rather than waiting to catch them in the act of leaving comments. I use this module on about two dozen production sites, in that "all" mode, because my thinking has always been to keep that garbage traffic off a site the moment it shows up. The risk is that it could punish innocent traffic with listed IPs, but that risk can be mitigated by adjusting the threshold. The module originally blocked anyone with any threat level greater than zero.

I'm adding this, https://drupal.org/node/354516#comment-6406368 , to the version of http:BL I'm using with clients, see as well: https://drupal.org/node/1248942#comment-6708832 . Although I'm looking for a different hook to use, as I'd only want it run when a module is enabled, not every bootstrap.

Try hook_install, as was recommended in the first link you sent at the top of this thread.

You can also refer to the code in the 7x version to see where you might add more logging to see what responses you get from p.h., but, again, be advised that logging is performance expensive in D6 because it comes at the tail end of the full bootstrap.

Three years ago or more I was unable to convince the creator of this module that the D5-to-D6 port had left functionality behind, so ran my own hacked version of it on my sites, and eventually that became the 6x-2 fork.

Overall I'm unconvinced that this is a captcha conflict. I've asked for evidence, you said you have it but didn't share, so color me skeptical.

bryrock’s picture

Status: Postponed (maintainer needs more info) » Closed (works as designed)

Changing status (closing)...

I just re-read "Just had CAPTCHA block 8 comment spams in 3 seconds that http:BL didn't blacklist." in http://drupal.org/node/1833648#comment-6714046, and that actually explains what is happening:

httpBL, in comment checking mode, checks IPs only when the comments are actually in the process of being saved (created or updated). In D6 this happens in hook_comment, during an insert or update. In D7, it happens during hook_comment_presave.

Using CAPTCHA short circuits (prevents) that comment creation process altogether, by adding an extra form-validation challenge before Drupal actually attempts to save the comment. If spammers fail that challenge, there is no attempt to create the comment, thus httpBL never gets a chance to look those IPs up and block them. The IPs will only be checked by httpBL if spammers successfully beat the CAPTCHA challenge (confirmed via testing), and move to the next step in an attempt to commit/write/save the comment. At that time httpBL will block the comment, if those IPs are found listed at Project Honeypot, even though they passed the CAPTCHA challenge.

Changing the module load weight is not going to resolve that conundrum, as each module is targeted at different points along the path to writing a comment. Getting CAPTCHA out of the way will allow all commenter IPs to always be checked, but it will not prevent spam from newer spammers not yet listed at Project Honeypot.

If you must use both httpBL and CAPTCHA, I'd suggest letting httpBL check all page access, and leave the comments to CAPTCHA . That way you ban known bad traffic as soon as it shows up on your site (who needs it?). CAPTCHA will then take care of those IPs not yet listed in project honeypot, but dumb enough or robotic enough to fail the CAPTCHA challenge.

I'd also recommend, if you aren't already, fully using and registering your honeypots (and donating mail servers) with Project Honeypot. That's the only way to catch those nascent spammers that are not yet listed, and get them reported and listed so they will be blocked on future visits.

Michael-IDA’s picture

Thanks for the info on CAPTCHA. Would have made this quicker if I'd checked back here sooner ;) Some of this is probably redundant. When I get a chance I'll submit a patch on D6 for the enhanced admin “Blacklisting Threshhold:” and “Blacklist Comment Spammers” (Although I think I posted the code for both in one of these threads)

Data Conclusion:
- Project Honey Pot does pass bad data.
- CAPTCHA get's called before http:BL, so set weight in system table to min “-1” so http:BL runs first.
- - excerpt is “-10” and there's no need for it to run on a bad IP, so “-15” for http:BL is probably okay.
- - (No other module in the two installs I'm playing with have a negative system weight.)

Usage Conclusion:
- Use http:BL to cut down most access to comment spammers.
- - Manually set http:BL system weight.
- Use CAPTCHA to catch bad data from httpbl. (Image CAPTCHA specifically, as math is easily brute forced bypassed.)
- Some spam comments still get through. (ARGH!)
- If your site can support the load, use Mollon instead of CAPTCHA. (but make sure to configure it correctly)
- - Note: Mollon has no way for a live human to “un-ban” themselves.

Raw data below:

With httpbl as -15 in system table.

dblog (60 odd “blocked by CAPTCHA” entries, at least 6 were the same IP, first and last below):
Friday, November 9, 2012 – 22:56
comment_form post blocked by CAPTCHA module: challenge "Image" (by module "image_captcha"), user answered "TooBad2", but the solution was "LNC5S".
58.22.70.195

Friday, November 9, 2012 – 23:00
comment_form post blocked by CAPTCHA module: challenge "Image" (by module "image_captcha"), user answered "TooBad68", but the solution was "NtbG3".
58.22.70.195

http://www.projecthoneypot.org/ip_58.22.70.195 (at ~2012-11-09 23:10)
58.22.70.195
The Project Honey Pot system has detected behavior from the IP address consistent with that of a comment spammer.
Threat Rating 12

Log file:
2012-11-09 09:56:36 : 58.22.70.195 : {removed}.195.70.22.58.dnsbl.httpbl.org.

Can't see a negative, but there didn't seem to be any “blocked by CAPTCHA” entries that were not shown in the logfile to be “clean” by Honey Pot.

With httpbl as 0 in system table.

dblog (multiple “comment_form post blocked by CAPTCHA” entries, last few were 192.210.59.250):

# grep -i "192.210.59.250" httpbl.log
2012-11-25 02:08:22 | 192.210.59.250 | {removed}.250.59.210.192.dnsbl.httpbl.org.

http://www.projecthoneypot.org/ip_192.210.59.250
consistent with that of a comment spammer.
Threat Rating 31

95.160.105.254
300+ user_login post blocked by CAPTCHA module, between 11/24/2012 14:40 and 11/24/2012 14:58
http://www.projecthoneypot.org/ip_95.160.105.254 = clean, “no data”

Other notes:

There are so many of these (1,000+), [user answered "TooBadNNN"] almost want to insta-ban any reply with “toobad”

Almost need a 'glue' module that combines the efforts of http:BL, CAPTCHA, and Mollon into a coherent whole.

Project adminrole has a method for running something on a module being enabled, probably will swipe it to re-set system weight to “run first” whenever a new module is enabled. (Don't want to run this on every bootstrap!)

PS: Yeah, I've personally registered / installed honeypots on one of my higher traffic sites. And I'll end up donating clients' MX Records as I install http:BL on their sites (clients don't usually want honeypots on their production sites, but they do need their own Honey Pot accounts).