Improve bot prevention with a better key and timestamp

It's certainly possible. The method the module uses is very easy to bypass if someone wanted to specifically target it. Only a small number of sites use it so it's rare to see a bot designed for it. Can I see the form in question so I can determine if it's actually bring y protected by this module?

It appears to me the bots have cracked it now. For about two weeks (I only recently adopted the module) spam was gone. Since three days, the bots are in full swing again.
This renders the module useless at the moment - at least for me.
Maybe it would be time to change the logic so the module takes the lead again before bots catch up (again)?
This will always be a race...

This is not a bug - the module still does exactly what the project page says that it does. And it never claimed to be able to stop or prevent bots entirely. If someone or some bot really wants to spam your site, they will be able to. This module will work for the other 99.9% of cases with generic spambots hitting your site. There are 16,000 sites using this module - the methods clearly works. But if a bot is written specifically for Antibot, it will have no problem getting through. The odds of that happening are just close to 0.

If you have some logs that shows exactly what the form payload is, that would be helpful for me to see.

We can use this ticket to brainstorm additional ways we can improve the bot prevention.

O.K.
Please explain what "Form Payload" means. I only know the word payload from HTTP connections / Oauth stuff I once did.
Would it be this?
https://stackoverflow.com/questions/23118249/whats-the-difference-betwee...

Referring to the exact, raw data that was POSTed on the form.

What we could do as a first step is to make the antibot key unique per Drupal site. That way it is not simply md5($form['#form_id']), which is the same result for every site. Something like drupal_hmac_base64($form['#form_id']), drupal_get_hash_salt()).

Not sure if the spammers have hard-coded the hashed form IDs or if they really execute the javascript, but this seems worth a try.

After that we could go further and implement a time-based antibot key. The key must be submitted with an extra antibot_timestamp form field and the timestamp must not be older than 1 day.

$antibot_timestamp = REQUEST_TIME;
$antibot_key = drupal_hmac_base64($form['#form_id']) . $antibot_timestamp, drupal_get_hash_salt());

Then the key rotates pretty quickly and everything still works with page caching. That should make the spammers angry or at least annoy them :)

I'll try to com up with a patch.

Here is a patch that introduces the idea with a timestamp from above.

What I also saw from our webserver logs is that bots don't use clean URLs. They send "POST /?q=user/register" requests instead of "POST /user/register" requests.

So as a next step we could check if the request uses clean URLs on sites that have it enabled anyway, block it otherwise. Should probably open a new issue for that.

Opened #3107696: Stop bots submitting with non-clean URLs.

Did you test with page caching enabled and using the form as an anonymous user? I tried using this method when the concept of the "key" was introduced and there was no way to get it to work with the timestamp because validation was impossible with caching.

Also, 7.x is no longer supported. I'll only accept 8.x patches.

Thanks for the effort. I like the idea of making the keys site-unique at the very least.

Yes, tested with page caching. Since the timestamp is part of the cached page that should not be a problem.

The timestamp is just used to rotate the antibot_key regularly so that spammers need to fetch it all the time.

I don't have time to work on the 8.x version, leaving that for someone else :)

After 2 days I can happily report that we were able to stop bots with this patch and #3107696: Stop bots submitting with non-clean URLs. Here is a combined patch file we are using in our drush make files.

@sepal pointed out that we should also uninstall the new variables, new patches here. The combined patch includes #3107696: Stop bots submitting with non-clean URLs again.

After running this for a bit more than a month we noticed that people can fill out the password reset form really fast if they use a password manager. So the 5 seconds is too long in that case. We still want to have antibot on that form, so I decided to make the timestamp checking configurable per form and just set it to 0 for the password reset form.

Attaching a re-roll of the patch along with a version combined with clean URLs patch.

Thanks Gregor! Not sure if I should set this to RTBC, as it is only D7.

We normally don't support 7.x at this point, but given that there are still lots of sites who are using it (7,446), I suppose I should apply this patch.

Patch #20 applied cleanly, and tested OK, will commit it to the 7.x branch.