Hi,

I really like the idea of this module (just trying it out now). I read your warning on the project homepage about getting an IP address banned from Google. It seems to me that even with 10 search queries and a depth of 100, that still equates to only 100 page views (assuming we don't increase the number of results per page, which would make it fewer page views). That doesn't seem like a lot to me for a 24 hour period - do you have any references for getting banned? I'd be happy to supply patches to stagger the requests via cron, or even just to put a small pause between requests.

Does that sound dangerous to you, or is that acceptable?!

Thanks for your work on the module!

A.

Comments

AndyF’s picture

Or maybe it would be better to rework the google crawler to use the custom search API?