Auto Regenerate Cache (pre-caching) Preemptive Cron Cache - throttle & crawl rate stats

This does 5 URL's at a time.

This does 5 at a time and calls it's self for unlimited URL crawlage.

Fine grained control over html, xml & json.

In order for this to be smart, I need a boost_crawler table. Once the crawler is done it will truncate the table, signaling to the worker thread(s) that it is done. This way I can grab URL's from boost_cache, url_alias, & nodes; and I should be able to make it use multiple threads. I can also then only crawl pages that are not cached currently.

id - serial - PK
extension - varchar 8
url - varchar 255

index on ID & extension.

@capellic
The crawler loads the html, it doesn't load the elements inside like javascript or images, so you don't have to worry about 3rd party stats. It will count on Drupal stats though, since that counts when the html is loaded.

Just tested the above patch on a live server and it works as advertised; which means the core of the crawler code works in drupal core. Thats good, it's like the batch API but it doesn't need a browser to call it's self. So my two step php hack works, now I need to make the crawler smarter.

Roadmap:

• Copy the URL's from the boost_cache table into the boost_crawler table. Pass a URL variable to keep track of the progress for large sites; transfer 10,000 URLs at a time to the boost_crawler table via LIMIT. Need to set ORDER BY filename in the SQL.
• Allow for a stop signal to be given

- Ship RC1 -

• Support multiple threads. This will be built with this in mind, but wont be enabled until it's been tested.
• Use the url_alias table to populate the boost_cache; that will assume all content in there is a html doc, unless it ends in .xml or /feed.
• Enable showing stats on the crawler
• Allow admin to better fine-tune some parameters of the crawler
• Have a real crawler, like what I have in #363077: Add spider to crawler - Cache entire site with new install.; auto discovers content not in the boost_cache or url_alias table, but is in a <a href=""></a> html tag.

Wow, thats some slow code... #3 took over twice as long to crawl a site as the crawler does. Some of it has to do with the sleep(1) but thats pretty awful. Doing a bootstrap to crawl every 10 pages is another source. I knew this would be a little bit slower, but I wasn't expecting this. For this to work well, there's a lot of work to do. I'll try refactoring the code (I made sure I wouldn't get a timeout), but if I can't get it's performance to improve, I'll be shipping RC1 without the cron crawler.

Still need to add in a stop crawler button & code. But this should be faster and it defaults to using 2 threads. Loads 25 url's per run.

Committed the above code with a stop button.

Right now the crawler is tided to boost quite heavily, so it will only hit URL's that it can grab from the boost_cache table. If you enable boost, hit the url's you want then disable the html file cache; the crawler will still grab those url's in the future. Right now boost could be split off into about 3 different modules, but it's much easier to develop for it when it's all in one codebase. Your request gives me another table to get URL's from which is cache_page
http://api.drupal.org/api/function/page_set_cache
It's a temp table so this wouldn't be that useful.

Here's the first bug... crawler finished and only did 1/2 the URL's here's an attempted fix. It's amazing how this doesn't work the same on all systems... gotta find code that does.

Figured out whats up. Each thread needs to wait a random amount of time otherwise things get messy.

committed this

Being able to get the crawl rate would be semi possible. It would be the average since the crawler started counting all threads & it would be quite inaccurate. Increased accuracy means a much slower crawler, since each thread would write to the database after each page was crawled. Having a "throttle" would also be possible using the usleep() function inside the crawler; but why wait 5 seconds between each request? Being able to set the number of threads is also needed.

php/cpu timeout is taken care of; it shouldn't happen, ever. Crawler starts right after cron is run.

@g10tto
The crawler hit's pages as an anonymous user, so if that page returns a 403, it doesn't really help anyone. Drupal's cache (what you see on the performance page) is for anonymous users as well. At the bottom of the boost project page are links to other performance/caching methods.

#363077: Add spider to crawler - Cache entire site with new install.
Here's a crawler I made thats independent of drupal. It's crawled a site with over 1,000,000 url's; it works, but not easy to setup.

...why Boost is restricted in this way, if such functionality regarding this thread is restricted to the module.

I'm assuming your talking about it only caching what anonymous users see. Anonymous is easy; there's only 1 version of each page, and with that, boost is still a very complicated module.