The attached file adds preliminary drush support to HTTPRL. It comes with two functions, both of which support nonblocking calls issued against self. This currently doesn't have support for external calls.

The first function is httprl-self-request . It is useful for requesting a path from drush so you can rebuild a page. It can be used as follows:

'examples' => array(
      'drush httprl-self-request node' => 
        'Request own node path.',
      'drush hsr --nonblocking --print' => 
        'Request front page of site in a non-blocking manner and print response',
    ),

The second function (which is evil) is called httprl-self-spider. It is used to seek out and spider all entities on a site via httprl self calls. It has support for non-entity based paths but they currently have to be defined. Examples of this are below:

'examples' => array(
      'drush httprl-self-spider' => 
        'Self request all known, web-accessible entities at their URL',
      'drush hss node --nonblocking' => 
        'Request all nodes in a nonblocking manner to prime cache',
    ),

drush hss without a type defined can be used to spider all known, url accessible entities on your site. This is an extreme version of https://drupal.org/project/drush_ecl because it not only loads the entities, it then requests them anonymously which prime the page cache fully. This is especially useful when used in conjunction with boost. For example...

drush cc all
drush hss --nonblocking

This will clear all caches, eliminating boost files; then it looks up all the nodes, users, field collections and whatever else you got going in there that has a defined url. It loads those objects in batches of 50, and issues connections against those items. After finishing with one entity type it will move on to the next.

If the item (because of permissions) is not able to be accessed by anonymous traffic, the message will simply fail. Issuing nonblocking at this scale against ones self can be dangerous (I downed a machine this way already in testing) as you are essentially committing a denial of service attack against yourself, so it is recommended to use drush hss node --nonblocking or drush hss without the nonblocking modifier. Both of these will limit the scope of "the attack". The end result is a site that is flushed of old data and completely primed. Stuff this in a cron job on your server at night and you should have a nice fresh site that no one has to bear the burden of rebuilding a page.

I tried to make a patch but as this is a new file git was mad at me. Just add this file to httprl and please jump all over me with modifications. Drush support for this module would be great imo.

CommentFileSizeAuthor
httprl.drush.inc.zip3.25 KBbtopro
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

btopro’s picture

Title: Add Drush support for command lin HTTPRL calls » Add Drush support for command line HTTPRL calls

misspelling in title

mikeytown2’s picture

Status: Needs review » Needs work

I'm not convinced that httprl core is the correct place to put a spider. A sub module seems like the correct place.

btopro’s picture

fair enough; do i have your blessing to roll some kind of httprl_drush_spider module as a separate project :)

mikeytown2’s picture

Yep :)

btopro’s picture

Status: Needs work » Closed (fixed)

Thank you for your blessing, done!

https://drupal.org/project/httprl_spider