We are running Drush in Drupal Gardens, which is a large (but not so large as to be ridiculous) multisite installation... In attempting to use Drush 3.0, we are finding that a single Drush command (on a single site) can take an incredibly long time - e.g., on the order of 20 minutes!

This appears to be due to #734270: Convenient aliases for running a command on groups of sites, which adds a scan of the sites directory to every Drush bootstrap via a call to drush_sitealias_create_sites_alias(). This can be extremely slow if you have hundreds or thousands of directories in that folder.

Since this functionality mostly only seems to be required for people who are using '@sites' on the command line or otherwise specifically requesting its use, this patch makes it so that the scan only happens in those situations. I think this makes sense, but it's possible it could cause a problem since the scan also results in populating various Drush static caches, so I'm not sure if the fact that that currently runs on every bootstrap is being relied on somewhere else. (I think probably not since the code is fairly new, but I haven't totally been able to follow the code flow here.)

Even if the patch does work correctly, it's a total hack though - so ideas for other approaches are welcome for that reason also :)

CommentFileSizeAuthor
#8 sites-alias-performance-2.patch2.75 KBgreg.1.anderson
#5 sites-alias-performance.patch2.76 KBgreg.1.anderson
#1 drush-avoid-expensive-sites-alias-801060-1.patch2.49 KBDavid_Rothstein
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

David_Rothstein’s picture

David_Rothstein CreditAttribution: David_Rothstein commented
FileSize
drush-avoid-expensive-sites-alias-801060-1.patch2.49 KB

Here is the patch.

moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented
Assigned: Unassigned » greg.1.anderson
Priority: Normal » Critical

The approach seems sound. I'm guessing that Greg has a more clever approach for detecting @sites than drush_get_context('argv')), '@sites')

moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented

I guess an alternate approach is to expand the @sites alias at the time it is used, instead of inspecting the request args.

greg.1.anderson’s picture

greg.1.anderson CreditAttribution: greg.1.anderson commented
Status: Needs review » Needs work

Yes, excellent suggestions. I'll work on fixing it up when I have time; maybe next week.

greg.1.anderson’s picture

greg.1.anderson CreditAttribution: greg.1.anderson commented
Status: Needs work » Needs review
FileSize
sites-alias-performance.patch2.76 KB

Try out this patch, and see if it makes things faster for you.

drush -r /path sa @sites and drush sa /path/@sites work, but drush sa @alias/@sites does not work. The last, though, is due to a separate issue not related to this patch (#811974: Relative aliases are broken).

greg.1.anderson’s picture

greg.1.anderson CreditAttribution: greg.1.anderson commented
Status: Needs review » Needs work

I was wrong, this patch breaks relative aliases.

greg.1.anderson’s picture

greg.1.anderson CreditAttribution: greg.1.anderson commented

@David: Could you test #5 and see if it fixes your performance issues, at least? I'll keep working on the full fix.

greg.1.anderson’s picture

greg.1.anderson CreditAttribution: greg.1.anderson commented
Status: Needs work » Needs review
FileSize
sites-alias-performance-2.patch2.75 KB

Okay, #5 was actually really close. Here's a patch that should actually work.

moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented
Status: Needs review » Reviewed & tested by the community

Minor nit below. Otherwise RTBC.

+++ includes/sitealias.inc	28 May 2010 23:37:52 -0000
@@ -384,7 +381,23 @@ function drush_sitealias_load_all($resol
+    $drupal_root = null;

use NULL instead of null

Powered by Dreditor.

greg.1.anderson’s picture

greg.1.anderson CreditAttribution: greg.1.anderson commented
Status: Reviewed & tested by the community » Fixed

Committed.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

David_Rothstein’s picture

David_Rothstein CreditAttribution: David_Rothstein commented

To follow up here (better late than never) - the committed patch does indeed fix the major performance issues.

We haven't tried it live yet because we are still running an older version of Drush (with the earlier patch applied).... but any patch that avoids calling drush_scan_directory() on the whole sites directory every request solves the problem - and I confirmed that this one does that :)

Thanks!