We have an installation with around 150 modules with around 50 features. If we reinstall the system it will take hours. This is because master is doing a complete cache flush and features resets everything on cache flush.

So this is like doing

  • drush en -y my_module1
  • drush cc all
  • drush fra
  • drush en -y my_module2
  • drush cc all
  • drush fra

If you do "drush master-execute -y --no-cache-clear" with the latest dev Version of master things will be 50 times faster. Maybe it is wise to add "$conf['features_rebuild_on_flush'] = FALSE;" to the settings.php as well. But it might not be necessary as there is no cache flush at all.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Kars-T’s picture

Title: Documentation about the "master-xecute" runs really slow problem » Write Documentation about the "master-xecute" runs really slow problem
Issue summary: View changes
geek-merlin’s picture

Title: Write Documentation about the "master-xecute" runs really slow problem » Enable multiple modules at once to save cache clears
Category: Task » Feature request

yes, that's indeed a pain.
i would go further and request that one cache clear should be the default.

the enable and disable commands both can eat multiple module names, so this will already do the trick.

EDIT: this should also stop the need for a drush-pm-fork. or do i miss something obvious?

derhasi’s picture

Title: Enable multiple modules at once to save cache clears » Write Documentation about the "master-xecute" runs really slow problem

@axel.rutz, no unfortunately that does not solve the problem. It may work with a module with few dependencies. But as Master is designed to controll all dependencies, this may lead to a really long "enable process" when initially setting up a site. This should not be done in a single php process, and that is all that is done by the core drush pm-enable command: it simply calls module_enable() and that does not implement any batch processing.

The fork problem can be resolved, when https://github.com/drush-ops/drush/pull/308 is implemented.

So this ticket is simply about documentation.

derhasi’s picture

Title: Write Documentation about the "master-xecute" runs really slow problem » Enable multiple modules at once to save cache clears
Issue summary: View changes
Related issues: +#2130239: Advanced options to control cache-clearing behavior?

On a second look, like @axel.rutz said, using the multiple modules at once might save cache-clears. As the drush-Team does/can not want to implement a skip cache clear for enabling modules, this should be the way to go. Therefore I'd revert the changes from #2130239: Advanced options to control cache-clearing behavior? and update the master-execute command.

derhasi’s picture

I just recognized, that bulk processing multiple modules will not completely solve the problems for Drupal 7, as we will still have some problems when modules shall be disabled and uninstalled. In Drupal 7 a module should be uninstalled right after it is disabled, before another module is disabled, so dependencies are not getting in the way. In addition we might have two cache flushes instead of one (or none). So fixing that might be more a task for v3. Will investigate that further.

  • derhasi committed 4d6b470 on 7.x-2.x
    Issue #2320641: Replace pm-disable fork with custom wrapper.
    
  • derhasi committed 522c83a on 7.x-2.x
    Issue #2320641: Replaces pm-enable fork.
    
  • derhasi committed 8f65bd2 on 7.x-2.x
    Issue #2320641: Provide wrapper for uninstalling multiple modules
    
  • derhasi committed c85ed1d on 7.x-2.x
    Issue #2320641: Only one cache clear per process.
    

  • derhasi committed 4d6b470 on 7.x-3.x
    Issue #2320641: Replace pm-disable fork with custom wrapper.
    
  • derhasi committed 522c83a on 7.x-3.x
    Issue #2320641: Replaces pm-enable fork.
    
  • derhasi committed 8f65bd2 on 7.x-3.x
    Issue #2320641: Provide wrapper for uninstalling multiple modules
    
  • derhasi committed c85ed1d on 7.x-3.x
    Issue #2320641: Only one cache clear per process.
    
derhasi’s picture

Status: Active » Fixed

In the latest 7.x-2.x release, I could not replace the pm fork completely. But I changed the implementation so, there will only be a single cache clear by default, or none at all, when --no-cache-clear is called.

geek-merlin’s picture

Great work!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

geek-merlin’s picture

Status: Closed (fixed) » Needs review
FileSize
2.42 KB

Yes, in an ideal world this would make everyone happy. But i had to notice that every now and then modules will cough or break if they are installed in the same request as others, often due to static caching, sometimes due to other caching. Sometimes the fix is a quite simple hook_module_enable implementation, sometimes it's way more complex, so to handle big deploys like 200+ modules, we need a workaround.

Patch flying in that adds quite-fine-grained options to have cache-clears before enabling specific modules.
I use it in production for a big deployment.

FluxSauce’s picture

Component: Documentation » Code
Status: Needs review » Needs work

Hey axel, I've got pretty much the same problem as you, and #11 did sort it for me. However, it's going to be pretty awkward to keep adjusting the build command each time. I think it'd be better if it was stored in the $conf array with the rest of the settings.

drush @sfg.sfg.dev master-execute -y --cache-clear-before=REMOVED_taxonomy,search_api

vs

$conf['master_modules']['cache_clear_before'] = array(
  'REMOVED_taxonomy',
  'search_api',
);

Thoughts?

geek-merlin’s picture

Status: Needs work » Needs review

> However, it's going to be pretty awkward to keep adjusting the build command each time.

So you might want to start using drushrc! Put something like this into sites/all/drush/drushrc.php and have fun!

$command_specific['master-exec'] = array(
  'verbose' => TRUE,
  'static-cache-clear-before' => '*', // Static cache clear is cheap, so do it before every module enable.
  'cache-clear-before' => 'rules,i18nviews', // These modules need DB cache clear before install.
);
$command_specific['features-revert-all'] = array(
  'verbose' => TRUE,
);
FluxSauce’s picture

Status: Needs review » Reviewed & tested by the community

I totally wasn't aware of that functionality! I have RTFM'd, tested and I confirm this is good to go. Thanks, axel.rutz!