The repo sync drush command (vcapi-parse-logs) is a potentially VERY long-running command. On util, it ran for 18 hours before exhausting 840M of memory and only getting through ~25% of all the d.o repositories. It desperately needs batching, as that'll hopefully reduce the amount of bloat and slowdown that occurs, as well as increase the speed through some concurrency (I hope, maybe?).

This is a good thing to do in general, though if we don't get the concurrency out of batched drush, then it's less useful. Still worth doing, but not as useful.

Comments

sdboyer’s picture

Title: Batch repository synchronization command » Batch repository synchronization drush command
Status: Active » Fixed

Yep, looks like they run concurrently. Wonder how drush manages the number of procs to create...oh well, it's in, so marking fixed. Will open subsequent issues if I notice big problems with it.

Status: Fixed » Closed (fixed)
Issue tags: -git phase 2, -git sprint 8

Automatically closed -- issue fixed for 2 weeks with no activity.

  • Commit 87ba390 on repository-families, drush-vc-sync-unlock by sdboyer:
    Issue #1016504: Batch repository synchronization drush command.
    
    

  • Commit 87ba390 on repository-families by sdboyer:
    Issue #1016504: Batch repository synchronization drush command.