Three things that would be awesome:

  • create an option for a dynamic number of spawned processes based on a specified CPU load. As each sub queue completes take read on CPU load and guestimate if another one (or more) should be spawned.
  • reporting as queue workers are completing. Currently I'm using a shell script around a queue runner to do this and I give myself information like:
    Completed 40 workers in 36 seconds. 1879641 to go at 67/minute.
Completed 40 workers in 35 seconds. 1879601 to go at 69/minute.
Completed 40 workers in 36 seconds. 1879561 to go at 67/minute.

    This helps me know how sad I should be about my queue speed.

  • a pony.

Comments

  • Dave Reid committed 078911c on 
    Issue #2358905 Added a rate per minute output to the result message.
dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

I added some rate reporting to the final results:

/var/www/drupal7.dev$ drush qrc concurrent_queue_debug --concurrency=10
local.d7   ] Processed 100 items from the concurrent_queue_debug queue in 2 seconds (2994/min). There are 0 remaining items.                                     [ok]
 * [@self.0] Processed 100 items from the concurrent_queue_debug queue in 2.97 seconds (2018/min). There are 0 remaining items.                                  [ok]
 * [@self.1] Processed 100 items from the concurrent_queue_debug queue in 3.97 seconds (1511/min). There are 0 remaining items.                                  [ok]
 * [@self.2] Processed 100 items from the concurrent_queue_debug queue in 4.99 seconds (1202/min). There are 0 remaining items.                                  [ok]
 * [@self.3] Processed 100 items from the concurrent_queue_debug queue in 5.96 seconds (1007/min). There are 0 remaining items.                                  [ok]
 * [@self.4] Processed 100 items from the concurrent_queue_debug queue in 6.91 seconds (868/min). There are 0 remaining items.                                   [ok]
 * [@self.5] Processed 100 items from the concurrent_queue_debug queue in 7.83 seconds (766/min). There are 0 remaining items.                                   [ok]
 * [@self.6] Processed 100 items from the concurrent_queue_debug queue in 8.87 seconds (677/min). There are 0 remaining items.                                   [ok]
 * [@self.7] Processed 100 items from the concurrent_queue_debug queue in 9.9 seconds (606/min). There are 0 remaining items.                                    [ok]
 * [@self.8] Processed 100 items from the concurrent_queue_debug queue in 10.88 seconds (551/min). There are 0 remaining items.                                  [ok]
Concurrently processed 1000 items from the concurrent_queue_debug queue in 11.73 seconds (5116/min). There are 0 remaining items.                                [ok]

I've been wanting to add some kind of 'progress' reporting. Ideas on how often it should report? Should it be a drush option? Is there a way I can make it show progress on the same line without having to generate a ton of messages?

I'm not sure I can work on spinning off a new process, because already I put in code to let it run as long as possible. The way this splits up the work I'm not sure it would make sense to do that either.

ponies++

  • Dave Reid committed 078911c on 7.x-2.x 
    Issue #2358905 Added a rate per minute output to the result message.