Retry failed tome:static-export-path commands

@jlatorre Thanks for the issue! This is a common problem with Tome, and from what I can tell is just Drupal/MySQL not being able to handle so many concurrent requests. I've been doing some light performance testing and profiling with Tome commands and, so far, Drupal is always the bottleneck.

In the short term, I would recommend tweaking the --process-count and --path-count options to find a combination that doesn't overload your site. A lower process count and higher path count would mean less concurrent MySQL queries, which may help you here.

In the longer term, I think some paths forward here are:

1. Continue to profile and audit Tome performance to see if there's something we can improve - I'd love any help here, and would welcome any tweaks to improve performance.
2. Write a patch to retry failed tome:static-export-path invocations at least one time, and maybe make the retry count configurable.

Thanks for the patch @jlatorre! This is basically how I would have implemented adding a configurable "retry-count" option to the static command. Do you think this change is going to be enough to get around your "MySQL server has gone away" issues?

Here's some review of your patch:

1. +++ b/modules/tome_base/src/ProcessTrait.php
   @@ -16,21 +16,44 @@ trait ProcessTrait {
   +  protected function runCommands(array $commands, $concurrency, $retry_count, $callback = NULL) {
   

   There are other commands that use this trait in tome_sync, they will also need updated so that they continue to function.

2. +++ b/modules/tome_base/src/ProcessTrait.php
   @@ -16,21 +16,44 @@ trait ProcessTrait {
   +    $filter_callback = function (Process $current_process) use (&$collected_errors, $callback, $retry_count) {
   +      $current_process->stop();
          $is_running = $current_process->isRunning();
   

   Why stop the current process here?

3. +++ b/modules/tome_base/src/ProcessTrait.php
   @@ -16,21 +16,44 @@ trait ProcessTrait {
   +          if (!$current_process->isSuccessful() && $retry == $retry_count) {
   

   Can we use strict equality here, i.e. ===?

4. +++ b/modules/tome_base/src/ProcessTrait.php
   @@ -16,21 +16,44 @@ trait ProcessTrait {
   +            $errors[] = "Error when running \"{$command}\" No more retry..." . PHP_EOL;
   +            $retry++;
   

   Thinking through the logic here, if the retry count was set to 2, I think this loop would only retry one time.

   Loop 1 - $retry 0, command is executed for the first time.
   Loop 2 - $retry 1, command is retried one time.
   Loop 3 - $retry 2, command is not retried again and this error is shown.

5. +++ b/modules/tome_base/src/ProcessTrait.php
   @@ -45,16 +68,16 @@ trait ProcessTrait {
   -        $process = new Process($command, isset($_SERVER['PWD']) ? $_SERVER['PWD'] : NULL);
   +        $process = new Process($command);
   ...
   -      usleep(50000);
   

   This lines shouldn't be changed - guessing the patch was just made against an older release of Tome.

6. +++ b/modules/tome_base/src/ProcessTrait.php
   @@ -45,16 +68,16 @@ trait ProcessTrait {
   +
   

   This newline can be removed.

7. +++ b/modules/tome_static/src/Commands/StaticCommand.php
   @@ -31,6 +31,10 @@ class StaticCommand extends CommandBase {
   +  /**
   +   * The default number of retry per failed process.
   +   */
   

   Add a newline above this block comment.

For the retry count - users will expect that if the retry-count option is set to "2", a failing command will be executed three times. One for the first execution, and then two retries. Does that make more sense?

This is looking good, thanks again @jlatorre. I'll tweak some things on commit but this should get into the next release.

+++ b/modules/tome_base/src/ProcessTrait.php
@@ -16,21 +16,43 @@ trait ProcessTrait {
+        while ($retry <= $retry_count && !$current_process->isSuccessful()) {

After testing this a bit, I found that Drush can have error output but still return 0, which means you have to check the error output as well.

Also, this while loop is blocking - meaning that retries of different processes can never happen in parallel.

Here's a new patch which moves all the retry checking to another callback before the normal process filter callback, and fixes some errors I found when testing #10.

This is in now - thanks for the help!