[meta] Improve repo locking & behavior on synchronization failures

related #1237764: Create custom exceptions (almost duplicate?)

@sdboyer: What do you think about this patch?

Not sure if we need VersioncontrolSynchronizationConcurrencyException, we already have VersioncontrolLockedRepositoryException.

And about errors field, not sure how we could catch many errors at the same time, if we use exceptions to communicate(so only one is used). I'm still in favor of having the field, but unless backend override the VersioncontrolRepository::sync*() it's not possible to have more than one error, and maybe that's fine :-)

Uhm, I was reviewing this, but it still needs work, will update soon. i.e. when repository is locked is not storing the right message on event sync log table.

Copying the commit message:

Issue #1642326: Recover from locked exception on repository syncEvent().

It is possible and necessary to do a recovery when syncing from an
event, otherwise the queue item is completely ignored.
To do that the queue item is en-queued again when the repository is
found locked.

Two main changes to be able to do that:

- A new abstract method is added to repository class,
generateRawData($event), which converts an event into the raw string
that the vcs backend hook script produces.
- Now, the repository generateCodeArrivalEvent() method can possibly
also receive a $data['elid'], which means the event was already
processed before, so a simple event load is expected in that case.

Naturally this needs a new method implementation and a change on the generateCodeArrivalEvent() method for all backends. I'm adding the patch for vc_git here for convenience.

On the other side, about the solution, I saw two alternatives:
- Use a global flag on repository table to indicate it needs a fullsync the next time.
- Re-enqueue a job to do sync of the same event again.

I found the second more natural, and that's what is implemented.

@sdboyer: What do you think about this? I would like to know your opinion.

bottom line is, in order to automatically retry, we'll need to set up some kind of escape hatch.

Based on the code from drupal_cron_run()

  foreach ($queues as $queue_name => $info) {
    $function = $info['worker callback'];
    $end = time() + (isset($info['time']) ? $info['time'] : 15);
    $queue = DrupalQueue::get($queue_name);
    while (time() < $end && ($item = $queue->claimItem())) {
      $function($item->data);
      $queue->deleteItem($item);
    }
  }

I think there is no standard way to deal re-added items to a queue, main system by design does not know about it(it just remove the item from queue after the worker callback is called), so I have just added a check for the number of times the event has been processed base on the versioncontrol_sync_log table entries and a new global variable versioncontrol_reposync_max_syncevent_retries.

Adding this here for bot to take a look, while I test it manually.

Also an interdiff for easier review, in summary:

• Move the locked syncronization error constant to the sync interface.
• Only enqueue the resync again on locked repository exception if we are under a limit of retries.
• Implement the new repository generateRawData() method on the test backend.

After trying this locally and making some minor changes(not really on ideas but on minor bugs) and some manual testing, I think this is ready.

Added to 7.x-1.x, now D6 backport is pending

Simple cherry-pick. BTW also added on d6 and d7 at vc_git the follow ups.