Avoid broken locks in commit processing

I haven’t been able to track down the error yet. I’m switching to trying to reproduce this in staging. My suspicion is something that handles the list of changed files in the commit OOMed. This could take a few hours.

Since the Git repo is working, please don’t pause committing for this. This also takes out the testing on commit. I queued up https://www.drupal.org/pift-ci-job/614401 manually, you may want to queue more.

In general, when a project gets out of sync, we don’t catch up the commit comments. We do usually get the Git history on Drupal.org back in line, but that OOMs for core. It can be manually populated with a little work.

I remembered we have versioncontrol_sync_log to potentially help here. I reviewed and found nothing useful.

I also broke the repo sync lock with UPDATE versioncontrol_repositories SET locked = 0 WHERE repo_id = 2;, so the next commit will have a full shot at syncing.

Looks like the recent commits are going through normally. The problem was most likely an out of memory error when processing the large commit, leaving the lock open after the process died.

mlhess and I have been making decent progress getting Git staging into a good state for testing. Commit hooks are working. We still need to resize a disk that filled up and test everything.

Staging’s Git is working well now.

Yesterday’s missed Git event sync happened after a Git event that took 109.03 seconds to process. It was retried 4 times while the previous event was processed, but in quick succession within 3.6 seconds.

https://cgit.drupalcode.org/versioncontrol/tree/includes/VersioncontrolR... is what needs some work. Beanstalk does support queue item delays, which would be appropriate here, https://cgit.drupalcode.org/drupalorg/tree/drupalorg/drupalorg.drush.inc....

Need to decide on:

• Hard-code Beanstalkd awareness into versioncontrol. Not technically correct, gets the job done.
• DrupalQueue has static variables and/or common object instances in it, have drupalorg sneak the beanstalkd_params in. Might be tricky.
• Implement our own queue class on top of Beanstalk that does this. Technically correct technical debt.
• Just put a sleep() call in. Queue portability?

We now have zero locked repositories with the new layer of reposync code for GitLab support, and are monitoring for new locked repositories. Since the cutover, we haven’t had any reposync issues logged.

This won’t fix the underlying race condition, but we’ll know it when it happens, and be more-able to recover these. The words I said in #10 still look good.

Hard-code Beanstalkd awareness into versioncontrol.

Going with this, since it is straightforward.

If the queue is Beanstalkd, this adds a configurable delay to the followup attempts. By default 5 seconds for the first attempt, then 10 for the second, etc.

This is now deployed on Drupal.org, we’ll see how effective it is over a few weeks.

In general, locked repositories were encountered much less frequently since the move to GitLab:

SELECT count(DISTINCT elid) events, substring(from_unixtime(start), 1, 7) month FROM versioncontrol_sync_log WHERE message LIKE 'Locked repository:%' AND start > unix_timestamp('2018-07-01') GROUP BY month;
+--------+---------+
| events | month   |
+--------+---------+
|     50 | 2018-07 |
|     51 | 2018-08 |
|     51 | 2018-09 |
|     72 | 2018-10 |
|     36 | 2018-11 |
|     56 | 2018-12 |
|     65 | 2019-01 |
|    215 | 2019-02 |
|    135 | 2019-03 |
|     23 | 2019-04 |
|     10 | 2019-05 |
|      9 | 2019-06 |
|     10 | 2019-07 |
|      4 | 2019-08 |
+--------+---------+

I’m doing a full sync on all 55 non-core repositories that hit a sync lock since April as a precaution. This is a bit overkill, since some of these would have synced on one of the 5 retries, it doesn't hurt to sync again.

We finally hit a couple syncs that ran into locked repositories, and this is working well.

A branch was updated on a project, which took 20 seconds. In the meantime, 7 seconds later, a tag was pushed too. That hit the lock, was retried 6 seconds later, also hit the lock, and was retried successfully 9 seconds after that.

#3078948: Git tags not available for new release creation hit a hard lock, the syncing crashed out with OOM. The retries were doomed, but they did get spaced out nicely.

@drumm, thanks for working on this.
I agree with your suggested solution: it seems like a non-usual case, and beanstalk awareness is the easier way to get it working.
This is now on 7.x-1.x.
We may want to wrap the condition with a module_exists() with the module providing the class.

Thanks!

I double checked and … instanceof ClassThatDoesNotExist just returns FALSE and does not throw any notices/etc, so I think this is fine without extra double checking.

@drumm Thanks for checking, it gives peace of mind :-)