Commit messages comment cron OOM

During DrupalCon, the cron-versioncontrol_repository_event_processor-drupal.org job did get stuck for ~4 days. Certain commits cause it to OOM and currently have to be killed manually. I've added my log of killed queue items to the issue summary.

I was able to re-run the queue worker on util for the three commits with code like:

versioncontrol_repository_event_processor_run_worker(array(
  'repo_id' => 2, // Core
  'elid' => db_query('SELECT elid FROM {versioncontrol_git_event_data} WHERE new_sha1 = :hash', array(':hash' => '122c9b4696f2e07b7afd6959ee34e57effed6557'))->fetchField(),
));

The comments now show up normally, although late. The code completed normally and quickly, so they were indeed not the cause of any problem, but collateral damage from last week's trouble.

Doing the same type of debugging on git7site with the killed queue items should run into the memory leak and let us get to the root cause.

I have not yet tried to replicated this, but I took a look at the related event data for killed items and I found a pattern(update: see file in next comment, it just looks bad inline here).
All of those events are adding a new branch to a drupal core repository clone in a sandbox.
So by best guess is that it is trying to parse the full history of the branch, which start at drupal core repository first commit, which takes time and probably can cause a OOM.

Adding a:2:{s:7:"repo_id";s:5:"18942";s:4:"elid";s:6:"968389";} to the killed list.

Adding

a:2:{s:7:"repo_id";s:5:"47158";s:4:"elid";s:6:"972767";}
a:2:{s:7:"repo_id";s:5:"47158";s:4:"elid";s:6:"972775";}

Adding

a:2:{s:7:"repo_id";s:5:"65569";s:4:"elid";s:6:"972993";}
a:2:{s:7:"repo_id";s:5:"35852";s:4:"elid";s:6:"973511";}

a:2:{s:7:"repo_id";s:5:"42576";s:4:"elid";s:6:"978983";}

a:2:{s:7:"repo_id";s:5:"68201";s:4:"elid";s:6:"981221";}

A couple of days ago, commit messages for the webfor 7.x-4.x branch stopped being automatically posted. I generally push to both the 7.x-4.x branch and 8.x-4.x branch at the same time. I am getting commit messages for the 8.x-4.x branch.

List of recent commit messages:
https://www.drupal.org/node/7404/commits

Here's an example issue:
https://www.drupal.org/node/2467877

Here the 7.x-4.x branch commit that doesn't have a comment in the issue:
http://cgit.drupalcode.org/webform/commit/?id=755af13a6964bc621ad8795f0e...

That looks like a separate issue, maybe related to #2329677: Branch missing in commit comment or another issue in https://www.drupal.org/project/issues/versioncontrol_project?text=commen.... Likely something in the logic in, or called by, http://cgit.drupalcode.org/versioncontrol_project/tree/versioncontrol_pr...

That was me. The versioncontrol event processor queue which posts these comments, among other things, was jammed up for quite awhile until I tracked down the root cause.

This seems to be happening again :-/ Last 6 commits haven't posted to the related issue: https://www.drupal.org/node/259843/commits

This is catching up now. A job needed to be deleted to unblock the queue.

We’ve caught up reposync for all Drupal.org repositories, including core sandboxes, and a few with things like thousands of files added or removed in a single commit. This is working well with the GitLab integration code we’ve built.

@drumm that is great to hear!