Not all commits showing up

Confirmed. This still isn't appearing as of Nov. 4

I can confirm this behavior.

On my project https://drupal.org/project/packaging. for example, I made NINE commits on 14 November but only TWO of them show up on https://drupal.org/node/1375146/commits

These were my commits on 14 November. Only the last two show up on the commits page:
http://drupalcode.org/project/packaging.git/commit/124fb9afdf5295235b14c...
http://drupalcode.org/project/packaging.git/commit/ea1220de3298c98b90924...
http://drupalcode.org/project/packaging.git/commit/f4dc7196f1d3bed82e9a3...
http://drupalcode.org/project/packaging.git/commit/b612df09668a2d20d28fa...
http://drupalcode.org/project/packaging.git/commit/ac628b3236e9367559260...
http://drupalcode.org/project/packaging.git/commit/c7fac7ea625521d38bf35...
http://drupalcode.org/project/packaging.git/commit/f26a8c8be14ee3f285913...
http://drupalcode.org/project/packaging.git/commit/332e84d2c1f69a8f876d3...
http://drupalcode.org/project/packaging.git/commit/4963c79e1a608e3447e50...

Here is another example of a commit not showing on the project listing. The commit was on 23rd November in this issue, #5
https://drupal.org/comment/8206099#comment-8206099

The actual commit is http://drupalcode.org/project/scheduler.git/commit/a1afb0b5bf7d82bd8379e...

But the list https://drupal.org/node/3292/commits does not show the commit from 23. Latest is from 18th Nov

Hope this helps

Jonathan

#2132659: [META] Problems with Project Release Packaging is not the parent issue, but it is related.

Changing the status back to 'Active' to avoid confusion, as we no longer user 'needs review' as confirmation step.

Not sure if this is related, but I made some commits/patches that were committed to a sandbox project. They do not show on my profile page in the "Projects" section: https://drupal.org/user/1579186 However, they do show on the "Your commits" tab https://drupal.org/user/1579186/track/code and on the project commits list https://drupal.org/node/1429904/commits

http://drupalcode.org/sandbox/amateescu/1429904.git/commit/3337650
http://drupalcode.org/sandbox/amateescu/1429904.git/commit/33797a3
http://drupalcode.org/sandbox/amateescu/1429904.git/commit/fd92de5
http://drupalcode.org/sandbox/amateescu/1429904.git/commit/01c0dff
http://drupalcode.org/sandbox/amateescu/1429904.git/commit/5929670

Other commits to a sandbox project that I created do show up on all three pages.

Since this seems to be a data loss-related issue (the data isn't really *lost* obviously, since it's in Git, but nevertheless) bumping to Major. It's also the kind of thing that ideally we'd have tests to cover so this sort of bug wasn't introduced.

Thanks for upping the priority. It expect this is related to the fact that -dev releases are stuck in the past, even when newer commits have been made, eg for Scheduler the latest -dev release is 18th Nov, but the following commits have been been made
27 Nov 8:51 http://drupalcode.org/project/scheduler.git/commit/ebfb1bb
27 Nov 8:41 http://drupalcode.org/project/scheduler.git/commit/dbfbef1
26 Nov 18:29 http://drupalcode.org/project/scheduler.git/commitdiff/46c1d4a
26 Nov 18:17 http://drupalcode.org/project/scheduler.git/commitdiff/c32b8390e
26 Nov 15:00 http://drupalcode.org/project/scheduler.git/commit/35739c5
22 Nov 17:45 http://drupalcode.org/project/scheduler.git/commitdiff/a1afb0b

None of these are showing on https://drupal.org/node/3292/commits - the latest is 18th November, which matches the -dev release date. If there is anything I can do to help, let me know.

Jonathan

I think my issue might actually be unrelated. Now that http://drupal.org/project/entityqueue was promoted to a full project I see the commit count in the "Projects" block on my profile now. It seems that block needs to be updated so that your commits on sandbox projects that weren't created by you are included?

Should I open a new issue for that, or is that within the realm of this issue?

There seem to be two new occurrences of this:
#2159285: Commits and -dev release for Panopoly not updating
#2158069: Development release not packaged after commits to Scheduler

Is there anything I can do to help with this issue? I gather that it is the cause of #2158069: Development release not packaged after commits to Scheduler

This is a new problem I can report seeing on the Demo Framework project since November '13.

I can confirm similar issues here, https://drupal.org/sandbox/chx/2105305

Commits are not appearing in the sidebar block or on my profile.

This issue is currently blocked on #2169079: Fix escaping in VersioncontrolGitRepositoryHistorySynchronizerDefault. I hope marvil07 will be able to take a look into it this week.

Updated the issue summary and started to list all affected projects.

Now that #2169079: Fix escaping in VersioncontrolGitRepositoryHistorySynchronizerDefault is fixed and pushed to Drupal.org, I tried deleting the Panopoly 7.x-1.x branch and re-pushing it (since that was what activated the script that imports commits in the past), however, commits newer than November 21st are still not showing up in Drupal.org:

https://drupal.org/node/1484528/commits

Is there anything else that I need to do? Or are we waiting on something else to happen or be fixed?

Thanks again for everyone's hard work on this issue!

The re-syncing described in #50 #2132327-50: Can't create release: git repository tag appears in repo browser, but not vcs tables is probably required.

I'm waiting a couple more hours for our git staging server to sync up. I did do it live successfully for a very small repo, https://drupal.org/project/infrastructure_drupalorg. That's promising, but I want to test a bit more for repos that people actually use. The syncing can be quite destructive if it decides to error out in the middle.

Fixed scheduler, panopoly, and df.

Syncing core sandboxes will need more work in versioncontrol* so that it does not run out of memory:

[drumm@git7staging htdocs]$ drush -r /var/www/git-dev.drupal.org/htdocs vc-sync 59327 --nobatch
Beginning synchronization of repository 2105305                                                                                                                                                            [ok]
PHP Fatal error:  Allowed memory size of 838860800 bytes exhausted (tried to allocate 328777 bytes) in /var/www/git-dev.drupal.org/htdocs/sites/all/modules/versioncontrol_git/includes/plugins/reposync/VersioncontrolGitRepositoryHistorySynchronizerDefault.class.php on line 454

Fatal error: Allowed memory size of 838860800 bytes exhausted (tried to allocate 328777 bytes) in /var/www/git-dev.drupal.org/htdocs/sites/all/modules/versioncontrol_git/includes/plugins/reposync/VersioncontrolGitRepositoryHistorySynchronizerDefault.class.php on line 454
Drush command terminated abnormally due to an unrecoverable error.                                                                                                                                         [error]
Error: Allowed memory size of 838860800 bytes exhausted (tried to allocate 328777 bytes) in
/var/www/git-dev.drupal.org/htdocs/sites/all/modules/versioncontrol_git/includes/plugins/reposync/VersioncontrolGitRepositoryHistorySynchronizerDefault.class.php, line 454

I found that using the --flush option didn't match tags back up to release nodes:

[drumm@git7staging ~]$ drush -r /var/www/drupal.org/htdocs vc-sync panopoly --flush --nobatch
Beginning synchronization of repository panopoly                                                                                                                                                           [ok]
WD vc_project: Release-attached label "7.x-1.0-alpha1" was missing after from-scratch resynchronization of repository "panopoly" (id: 35328).                                                              [error]
WD vc_project: Release-attached label "7.x-1.0-beta1" was missing after from-scratch resynchronization of repository "panopoly" (id: 35328).                                                               [error]
WD vc_project: Release-attached label "7.x-1.0-beta2" was missing after from-scratch resynchronization of repository "panopoly" (id: 35328).                                                               [error]
WD vc_project: Release-attached label "7.x-1.0-beta3" was missing after from-scratch resynchronization of repository "panopoly" (id: 35328).                                                               [error]
WD vc_project: Release-attached label "7.x-1.0-beta4" was missing after from-scratch resynchronization of repository "panopoly" (id: 35328).                                                               [error]
WD vc_project: Release-attached label "7.x-1.0-beta5" was missing after from-scratch resynchronization of repository "panopoly" (id: 35328).                                                               [error]
WD vc_project: Release-attached label "7.x-1.0-beta6" was missing after from-scratch resynchronization of repository "panopoly" (id: 35328).                                                               [error]
WD vc_project: Release-attached label "7.x-1.0-rc1" was missing after from-scratch resynchronization of repository "panopoly" (id: 35328).                                                                 [error]
WD vc_project: Release-attached label "7.x-1.0-rc2" was missing after from-scratch resynchronization of repository "panopoly" (id: 35328).                                                                 [error]
WD vc_project: Release-attached label "7.x-1.0-rc3" was missing after from-scratch resynchronization of repository "panopoly" (id: 35328).                                                                 [error]
WD vc_project: Release-attached label "7.x-1.0-rc4" was missing after from-scratch resynchronization of repository "panopoly" (id: 35328).                                                                 [error]
WD vc_project: Release-attached label "7.x-1.0-rc5" was missing after from-scratch resynchronization of repository "panopoly" (id: 35328).                                                                 [error]
WD vc_project: Release-attached label "7.x-1.x" was missing after from-scratch resynchronization of repository "panopoly" (id: 35328).                                                                     [error]
There was an error during the synchronization, review the sync log table for details.                                                                                                                      [error]

So I didn't use that option. New tags do get pulled in, ready for releases.

Moving down to normal since this is not blocking any releases.

There can be several reasons for a commit not showing up(version control operation row is missing).

First, let me point some details on how synchronization works on a git push(event synchronization):
- A user push some code (the usual case is one commit in one branch, but it can be multiple commits in multiple branches, i.e. the whole 7.x branch of drupal core from the start).
- A new item is added to the versioncontrol_reposync queue.
- The queue runner, process-waiting-queue drush command in d.o case, is executed.

AFAIK the most common case is that versioncontrol reposync queue worker failed execution, and the usual case is memory limit is reached.

In those cases we need to run synchronization manually via the vc-sync drush command(full synchronization, instead of push only data changes), which is what drum has done for the fixed projects.

Sadly, in the case of really big git repositories, memory limit will still applies, so yes, the way to solve this would be to #2226443: Reduce memory usage on default reposync plugin full synchronization.

That would probably explain why the IMP sandbox has issues: https://drupal.org/sandbox/chx/2105305 since it's got all the history for core.

Not sure which queue is the best for this issue, please move somewhere if this isn't the one. Thanks!

Since this is a general concern, I guess it's ok to move it back to QA.

I have made several adjustments on #2226443: Reduce memory usage on default reposync plugin full synchronization, which let run most full sync cases ok. They still take long, but as mentioned on #2226443-5: Reduce memory usage on default reposync plugin full synchronization that's because there are a lot of exec's to do to sync a lot of git data.

So, drumm, if you are ok to having a drush sync process take a lot of time, we can probably fix imp sandbox.
Please make sure we deploy versioncontrol and versioncontrol_git before trying to run that.
Other un-sync'ed projects should be straight forward(no so much time, no so much memory) to sync.

I've deployed both projects and did a final test resyncing https://git7site.devdrupal.org/node/1819382/commits, which took 121m. I'm starting the resyncs on production.

https://drupal.org/node/1819382/commits has been resynced and looks okay. There are 26 remaining.

The last 2 are resyncing now. sandbox/chx/2105305.git and sandbox/sun/1255586.git

The stuck repos are synced now.

There are some new locked repos, which are syncing normally as far as I know.

I checked on the commits I reported in #4, and the situation is unchanged - I made 9 commits that day but only two show up on my list of commits.

I resynced the packaging project.

So this has to be done on a per-project basis? Can't the resynching be done on all projects, since it would be extremely difficult to go back and try to manually identify specific projects with missing commits?

This could be a different issue, but should the commits show up on our user profiles once they've appeared on the project page? Eg, my commits appeared at sandbox/chx/2105305.git but not on my profile.

@TR - Re-syncing all projects would be a major undertaking. Usually, we can identify problem repos by being hung in a locked state. Looking further into how packaging got into whatever state it was in would not be worth it, until we see a pattern of it happening. I didn't correct it right away since I was going by the issue summary.

@benjy - Yes, that is a different issue. versioncontrol_project_user_view() adds the commits to user pages. It only shows sandbox commits to sandboxes you own.

I also am not seeing my commits show up on the project page. I'm also not clear from this thread what 're-syncing' means in this context. Does anyone have any suggestions on how to make my commits show up on the product page?

This is the project:

https://www.drupal.org/sandbox/mosewrite/2137455

And I have made two commits in the last few days Neither are showing up. Screencap attached.

--Marshall

I'll follow up at #2345799: Commits not showing up in drupal. org. Thanks for including the URL here.