Related to #1003576: Add capacity to create repositories to a versioncontrol submodule

We still need to create git repositories on disk using some kind of a repository manager even if it can't be abstracted enough for it to live in vcs_api itself.

Lets create a git repo manager that provides a drush command for creating and removing repositories.

CommentFileSizeAuthor
#9 1018942-repo-manager.patch12.11 KBsdboyer
#5 1018942-updated-create-repo-manager-4.patch12.04 KBtizzo
#4 1018942-updated-create-repo-manager-2.patch13.26 KBtizzo
#2 1018942-updated-create-repo-manager.patch11.54 KBtizzo
#1 1018942-initial-create-repo-manager.patch8.93 KBtizzo

Comments

tizzo’s picture

tizzo commented
Assigned: Unassigned » tizzo
Status: Active » Needs work
StatusFileSize
new 1018942-initial-create-repo-manager.patch8.93 KB

I coded this up fast and haven't tested it (or even turned it on). It's incomplete and broken but it's a start, I'll update as I continue to develop this.

tizzo’s picture

tizzo commented
StatusFileSize
new 1018942-updated-create-repo-manager.patch11.54 KB

Updated repo manager module for versioncontrol_git

tizzo’s picture

tizzo commented
Status: Needs work » Needs review

This definitely still needs a little tweaking but the core functionality is there.

This patch provides the ability to have either nodeapi or drush manage git repositories via the drupal queue.

Nodeapi enqueues operations and drush processes them.

tizzo’s picture

tizzo commented
Assigned: tizzo » Unassigned
StatusFileSize
new 1018942-updated-create-repo-manager-2.patch13.26 KB

I found a few bugs and stomped them out.

I think this should be working, we'll still need to get a slimmed down version running for the always-on queue. But I think this should be pretty close.

tizzo’s picture

tizzo commented
StatusFileSize
new 1018942-updated-create-repo-manager-4.patch12.04 KB

Messed up that last patch, re-rolling

Anonymous’s picture

Anonymous (not verified) commented

trying to tie all the bits together here, as well as review this patch.

for the tie-it-all-together bits:

* i've created a small module to process a queue where the worker will wait indefinitely for a job, and keep processing jobs indefinitely.

https://github.com/justinrandell/Drupal-Waiting-Queue

* we should implement hook_drupal_queue_load_classes() so we can use drupal_queue_get().

* we should implement hook_cron_queue_info() so we can set 'worker callback' somewhere. (wow, that's a badly named hook, and yeah, we could just have waiting_queue use one of its own hooks, but for now it seems cleaner to use drupal_queue's code seeing as we are already relying on it.

* sdboyer recommended drupalorg_versioncontrol module for that, so i'll create an issue and patch for that

now, for this patch.

+  // Create the repository on disk
+  `mkdir $repository_path`;
+  `git init --bare $repository_path`;

that looks like it lives in a VersionControlGitRepository method somewhere.

+  // Create, configure and save versioncontrol entity.
+  $repository = new VersioncontrolGitRepository();
+  $repository->build($data['repository_data']);
+  $repository->insert();

build() is implemented by VersionControl, and sdboyer said in IRC: "sdboyer: beejeebus: build() is a VersioncontrolEntityInterface method, and is invoked by VersioncontrolBackend::buildEntity(), it should really never be called directly" and "sdboyer: beejeebus: one of VersioncontrolBackend's roles is as a factory", so it looks like we need to a) add the bare calls to make directories and call git commands into VersionControlGitRepository(), and call in to the write methods.

i don't know the VersionControl code well enough to say what the right way is just yet, but i'll get to that unless someone else beats me to it.

also, i'm not 100% of the relation between this ticket and #893114: Implement job queue system for creating new Git repos based on projects, but i suppose that's the meta-ticket for this one?

Anonymous’s picture

Anonymous (not verified) commented 
+          drupal_queue_include();
+          $queue = DrupalQueue::get('versioncontrol_git_repo_manager');

should be just drupal_queue_get(), which calls in to drupal_queue_include().

Anonymous’s picture

Anonymous (not verified) commented
Status: Needs review » Needs work

created this issue for calling in to this module from drupal.org code: #1022540: add a queue for processing git repo jobs.

sdboyer’s picture

sdboyer commented
Assigned: Unassigned » sdboyer
StatusFileSize
new 1018942-repo-manager.patch12.11 KB

New patch, mostly just fixing spelling errors and using the proper method for creating a repo object. Something that needs doing but I didn't do yet is to add some error handling to the worker. We don't want to erroneously insert a repo into the db if it's creation on disk didn't work, and we also want to watchdog() it if repo creation failed.

Next step, though, is to refactor this a fair bit to use a new ctools plugin for actually doing the work of creating the repo. That's something mikey_p and I talked about, and it's really preferable because there are lots of options, and certainly d.o-specific things, which the repo manager will need to take into account, and trying to make the job interface rich enough that it could handle it all via parameters would just be crazy.

tizzo’s picture

tizzo commented

Committed! http://drupal.org/cvs?commit=479574

mikey_p’s picture

mikey_p commented

I found the following snippet worked fine for saving permissions after updating {versioncontrol_project_projects}.repo_id

$node = node_load($NID);
$perms = array_fill_keys(array_keys(project_permission_load($node)), 1);
project_maintainer_save($node->nid, $node->uid, $perms);
mikey_p’s picture

mikey_p commented

Wanted to add that this could be node_load caching if the node is already loaded for some reason, before {vc_project_projects} was updated.

$node = node_load($NID, NULL, TRUE);
$perms = array_fill_keys(array_keys(project_permission_load($node)), 1);
project_maintainer_save($node->nid, $node->uid, $perms);

Should definitely do it.

tizzo’s picture

tizzo commented

That still didn't work... Assigning to mikey_p though I'm not sure he has time to address this this weekend.

tizzo’s picture

tizzo commented
Title: Create git repository manager submodule » Git repository manager submodule doesn't properly associate new repos with projects

updating title

sdboyer’s picture

sdboyer commented
Title: Git repository manager submodule doesn't properly associate new repos with projects » Create git repository manager submodule
Status: Needs work » Fixed

Truth is that we'd probably fixed this problem a few times today...but didn't know it because the supervisord script just kept on running using the same old code. So the new code was never parsed into memory. Zoiks. Anyway, I fixed & verified the problem with the plugins & auth permissions not being properly set, finally. I did it in a way that is HORRIBLY d.o-specific, and committed it, but that's OK because it's really clear at this point that the logic is going to be moved into a plugin. So when we plugin-i-fy it (very early next week), we'll move the logic out into drupalorg. But for now, this works.

sdboyer’s picture

sdboyer commented
Title: Create git repository manager submodule » Git repository manager submodule doesn't properly associate new repos with project

Setting title back, I had a really old cached version when I commented.

Status: Fixed » Closed (fixed)
Issue tags: -git phase 2, -git sprint 8

Automatically closed -- issue fixed for 2 weeks with no activity.

Issue tags: +git phase 2, +git sprint 8

Restoring issue tags, see #2125755: System messages removed all issue tags during D7 upgrade.

  • Commit 53c60f1 on 6.x-2.x, fix-invalid-default-branches, fullsync-memory by tizzo: 
    task #1018942 by sdboyer, tizzo: Added repository manager submodule
  • Commit 1dcc42a on 6.x-2.x, fix-invalid-default-branches, fullsync-memory by sdboyer:
    Issue #1018942: properly initialize and associate new repos with their...