I have a makefile that was working nicely with beta 2, and after upgrading to beta 6 yesterday now has an error.

I'm defining some custom theme repo's in my make file like so, with 'private' uri's:

projects[perlucida][type] = theme
projects[perlucida][download][type] = git
projects[perlucida][download][url] = "git@devel.perlucida.com:themes/perlucida.git"
projects[hill_logan][type] = theme
projects[hill_logan][download][type] = git
projects[hill_logan][download][url] = "git@devel.perlucida.com:themes/hill_logan.git"

But during the make I get an error for these:

Unable to clone perlucida from git://devel.perlucida.com/themes/perlucida.git.             [error]
Unable to clone hill_logan from git://devel.perlucida.com/themes/hill_logan.git.           [error]

Notice the change from what I believe is a private git uri to a public one. The former works, the latter doesn't (even outside of drush_make).

CommentFileSizeAuthor
#11 697018_giturl-3.patch3.45 KBadrinux
#9 697018_giturl-2.patch3.4 KBadrinux
#6 697018_nowc.patch2.7 KBanarcat
#2 git-url-no-rewrite.patch1.88 KBboztek

Comments

adrinux’s picture

adrinux commented

Been doing some research :)

It seems this problem was introduced in this commit http://drupal.org/cvs?commit=318778
Specifically this line:

 $url = $wc ? "git@{$url_ar[0]}:{$url_ar[1]}" : "git://{$url_ar[0]}/{$url_ar[1]}";

The result of which is that unless --working-copy is passed to drush when calling make the url is converted regardless.
So a workaround for now is to call drush with --working-copy:

drush --working-copy make mymakefile.make mydestination/here

But that means an entire makefile (in my case entire aegir platform) is treated as working-copy, it's no longer possible to clone in one project via export and another as a working copy. Anybody using ssh to access their repos and not running git-daemon is forced to use --working-copy or their git clones will fail.
(Or is there some way of adding --working-copy in the makefile?)

Basically this approach doesn't seem fine grained enough.

Short of reverting http://drupal.org/cvs?commit=318778 I have no idea how to fix this.

boztek’s picture

boztek commented
StatusFileSize
new git-url-no-rewrite.patch1.88 KB

There clearly needs to be work done on the git protocol handling but for a simple fix I have the following patch with what I think are more sensible defaults.

It stops the rewriting of git download urls whenever --working-copy is missing. It instead rewrites public urls to private ones _only_ when --working-copy is requested but maintains the public url fallback so that if a private url fails (which could easily happen if you're forcing all urls to private with the working copy option) but the same repo is accessible over git:// then it uses that.

boztek’s picture

boztek commented
Status: Active » Needs review
adrinux’s picture

adrinux commented
Status: Needs review » Needs work

This fixes my initial problem but not in an ideal way.

Here's my simple test make file. Note one 'private' git repo, one 'public'.

core = 6.x
projects[] =  drupal

; Themes custom
projects[perlucida][type] = theme
projects[perlucida][download][type] = git
projects[perlucida][download][url] = "git@devel.perlucida.com:themes/perlucida.git"

projects[hostmaster][type] = profile
projects[hostmaster][download][type] = git
projects[hostmaster][download][url] = "git://git.aegirproject.org/hostmaster.git"

When not using --working-copy both downloads succeed straight off – but of course no .git folder is kept.

php /var/aegir/drush/drush.php make /var/aegir/builds/d6_test.build /var/aegir/platforms/d6_test_01
Project information for drupal retrieved.                                            [ok]
drupal downloaded from http://ftp.drupal.org/files/projects/drupal-6.15.tar.gz.      [ok]
perlucida cloned from git@devel.perlucida.com:themes/perlucida.git.                  [ok]
hostmaster cloned from git://git.aegirproject.org/hostmaster.git.                    [ok]
[rest of output trimmed off]

When using the --working-copy flag both downloads succeed, but the public repo is first approached via ssh, resulting in being challenged for a password three times (in this case, I also had to type 'yes' to accept the RSA key fingerprint on my first try).

php /var/aegir/drush/drush.php --working-copy make /var/aegir/builds/d6_test.build /var/aegir/platforms/d6_test_01-wc

Project information for drupal retrieved.                                            [ok]
drupal downloaded from http://ftp.drupal.org/files/projects/drupal-6.15.tar.gz.      [ok]
perlucida cloned from git@devel.perlucida.com:themes/perlucida.git.                  [ok]
git@git.aegirproject.org's password: 
git@git.aegirproject.org's password: 
git@git.aegirproject.org's password: 
hostmaster private url failed, using public.                                         [ok]
hostmaster cloned from git://git.aegirproject.org/hostmaster.git.                    [ok]
[rest of output trimmed off]

I can think of other outcomes here too: a server using fail2ban or similar can block IP's following failed authentication attempts – if that were to happen the public download would fail too.

I'm still of the opinion that drush_make is trying to be too clever about git urls, the more sensible option – use the url provided and error if it fails – seems better than trying to second guess the user or the server setup.

Having talked with mig5 in #aegir I'd also say that the 'private' vs 'public' nomenclature is misleading. What's been described as private is in fact git over ssh, whilst 'public' is via the git protocol (served by git-daemon usually) – it's perfectly possible to push to git-daemon too (there's a config option that's disabled by default because there's no authentication, you can set up anon push too [1]) although it's not done often. This makes a nonsense of drush_make's current approach to git urls and --working-copy.

(It'd be nice to have a per download --working-copy option – it relates to this and would greatly simplify matters I think.)

[1] http://ikiwiki.info/tips/untrusted_git_push/

boztek’s picture

boztek commented

Fixing that behaviour is out of the scope of drush_make though don't you think?

Basically you should never have ssh urls in a make file for a server you don't have access to. If you had ssh access to that server then the fallback would work smoothly if for instance you didn't have write access to that particular repo.

I agree that better git support is required and am willing to work on it. I particularly am interested in restoring local repo support when in dev mode to test build and make files without downloading everything over the network. There seem to be two parts to this though; the first is to make --working-copy more resilient to different protocol options and the second is to have some sort of repo caching option. We should probably attack the first part before the second.

anarcat’s picture

anarcat commented
Status: Needs work » Needs review
StatusFileSize
new 697018_nowc.patch2.7 KB

I disagree: I think that drush make is trying to be too witty here. First off, --working-copy has nothing to do with the URL: it just says you want to keep the .git directory or not, it should not have the side effect of ... changing the upstream URL, which is no small thing!

I have rolled a patch that stops this sillyness and just tries the "alternate url" of the one provided if it fails. The alternate url is defined as the "other" way to access the repo: if you're specified git:// and it failed, it will try git@ and vice versa.

--working-copy is then used only to determine if we keep the .git directory or not, which is what it's about.

Note that this gives indeterminate results if the url starts with foo@ (ie. with something else than git:// or git@), but I think the original code already has this problem.

"You assume too much." ;)

adrinux’s picture

adrinux commented
Status: Needs review » Needs work

Thanks @anarcat.

This takes a sensible approach. I retested with my makefile and tests as described in comment#4 above, everything now works as expected. Including retaining .git if working copy is specified.

I also tested how well it failed by deliberatly using a makefile with incorrect urls too:

core = 6.x
projects[] =  drupal

; Themes custom
projects[perlucida][type] = theme
projects[perlucida][download][type] = git
;projects[perlucida][download][url] = "git@devel.perlucida.com:themes/perlucida.git"
projects[perlucida][download][url] = "git://devel.perlucida.com/themes/perlucida.git"

projects[hostmaster][type] = profile
projects[hostmaster][download][type] = git
;projects[hostmaster][download][url] = "git://git.aegirproject.org/hostmaster.git"
projects[hostmaster][download][url] = "git@git.aegirproject.org:hostmaster.git"

Here are the results, with --working-copy and without:

php /var/aegir/drush/drush.php --working-copy make /var/aegir/builds/d6_test.build /var/aegir/platforms/d6_test_03-wc
Project information for drupal retrieved.                                            [ok]
drupal downloaded from http://ftp.drupal.org/files/projects/drupal-6.15.tar.gz.      [ok]
Unable to clone perlucida from .                                                     [error]
git@git.aegirproject.org's password: 
git@git.aegirproject.org's password: 
git@git.aegirproject.org's password: 
Unable to clone hostmaster from .                                                    [error]
An error occurred at function : drush_drush_make_make          

 php /var/aegir/drush/drush.php make /var/aegir/builds/d6_test.build /var/aegir/platforms/d6_test_03
Project information for drupal retrieved.                                            [ok]
drupal downloaded from http://ftp.drupal.org/files/projects/drupal-6.15.tar.gz.      [ok]
Unable to clone perlucida from .                                                     [error]
git@git.aegirproject.org's password: 
git@git.aegirproject.org's password: 
git@git.aegirproject.org's password: 
Unable to clone hostmaster from .                                                    [error]
An error occurred at function : drush_drush_make_make

So it seems error reporting needs some work and it fails to try the alternative url.

Marking this needs work while I look at this closer.

adrinux’s picture

adrinux commented

I'm reworking anarcat's patch. More later.

adrinux’s picture

adrinux commented
StatusFileSize
new 697018_giturl-2.patch3.4 KB

Tiny changes to anarcat's patch, still work-in-progress, attached.

+ removed incorrect and no longer applicable comments
+ corrected some variable names so the git url is correctly reported on successful cloning

- still doesn't fail-over to alternative url, because $alt_url is not being created correctly and is just 'git@' or 'git://' without the actual location.

adrinux’s picture

adrinux commented
Status: Needs work » Reviewed & tested by the community

Working patch. Basically a finished version of anarcat's. Changes from anarcat's patch:

+ removed incorrect and no longer applicable comments
+ corrected some variable names so the git url is correctly reported on successful cloning
+ reverted part of the alternative url generation so it actually works

This fixes my original issue, it retains the feature of trying the alternative git url syntax if the provided one fails and it respects the working copy tag.

Tested and working.

Marking this RTBC.

adrinux’s picture

adrinux commented
StatusFileSize
new 697018_giturl-3.patch3.45 KB

and of course the patch itself. sigh...

dmitrig01’s picture

dmitrig01 commented
Status: Reviewed & tested by the community » Fixed

comitted, thanks

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.