1. Place a work area inside a symlinked directory. My real directory was ~/Documents/workspace, the symlinked dir was ~/workspace. So working with ~/workspace/drupal, which is actually ~/Documents/workspace/drupal
  2. cd ~/workspace/drupal; drush dl --package-handler=wget some_module. You get the message Project field_reference (7.x-1.x-dev) downloaded to /Users/rfay/workspace/commerce/sites/all/modules/field_reference.
  3. Look in /modules, and there's a module directory there as well, with a .info in it.
  4. The .gitmodules points to the path in /modules, instead of what's in sites/all/modules

This is with MacOS Lion; using git 1.7.5.4 from xcode, but it seems the git version was not a factor, nor the OS.

Comments

rfay’s picture

After #1314554: dl --destination sets a relative path for project_install_location lands, this still remains an issue, easily recreatable on MacOS Lion.

moshe weitzman’s picture

Status: Active » Postponed (maintainer needs more info)

I'm on Leopard at the moment instead of Lion and git 1.7.8.3. Cannot reproduce. Please post the the full log here when run with --debug.

It would be good to know if the downloaded module is already in the reference cache when this bug is triggerred.

rfay’s picture

Status: Postponed (maintainer needs more info) » Active
FileSize
7.5 KB

I turned off caching completely and rebuilt my repo without the caches as described in git clone --reference considered harmful, which, of course, was an outcome of your mention of the reference cache :-)

Attached is the log of a drush dl; After the dl I have this on a git status:

$ git status
# On branch site
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	modified:   .gitmodules
#	new file:   modules/backup_migrate
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	sites/all/modules/backup_migrate/
moshe weitzman’s picture

Wow, thats quite a 180 on reference cache. --cache is off by default for git_drupalorg so folks won't be getting it unknowingly. They do get it when they do a drush make without --working-copy which is pretty neat IMO. They aren't ending up with .git dirs anyway so it makes sense to turn on --cache by default.

As for this submodule bug, I dunno. I probably won't look into it more myself but hopefully someone will.

greg.1.anderson’s picture

Component: PM (dl, en, up ...) » Documentation
Status: Active » Needs review
FileSize
877 bytes

I have used $options['cache'] = TRUE; to good effect; my normal workflow does not bring me into any of the dangerous territory. The blog post is great, though; best to be sure everyone is forewarned; it was almost a coincidence more than a plan that I had no trouble.

Here is a suggested documentation patch for examples.drushrc.php to alert users to the potential issues.

greg.1.anderson’s picture

n.b. #5 does not address the issue with --gitsubmodule, so perhaps it should have been a separate issue.

moshe weitzman’s picture

Status: Needs review » Active

Committed the docs patch. I actually removed pm-clone because it was too thin and awkward of an alias without its --cache . And the new docs are a little awkward since they suggest making a global 'cache' option, instead of command-specific. I did not change this though.

Moving this to active so someone can look into submodule problem.

opdavies’s picture

I'm having a similar issue, although I'm doing this directly on my remote web server running CentOS. I've cloned Pressflow 6 from Github which worked fine. I've made the following amends to my drushrc.php file so that it uses Git submodules by default.

$options['package-handler'] = 'git_drupalorg';
$options['gitsubmodule'] = TRUE;

I've downloaded admin_menu using the following command, and this seems also to work fine as the module is in the correct place.

[root@homer all]# drush dl admin_menu
Project admin_menu (6.x-1.8) downloaded to /Websites/pnt.oliverdavies.co.uk/pressflow/sites/all/modules/admin_menu. 

However, my .gitmodules file says something different:

[submodule "all/modules/admin_menu"]
        path = all/modules/admin_menu
        url = git://git.drupal.org/project/admin_menu.git

There is now also an additional 'all' directory within the root of this installation that also contains a modules/admin_menu directory that contains a README.txt file and a .git file.

This doesn't happen if I add a submodule using the usual Git method.

[root@homer pressflow]# git submodule add --branch 7.x-3.x git://git.drupal.org/project/views.git sites/all/modules/views

opdavies’s picture

I've tried this on a Ubuntu virtual machine and can confirm that I don't get the issue then.

opdavies’s picture

I've changed my git clone code slightly, and now it seems to be working OK on Mac OS X Lion. This could, of course, be a coincidence - but I can't think of anything else that I've changed.

No luck with CentOS yet though. :(

Original:

git clone git://git.drupal.org/project/drupal.git httpdocs

New:

git clone http://git.drupal.org/project/drupal.git httpdocs

rfay’s picture

@opdavies, it sounds like you're off topic. And I assume you mean "git clone". There should be no difference whatever in what protocol you use for accessing the git repository.

opdavies’s picture

Yes, 'git clone' of course (I've updated my original post).

It just seems strange that it's working for me now on OS X Lion when it wasn't before and that's the only thing that I can think of.

rfay’s picture

Just a note that this problem remains in current lastest in master branch. It will eventually become important enough for action as people use Lion and git submodule, I imagine.

jonhattan’s picture

Component: Documentation » PM (dl, en, up ...)

I'm using git 1.7.10 on debian testing and can't reproduce.

Drush adds the downloaded project as a submodule in package_handler_post_download(), at commands/pm/package_handler/git_drupalorg.inc

For a project downloaded to sites/all/modules, the sequence used to obtain the superproject path is:

jonhattan@croatan:/tmp/drupal-7.14$ cd sites/all/
jonhattan@croatan:/tmp/drupal-7.14/sites/all$ git rev-parse --show-toplevel
/tmp/drupal-7.14

It comes from #999480: Git Submodules and Drush temp directories. Previously, git rev-parse --git-dir was used instead of git rev-parse --show-toplevel.

jonhattan@croatan:/tmp/drupal-7.14/sites/all$ git rev-parse --git-dir
/tmp/drupal-7.14/.git

If I'm in the track --show-toplevel gives you a relative path and perhaps --git-dir is the option we need.

rfay’s picture

I'm wondering if this is actually a git version problem. If so I think it will show up on other platforms before long.

I happened to notice an item on stackexchange that pointed to a git mailing list discussion (I think) talking about confusion with "modules" directory and git.

So I updated to git 1.7.10.4, and the symptoms have changed somewhat.

Now the drush dl'ed submodule gets added to l/modules/<modulename>

drush dl -v --package-handler=git_drupalorg views 
Initialized Drupal 7.15-dev root directory at /tmp/drupal               [notice]
Loading version_control engine.                                         [notice]
Loading package_handler engine.                                         [notice]
Loading release_info engine.                                            [notice]
Downloading release history from                                        [notice]
http://updates.drupal.org/release-history/views/7.x
Executing: which wget
Executing: wget -q --timeout=30 -O /private/tmp/download_fileUyNROj http://updates.drupal.org/release-history/views/7.x
Downloading project views to /tmp/drush_tmp_1338990340_4fcf5f04ecce4    [notice]
...
Executing: git clone  --verbose --progress git://git.drupal.org/project/views.git /tmp/drush_tmp_1338990340_4fcf5f04ecce4/views
Executing: git tag -l 7.x-3.3
Executing: git checkout  7.x-3.3
Using destination directory /tmp/drupal/sites/all/modules               [notice]
Executing: svn info /tmp/drupal/sites/all/modules
Executing: bzr root /tmp/drupal/sites/all/modules
Project views (7.x-3.3) downloaded to                                [success]
/tmp/drupal/sites/all/modules/views.
Executing: git rev-parse --show-toplevel
Executing: git submodule add  git://git.drupal.org/project/views.git l/modules/views
Project views contains 2 modules: views, views_ui.
Command dispatch complete                                               [notice]
rfay@Randys-MacBook-Air:/tmp/drupal$ git status
# On branch 7.x
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	new file:   .gitmodules
#	new file:   l/modules/views
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	sites/all/modules/views/
rfay’s picture

I decided to see if this was related to the git version, so built git 1.7.7.2, which is a longstanding version working (well) with drush on Ubuntu 10.04.

But the behavior and the l/ directory is the same.

Note that drush explicitly adds the submodule in the l/ directory. Why is that?

Executing: git submodule add  git://git.drupal.org/project/views.git l/modules/views
drush dl -v --package-handler=git_drupalorg views 
Initialized Drupal 7.15-dev root directory at /tmp/drupal               [notice]
Loading version_control engine.                                         [notice]
Loading package_handler engine.                                         [notice]
Loading release_info engine.                                            [notice]
Downloading release history from                                        [notice]
http://updates.drupal.org/release-history/views/7.x
Executing: which wget
Executing: wget -q --timeout=30 -O /private/tmp/download_fileYcDmv0 http://updates.drupal.org/release-history/views/7.x
Downloading project views to /tmp/drush_tmp_1338992003_4fcf65834749e    [notice]
...
Executing: git clone  --verbose --progress git://git.drupal.org/project/views.git /tmp/drush_tmp_1338992003_4fcf65834749e/views
Executing: git tag -l 7.x-3.3
Executing: git checkout  7.x-3.3
Using destination directory /tmp/drupal/sites/all/modules               [notice]
Executing: svn info /tmp/drupal/sites/all/modules
Executing: bzr root /tmp/drupal/sites/all/modules
Project views (7.x-3.3) downloaded to                                [success]
/tmp/drupal/sites/all/modules/views.
Executing: git rev-parse --show-toplevel
Executing: git submodule add  git://git.drupal.org/project/views.git l/modules/views
Project views contains 2 modules: views, views_ui.
Command dispatch complete                
jonhattan’s picture

Drush fails to get the full path of the superproject. Please can you post the output of git rev-parse --show-toplevel and git rev-parse --git-dir from /tmp/drupal/sites/all per #14 ?

rfay’s picture

Thanks for looking, @jonhattan

#17:

rfay@Randys-MacBook-Air:/tmp/drupal$ cd sites/all/
rfay@Randys-MacBook-Air:/tmp/drupal/sites/all$ git rev-parse --show-toplevel
/private/tmp/drupal
rfay@Randys-MacBook-Air:/tmp/drupal/sites/all$ git rev-parse --git-dir
/private/tmp/drupal/.git
rfay@Randys-MacBook-Air:/tmp/drupal/sites/all$ 
jonhattan’s picture

is /tmp a symlink or what's going on?

if you intersect this:

/tmp/drupal/sites/all/modules
/private/tmp/drupal

you get l/modules so the submodule path is l/modules/views

jonhattan’s picture

specifically:

$project['full_project_path'] = /tmp/drupal/sites/all/modules/views';
$superproject = '/private/tmp/drupal';
substr($project['full_project_path'], strlen($superproject) + 1);

For reference: http://api.drush.org/api/drush/commands%21pm%21package_handler%21git_dru...

rfay’s picture

It's a MacOS thing. /tmp is a symlink to /private/tmp

So I did it in ~/tmp instead, and lo and behold, with the old git (1.7.7.2) drush dl --package-handler=git_drupalorg works fine:

rfay@Randys-MacBook-Air:~/tmp/drupal$ drush dl -v --package-handler=git_drupalorg views
Initialized Drupal 7.15-dev root directory at /Users/rfay/tmp/drupal    [notice]
Loading version_control engine.                                         [notice]
Loading package_handler engine.                                         [notice]
Loading release_info engine.                                            [notice]
Downloading release history from                                        [notice]
http://updates.drupal.org/release-history/views/7.x
Executing: which wget
Executing: wget -q --timeout=30 -O /private/tmp/download_filenY9S7c http://updates.drupal.org/release-history/views/7.x
Downloading project views to /tmp/drush_tmp_1339014717_4fcfbe3d60a5c    [notice]
...
Executing: git clone  --verbose --progress git://git.drupal.org/project/views.git /tmp/drush_tmp_1339014717_4fcfbe3d60a5c/views
Executing: git tag -l 7.x-3.3
Executing: git checkout  7.x-3.3
Using destination directory /Users/rfay/tmp/drupal/sites/all/modules    [notice]
Executing: svn info /Users/rfay/tmp/drupal/sites/all/modules
Executing: bzr root /Users/rfay/tmp/drupal/sites/all/modules
Project views (7.x-3.3) downloaded to                                [success]
/Users/rfay/tmp/drupal/sites/all/modules/views.
Executing: git rev-parse --show-toplevel
Executing: git submodule add  git://git.drupal.org/project/views.git sites/all/modules/views
Project views contains 2 modules: views, views_ui.
Command dispatch complete                                               [notice]
rfay@Randys-MacBook-Air:~/tmp/drupal$ git status
# On branch 7.x
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	new file:   .gitmodules
#	new file:   sites/all/modules/views
#

Now works also with 1.7.10.4:

rfay@Randys-MacBook-Air:~/tmp/drupal$ drush dl  -v --package-handler=git_drupalorg views
Initialized Drupal 7.15-dev root directory at /Users/rfay/tmp/drupal    [notice]
Loading version_control engine.                                         [notice]
Loading package_handler engine.                                         [notice]
Loading release_info engine.                                            [notice]
Downloading release history from                                        [notice]
http://updates.drupal.org/release-history/views/7.x
Executing: which wget
Executing: wget -q --timeout=30 -O /private/tmp/download_fileQuegLR http://updates.drupal.org/release-history/views/7.x
Downloading project views to /tmp/drush_tmp_1339014917_4fcfbf0583362    [notice]
...
Executing: git clone  --verbose --progress git://git.drupal.org/project/views.git /tmp/drush_tmp_1339014917_4fcfbf0583362/views
Executing: git tag -l 7.x-3.3
Executing: git checkout  7.x-3.3
Using destination directory /Users/rfay/tmp/drupal/sites/all/modules    [notice]
Executing: svn info /Users/rfay/tmp/drupal/sites/all/modules
Executing: bzr root /Users/rfay/tmp/drupal/sites/all/modules
Project views (7.x-3.3) downloaded to                                [success]
/Users/rfay/tmp/drupal/sites/all/modules/views.
Executing: git rev-parse --show-toplevel
Executing: git submodule add  git://git.drupal.org/project/views.git sites/all/modules/views
Project views contains 2 modules: views, views_ui.
Command dispatch complete                                               [notice]
rfay@Randys-MacBook-Air:~/tmp/drupal$ git status
# On branch 7.x
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	new file:   .gitmodules
#	new file:   sites/all/modules/views
#
rfay@Randys-MacBook-Air:~/tmp/drupal$ 

Now with git 1.7.8.4, which is the version that has dogged me for months...
I ALSO can't recreate the problem in this simple environment:

rfay@Randys-MacBook-Air:~/tmp/drupal$ git --version
git version 1.7.8.4
rfay@Randys-MacBook-Air:~/tmp/drupal$ drush dl  -v --package-handler=git_drupalorg views
Initialized Drupal 7.15-dev root directory at /Users/rfay/tmp/drupal    [notice]
Loading version_control engine.                                         [notice]
Loading package_handler engine.                                         [notice]
Loading release_info engine.                                            [notice]
Downloading release history from                                        [notice]
http://updates.drupal.org/release-history/views/7.x
Executing: which wget
Executing: wget -q --timeout=30 -O /private/tmp/download_fileFhzfRx http://updates.drupal.org/release-history/views/7.x
Downloading project views to /tmp/drush_tmp_1339015311_4fcfc08fe971b    [notice]
...
Executing: git clone  --verbose --progress git://git.drupal.org/project/views.git /tmp/drush_tmp_1339015311_4fcfc08fe971b/views
Executing: git tag -l 7.x-3.3
Executing: git checkout  7.x-3.3
Using destination directory /Users/rfay/tmp/drupal/sites/all/modules    [notice]
Executing: svn info /Users/rfay/tmp/drupal/sites/all/modules
Executing: bzr root /Users/rfay/tmp/drupal/sites/all/modules
Project views (7.x-3.3) downloaded to                                [success]
/Users/rfay/tmp/drupal/sites/all/modules/views.
Executing: git rev-parse --show-toplevel
Executing: git submodule add  git://git.drupal.org/project/views.git sites/all/modules/views
Project views contains 2 modules: views, views_ui.
Command dispatch complete                                               [notice]
rfay@Randys-MacBook-Air:~/tmp/drupal$ git status
# On branch 7.x
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	new file:   .gitmodules
#	new file:   sites/all/modules/views
rfay’s picture

Title: drush dl (--gitsubmodule) downloads extra pieces into /modules (and points there with .gitmodules) » drush dl (--gitsubmodule) downloads extra pieces into /modules (and points there with .gitmodules) when symlinked parent dir

OK, maybe we're making progress here, and maybe someday we'll understand this.

I use ~/workspace/sitename normally, but on my MacOS env, ~/workspace is a symlink to ~/Documents/workspace

So taking the exact same repo we've been working with and putting it in ~/workspace, we have the fail again:

rfay@Randys-MacBook-Air:~/workspace/drupal$ drush dl -v --package-handler=git_drupalorg views
Initialized Drupal 7.15-dev root directory at                           [notice]
/Users/rfay/workspace/drupal
Loading version_control engine.                                         [notice]
Loading package_handler engine.                                         [notice]
Loading release_info engine.                                            [notice]
Downloading release history from                                        [notice]
http://updates.drupal.org/release-history/views/7.x
Executing: which wget
Executing: wget -q --timeout=30 -O /private/tmp/download_fileN38FXJ http://updates.drupal.org/release-history/views/7.x
Downloading project views to /tmp/drush_tmp_1339016702_4fcfc5fe46ffe    [notice]
...
Executing: git clone  --verbose --progress git://git.drupal.org/project/views.git /tmp/drush_tmp_1339016702_4fcfc5fe46ffe/views
Executing: git tag -l 7.x-3.3
Executing: git checkout  7.x-3.3
Using destination directory                                             [notice]
/Users/rfay/workspace/drupal/sites/all/modules
Executing: svn info /Users/rfay/workspace/drupal/sites/all/modules
Executing: bzr root /Users/rfay/workspace/drupal/sites/all/modules
Project views (7.x-3.3) downloaded to                                [success]
/Users/rfay/workspace/drupal/sites/all/modules/views.
Executing: git rev-parse --show-toplevel
Executing: git submodule add  git://git.drupal.org/project/views.git modules/views
Project views contains 2 modules: views, views_ui.
Command dispatch complete                                               [notice]
rfay@Randys-MacBook-Air:~/workspace/drupal$ cd sites/all/
rfay@Randys-MacBook-Air:~/workspace/drupal/sites/all$ git rev-parse --show-toplevel
/Users/rfay/Documents/workspace/drupal
rfay@Randys-MacBook-Air:~/workspace/drupal/sites/all$ git rev-parse --git-dir
/Users/rfay/Documents/workspace/drupal/.git
rfay@Randys-MacBook-Air:~/workspace/drupal/sites/all$ git --version
git version 1.7.10.4

Ah finally, some closure on this! It turns out to have to do with symlinked parend directories, probably not MacOS. Here's the exact same thing in the same directory (but not using the symlink). Now it's in ~/Documents/workspace:

rfay@Randys-MacBook-Air:~/Documents/workspace/drupal$ drush dl -v --package-handler=git_drupalorg views
Initialized Drupal 7.15-dev root directory at                           [notice]
/Users/rfay/Documents/workspace/drupal
Loading version_control engine.                                         [notice]
Loading package_handler engine.                                         [notice]
Loading release_info engine.                                            [notice]
Downloading release history from                                        [notice]
http://updates.drupal.org/release-history/views/7.x
Executing: which wget
Executing: wget -q --timeout=30 -O /private/tmp/download_file1YJTMZ http://updates.drupal.org/release-history/views/7.x
Downloading project views to /tmp/drush_tmp_1339016979_4fcfc71335dfa    [notice]
...
Executing: git clone  --verbose --progress git://git.drupal.org/project/views.git /tmp/drush_tmp_1339016979_4fcfc71335dfa/views
Executing: git tag -l 7.x-3.3
Executing: git checkout  7.x-3.3
Using destination directory                                             [notice]
/Users/rfay/Documents/workspace/drupal/sites/all/modules
Executing: svn info /Users/rfay/Documents/workspace/drupal/sites/all/modules
Executing: bzr root /Users/rfay/Documents/workspace/drupal/sites/all/modules
Project views (7.x-3.3) downloaded to                                [success]
/Users/rfay/Documents/workspace/drupal/sites/all/modules/views.
Executing: git rev-parse --show-toplevel
Executing: git submodule add  git://git.drupal.org/project/views.git sites/all/modules/views
Project views contains 2 modules: views, views_ui.
Command dispatch complete                                               [notice]
rfay@Randys-MacBook-Air:~/Documents/workspace/drupal$ git status
# On branch 7.x
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	new file:   .gitmodules
#	new file:   sites/all/modules/views
#

Retitling to indicate the actual source of the problem.

Updated the issue summary.

rfay’s picture

Issue summary: View changes

Simply minor finesse.

jonhattan’s picture

Status: Active » Needs review
FileSize
932 bytes

Replicated in Debian:

cd /tmp
mkdir real-workspace
ln -s real-workspace symlinked-workspace
cd symlinked-workspace
drush dl --package-handler=git_drupalorg drupal
cd drupal-7.14
drush dl -v --package-handler=git_drupalorg --gitsubmodule token

log:

Project token (7.x-1.1) downloaded to /tmp/symlinked-workspace/drupal-7.14/sites/all/modules/token.                                                                                                     [success]
Executing: git rev-parse --show-toplevel
Executing: git submodule add  git://git.drupal.org/project/token.git 7.14/sites/all/modules/token

Note that the directory it tries to add as submodule depends on the strlen of the directory and the symlink.

# git status
# Not currently on any branch.
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	new file:   .gitmodules
#	new file:   7.14/sites/all/modules/token
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	sites/all/modules/token/

Attached a simple fix.

rfay’s picture

Status: Needs review » Reviewed & tested by the community

Works for me.

Thanks so much for hanging with me on this long-term issue. Sure glad you mentioned the symlink back in #19.

jonhattan’s picture

I'm reluctant to commit this fix by now. We need tests to cover several scenarios with symlinks, and perhaps the fix here is other: to use realpath'ed paths since early stages.

* #1539760: Drupal root getting miscalculated when sites dir is a symlink
* #721064: Remove dangerous getcwd() and use drush_cwd() instead

I'll write some tests in short.

moshe weitzman’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs tests
Lukas von Blarer’s picture

Lukas von Blarer’s picture

Works perfectly! Thank you sooo much!!! ;)

moshe weitzman’s picture

Assigned: Unassigned » jonhattan
greg.1.anderson’s picture

Version: » 8.x-6.x-dev
Status: Needs work » Closed (won't fix)
Issue tags: +needs migration

This issue was marked closed (won't fix) because Drush has moved to Github.

If desired, you may copy this bug to our Github project and then post a link here to the new issue. Please also change the status of this issue to closed (duplicate).

Please ask support questions on Drupal Answers.

greg.1.anderson’s picture

Issue summary: View changes

Updated to mention a symlink problem, not a MacOS problem.