Overview
Some modules require you to copy not the whole unzipped archive, but just a subtree from within the archive to the module's directory.
Without the attached patch, it is for example not possible to completely assemble a site that uses the amfphp module.
The attached patch adds a subtree
option to POST and GET downloads. If the download is an archive and the subtree option is specified, only the "subtree" path from within the archive will be copied to the target destination.
Example
The amfphp module requires you to download the amfphp library. The directory structure of the unzipped amfphp library looks like this:
./amfphp-1.9.beta.20080120/
./amfphp-1.9.beta.20080120/amfphp/
./amfphp-1.9.beta.20080120/amfphp/gateway.php
./amfphp-1.9.beta.20080120/amfphp/json.php
./amfphp-1.9.beta.20080120/amfphp/...
Now, the amfphp Drupal module requires you to copy the "amfphp" subdirectory to the module's path, so that you get
sites/all/modules/amfphp/amfphp/gateway.php
This is currently not possible with drush_make. If setting directory_name
to amfphp
, it is still nested one level too deep, so you get sites/all/modules/contrib/amfphp/amfphp/amfphp/gateway.php
So with this patch and
libraries[amfphp][download][type] = "get"
libraries[amfphp][download][url] = "http://downloads.sourceforge.net/project/amfphp/amfphp/amfphp%201.9%20beta2/amfphp-1.9.beta.20080120.zip?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Famfphp%2Ffiles%2F&ts=1285152291&use_mirror=ignum"
libraries[amfphp][download][subtree] = "amfphp"
libraries[amfphp][directory_name] = "amfphp"
libraries[amfphp][destination] = "modules/contrib/amfphp"
I succesfully get the required sites/all/modules/contrib/amfphp/amfphp/gateway.php
structure.
This patch is sponsored by Xarxa Media GmbH.
Comments
Comment #1
dmitrig01 CreditAttribution: dmitrig01 commentedCould we have a test for this? See README.txt for examples
Comment #2
dmitrig01 CreditAttribution: dmitrig01 commentedI think it would also be great if it could grab an individual file from the archive - this is something I'd be willing to code myself, just not at this very moment.
Comment #3
dmitrig01 CreditAttribution: dmitrig01 commentedthis is required by #840540: Specify a patch from local file system
Comment #4
dmitrig01 CreditAttribution: dmitrig01 commentedthis is hardly a bug
Comment #5
nicholas.alipaz CreditAttribution: nicholas.alipaz commentedJust subscribing since this effects me too. I don't have too much time to work on this right now, but may look at it next month... good luck if anyone can get to it sooner.
Comment #6
nicholas.alipaz CreditAttribution: nicholas.alipaz commentedJust checking back in about a month later to say I still don't have much time to look at this, but it is still very high on my wishlist.
Comment #7
bwood CreditAttribution: bwood commentedsubscribing.
Comment #8
dmitrig01 CreditAttribution: dmitrig01 commentedAlso, this should happen at a higher-up level than the download_file -- it sholud be applicable to any download method.
I'm thinking of revising drush make's workflow, so it does the following steps:
- For every project, download its release information from d.o, and directly after, download the file itself, and verify the md5 checksum
- Recurse, doing the above again
- Apply patches (splitting this out is the only real way to fix #634458: Cannot apply a patch up the recursion tree)
- Move files into place, moving files, subtrees and entire projects
Comment #9
smls CreditAttribution: smls commentedsubscribe. (I was just about to file the same feature request!)
This would not only be useful for archives, but also for repository checkouts - and both for projects (modules/themes) and for libraries.
I have two real-life use-cases right now:
library/
sub-folder inside this archive is the actual library which needs to be copied to the appropriate location in the Drupal tree, not the whole archive content (which includes several additional sub-folders likeart/
,benchmarks/
,docs/
...)It would be great to have this functionality in Drush Make.
An added bonus would be if each archive/repository would only be actually downloaded once, even if multiple sub-folders from it are referenced in the makefile.
Comment #10
aidanlis CreditAttribution: aidanlis commentedSubcribe, I have the same use case as #9.
I have a git repository on github with all of the custom modules for a particular site. At the moment I'm not sure how I can use drush make to fetch each of those modules.
Cheers,
Comment #11
simeSubscribe as #1020950: symlinking or alternative workflow marked duplicate. I'd like to have a location separate from Drupal site, eg a repository, and have Drush symlink from /modules and /themes to subtrees in that location.
I'm seeing this issue as different so far. This issue seems to not care about what happens to the download as long as the different subtrees bits are in place at end. But waiting to see what dmitrig01 might cook up along with the giant chicken eggs.
Comment #12
dmitrig01 CreditAttribution: dmitrig01 commentedWIN :D
Comment #13
ao2 CreditAttribution: ao2 commentedSubscribing.
Just to clarify #9: for HTMLPurifier, we need to get the content of the
library
subdir into for examplesites/all/libraries/htmlpurifier
, not the subdir itself.Comment #14
ao2 CreditAttribution: ao2 commented@Frando: after looking at the patch I think maybe it can be made simpler if we use the
subtree
option when building up the$directory
path under$tmp_path
. Would you take a look at this possibility?Ah, and your patch contains an unrelated change to
drush_make_download_git
in the last hunk.Thanks,
Antonio
Comment #15
ao2 CreditAttribution: ao2 commentedIn the attached file there are several patches to solve this issue in drush_make-6.x-3.x branch.
I hope this could be merged after some review.
An brief overview of the patches:
Detailed commit messages are in the file.
The whole patchset can be applied with
git am
to get the separate commits preserving the commits messages and authorship.Comment #16
ao2 CreditAttribution: ao2 commentedmmh, the commit message in the last patch from #15 is wrong about the location of htmlpurifier library, but anyways let me know what do you think of the approach and then I can send a fixed patchset later.
Comment #17
izmeez CreditAttribution: izmeez commentedsubscribing
Comment #18
patcon CreditAttribution: patcon commentedLink to a use-case involving management of customized starterkit subthemes (Omega, Zen, etc).
#1074748-2: Get subtree with git
Comment #19
ao2 CreditAttribution: ao2 commentedHi,
if you can suggest a use case other than amfphp (I cannot test it and htmlpurify does not need this as I originally thought), I will retouch the commit message and repost the patchset along with #1074748: Get subtree with git, so hopefully we can get this merged and close the issue.
Thanks,
Antonio
Comment #20
ao2 CreditAttribution: ao2 commentedOpened an issue which would make this one easier to solve, look at #1212834: Decouple "download" and "copy" operations..
Comment #21
imp7 CreditAttribution: imp7 commentedI just came across this limitation of drush_make with amfphp aswell, frustrating i cant build automatically out of the box. I am sure this will help other projects.
Comment #22
ao2 CreditAttribution: ao2 commented@imp7, have you tried the patchset? If you do, can you paste here the makefile snippet relative to amfphp? So I can use that one in the documentation.
Thanks,
Antonio
Comment #23
mrfelton CreditAttribution: mrfelton commentedStubmbled accross this issue trying to build up the corrct directory/file structure for the wurfl library. So far I have this:
which downloads the wurfl library, grabs the web_brosers_xml.patch file and creates a cache directory. One of the last things that is needed is to copy the examples/resources/wurfl-config.xml file out to the resources subdirectory. It sounds like this patch would allow for that. Going to test now.
Comment #24
mrfelton CreditAttribution: mrfelton commentedAttached is the patch in #15 updated to apply to the latest code in git (I also needed the patch in #1248686: Error: Unsupported operand types in drush_make.drush.inc ). However, it's not working for me. With the following added to my drush make file:
I simply end up with an empty directory at libraries/wurfl2. I've tried with
libraries[wurfl_config][download][subtree] = "examples"
andlibraries[wurfl_config][download][subtree] = "examples/resources"
(which is what I actually want. Both result in an empty directory.Comment #25
mrfelton CreditAttribution: mrfelton commentedOh, and the updated patch.
Comment #26
mrfelton CreditAttribution: mrfelton commentedAttached patch reworks things slightly so that it works - I've been able to use this with amfphp as well as the following code for wurfl:
The problem was that the code for grabbing the subdirectory wasn't properly accounting for the name of the downloaded and extracted directory (which can be different from the name of the library). This version of the patch moves that logic into the
_drush_make_download_file_move
function, which also reduces on duplication.This applies to the latest in git, however you will need the patch from #1248686: Error: Unsupported operand types in drush_make.drush.inc too.
Comment #27
ao2 CreditAttribution: ao2 commented@mrfelton thanks, I'll take a look at your updates as soon as I have some more time.
Ciao,
Antonio
Comment #28
helmo CreditAttribution: helmo commented[ Powered by #1115636: Issue Macros and Templates - Drush Make]
Drush make is being merged into drush core (discussed in issue:#1310130: Put drush make in drush core)
This means that the issue queue is also moving. The drush queue has a component 'Make' especially for drush_make.
More information will be posted on the drush_make and drush project pages.
Comment #29
jhedstromThis will need to be re-rolled.
Comment #30
jhedstromShould also have at least one test.
Comment #31
Garrett Albright CreditAttribution: Garrett Albright commentedMy attempt at a reroll. It was pretty painful; had to reapply a lot of stuff by hand. It most certainly doesn't work; in particular, I can't quite follow what _make_download_file_move() is trying to do either before or after the patch (boo for inadequately-commented code).
Comment #32
dwwThis would help with fully-packaged distributions on d.o. See #1481158: how to deal with problem third party libraries?. #1482634: Specify what directory to extract in downloaded libraries was just marked as duplicate, but has a little bit of useful discussion if anyone's interested. ;)
In short, +1 ;)
Comment #33
jgraham CreditAttribution: jgraham commentedRe-roll of patch in 26, this is mostly the same except I attempted to keep the file move logic equivalent to what is currently in drush.
FWIW some library stubs that I tested with;
With the patch applied the nested directories fullcalendar and nivo-slider respectively were pulled to the target directory with nothing else.
Comment #34
helmo CreditAttribution: helmo commented@jgraham: The README.txt part seems to have gone missing.
And I agree that this deserves a testcase, see patch.
Comment #35
jhedstromI committed #34 with one small variable name change. Glad to have this in. Thanks everybody!