Allow use of subtrees or individual file from archives instead of the whole archive

Could we have a test for this? See README.txt for examples

I think it would also be great if it could grab an individual file from the archive - this is something I'd be willing to code myself, just not at this very moment.

this is required by #840540: Specify a patch from local file system

this is hardly a bug

Just subscribing since this effects me too. I don't have too much time to work on this right now, but may look at it next month... good luck if anyone can get to it sooner.

Just checking back in about a month later to say I still don't have much time to look at this, but it is still very high on my wishlist.

subscribing.

Also, this should happen at a higher-up level than the download_file -- it sholud be applicable to any download method.

I'm thinking of revising drush make's workflow, so it does the following steps:
- For every project, download its release information from d.o, and directly after, download the file itself, and verify the md5 checksum
- Recurse, doing the above again
- Apply patches (splitting this out is the only real way to fix #634458: Cannot apply a patch up the recursion tree)
- Move files into place, moving files, subtrees and entire projects

subscribe. (I was just about to file the same feature request!)

This would not only be useful for archives, but also for repository checkouts - and both for projects (modules/themes) and for libraries.
I have two real-life use-cases right now:

• a custom-made Drupal theme located in a sub-folder of a website project's git repository
• the HTML Purifier library (needed by the d.o module of the same name), which can be downloaded through a tar.gz archive, but only the library/ sub-folder inside this archive is the actual library which needs to be copied to the appropriate location in the Drupal tree, not the whole archive content (which includes several additional sub-folders like art/,benchmarks/, docs/...)

It would be great to have this functionality in Drush Make.

An added bonus would be if each archive/repository would only be actually downloaded once, even if multiple sub-folders from it are referenced in the makefile.

Subcribe, I have the same use case as #9.

I have a git repository on github with all of the custom modules for a particular site. At the moment I'm not sure how I can use drush make to fetch each of those modules.

Cheers,

Subscribe as #1020950: symlinking or alternative workflow marked duplicate. I'd like to have a location separate from Drupal site, eg a repository, and have Drush symlink from /modules and /themes to subtrees in that location.

I'm seeing this issue as different so far. This issue seems to not care about what happens to the download as long as the different subtrees bits are in place at end. But waiting to see what dmitrig01 might cook up along with the giant chicken eggs.

WIN :D

Subscribing.

Just to clarify #9: for HTMLPurifier, we need to get the content of the library subdir into for example sites/all/libraries/htmlpurifier, not the subdir itself.

@Frando: after looking at the patch I think maybe it can be made simpler if we use the subtree option when building up the $directory path under $tmp_path. Would you take a look at this possibility?

Ah, and your patch contains an unrelated change to drush_make_download_git in the last hunk.

Thanks,
Antonio

In the attached file there are several patches to solve this issue in drush_make-6.x-3.x branch.
I hope this could be merged after some review.

An brief overview of the patches:

1. cleanup _drush_make_download_file_move()
2. cleanup the use of '/__unzip__' suffix
3. Remove unused $main_directory variables
4. Implement the [download][subtree] option

Detailed commit messages are in the file.

The whole patchset can be applied with git am to get the separate commits preserving the commits messages and authorship.

mmh, the commit message in the last patch from #15 is wrong about the location of htmlpurifier library, but anyways let me know what do you think of the approach and then I can send a fixed patchset later.

subscribing

Link to a use-case involving management of customized starterkit subthemes (Omega, Zen, etc).
#1074748-2: Get subtree with git

Hi,

if you can suggest a use case other than amfphp (I cannot test it and htmlpurify does not need this as I originally thought), I will retouch the commit message and repost the patchset along with #1074748: Get subtree with git, so hopefully we can get this merged and close the issue.

Thanks,
Antonio

Opened an issue which would make this one easier to solve, look at #1212834: Decouple "download" and "copy" operations..

I just came across this limitation of drush_make with amfphp aswell, frustrating i cant build automatically out of the box. I am sure this will help other projects.

@imp7, have you tried the patchset? If you do, can you paste here the makefile snippet relative to amfphp? So I can use that one in the documentation.

Thanks,
Antonio

Stubmbled accross this issue trying to build up the corrct directory/file structure for the wurfl library. So far I have this:

; Grab the wurfl library.
libraries[wurfl][download][type] = "get"
libraries[wurfl][download][url] = "http://downloads.sourceforge.net/project/wurfl/WURFL%20PHP/1.3/wurfl-php-1.3.1.tar.gz?r=&ts=1318333899&use_mirror=dfn"

; Get the latest web_browsers_patch.xml file.
libraries[wurfl_web_browsers][download][type] = "file"
libraries[wurfl_web_browsers][download][url] = "http://wurfl.sourceforge.net/web_browsers_patch.xml"
libraries[wurfl_web_browsers][download][filename] = "wurfl_web_browsers.xml"
libraries[wurfl_web_browsers][destination] = "libraries/wurfl"
libraries[wurfl_web_browsers][directory_name] = "resources"
libraries[wurfl_web_browsers][overwrite] = TRUE

; Create a cache directory.
; FIXME: This needs to be made writable by the web user.
libraries[wurfl_cache][destination] = "libraries/wurfl"
libraries[wurfl_cache][directory_name] = "cache"

; FIXME: Need to copy examples/resources/wurfl-config.xml to resources/wurfl-config.xml

which downloads the wurfl library, grabs the web_brosers_xml.patch file and creates a cache directory. One of the last things that is needed is to copy the examples/resources/wurfl-config.xml file out to the resources subdirectory. It sounds like this patch would allow for that. Going to test now.

Attached is the patch in #15 updated to apply to the latest code in git (I also needed the patch in #1248686: Error: Unsupported operand types in drush_make.drush.inc ). However, it's not working for me. With the following added to my drush make file:

; Copy examples/resources/wurfl-config.xml to resources/wurfl-config.xml
libraries[wurfl_config][download][type] = "file"
libraries[wurfl_config][download][url] = "http://downloads.sourceforge.net/project/wurfl/WURFL%20PHP/1.3/wurfl-php-1.3.1.tar.gz?r=&ts=1318333899&use_mirror=dfn"
libraries[wurfl_config][download][subtree] = "examples"
libraries[wurfl_config][destination] = "libraries/wurfl2"

I simply end up with an empty directory at libraries/wurfl2. I've tried with libraries[wurfl_config][download][subtree] = "examples" and libraries[wurfl_config][download][subtree] = "examples/resources" (which is what I actually want. Both result in an empty directory.

Oh, and the updated patch.

Attached patch reworks things slightly so that it works - I've been able to use this with amfphp as well as the following code for wurfl:

; Grab the wurfl library.
libraries[wurfl][download][type] = "get"
libraries[wurfl][download][url] = "http://downloads.sourceforge.net/project/wurfl/WURFL%20PHP/1.3/wurfl-php-1.3.1.tar.gz?r=&ts=1318343733&use_mirror=switch"

; Copy examples/resources/wurfl-config.xml to resources/wurfl-config.xml
libraries[wurfl_config][download][type] = "get"
libraries[wurfl_config][download][url] = "http://downloads.sourceforge.net/project/wurfl/WURFL%20PHP/1.3/wurfl-php-1.3.1.tar.gz?r=&ts=1318343733&use_mirror=switch"
libraries[wurfl_config][download][subtree] = "examples/resources"
libraries[wurfl_config][destination] = "libraries/wurfl"
libraries[wurfl_config][directory_name] = "resources"

; Get the latest web_browsers_patch.xml file.
libraries[wurfl_web_browsers][download][type] = "file"
libraries[wurfl_web_browsers][download][url] = "http://wurfl.sourceforge.net/web_browsers_patch.xml"
libraries[wurfl_web_browsers][destination] = "libraries/wurfl"
libraries[wurfl_web_browsers][directory_name] = "resources"
libraries[wurfl_web_browsers][overwrite] = TRUE

The problem was that the code for grabbing the subdirectory wasn't properly accounting for the name of the downloaded and extracted directory (which can be different from the name of the library). This version of the patch moves that logic into the _drush_make_download_file_move function, which also reduces on duplication.

This applies to the latest in git, however you will need the patch from #1248686: Error: Unsupported operand types in drush_make.drush.inc too.

@mrfelton thanks, I'll take a look at your updates as soon as I have some more time.

Ciao,
Antonio

[ Powered by #1115636: Issue Macros and Templates - Drush Make]

Drush make is being merged into drush core (discussed in issue:#1310130: Put drush make in drush core)
This means that the issue queue is also moving. The drush queue has a component 'Make' especially for drush_make.

More information will be posted on the drush_make and drush project pages.

This will need to be re-rolled.

Should also have at least one test.

My attempt at a reroll. It was pretty painful; had to reapply a lot of stuff by hand. It most certainly doesn't work; in particular, I can't quite follow what _make_download_file_move() is trying to do either before or after the patch (boo for inadequately-commented code).

This would help with fully-packaged distributions on d.o. See #1481158: how to deal with problem third party libraries?. #1482634: Specify what directory to extract in downloaded libraries was just marked as duplicate, but has a little bit of useful discussion if anyone's interested. ;)

In short, +1 ;)

Re-roll of patch in 26, this is mostly the same except I attempted to keep the file move logic equivalent to what is currently in drush.

FWIW some library stubs that I tested with;

libraries[fullcalendar][download][type] = get
libraries[fullcalendar][download][url] = http://arshaw.com/fullcalendar/downloads/fullcalendar-1.5.3.zip
libraries[fullcalendar][download][sha1] = c7219b1ddd2b11ccdbf83ebd116872affbc45d7a
libraries[fullcalendar][download][subtree] = fullcalendar

libraries[nivo-slider][download][type] = get
libraries[nivo-slider][download][url] = https://github.com/downloads/gilbitron/Nivo-Slider/nivo-slider2.7.1.zip
libraries[nivo-slider][download][sha1] = bd8e14b82f5b9c6f533a4e1aa26a790cd66c3cb9
libraries[nivo-slider][download][subtree] = nivo-slider

With the patch applied the nested directories fullcalendar and nivo-slider respectively were pulled to the target directory with nothing else.

@jgraham: The README.txt part seems to have gone missing.

And I agree that this deserves a testcase, see patch.

I committed #34 with one small variable name change. Glad to have this in. Thanks everybody!