how to deal with problem third party libraries? [#1481158]

First thanks to all the hard work surrounding packaging on drupal.org its awesome to see fully packaged distributions available. :)

We have a couple libraries for http://drupal.org/project/julio that suffer from some packaging and release engineering issues eg. no version tags on the external libraries
#1472476: FullCalendar library packaging/build issues
#1465816: nivo slider js library packaging/build issues

As such our distribution has some issues until a user adjusts the built library paths.

I have attempted to follow up in the respective issue queues with the external library maintainers, but have received no response. In the case of colorbox there has been no response in over a year.

Personally I would prefer to use tagged releases from the original maintainer. This does not appear to be a viable option, at least in a timely fashion. Unless the drupal.org packaging script un-mangles bad packages, which I think is an awful unmaintainable idea, I don't see much of a way around this with the exception of forking where possible. I'm not excited about this approach, but it seems like the only reasonable way forward.

For example I have forked and tagged for colorbox:
https://github.com/jgraham909/colorbox
related whitelist ticket here;
#1403884: Add ColorBox to packaging whitelist

and for nivo-slider:
https://github.com/jgraham909/Nivo-Slider
related whitelist ticket here;
#1404316: Add jquery Nivo Slider

Does it make sense to add these forks as whitelists?

Unfortunately I don't see these libraries being the only ones that we will have to adjust for.

Comments

Comment #1

geerlingguy CreditAttribution: geerlingguy commented 14 March 2012 at 19:08

For FullCalendar, please see and subscribe to/support #1472436: Allow "Path to FullCalendar" Setting to use Libraries API.

Basically, the best way to fix some of these issues is to get the actual modules (like FullCalendar and whatever uses Nivo Slider) to use the Libraries API appropriately. I asked about the ability to have an install profile automatically move a downloaded library to another location in the Drupal.org drush queue (see #1429500: How to place downloaded library in /sites/all/libraries/[directory]?), but that's probably not going to happen.

Comment #2

redndahead CreditAttribution: redndahead commented 14 March 2012 at 19:35

The problem is even with libraries support it still doesn't work. The tar files extract to something like full_calendar-1.5.2/fullcalendar/fullcalendar.js There is no way around this.

The only way I can see us being able to work around these issues is if we put something into drush make that can mess with the contents of the tar/zip file during the extraction. Maybe a command line option that gets passed to the extractor. Probably a better issue for the drush queue.

Comment #3

jgraham CreditAttribution: jgraham commented 14 March 2012 at 19:35

While the use of libraries API is certainly relevant to this issue I think the packaging of external libraries is also an issue.

I wanted this issue to focus on the external libraries packaging issues. Sorry if the linked issues confused this I just wanted to provide relevant information where possible. As such fullcalendar is perhaps a poor example as it does not currently have libraries API integration.

Focusing on nivo slider might be better to illustrate the issue at hand;
#1465816: nivo slider js library packaging/build issues

This module has libraries API integration. However the external library packaging is inconsistent. In 2.6 it was packaged such that there was no nested directories so that when you unzipped it you received a non-nested directory structure. In subsequent releases the nivo-slider directory was nested such that you get 'nivo-slider/nivo-slider' rather than 'nivo-slider'. Correct me if I'm wrong but libraries API doesn't solve this issue of my library is at X but the relevant files may be at 'X/nivo-slider/FILE' or 'X/nivo-slider/nivo-slider/FILE' depending on the packaging flavor of the week.

Like I mentioned in the initial post this is the issue at hand.

Additionally even if fullcalendar did receive libraries API integration this issue would still be present for distributions as according to tim.plunkett in #1472436: Allow "Path to FullCalendar" Setting to use Libraries API it sounds like he has followed up with the library maintainer.

plugin zip contains an entire copy of jquery and jquery ui, as well as possibly insecure demos. I've requested that a unbundled version is provided, but I haven't heard back yet.

It remains if Tim would accept a patch that set the library path under the assumption of the nested directory structure implying that the extra jquery and demos dirs would be present.

Fundamentally this is an external libraries packaging issue. I would like to consider libraries API integration as a separate (but related) issue. While that solves some of the issues it does not (unless I'm mistaken) handle the situation where specific files may be at arbitrary depths in the library directory.

Comment #4

bonobo CreditAttribution: bonobo commented 14 March 2012 at 19:56

One of the causes of this issue has to do with how external libraries are maintained.

There have been cases where the directory structure of the library changes between versions, or contains extraneous code (demos, etc) that aren't needed for the 3rd party library to be used.

Libraries API integration is a good best practice, but even if every module implemented the Libraries API (and as is clear from FullCalendar, they don't) this wouldn't solve the issue of 3rd party libraries with directory structures that change between versions. Libraries API integration also wouldn't solve the issue of cruft code (demo code, etc).

A decent middle ground would be convincing the 3rd party library maintainers to provide an unbundled version, but that "solution" requires 3rd party library maintainers to provide the unbundled version.

Possibly, the d.o packaging script could (theoretically) account for this, but (IMO) this is not a realistic feature request, and would get unmaintainable quickly, as the problems caused by 3rd party libraries will occur arbitrarily and unpredictably over time.

Creating forked versions (that get whitelisted) of the 3rd party libraries is also a less than ideal option, but given the other options here it seems like the most acceptable stopgap.

Any other ideas on ways to address this?

Comment #5

redndahead CreditAttribution: redndahead commented 14 March 2012 at 19:59

I have created an issue in drush for this. #1482634: Specify what directory to extract in downloaded libraries

Comment #6

geerlingguy CreditAttribution: geerlingguy commented 14 March 2012 at 22:51

Status:

Active

» Closed (works as designed)

A decent middle ground would be convincing the 3rd party library maintainers to provide an unbundled version, but that "solution" requires 3rd party library maintainers to provide the unbundled version.

It's hard enough getting Drupal module maintainers to change things to help us out... I'd definitely not bank on getting 3rd party maintainers to change things to help us (or anyone besides themselves) out. Though it's always nice to ask :)

Possibly, the d.o packaging script could (theoretically) account for this, but (IMO) this is not a realistic feature request, and would get unmaintainable quickly, as the problems caused by 3rd party libraries will occur arbitrarily and unpredictably over time.

Definitely not going to be an automated process. But I could foresee something like #1482634: Specify what directory to extract in downloaded libraries happening at some point. Just don't hold your breath, and help code a solution if you can :)

Creating forked versions (that get whitelisted) of the 3rd party libraries is also a less than ideal option, but given the other options here it seems like the most acceptable stopgap.

Since it's open source, feel free to do it... but know that you'd be less likely to get a whitelist node made for arbitrary forks of popular 3rd party libraries that don't seem to be affiliated with the original source. The best stopgap right now might be to either set the path (in FullCalendar's case) explicitly for the version of the library you're checking out using variable_set() in your install profile's setup code, or to instruct your install profile users to change the name of the directory. Definitely not ideal, but until something better...

I'm marking this closed, just because I think we've reached whatever conclusions this thread can come to.

Comment #7

dww

we/he/they

CreditAttribution: dww commented 14 March 2012 at 23:19

Note: #919224: Allow use of subtrees or individual file from archives instead of the whole archive seems to have support from the Drush maintainers. I'd be happy to start supporting that in drupalorg_drush once it's available in Drush core. That seems like the best solution here. I'd be -1 to forking libraries just to deal with inconsistent packaging and whitelisting those forks.

CreditAttribution: dww commented 14 March 2012 at 23:59

I opened #1482888: Support 'subtree' download attribute for libraries to track support for 'subtree' in drupalorg_drush. Interested parties are invited to follow that. ;)

Cheers,
-Derek