Problem/Motivation

This module includes multiple updated versions of core provided JavaScript libraries (which are not altered). They are actively maintained and easy to find in other locations on the internet.

Ultimately it does not adhear to [the current] Policy for 3rd party libraries on Drupal.org.

This is currently blocked on a higher level issue discussion: #1856762: Revisit and Redefine Drupal's Policy on hosting 3rd party files and/or files that is not strictly GPL.

Proposed resolution

TBD

Remaining tasks

TBD

User interface changes

TBD

API changes

TBD

Original report by @kreynen

This module includes 3 versions of JQuery and JQuery UI that aren't altered, are actively maintained, and easy to find in other locations on the internet.

http://drupalcode.org/project/jquery_update.git/tree/refs/heads/7.x-2.x:...
http://drupalcode.org/project/jquery_update.git/tree/refs/heads/7.x-2.x:...

This is a clear violation for the current Policy for 3rd party libraries on Drupal.org.

My goal is not to see this module changed, but to illustrate popular modules that violate the current policy in an effort to get the policy revised so that making installs easier for users is a legitimate reason for including a 3rd party library in a module. This is related to...

#1856438: Provide project bundling exception for custom build of CKEditor

Files: 

Comments

kreynen’s picture

gisle’s picture

Issue summary:View changes
Issue tags:+licensingpolicy

Added tag.

markcarver’s picture

Title:Violation of policy for 3rd party libraries on Drupal.org?» [policy][jquery_update] 3rd party libraries
Category:Bug report» Task
Priority:Major» Normal
Issue summary:View changes
Status:Active» Postponed
Parent issue:» #1856762: Revisit and Redefine Drupal's Policy on hosting 3rd party files and/or files that is not strictly GPL

I am marking this postponed on the related issue as we really cannot do much until an official stance has been decided (either way).

FWIW, I believe (not entirely sure) the reasoning behind why jquery_update includes the jQuery and jQuery UI versions is because their [predecessing] code is already bundled with core (thus already vetted?).

gisle’s picture

For the record: There is no such thing as "already vetted" in the current policy.

The current policy says that you need an explicit exception granted to include third party materials in a distro (the exception must be applied to the distro, not the material), and there is no record of this project ever being granted one. Further, since the libraries are easily available from the jQuery website, and to download and install them only requires a separate installation step, an exception should not be granted under the current policy - see https://www.drupal.org/node/422996 and https://www.drupal.org/node/66113 where this is explained.

However, I agree with postponing this until #1856762: Revisit and Redefine Drupal's Policy on hosting 3rd party files and/or files that is not strictly GPL is sorted out.

To anyone wondering what is going on: We try to tag every issue with a possible license violation with the tag licensingpolicy to get an overview over how the current policy is enforced (or, not to beat around the bush: not enforced).

Please note: No immediate enforcement is planned (as should be obvious: nobody has tried to enforce this since it was opened in 2012). We first need to get an overview of the current situation. Then we need to decide what to do about it.

markcarver’s picture

For the record: There is no such thing as "already vetted" in the current policy.

Which is why I added a question mark at the end. I know there isn't such a thing in the policy. However, this specific module is rather unique in that it is replacing existing core libraries; hence probably why it includes them. Not saying I agree either way TBH and have very little interest in starting a comment war. Just offering a possible reason is all.

kreynen’s picture

This was actually vetted (as much as anything was), but rejected back in 2007 when developers used the development mailing list to discuss this type of issue. At least some of the maintainers of JQuery Update have been knowingly violating the policy since that discussion. The issue with JQuery Update came up when I was trying to get an official exception to the policy to commit WYSIWYG related javascript so that the TinyMCE would load something for users out "of the box" that they could then replace with a full blown version of TinyMCE they downloaded from Moxiecode later. This was before the Library module when most module maintainers instructed users to add the library to the module dir. TinyMCE was the most popular WYSIWYG module on Drupal.org at the time and the issue queue was full of "I don't see TinyMCE after enabling the module" and "I updated and TinyMCE is now gone". The code I was asking to commit was originally licensed as LGPLv2, but even after the team at Moxiecode agreed to allow it to be re-licensed as GPLv2 or later the answer was still no

A number of reasons were given including that the additional code was going to make CVS unmaintainable. That was obviously an excuse from someone who just didn't want this code in the Drupal repo.

I was told in no uncertain terms that if I committed the code, my CVS privileges would be revoked. As a result, I stopped updating TinyMCE.

You can read my recap of that discussion or the entire thread , but @jjeff's input towards the end of the discussion still rings true today...

What exactly does "foreign code" mean?!? If we write it from scratch,
it's not foreign, right? What if we copy some of the code from
another GPL project? What if we only slightly modify another GPL
project, perhaps just enough to make it work with Drupal? Well then
it can't be downloaded from the original source. Are you telling me
that this can't be included in the contrib repository? This sort of
modification is the entire spirit of the GPL! It's about freedom and
growth. By not allowing it into the repository, we are actually
diverging from the spirit of the GPL.

How modified does a project need to be in order to be considered
"native" Drupal code? I would submit that it DOES NOT need to be
modified at all in order to be considered Drupal code. In the case of
Drupal 5's JQuery, Drupal includes JQuery version 1.0.3 which is now
fully deprecated in the JQuery community. Many of the JQuery plug-in
maintainers do not continue to distribute plug-ins that function with
1.0.x and so there is no way to link to these plugins and ask users
to download them elsewhere. So in the form of the old versions of
these plugins, we have unmodified code, that for all intents and
purposes, only works with Drupal and can only be found in the Drupal
code repository.

We have a great resource with the Drupal CVS repository. I think that
we should be as accepting as possible in order to foster creativity
and growth. The GPL and Source Forge both provide examples as to the
spirit of this openness. I think we should take notes. The larger
that Drupal becomes, the more "gray area" code will emerge. I believe
the rules for code inclusion should be very very simple: All code
must be GPLed.

Don't get me wrong, I would be ALL FOR allowing LGPLed code into the
repository, but I also understand that the line needs to be drawn
somewhere. But I think when people talk about *not* allowing at-one-
time-external GPLed code into the Drupal repository, they're simply
on crack.

Unfortunately the "crackheads" got their way and the policy went unchanged and inconsistently enforced for 5 more years until @Jen Lamption pushed for a revision to the policy again in 2012 based on work @quicksketch was doing with CKEditor. At that point it was obvious Drupal.org had gotten out of sync with the approach other projects were taking, but no changes were made after weeks of discussion... so here we are again.

Since https://www.drupal.org/project/jqmulti is able to provide similar functionality to jquery_update without violating the policy, it's obviously possible to do that. There is no technical reason for giving JQuery Update an exception. It doesn't meet any of the current criteria.

Back in 2012, JQuery forced Drupal to update our approach to licensing when that team stopped dual licensing JQuery as MIT and GPLv2. If Drupal.org didn't adapt our policy to allow MIT and other lax licenses, we couldn't have committed newer, MIT-only licensed version of JQuery. We would have had to start asking users to download JQuery separately and place it in the correct directory before they even installed Drupal.

Based on this history, it seems logical to use JQuery Update as a test case to form new policies around. If "because it's easier to install the module" is a good enough reason to grant JQuery Update an exception to commit copies of JQuery to be distributed with the module, then it should be good enough for any other contrib module to get an exception.

It would obviously be better if modules could get the libraries they depend on out of their module directory and into sites/all/libraries using some yet to be defined .make/post enable/packaging process, but that's really separate issue from whether the code is allowed in git right now.

markcarver’s picture

Title:[policy][jquery_update] 3rd party libraries» [jquery_update] 3rd party libraries

I, perhaps, unintentionally misconstrued this issue by placing [policy] in the front of the title when we should in fact be discussing it in the parent issue. @kreynen, I think your comment (which has been very helpful btw and I agree with, mostly) would be more beneficial to everyone as a whole over there. I think we should keep this issue specifically reserved for actually implementing whatever is decided over there (if anything).

kreynen’s picture

While the process and policy is still being reviewed, I'm starting to contribute patches that add the ASSETS.yml file using the proposed format to projects I believe will qualify for an "official LWG exception" to both test the format and get more eyes on it.

kreynen’s picture

So I was going to contribute a patch for the ASSETS.yml, but then I found #2534056: Remove unused and NON-MIT/GPL licensed or compatible code.

This is why even the best intending maintainers should not be allowed to commit 3rd party code without some oversight. You've now distributed code to > 400K Drupal sites that restricts them from doing anything evil... whatever that means.

kreynen’s picture

Status:Postponed» Needs review
StatusFileSize
new2.08 KB

Now that #2534056: Remove unused and NON-MIT/GPL licensed or compatible code has been removed, this is what I think the ASSETS.yml would look like. The entire ASSETS.yml structure still needs review.

markcarver’s picture

Version:7.x-2.x-dev» 7.x-3.x-dev
StatusFileSize
new21.19 KB

This is an awesome start! Since it sounds like y'all haven't yet "officially" formalized/adopted this format, let me make a quite a few suggestions in the form of this new patch. I'll then summarize in a follow-up comment.

markcarver’s picture

StatusFileSize
new20.92 KB

Forgot to change the "approval" format (after I had gone through it again).

markcarver’s picture

First let me start of by stating some of the issues I had found with the patch in #10:

  1. ASSETS.yml vs. LWG_EXCEPTION.yml
    Naming a file "assets" can easily be misunderstood and/or conflict with existing architecture/design patters (especially in future D8 projects) and has the potential to be extremely problematic. It also does not really describe what the file is about or actually used for. I think naming it to something "LWG" related here is necessary.
  2. +++ b/ASSETS.yml
    @@ -0,0 +1,54 @@
    +replace/jquery:
    +  type: code
    +  title: jQuery JavaScript Library
    +  approved: https://www.drupal.org/node/1857042
    +  attributionText: John Resig
    +  attributionUrl: http://jquery.org/
    +  license: MIT or GPL Version 2 license
    +  rights: http://jquery.org/license
    +  source: https://github.com/jquery/jquery
    +replace/jquery:
    +  type: code
    +  title: Sizzle CSS Selector Engine
    +  approved: https://www.drupal.org/node/1857042
    +  attributionText: The Dojo Foundation
    +  attributionUrl: http://sizzlejs.com/
    +  license: MIT, BSD, and GPL Licenses
    +  rights: https://github.com/jquery/sizzle/blob/master/LICENSE.txt
    +  source: https://github.com/jquery/sizzle/

    Having duplicate keys (replace/jquery) will cause issues if this is turned into an associative array. The second instance will always overwrite the first.

    Also, the Sizzle library (code) isn't explicitly used in this project (i.e. no actual sizzle.js file). The code used in jQuery is covered by jQuery's license because it has effectively transformed Sizzle's code into it's own. There's no obligation to explicitly specify Sizzle's license.

  3. +++ b/ASSETS.yml
    @@ -0,0 +1,54 @@
    +  approved: https://www.drupal.org/node/1857042

    This really should be a list of whom approved it from LWG and in which issue.

    It should be a list because there can be (and should be) multiple issues when code is re-evaluated anytime the files have been altered.

  4. +++ b/ASSETS.yml
    @@ -0,0 +1,54 @@
    +  attributionText: M. Alsup

    This really should just be a list of "authors". There can be more than one.

  5. +++ b/ASSETS.yml
    @@ -0,0 +1,54 @@
    +  attributionUrl: http://malsup.com/jquery/form/

    Simple rename to "url".

  6. +++ b/ASSETS.yml
    @@ -0,0 +1,54 @@
    +  license: MIT and GPL licenses

    This should be split into a list of "licenses" because there can be more than one.

  7. +++ b/ASSETS.yml
    @@ -0,0 +1,54 @@
    +  rights: https://github.com/malsup/form#copyright-and-license

    Simple rename to "license"

  8. +++ b/LWG_EXCEPTION.yml
    @@ -0,0 +1,433 @@
    +# The following 3rd-party libraries or code have been approved for inclusion
    +# in this project's code repository by the Drupal Licensing Working Group
    +# (https://www.drupal.org/project/drupal_lwg).
    +#
    +# ANY and ALL alterations to code that affect the listed files below MUST
    +# be re-submitted to the Drupal Licensing Working Group for re-approval
    +# before the changes can be committed to this project's code repository.

    I just added this, but I think it is important to show some sort of disclaimer of how to treat these 3rd party libraries and why this file exists.

  9. +++ b/LWG_EXCEPTION.yml
    @@ -0,0 +1,433 @@
    +  files:
    +    - replace/jquery/2.1/jquery.js
    +    - replace/jquery/2.1/jquery.min.js
    +    - replace/jquery/2.1/jquery.min.map
    +    - replace/jquery/1.11/jquery.js
    +    - replace/jquery/1.11/jquery.min.js
    +    - replace/jquery/1.11/jquery-1.11.2.min.map
    +    - replace/jquery/1.10/jquery.js
    +    - replace/jquery/1.10/jquery.min.js
    +    - replace/jquery/1.10/jquery-1.10.2.min.map
    +    - replace/jquery/1.9/jquery.js
    +    - replace/jquery/1.9/jquery.min.js
    +    - replace/jquery/1.9/jquery.min.map
    +    - replace/jquery/1.8/jquery.js
    +    - replace/jquery/1.8/jquery.min.js
    +    - replace/jquery/1.7/jquery.js
    +    - replace/jquery/1.7/jquery.min.js
    +    - replace/jquery/1.5/jquery.js
    +    - replace/jquery/1.5/jquery.min.js
    +  versions:
    +    - 2.1.4
    +    - 1.11.2
    +    - 1.10.2
    +    - 1.9.1
    +    - 1.8.3
    +    - 1.7.2
    +    - 1.5.1

    files and versions - It is important to know which "files" and "versions" have been approved for inclusion within the project. Code can drastically change between versions, including licenses. I'm not entirely sure, but I would imagine that this would allow projects to "go back" to a previous version if they really needed to (without approval?).

    Personally, I don't necessarily enjoy the fact that the jQuery UI library has essentially bloated this entire file because of the above added properties, but I do think it is necessary. It just goes to show how much code/files are actually being used from that library in this project. It may actually help remove/move the entire thing or certain files in the future? I'm not sure yet.

  10. +++ b/LWG_EXCEPTION.yml
    @@ -0,0 +1,433 @@
    +jquery.cookie:
    ...
    +  files:
    +    # TODO: This should be moved to its own folder, not inside jQuery UI.
    +    - replace/ui/external/jquery.cookie.js

    Follow-up created: #2536698: Move and update jquery.cookie

gisle’s picture

markcarver, thank you for your comments and suggestions!

It is correct that the file format that we're going to use to document approval of third party materials in the repo is not finalized yet. We're trying out the proposed format on existing projects (like yours) to learn more about its suitability and usability (or lack thereof).

The primary function of this file is to make sure that a repo on Drupal.org contains a record when an exception to the Git Repo Usage Policy is granted by the LWG. Having a file in the repo that points (by means of URL) to the issue in the LWG issue queue will make it possible to verify (if somebody in the future requests documentation) that the exception has been granted. Any details (who granted it, special conditions, etc.) need to be looked up in the issue queue.

The secondary function of this file is to make it simple for downstream recipients to locate license and attribution metadata for third party materials. We should do this for all third party materials, but this metadata is particularly important if the third party non-code asset comes with a license that is not GPLv2+ and also have specific attribution requirements (like the CC suite of licenses).

In the Apache 2.0 license, attribution metadata must be preserved in downstream derivative works in the form a cumulative "NOTICE" text file (if it is initially present). Here is how the Apache 2.0 license describes this file:

If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.

A short note about our choice of a data serialization format (YAML) for this metadata, instead of free text (like Apache 2.0):

The LWG could have opted for a similar scheme as the "NOTICE" text file used by Apache 2.0. But we think that the YAML format, consisting og key-value pairs, provides better guidance to users about what information to include (when the keys that are required are pre-defined).

Your version for this metadata file for the JQuery Update project is 938 lines (kreynen's version is 149 lines). That is a lot of lines. I think it is going to be very difficult to gain community acceptance for this scheme if the level of detail you propose to supply is going to be the norm.

I think we need to focus on providing enough information to satisfy present and future documentation requirements - but not more. Up to now, exceptions have been granted ad-hoc and no documentation has been kept. In the future, we need to have some documentation in place. But insisting that developers maintain a complete audit trail of any third party materials is going to be perceived as needless bureaucracy. IMHO, we should not do this unless it is deemed absolutely necessary (and I do not think it is).

I am now going to respond to all your 10 points:

  1. The file name: ASSETS.yml vs. LWG_EXCEPTION.yml.
    This is a valid concern. I am not sure LWG_EXCEPTION.yml is the right choice. While recording an exception is the primary function of this file, the secondary function (providing metadata about assets to downstream recipents) should also follow from the name. What about LWG_NOTICE.yml?
  2. Duplicate keys.
    This is a valid concern, the format needs to disallow duplicate keys.
  3. Include the name of who approved it.
    I think this information does not belong in this file. All information about the exception should be in the linked LWG issue. In practice, I believe the metadata file will be placed in the repo when an exception issue is opened (before the exception issue is resolved - so the name is not known yet). You shouldn't have to edit it after the request have been approved.
  4. Introduce key authors to list multiple authors.
    You may list multiple authors separated by commas/conjunctions as a single value assigned the key attributionText (from the Open Data Rights Statement Vocabulary vocabulary).
  5. attributionUrl vs url
    The element attributionUrl comes from CC RDFa vocabulary. It has been decided that the LWG should not invent its own metadata vocabulary, but leverage existing vocabularies whenever possible.
  6. Multiple licenses.
    You may list multiple licenses separated by commas/conjunctions as a single value assigned the key license (from the DCAT vocabulary).
  7. rights vs license
    The element rights comes from Open Data Rights Statement Vocabulary vocabulary. It has been decided that the LWG should not invent its own metadata vocabulary, but leverage existing vocabularies whenever possible.
  8. Add disclaimer and explanation in comment block at the top the file.
    Having some information at the beginning explaining this file is good idea. However, I don't think our policy shall be that any changes must be re-approved by the LWG. The LWG are going to trust you after your exception has been approved. If the third party code base switches licenses, the vetted maintainer will be expected to deal with this in a responsible manner, including requesting guidance from the LWG if he/she thinks this is required. But we're not going to put in place some horrible bureaucracy where the LWG is going to block new releases until it can get its act together.
  9. Files and versions.
    I do not think we should do this. It will put a horrible documentation burden on maintainers. I do not diagree that it is important to know which "files" and "versions" have been approved for inclusion within the project, but inclusion of files is automatically tracked by Git, so this can be recontructed from the timestamps in the Git log if required at one point.
  10. Have comments about planned changes to the repo.
    I do not think this (TODO comments) should go in the metadata file.
markcarver’s picture

Status:Needs review» Needs work

Your version for this metadata file for the JQuery Update project is 938 lines (kreynen's version is 149 lines). That is a lot of lines. I think it is going to be very difficult to gain community acceptance for this scheme if the level of detail you propose to supply is going to be the norm.

Isn't that kind of the point? To discourage people from committing 3rd party libraries in the first place. This isn't a matter of convenience. If it were, we wouldn't have this process to begin with. This entire issue is about ensuring that we are documenting which compatible external files are being packaged with a GPLv2 licensed release on d.o and removing the ones that aren't.

But insisting that developers maintain a complete audit trail of any third party materials is going to be perceived as needless bureaucracy. IMHO, we should not do this unless it is deemed absolutely necessary (and I do not think it is).

I completely disagree. Just the fact that Drupal now has a Licensing Working Group is proof of said "bureaucracy". I'm not opposed to this at all, but if we're going to do this... we may as well do it correctly the first time around and establish proper protocol. This means keeping records of what we are actually providing "exceptions" FOR.

Is it large, sure... but it should be IMO. It should be a comprehensive record, not just "details". Before I posted my revision, this issue didn't actually show which files are being "approved" for this "exception"... just the "library names". What would stop someone in the future to just arbitrarily throw in a file in a said library structure (I have seen that, more than I would care to admit)? What exactly are we approving then? Just the concept of the library? I was under the impression that the code is what we're approving an exception for. Where that lives matters.

Furthermore, I suppose I'm also thinking long term here. I know this can seem like a seemingly daunting task at first having to do everything manually. However, by specifying these files in a file like this, we could (in theory) make this part of our Drupal CI. This would allow us to actually track these files and allow maintainers to get valuable feedback if a patch contains extra code/files that don't match what's in this one. It would allow accidents, like my theory on how #2534056-2: Remove unused and NON-MIT/GPL licensed or compatible code happened, to be prevented more easily.

Also, the creation and/or modification of this file would be a rather rare event. Initially it can be a pain for some, sure, if they're including a huge library like jQuery UI... but the majority of libraries are only a few files at most. I think y'all are missing the point of why this project's file is going to be so large: this project is truly the exception, definitely not the rule. We are adding 6 separate external libraries (some with multiple versions). Most projects would not, nor should not, do this.

The main reason that this project is the way it is, is because it has evolved this way, plain and simple. Before proper "asset" handling was established in the community as "Best Practices™" and before 3rd party licensing was truly "wrangled in". This project was "allowed" because... core already did it, so "it must be ok". So instead of looking at how expensive we might treat this for all projects and how this may affect this particular project, we should instead be focused on establishing proper protocol.

If anything, I would also argue that this actually provides a legitimate justification for denying more addition of code by maintainers of this project (and even the removal of existing code in the future). The more external code projects put in, the more you have to keep actual records of it. An example: #2341259-9: Allow detection of different jQuery UI libraries in sites/*/libraries/jquery.ui, which wanted to add nearly 8MB of new code to affectively "switch" jQuery UI versions. This is an unacceptable direction of this project and makes it even more difficult to maintain, not this particular file.

  1. What about LWG_NOTICE.yml?

    This is certainly fine by me. All I really think that matters is to just have something with LWG in it.

  2. Ok
  3. Include the name of who approved it.
    I think this information does not belong in this file. All information about the exception should be in the linked LWG issue. In practice, I believe the metadata file will be placed in the repo when an exception issue is opened (before the exception issue is resolved - so the name is not known yet). You shouldn't have to edit it after the request have been approved.

    Knowing who approved something is definitely important. In fact, I would argue that we should probably not just put the username (which can change) but the individual's REAL name, resembling what we already do in MAINTAINERS.txt. Having "just a link to the issue" does not convey relevant information, it defers it and can sometimes be lost (username changes, user deleted, etc.). There shouldn't be any need to "edit after the request has been approved" either, it should be part of the issue and the LWG member involved should in fact sign-off of the issue by adding these lines. I didn't add these lines, @kreynen did in #10... I simply moved it around and gave it more contextual information. So I propose that we instead, in fact, expand this to read like MAINTAINERS.txt, where instead of a link to the user it links to the issue where it was approved instead:

      approval:
        - Kevin Reynen 'kreynen' (https://www.drupal.org/node/1857042)
  4. See response in #7
  5. See response in #7
  6. See response in #7
    1. separated by commas/conjunctions as a single value

      This format is certainly fine for licenses. However as y'all have already pointed out, this file isn't a license. It is meant to convey information that can be easily consumed, as you said, in a serialized data format (which is why YAML was chosen). If that is the case, then let's actually utilize said data format so it can be easily and automatically parsed.

    2. property names

      I'm all for "using terms found elsewhere", however the rational behind these property names cited THREE different sources (which mostly deals with how things should be worded in licenses). This file is to be consumed by the Drupal community and the Drupal project it affects in an effort to deal with 3rd party code/licenses that should normally not be "allowed" in d.o hosted projects. Using concise and appropriate terminology here is OK (i.e. authors makes a hell of a lot more sense than attributionText).

  7. Having some information at the beginning explaining this file is good idea. However, I don't think our policy shall be that any changes must be re-approved by the LWG. The LWG are going to trust you after your exception has been approved.

    Then I'm confused... what is the point of the LWG then? Is it not because: we currentlyDON'T have faith in people doing the right thing? Licensing is hard and quite confusing at times. It isn't something that most projects (or their maintainers) will necessarily know if they're doing "right" or not. It really isn't a "bad thing" to have someone from this group sign-off on any code that touches existing exceptions. In fact, it's a GOOD thing because then we're following procedure and ensuring that there isn't any unintentional code added, changed or removed like I discovered in #2534056-2: Remove unused and NON-MIT/GPL licensed or compatible code which was added in a fell-swoop commit of #1947438: Update jQuery UI to the latest version (1.10.2) by a maintainer. Ensuring that it must be "signed-off" by LWG would ensure a second level of eyes on code that can easily be snuck in (even on previously "trusted" projects).

    If the third party code base switches licenses, the vetted maintainer will be expected to deal with this in a responsible manner, including requesting guidance from the LWG if he/she thinks this is required.

    That's all well and good, but really not practical. Either we treat the LWG as an official "authority" (so to speak, like we do with security issues) or there's really no point in this, is there? Ignoring, mismatching or even removing licensing has serious and sometimes costly consequences. This isn't something we should tread lightly with. If we're going to do this, then it needs to be done correctly. I am certainly fine (and I'm sure others would be too) with putting a little extra effort into the projects that need or want external libraries in them if it means no one is legally liable at the end of the day and everything is fine.

    But we're not going to put in place some horrible bureaucracy where the LWG is going to block new releases...

    Being bureaucratic isn't inherently "horrible". It is born out of a necessity to govern with checks and balances for things that normally would be overlooked and run wild. The fact that the LWG has been created is evidence that such a system is needed and I certainly agree. There is no need to apologize for it, nor diminish its authority before it has even really left the gate. I'm OK with being a responsible project maintainer and allowing the LWG to govern an exception IT has granted.

    Also, no one is saying that this has to "block releases". In fact, that should not ever be the case. There will be previous releases that will ALWAYS have incorrect license or include non-approved code (as #9 points out). This is just the nature of the beast, the best we can ever hope for is moving forward correctly in the future.

    ...until it can get its act together

    Isn't that what I'm helping you with now? It's not a perfect system now, no, but that is certainly what I am trying to help you create. Something that actually has meaning and value, not just some filer text that can be arbitrarily created, wishy-washy and lacking any true context.

  8. I do not think this (TODO comments) should go in the metadata file.

    I agree. I will remove that comment in the next iteration. I was simply trying to point out (to y'all) that where the file lives is technically wrong, which only this issue has brought to light. This is just proof of why keeping record of the versions and files of external libraries is important. It forces any maintainer (including new ones) to actually know what the project is doing with said external libraries. I only recently became a co-maintainer and do not know the full history of the project. Having to step through all this manually has definitely been a learning experience (one I am happy to learn btw). I could only imagine if I had this file when I started to dig into it how much easier it would be to know exactly what external libraries this project added/replaced.

Setting to CNW per above comments. Will await feedback before iterating on latest patch.

kreynen’s picture

@markcarver I haven't digested all of this yet, but this is exactly the type of conversation I was hoping to start with these patches. Drupal has always been ahead of most other projects when it comes to licensing, so we don't have to many examples to use as a reference. We do want to get this right, but in this case that means finding an acceptable balance between the information we require from Drupal project maintainers and what they are willing to provide. Obviously if we make this too difficult, developers will find alternatives to hosting a project on Drupal.org. We want the fact that projects are vetted by the LWG to be a benefit to hosting on Drupal.org... not the reason projects move to GitHub.

markcarver’s picture

Drupal has always been ahead of most other projects when it comes to licensing, so we don't have to many examples to use as a reference.

I'm confused by this. What exactly do you mean by "Drupal", core or the community? In either case, I would have never imagined we were "ahead" of anyone in the respect of licensing. It has only been recently that this has been a more prevalent and enforced issue, at least from my perspective. The reason there's no "examples" is because this issue really hasn't been that well (or rather consistently) addressed or enforced, if and when issues arose. I could be mistaken though.

Obviously if we make this too difficult...

I really don't see how this is being "too difficult" I recreated this file in less than 15min (and that was also re-formatting it a bit so it would utilize the YAML syntax a bit better). It's actually taken more time talking about this file than creating the file itself.

I would even be willing to help develop a drush command (or even figuring out some other way, maybe on d.o, to automate the creation of this file). This doesn't have to be a manual process, at least with file detection aspect, but it's inclusion is rather important from a "record keeping" perspective.

So my point remains: it seems a little odd that projects are required (by LWG) to a file to their repo for "record keeping" purposes, but then don't actually "record" what exactly it is we're approving. If we do not explicitly state which files from the external code are approved exclusions, then we are essentially leaving that door up for interpretation and are allowing the entire generalized concept of said library to be part of that exclusion. We are then at the mercy of said external library if they decide to suddenly add some arbitrary file (jshint.js in jQuery UI for example) and then it makes it's way in our repo and "prevents anyone from doing evil".

Now, I'm not saying that we can't have it without the files... but it seems a rather moot point if we don't (i.e. approve this library, 3 months later... some new file is added under the guise of said library). I'm also saying that we can't have the "trust system" here, but why even bring it to that? Why not just make it a sane system that is very clean on what we're allowed to do and not do?

... developers will find alternatives to hosting a project on Drupal.org

Developers already do this, but it has little to do with licensing (from my perspective anyway). I moved Dreditor off of d.o, but that was because it offered a better CI with Travis than d.o can provide. Maybe someday that will change. All my "drupal" related projects (modules/themes) are on d.o. Determining where to host a project has too many factors to pinpoint on just one issue, especially this one.

We want the fact that projects are vetted by the LWG to be a benefit to hosting on Drupal.org... not the reason projects move to GitHub.

I was unaware that was the goal of LWG. It has always been my understanding that d.o frowned upon 3rd party libraries (with a few notable exceptions) and it has been this way for years. It hasn't been until recently that LWG was formed and a more organized approach to this issue has been taken. I was also under the impression that this was about recording what external code/libraries are being, essentially, forked and hosted on d.o.

The only "true" benefit of hosting on d.o is that you have the Drupal community at your fingertips and everything is interconnected. I don't see how LWG is a strong selling point, no offense.

markcarver’s picture

P.S. I keep forgetting to mention a correction that the patch in #12 is only 434 lines, not 938 as #14 mentions. While, yes, this is still a large line count... it's also rather unique to jquery_update only. Most projects would likely only be a single library entry and total ~20-40 lines, mimicking something along the lines of jquery.bbq or jquery.cookie.

Crell’s picture

Regarding approval, I question putting an individual's name on the approval. In practice, I would want such exceptions granted by the LWG in some regular periodic meeting, as a group. We want to put an end to "well this one person said yes over here, but didn't check with anyone else."

markcarver’s picture

Approval shouldn't be based on personal opinion or bias, it should be consistent across ALL projects by the policies and procedures laid out in the LWG (which is what I thought this group was created for). I don't see how deciding issues in a group would change this (unless it were a severely controversial/heated issue). Even then, I still don't see how that can come up when this is dealing with external code in where licenses are either a) compatible with GPLv2 or b) not compatible... seems rather cut and dry?

So I would hope that ALL members of this group follow the same policies and procedures. If something isn't right, shouldn't we know whom to contact 1) the LWG member who approved it and 2) the individual/maintainer who committed the code rather quickly? This seems like relevant information and would certainly save time trying to sift through issues/git history to try and track down when and where something was introduced and by whom.

I'm not saying that this to be malicious or make it sound like there will be "capital punishment", mistakes happen... I get that. However if this file is to truly be for "record keeping" purposes... I would imagine that we should be adding this kind of relevant information. Am I wrong?

----

I'd also like to make it clear that I am definitely not trying to start any argument or debate around every little topic. I'm simply playing the devils advocate here since this is proposing adding a required/mandated file from the LWG for projects like this. That's all.

I'm certainly OK with adding more or even removing the information I added if it's truly deemed irrelevant or difficult in the long run. Just trying to give feedback (as was asked) and bring up valid points with how I define "record keeping".

I'm also trying to see how this file will possibly be consumed in the future and its potential in automated processes like Drupal CI. So while I ask that what I say is taken under serious advisement (I know I'm probably not the only one who thinks this way), know that in the end I of course will definitely defer to any decision or judgment the LWG makes :D

gisle’s picture

First. Apologies for getting the line count wrong. You're right: The patch attached #12 is 434 lines (and the patch attached #10 is 54 lines, not 149). I copied the wrong column from the output of wc.

Is it large, sure... but it should be IMO. It should be a comprehensive record, not just "details".

We already have a comprehensive record of what files and versions are added at different times. This is the project's history as automatically recorded in the git log. Requiring maintainers to also have this record in the metadata file will IMHO create a lot of extra work.

Before I posted my revision, this issue didn't actually show which files are being "approved" for this "exception"

This is correct, but in case of this being disputed, we need only look at a snapshot of the Git repo as it existed at the time when the exception was granted, and that would show the list. In my experience, the manually maintained version of duplicated data will rot.

What would stop someone in the future to just arbitrarily throw in a file in a said library structure (I have seen that, more than I would care to admit)?

Nothing can stop that. Even if the kept a comprehensive record inside the metadata file, people would do this.

What exactly are we approving then? Just the concept of the library? I was under the impression that the code is what we're approving an exception for. Where that lives matters.

We need to find the correct level of diligence if this is going to work.

In the past, no documentation was created when an exception was granted. This is going to end. But how much documentation is required to let the LWG to do its job in a consistent and transparent fashion? Where should this documentation live?

The "exception" requirement was not introduced by the LWG, it has been part of our Git Repo policy since day 1. In the past, no documentation was created when an exception was granted. People met face to face, or on IRC, and cordial agreements was made. This does not scale very well to a community of thousands. The LWG wants to put in place a transparent and persistent documentation of the exception agreement. That documention is the comments below the exception request in the LWG issue queue. If you look up an exception request issue, you shall find the dates, the names of individuals involved, the licenses involved, and the grounds for granting (or rejecting) an exception request. The metadata file is not the documentation. However, to make it simple to locate the documentation in the future without searching, it it is required to have an URL pointing to the documentation, but I see no grounds for keeping other data about the exception agreement in this file. I think we should apply the principle of Occam's razor to the metadata file format.

Then I'm confused... what is the point of the LWG then? Is it not because: we currently DON'T have faith in people doing the right thing?

I hope not. There is of course no single point behind the creation of the LWG, but I believe that one of these points was the observation that Drupal.org Webmasters (who where previously tasked with enforcing the Git Repo Usage Policy) did not much like doing it. As a result, some reported policy violations were acted upon, others were ignored - with no transparency and not much of a pattern.

That's all well and good, but really not practical. Either we treat the LWG as an official "authority" (so to speak, like we do with security issues) or there's really no point in this, is there?

I think we're talking past each other here. The LWG has the authority to speak (and to act, such as removing a noncompliant repo).

However, the LWG is not going to be a licensing STASI, keeping a 24/7 surveillance on the repos. I do not have the capacity, nor the inclination (speaking for myself, but I'd be surprised if other LWG-members feel different).

The LWG hopes to bring consistency, transparency and persistency to the exception granting process. The LWG will coordinate the process when somebody requests an exception, or when a breech of Git Repo Usage Policy is reported to us. I hope our main task will be to provide guidance to maintainers about licensing issues, and that this guidance will result in people doing the right thing. I don't think maintainers that fails to comply with our licensing policy do so on purpose. But the policy is complex and some may fail to comply simply because they're not aware of what is required. After going through the process of being granted an exception, that person shall know how to do the right thing, and will go on doing it.

I will conclude saying that I really appreciate you're taking the time to provide solid feedback. I should add that what we've had other feedback, and the extra workload required to provide metadata is one of the major objections we've encountered.

kreynen’s picture

I'd also like to make it clear that I am definitely not trying to start any argument or debate around every little topic. I'm simply playing the devils advocate here since this is proposing adding a required/mandated file from the LWG for projects like this. That's all.

I think that's understood and appreciated by everyone. I'd still classify this as healthy debate.

I would have never imagined we were "ahead" of anyone in the respect of licensing.

By clinging to restrictive policies for our CVS/Git policy as well as continuing to maintain our own repository, the Drupal community has the opportunity to provide downloads of projects that create very little risk to license conscience users and organizations.

One goal of the Drupal project has always been to keep everything downloaded from Drupal.org compatible with GPLv2. As the number of contributors grew, the community's ability to maintain GPLv2 compatibility suffered and a lot of code made its way into projects that wasn't strictly GPL compatible. The LWG has already spent months cleaning up those issues. While you can still find licensing violations if you go looking for them, the LWG has either dealt with the unresolved issues in the backlog or postponed the issue while drafting updates the Drupal.org policies and redefining the process to allow or reject including the questioned code or asset in a project.

While someone can leagally create a downloadable package that includes Drupal as GPLv3 along with Apache2 licensed PHP and commercially licensed Javascript, images, fonts, etc, users should never accidentally find themselves running code like that in projects they download from Drupal.org. While WordPress references the Drupal project's belief that all modules and themes are derivatives of the core project and inherit the GPLv2 and later licensing, I would challenge anyone to download 10 random projects or themes from https://wordpress.org/plugins/ and https://wordpress.org/themes/ without finding a license combination that would limit the freedoms the GPL allows. While the WordPress download situation makes a great framework to build a licensing violation bingo game, it isn't ideal for anyone who really wants to run or contribute to a GPL licenses project.

And as you've seen with jQuery, GitHub can be the wild west of licensing.

That's what I mean by "ahead of other projects". There's still a lot of work to do, but we have the infrastructure that makes this possible. So in that sense by refusing to change, we are still ahead.

I would even be willing to help develop a drush command (or even figuring out some other way, maybe on d.o, to automate the creation of this file).

While drush support would be useful, since a large number of projects that require 3rd party exceptions may end up being themes dealing with fonts and images, I think a UI might be more useful. I've added #2537948: Add form to generate asset list. #2537954: Better support for multiple licenses is also directly related to jquery_update since several of these files have multiple licenses.

I going to push for the LWG to meet next week. Hopefully we can dig into this feedback and get the rest of the proposed changes published for more community feedback.