Support for distributions

Duplicate of webflos issue opened with an example.

Not a duplicate. Other issue was just for config_installer, an install profile, and this is how we determine how to support distributions moving forward.

Plan: Encourage distributions users to have a scaffold project template. This should live on anywhere that packagist understands. The namespace on packagist should be anything thats not drupal. Update scaffold building documentation/ examples/ maybe provide a starterkit.

Distributions on drupal.org need a new field that is the namespace on packagist as well as the URL of where the scaffold lives.
Scaffolding should have links to packages.drupal.org as well as the distro can for dev purposes.

Composer facade will process distributions that follow this pattern of distros as provide that metadata properly.

Packaging will repackage distros on commit

Generate a drupal.org license whitelist of strings compatible with gpl (see this list: https://spdx.org/licenses/))

Involves doing a composer create project namespace, and then composer licenses to check to make sure all licenses are on our whitelist.
And then creating a tarball.

So a whitelist of licenses vs. regex of paths?

Yep. We can build the distro, and check if it would ship with any incompatible licences, and if so, we would only be able to offer the source, and not a packaged version of the distribution. But this would only apply to distributions that would be built with composer, not ones built with drush make, since there isnt a licensing tool available for that build process.

Where do you envision the list of license compatibility living? Currently the whitelist regex are nodes. The closest thing we have to a list is https://www.drupal.org/node/1475972#gplv2-compatible-licenses

There have been conversations about creating a composer plugin to check license compatibility, but Drupal's interpretation of GPL compatibility that is different than WordPress or Google. I think the list of compatible licenses would need to be provided by Drupal.org, but as code or content? Adding to the complexity is that Composer supports OR in the license definition, so the license check isn't going to be a simple array comparision.

I started #2816757: Impact of Using Composer to Build D8 Distributions in the LWG queue to discuss the licensing issues, but in general I think this is a smart move. Most distributions have development workflows and testing that it would make little sense for Drupal.org to try to match. This change would also give us more flexibility and a clearer distinction between what is distributed from Drupal.org and what is checked into Drupal's git repo when it comes to licensing policy.

We'll probably keep it something simple like a variable set that we keep in code. A licensing whitelist is something that is very unlikely to change - there aren't new, compatible licenses that crop up on a frequent basis, so it'd be a matter of opening an issue in the infra queue to add one.

Composer outputs the license data in json format, so its relatively easy to parse, and if we detect an OR or an array, we can check if either of the disjunctive licenses match.

I do not think a composer plugin is necessary, its already built into composer to display what the licenses are, which is all we need to know to be able to determine if we can distribute the packaged code.

there aren't new, compatible licenses that crop up on a frequent basis

For licenses that are commonly used on PHP projects and libraries, we've probably only added 3 or 4 licenses since the packaging whitelist project started. Unfortunately we deprecated more than that where approval was based on the "or later" clause of the license of Drupal's licensing. These non-GPLv2 compatible licenses were added to the whitelist entry content type's license taxonomy. I'm assuming those terms were added or maintained by infrastructure and still includes Apache2, AGPLv3, and GPLv3. Once those whitelist entries were approved, revoking them became problematic.

For additional, non-code assets the list of licenses we find in projects or people have requested that are common in Composer and are allowed to be included with other popular GPLv2+ projects are changing far more often.

The LWG has spent hours reviewing "GPL friendly" licenses and never come close to reviewing them all. I don't know of any project that maintains a thorough GPLv2+ license compatibility list.

Wasn't that a one time misunderstanding of how GPLv2+ worked? i.e. Not realizing that there is no such thing as a GPLv2+ license, and that what we are actually doing is offering all of our code under a disjunctive license (OR) of GPLv2, OR GPLv3 OR GPLv4 if it comes into existance?

We dont need to spend any time whatsoever on determining what licenses are compatible or not. GNU has already provided a list: every single license on this page in green (https://www.gnu.org/licenses/license-list.en.html#GPLCompatibleLicenses), will be on the whitelist, with the exceptions of the ones that specifically say they are not compatible with GPLv2. We'll only have to worry about reviewing licenses when exceptions come up. The vast majority of the time, people are using common licenses.

Wasn't that a one time misunderstanding of how GPLv2+ worked?

I was only involved as a developer requesting whitelist entries when this happened, but I think some people thought we'd allow any license that was compatible with GPLv2 or GPLv3 since Drupal can be distributed under either license.

The bigger problem was that the DA and the shops that put time into packaging third party assets on Drupal.org didn't really understand licensing and left a mess for someone else to clean up. I'd like to avoid a repeat of the packaging whitelist since we really don't have many licensing savvy community members willing to volunteer their time.

We dont need to spend any time whatsoever on determining what licenses are compatible or not.

I can't express strongly enough that licensing is rarely that simple.

If everything in Packagist was a PHP library it would be easier, but it isn't. Are we going to allow distributions to include https://packagist.org/packages/fortawesome/font-awesome ? https://github.com/FortAwesome/Font-Awesome/blob/master/composer.json lists the license as SIL-OFL and MIT. You have to read https://github.com/FortAwesome/Font-Awesome/blob/master/README.md to know that MIT only applies to CSS, LESS, and Sass files.

Read https://www.gnu.org/licenses/license-list.en.html#SILOFL

Is that compatible with GPLv2 and GPLv3 or not?

Does it even matter since non-code assets don't have to be GPL compatible to be distributed with GPL code?

Hopefully you are starting to see why this isn't going to be as easy as comparing the license value from the composer.json with a list of licenses compatible with both GPLv2 and GPLv3.

The vast majority of the time, people are using common licenses.

This is a PHP code centric view of licensing. Most distributions want things like fonts, images, and content and the people creating those assets are now using the license most common for that asset... which are often not GPL compatible... but in many cases don't need to be.

You are really trivializing the amount of time and effort the LWG has spent discussing this and cleaning up the mess the "licensing to simple" approach has caused in the past.

I've been dealing with licensing and building distributions in Drupal for years. It is not nearly as simple as you are trying to make it. With this approach, you are either going to fail to provide the service distribution maintainers want or fail to distribute properly licensed distributions.

++ to the approach outlined in #4. Cross posting #2763017: Dependencies of dependencies aren't installed when both dependencies are pulled from packages.drupal.org which I think is also currently a blocker.

This issue is actually the blocker for that issue. We currently do *not* process dependencies of distribution - hence why they are not yet "supported". I'll add more details to the other issue too.

I'm just going to start adding related issues as they come up.

#2828209: Add php-font-lib package is primarily php that includes a copy of the LGPL-2.1 license, but claims it is LGPL-3.0 in its composer.json

To pass any kind of automated license compatibility check, the project would need to update https://github.com/PhenX/php-font-lib/blob/master/composer.json#L6 to be either the deprecated LGPL-2.1+ or the License Expression Syntax...

{
    "license": "(LGPL-2.1 or LGPL-3.0)"
}

That still isn't going to catch the fact that the project includes a font which may have been designed for Intel with no license. It is VERY unlikely that font was licensed under any version of LGPL.

IN #11, kreynen wrote:

Read https://www.gnu.org/licenses/license-list.en.html#SILOFL
Is that compatible with GPLv2 and GPLv3 or not?

First - to lay this red herring to rest: The the FSF is the authoritative source on these things and it is pretty clear from colour coding (orange dashes) on the page you link to that the FSF does not consider the SIL OFL to be GPL compatible. (But this particular point is irrelevant, please read on.)

That statement is only relevant if the GPL requires the file in question to be under the GPL. In this particular case, it is clear from the GPLv2 license text that this is not a requirement. Here is the relevant part:

In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. (my emphasis).

So GPL compatibility is not a requirement for assets (such as fonts and images), since such assets would be stored and distributed along with our GPL-licensed PHP code as a mere aggregate.

And the SIL OFL FAQ statement about GPL compatibility is simply a restatement of the "mere aggregation" in the GPL:

Question: 1.2 Can the fonts be included with Free/Libre and Open Source Software collections such as GNU/Linux and BSD distributions and repositories?
Answer: Yes! Fonts licensed under the OFL can be freely included alongside other software under FLOSS (Free/Libre and Open Source Software) licenses. Since fonts are typically aggregated with, not merged into, existing software, there is little need to be concerned about incompatibility with existing software licenses. You may also repackage the fonts and the accompanying components in a .rpm or .deb package (or other similar packaging formats or installers) and include them in distribution CD/DVDs and online repositories.

So according to both the FSF and SIL, a font is merely aggregated with Drupal software, not merged into the software. IMHO, it should be permitted in our Git repo and our distributions.

All that remains is for uschange our Git Repo Policy to allow for the "mere aggregation" exception in the GPL. Two years ago, I joined the LWG with the ambition to get our Git Repo Policy amended to make room for the "mere aggregation" exception in the GPL. Unfortunately. I haven't managed to make progress on that, but I have not given up.

Relating https://github.com/drush-ops/drush/issues/2528 "we are removing make from Drush 9 and recommending that folks use Composer instead"

Plan: Encourage distributions users to have a scaffold project template.

@Mixologic is there any documentation available for the recommended best practice as a distribution maintainer? I am starting to port my distribution to Drupal 8.

I found a few examples of other D8 distros doing this, I think:

https://packagist.org/packages/acquia/lightning-project
+
https://packagist.org/packages/acquia/lightning

https://packagist.org/packages/goalgorilla/social_template
+
https://packagist.org/packages/goalgorilla/open_social

These distros are also including Drush Make files - I assume that is so that they can continue to be packaged on drupal.org currently? Notably they both also recommend using Composer on their project page descriptions.

@m.stenta I can speak for Lightning... Yes, we include the makefiles so that Lightning can be packaged by D.O. Although we do discourage the use of the tarball since it makes it difficult to update dependencies and you don't get some of the non-drupal dependencies like php-twitter-api. But some services (like dev desktop) still use that tarball, although that one in particular is hopefully changing soon.

The makefiles are automatically generated from our composer.lock file. See https://github.com/acquia/lightning/blob/8.x-2.x/src/Composer/Package.php

Our scaffold project is pretty minimal. Ultimately we recommend using acquia/blt-project instead of acquia/lightning-project - but that can be overwhelming for users just getting started which is why our project page points to lightning-project. In theory, you should be able to just require acquia/lightning in any drupal scaffold project including drupal-composer/drupal-project.

@balsama - Thanks! Very informative!

Note: If this were supported, then https://github.com/acquia/reservoir and https://github.com/contentacms/contenta_jsonapi could also have a tarball available on d.o.

I've burned out after going in circles on this for > 2 years. I'm taking a break for a least a few weeks from maintaining https://www.drupal.org/project/drupalorg_whitelist. I've already informed @megansanicki and @hestenet, but the practical impact of 1 person taking a break is now that improvements and security fixes for distributions that require changes to the whitelist on Drupal.org will not happen.

There is just no one left willing to deal with these policies.

In just the last 2 weeks I've rejected a FFW/OpenY request to add a font that would have been allowed under the tweaks the LWG proposed and a request from Lullabot/Contenta to aprove an Apache-2.0 library which can't be approved unless Drupal.org is going to allow distributing Drupal as GPL-3.0.

My 2¢ is to make this change for D8 ASAP and leave it up to the distribution maintainers to add either the GPL-2.0 or GPL-3.0 license at the root of their project based on what they've packaged with the distribution and call it a day. I don't belive there is enough volunteer bandwidth who understand or care about Drupal.org's draconian GPL-2.0 compatibility requirements.

If Dries really wants to keep everything distributed from Drupal.org strictly GPL-2.0 compatible, he's going to have to figure out another way to make this happen.

I wanted to add here that this is really dissapointed thing in terms of Drupal Distribution development. We're blocked with this policy for 1 month and due to this we had to disable all releases on drupal.org and redirect people to GitHub and composer workflow.

Please let us know how we can help to push this forward and add tarballs back to drupal.org page.

Another project with Drupal 8 distributions, that drush make cannot download libraries outside of profile. E.g. Slick library shoud be placed in /libraries folder and eventually we get not full distribution through drupal.org.

If there's separate topic for this, please point me out.

@sanchiz #2281721: Update Drupal.org's drush make for distro packaging so distros are not forced to nest contrib projects inside contrib projects may be the issue that addresses that

Tentatively escalating to major. It sounds like more and more distributions are being forced to move off of Drupal.org because of this issue not being done.

When we say "Support distributions", I think there are two main components:

1. Make the distribution available on the D.O packagist (composer require drupal/my_profile)
2. Make the packaging script produce suitable tarballs for download

For the first problem, we can't currently use the D.O packagist because it ignores dependencies of profiles. See #2763017: Dependencies of dependencies aren't installed when both dependencies are pulled from packages.drupal.org. Most (all?) distro maintainers work around this by submitting their distros to the main PHP packagist and namespacing them with their own organization (see burdamagazinorg/thunder, acquia/lightning, etc).

The second problem is mainly due to the packaging script using legacy makefiles to build the tarball. There are two main problems with that (in addition to the fact that modern versions of Drush have dropped support for it):

• Composer dependencies of modules aren't pulled in (e.g. drupal/media_entity_twitter has a dependency on j7mbo/twitter-api-php - and drush make has no way to pull that dependency in)
• It sets users up with an unmaintainable codebase. Since it wasn't built with composer in the first place, they have no easy way to update/resolve their dependencies.

I'd be very happy to resolve the packaging script problem by following the pattern that GitHub has established by allowing maintainers to provide their own binary file per release. So distro maintainers would be responsible for generating their own tarball. If we're worried about hosting code that isn't properly licensed, we could say that the binary file has to be hosted elsewhere and only provide a URL field.

Actually, related to the issue being able to have a parent/base profile (#1356276: Allow profiles to define a base/parent profile), the use case where only the codebase, without packaged dependencies, should be supported. Offering an install profile as purely the codebase (as a composer package), should work perfectly too, since all dependencies will be installed in the modules/contrib folder. Dependency conflicts will be resolved properly then too.

This is quite urgent for distribution maintainers, since all "Drupal.org Library Packaging Whitelist" maintainers stepped down. @kreynen told me last week, that he was the last one and now also has resigned. This means we are not able to add new libraries to our distributions.

kreynen was active as of last november 2018, he assisted me with the swiper javascript library whitelisting.

I'd say that finding "Drupal.org Library Packaging Whitelist" maintainers should be made an urgent priority.

I am not aware of kreynen having resigned as maintainer of the whitelist or as chair of the LWG. He was active in the issue queue as recent as two months ago: https://www.drupal.org/project/drupalorg_whitelist/issues/3062787 - as per comment #22 he took a temporary break about two years ago.

For the record: The whitelist is maintained not only by kreynen, but by the LWG; https://www.drupal.org/project/drupal_lwg - in addition to kreynen (chair), its members are Shawn DeArmond and myself.

That being said, I think the LWG could do with more members. There should at least be five. Members of the LWG are selected and promoted by the Drupal Association.

It would also be nice if members of the LWG had permission to edit the taxonomy of allowed licenses (see #2307465: Update Whitelist License Taxonomy to Match Allowed Licenses). It just plain wrong, and is creating problems because people creating whitelist entries think it is accurate. The LWG made its decision five years ago about what licenses to permit, but it has not progressed since then since nobody that cares about this has the permission to edit those tags.

Edit: Kiamlaluno finally did the required tag edits. Much obliged.

this is not only a 7.x issue, but also an 8.x issue.

see the referenced issues for 'tarball installation' of wxt 8.x
https://www.drupal.org/project/wxt

We still need someone to replace @kreynen or else automate this process of greenlisting dependencies.

Wouldn't it be quite easy to automate the process of greenlisting dependencies? Check for the presence of the GPL license, if found, automatically approved pending a form where maintainer accepts responsibility and a captcha confirms a human requested.

In this way , can check for gpl and or other FSF compatible licenses, approved quickly with no less bottlenecks on having one super god like person deciding.

The process is currently backwards, it should be the other way around. Blocking dependencies should be the process that needs human interaction.

Looking at the issue summary, I think this has drifted away from that, but just to address recent concerns about the allowdelist maintenance status:

joseph.olstad wrote:

this is not only a 7.x issue, but also an 8.x issue.

see the referenced issues for 'tarball installation' of wxt 8.x
https://www.drupal.org/project/wxt

Why do you think this still is an issue? There now is a LWG, and the LWG actually maintains this list. We could do with a few more hands, but Kreynen and myself are able to cope.

As for the tarball Installation of https://www.drupal.org/project/wxt - that way of installing has been deprecated (and good riddance).

joseph.olstad wrote:

We still need someone to replace @kreynen or else automate this process of greenlisting dependencies.

However …

• Kreynen is still around, making sure that licensing issues get a timely response (last seen one week ago).
• And for the record: I am still around, and also try to make sure that licensing issues get a timely response.
• There are no "Active" allowlisting requsts in Drupal.org Library Packaging Allowlist.

joseph.olstad wrote:

Wouldn't it be quite easy to automate the process of greenlisting dependencies?

No, it would not. For one thing, there is no such thing as a "GPL" license, and in the past (before the LWG took charge of this) we had a lot of erroneously allowlisted entries inserted by people who believed that there was.

Please read the instructions for reviewing allowlist requests, to get an idea of the process. I don't think this is something that can be easily automated.

The rumors of my death have been greatly exaggerated... kind of.

After spending the last year and half shifting away from Drupal/CMS development to Salesforce/CRM/microsite development, I just accepted a new role at the University of Colorado as a Senior (only) Drupal Developer for the Office of Digital Education. So while the amount of time I've been able to justify spending on Drupal.org related issues has been limited, that will change next month as I dig back into Drupal full time.

As far as support for distributions on Drupal.org, I have some thoughts on that after working with deploying non-Drupal PHP and Next.js projects Heroku as to how this might work. I only recently started tuning back into Drupal with the D9 release. We were curious if this release could be deployed on Heroku. It largely just worked. Maybe I just needed a break, but I was really impressed by the progress around drupal-scaffold (we have the option to symlink parts of a codebase again!), layout building and media handling.

Licensing checks have gotten easier at some level with https://getcomposer.org/doc/03-cli.md#licenses and https://docs.gitlab.com/ee/user/compliance/license_compliance/, but I think we need to step back and revisit the distributions that are still active and what tooling on Drupal.org would really help the teams maintaining distributions.

I thought about this and from my point of view, when going away from tarballs, having a distribution specific project (f.e. https://github.com/thunder/thunder-project) available as a package through packages.drupal.org (here: drupal/thunder-project) and represented at the project page (https://www.drupal.org/project/thunder) seems to be the most coherent solution.

Hosting the profile there is sort of misleading, since you cannot really do anything with a profile without creating a specific project depending on it, nor does it replace the tarball.

Unless I'm very much mistaken there doesn't seem to be any way to install a distribution hosted on drupal.org using composer.
I would expect it to be easily possible using
composer create-project drupal/my_project
but that yields
Could not find package drupal/cforge with stability stable.
However all the examples I can find of one line composer commands involve a non-drupal repository.
At the moment I'm at a loss as to how to tell non-experts the proper way to install my profile.
[addendum]
Things improved dramatically when I removed the makefiles from my profile. I can now at least build and distribute a composer file which does the installation.

Okay, so closing out some old issues.

Distributions now have a whole different support model on drupal.org.

All distributions have had their invalid metadata removed from packages.drupal.org.

If a distribution maintainer wishes to use a composer starter template, which has been the pattern for years now, they can create a "General Project", and upload their composer.json for the starter template.

If the project matches the criteria here:
https://www.drupal.org/docs/develop/managing-a-drupalorg-theme-module-or...

It will automatically be submitted to packagist.org under the drupal/ namespace, and will be installable by any user with `composer create-project drupal/distro_machine_name`

Drupal.org will not offer any form of distribution packaging for these general project distros.

Great work guys
I have an old distribution which was created as a 'distribution', naturally.
It doesn't work with composer as there is some problem with version numbering.
https://www.drupal.org/project/cforge
Do I now need to clone it into a 'general project'?
And if so that means changing the name, does it not?
Which implies renaming all my hooks, does it not?
I'll also need to do something about upgrade versions will i not?

It seems necessary to update the text on
https://www.drupal.org/docs/develop/managing-a-drupalorg-theme-module-or...
and
https://www.drupal.org/node/add
does it not?

Please open a separate issue for support, this issue has been closed for months.

In many cases, projects can be changed from one type to another without the need to make a new project with a new name. That would be an infrastructure project issue, since it requires server access to do.

Maybe this is fixed in the narrow project_composer sense but if someone is looking for a distribution of Drupal they will go to Build » Distributions not Build » General. So distributions that are set up the proper, modern way— people won't find them.

Is there an issue where fixing that aspect is discussed?