With Drupal 8 now released, and based on some good discussions had so far in this thread and elsewhere, we have a plan to begin to move forward with composer support, particularly the Drupal.org components. There is certainly additional discussion and planning to be done - but it would be best to focus those conversations in the appropriate metas below.

Meta Issues for Composer Support

#2576285: Drupal.org Composer Service (façade)
#2485011: Infrastructure Requirements to support Composer based workflows
#2622420: Site-builders should be able to use projects with composer dependencies
#2622400: Explore ways to make Composer support more robust

Next steps

  • The initial roll out of the Drupal.org Composer Facade is complete, and Drupal.org composer repositories are now available for Drupal 7 and Drupal 8. Documentatoin is available here: https://www.drupal.org/node/2718229
  • Bug fixes to the alpha release
  • Stable release
  • Drupal Core should be updated to use D.O's composer endpoints
  • Feature enhancements, as outlined in the plan issues above

History

We've been going back and forth in various issues about what the path forward should be for "official" Composer support for Drupal extensions. Hopefully, we can come to an agreement here before we go too much further down any one path.

I've attempted to summarize the arguments here, though I recognize that I may be biased. If you have any other arguments for any of the solutions (or other solutions that we haven't considered), please add to the table

Must haves:

  • Expose Drupal extensions to Composer
  • Survive the load that the Drupal community will put on it
  • Be updated as quickly as possible
  • Easy to use (especially for those new to Drupal)

Nice to haves:

  • Be updated instantly
  • Avoid extra repo lines in composer.json if possible
  • Ideally allow us to notify site owners/developers when they're doing something that has weird licensing implications if we can't outright prevent it
  • Support the wider PHP community (rather than just Drupal)

Summary of options

Solution Pros Cons
Pushing data directly to packagist
  • The standard and official source for composer packages
  • Packagist is where all the other packages are that Drupal already uses (drupal/core, symfony, phpunit etc)
  • Packagist devs seem to be willing to accept new features from us
  • Does not require any additional repo lines in composer.json - packagist.org is the default
  • Embraces the PHP ecosystem
  • Any effort / time / money / support benefits the whole PHP community
  • Increases package count
  • Familiar to PHP community
  • Supported by Packagist / composer maintainer
  • Gives Packagist more visability
  • API on packagist already contributed to by Drupal community for this requirement
  • d.o will soon depend on packagist for packaging core
  • d.o already has a testbot that depends on packagist
  • "Fast and reliable" isn't really promised by Packagist (but this is something we can help with)
  • Adds yet another place that modules are undiscoverable (we would still use Drupal.org for discovering modules)
  • Increases package count, releases, and traffic for Packagist in a pretty significant way - causes a problem for other people (forces packagist to have a more robust infrastructure, with Drupal's support)
  • Pushes many projects (modules, themes, profiles, drush extensions) that aren't at all applicable to the wider PHP community (as so many existing projects)
  • Composer docs say that it may be a good idea not to do this (although they are very must open to it)
  • Requires us to build a queueing mechanism in the event we try to push data to Packagist and the request fails/packagist is down
  • Requires us to fix semantic versioning in contrib modules to something that will work (which should be done anyway)
  • Requires us to encourage module developers to add composer.json to their repos. Module would not be usable by composer without it. (which they should already be familiar with from any other PHP project)

packagist.drupal-composer.org
  • Already works
  • Drupal community supported
  • Does not require contrib maintainers to add a composer.json to their project
  • Relies on scraping data from Drupal.org
  • Unique to Drupal (drupalism)
  • May not be trusted by enterprise users
  • Requires adding a repo to your sites root composer.json for packagist.drupal-composer.org
  • Needs supporting

Serving repo JSON data directly from Drupal.org
  • Very straightforward engineering effort
  • Doesn't require parsing through all the repos (most data is already in db - except composer.json files in the repos themselves)
  • May not require contrib maintainers to add a composer.json to their project unless they actually have dependencies.
  • Updates instantly
  • Leverages the existing infrastructure for Drupal.org (reliable, fast, geographically distributed)
  • Doesn't wholly rely on a third party service (mitigates outage risks)
  • Allows us to continue to track download stats and site analytics
  • Allows us to resolve the subproject problem without having to delete all the drupal projects from packagist and re-create them - we can just change the JSON we're serving
  • Allows us to 'fix' semantic versioning for composer without having to fix every piece of code on drupal.org that makes assumptions about the versioning scheme
  • Could implement licensing checks of dependencies in drupal modules(?)
  • Requires adding a repo to your sites root composer.json for packages.drupal.org
  • May seem unfamiliar / unusual to non-drupal users / Not standard practice
  • Needs supporting
  • Any non-drupal dependencies would still require metadata from packagist.org
  • Unique to Drupal (drupalism)
  • Non-standard implementation (to output standard json)

Pushing data to a self hosted Packagist/Satis instance
  • Familiar interface
  • More control
  • Leverages the existing infrastructure for Drupal.org (reliable, fast, geographically distributed)
  • Doesn't rely on a third party service (mitigates outage risks)
  • Allows us to continue to track download stats and site analytics
  • Relies on a fairly lengthy process to update data from d.o database
  • Another application to maintain
  • Requires adding a repo to your sites root composer.json for packages.drupal.org
  • Unique to Drupal (drupalism)

Comments

cweagans created an issue. See original summary.

webflo’s picture

Issue summary: View changes
cweagans’s picture

I'd also like to add that if drupal/drupal remains on packagist, we can add a drupal-specific repository by default simply by committing it to core (assuming people are using `composer create-project`).

markcarver’s picture

Comments from #2547617: Notify Packagist about repository update:

especially given that the vast majority of our extensions right now are for Drupal 7

something like 20k-25k packages including themes and distros and drush extensions

Very true.

So, inspite of the effort of #2547385: [PR effort] Contrib maintainers should add composer.json (which is basically saying we should "mandate" a composer.json file in each repo), I think it would make more sense to simply "check" if a project's repo first has a composer.json and then, only then, do we push to packagist.

If a project has added a composer.json, they are basically declaring that they're aware of the "composer mentatlity" and is likely to be more abstract in nature. To me, this sounds like the most organic and sensible solution we could hope to attain.

cweagans’s picture

That would mitigate a lot of the problems with pushing a large volume of packages to Packagist, however, I still don't think that would be entirely useful. A Drupal project maintainer (even one that's aware of the Composer Way) is still going to be writing Drupal code. The majority of their module might be some external libraries, but it's still Drupal code in the d.o repo. I don't understand why that'd be valuable to i.e. a Symfony developer or a Laravel developer.

markcarver’s picture

The majority of their module might be some external libraries, but it's still Drupal code in the d.o repo. I don't understand why that'd be valuable to i.e. a Symfony developer or a Laravel developer.

I think there's too many hypotheticals here. Just because someone decides to install a "Drupal" package, doesn't mean they have to use the entire thing. They can choose to use a class or two for their specific needs. The fact that it also a "Drupal" project is of no consequence.

Regardless, pushing repositories from Drupal to packagist is more about simply being able to do the following, natively:
composer require drupal/<project>:<version>

Instead of some other newfangled approach that would:
a) require more from the d.o infra team to support
b) introduce yet another Drupalism to new comers (who likely already use composer) requiring them to add an addition "repo" to their composer.json file

Mixologic’s picture

introduce yet another Drupalism to new comers (who likely already use composer) requiring them to add an addition "repo" to their composer.json file

That isn't a drupalism, thats a composerism. You have to add repos to your root composer.json in order to support pulling packages from *anywhere* other than packagist.org. -This includes any custom modules you'll inevitably create that are not for public consumption. So everybody will be adding repo's to their root file anyhow. I dont see how adding another single line to your root project is all that big of a burden.

And there is nothing newfangled about serving up our own json. Its exactly why there are suggestions in composer's documentation that this be done: https://getcomposer.org/doc/05-repositories.md#hosting-your-own - see the part about separate ecosystem.

cweagans’s picture

I think there's too many hypotheticals here. Just because someone decides to install a "Drupal" package, doesn't mean they have to use the entire thing. They can choose to use a class or two for their specific needs. The fact that it also a "Drupal" project is of no consequence.

Alright, I suppose that's fair.

Regardless, pushing repositories from Drupal to packagist is more about simply being able to do the following, natively:
composer require drupal/project:version

Right, and that's fine. If we commit the repo URL directly to Drupal core, if somebody does `composer create-project drupal/drupal`, they'll have the repo pre-configured and will be able to do that. If they are setting up their composer project manually for whatever reason, they they have a one-time change to add the repo, which is really not that bad of a thing, IMO. We have to do it at work anyway because we have an internal Satis instance for some of our non FOSS code, so having to add one more repo isn't that much more work.

To your points:
a) I'm not sure that's the case. There's at least one hurdle to consider with pushing to Packagist: we'd have to build some kind of queueing mechanism for the data that gets sent. If packagist is down or unreachable, we want to be able to retry that request later and not just silently fail (thereby letting Packagist continue running with stale data). We may already have a mechanism to do this, but if not, that's something the infra team would be responsible for.
b) Drupalism, maybe. As I said, at my day job, it's pretty rare that we don't have to add an extra repo to a composer project, so I don't really agree. I'm sure it's clear by now, though, that I think the one extra step is justified in this case, given the benefits.

webchick’s picture

IMO if a module has one or two classes that are of value outside their module, the "Newfangeldy PHP/Packagist Way™" would be to make that class/es a separate library—most likely on GitHub, until/unless #2474007: Add a "library" node type to drupal.org is done—and send that library to Packagist, then reference it from their module's composer.json. Symfony bundles that wanted to use it would do the same. This is effectively the model that Drupal Commerce is embracing in their 8.x version.

I was originally on board with this vision to push all Drupal projects to Packagist because getting off the island FTW, but #2547385-41: [PR effort] Contrib maintainers should add composer.json makes some very compelling points. The official docs seem to be bold-face telling us not to do this, since Drupal-specific code is generally of little to no value outside of Drupal, same as WordPress plugins are of little to no value outside of WordPress. My read is that if there is code in there of value outside of $framework, it should be a separate Packagist library on its own.

As far as this issue goes, I think we do ultimately need a *.drupal.org solution, I'm pretty ambivalent to what it is. That enterprises are cool trusting external sites run by some random developers (well, not random, but :)) is actually quite shocking to me.

Mixologic’s picture

Issue summary: View changes
cweagans’s picture

Dunno if I've mentioned in this issue or not: Mixologic was kind enough to create a dev site for me, so I'm going to work on a module that can serve the package JSON data directly from Drupal.org over the weekend. Essentially, it'll be the same data that Satis or Packagist generates, but ours will come directly from the d.o database. So basically, we'd have https://www.drupal.org/composer/packages.json and a bunch of "child" JSON endpoints that do the same thing as https://packagist.org/packages.json and associated URLs. I already checked and the d.o database already has information about dependencies and version constraints for every Project node, so it's literally a matter of getting the data out of the db into the right JSON schema and outputting it somewhere.

Just wanted to put that out there, lest somebody think I was yelling at other people to do different work. :)

I think that this will allow us to not have composer.json in Drupal.org repos. The only real reason it's required for Packagist is so that it can go download your repo and look at composer.json on every branch to gather info about the project. If we already have that data and can generate the repository JSON without digging through every repository, that makes *all* Drupal packages available to composer, albeit through a separate repo.

Crell’s picture

*sigh* Copying my comment from the previous issue:

Symfony, Zend, Cake, and many other project-specific packages live on Packagist. Most of the major frameworks now distribute themselves and their add-ons via Packagist. If the composer docs say you don't need to do that, well, no one is listening to them. :-)

This is also about enabling Drupal site owners/builders to setup and manage their Drupal site entirely through Composer/Packagist. That means even if we weren't using Packagist.org, we would still need to have composer.json files in the repo so that Composer could handle dependencies properly.

Mixologic’s picture

From an infrastructure standpoint, we would ideally be serving static json files - i.e. not a module that dynamically generated the json per request. I was looking at satis - it seems to create exactly what we need - a static site of all the composer.json files. The main issue is whether or not we can process a satis.json file fast enough for the 100,000 or so releases we have. But even if satis isnt the right approach, perhaps all that needs to happen is to piggyback off the work in https://www.drupal.org/node/2547617, but instead of posting to packagist.org's api, we output the static json needed for the individual project. This is not a lot of custom code, nor is it much to maintain. Nor does it preclude us from, at some point in the future, switching over to using packagist.org if the politics demand it.

webflo’s picture

I think that this will allow us to not have composer.json in Drupal.org repos.

But please don´t forget other use-cases. Modules should have the ability to provide their own composer.json with external dependencies.

timmillwood’s picture

Issue summary: View changes

There seems a lot of negativity towards Packagist:

"Fast and reliable" isn't really promised by Packagist

It seems every other PHP project manages well enough

Adds yet another place that modules are undiscoverable

Packagist won't be a place to "discover" drupal modules, it will just be a place for composer to grab the module info from.

Increases package count

After speaking to Jordi (packagist / composer maintainer) this is one of the major plus points to him for us using packagist

Pushes many projects (modules, themes, profiles, drush extensions) that aren't at all applicable to the wider PHP community

As do many other PHP projects

Requires us to build a queueing mechanism in the event we try to push data to Packagist and the request fails/packagist is down

I don't think this is a requirement. No other package listed on packagist has a system like this incase packagist is down.


To me any non-packagist solution for listing drupal modules for composer to consume seems crazy.

Mixologic’s picture

Crell: there is zero doubt that packagist composer will be the way forward. This particular issue is to hammer out who's going to serve the json. I strongly support the idea that drupal.org infrastructure team handle that responsibility, primarily from an infrastructure and load perspective. We've already got all the groundwork in place to support massive amounts of requests to static files (updates.drupal.org gets hit by 1.2 million sites a week, with about 40 million total requests ~ 12 TB of data). If drupal 8 takes off, and composer becomes the defacto way you extend/update your site (including some tool for sitebuilders) then there will be a tremendous amount of traffic hitting packagist.org. Those 1.2 million sites that hit us every week? about 250,000 of them turn out to be transient sites - dev/test/simplytest.me/testbots/travis and any number of reasons to automatically build a drupal site beyond just making a site. packagist.org should prepare for an additional 10 million hits a week, just from our symphony packages alone, nonwithstanding the amount of drupal modules they would then additionally have to serve.

Another reason for us to handle this is we are not dependent upon any third party to work with us on translating our current situation that isnt supported by composer, into something that is: semantic version numbers, lack of composer.json in the repos, submodules and subtree splits - we can work on those features until we've reached parity with the rest of the php ecosystem without relying on packagist.org to be responsive to whatever our requests are.

derhasi’s picture

"Fast and reliable" isn't really promised by Packagist (suggests Toran Proxy)

With Toran Proxy there is a "Fast and reliable" solution. In addition, the whole PHP Community would benefit from improvements for Packagist.

Adds yet another place that modules are undiscoverable (search mailchimp on Packagist - what do you see?)

That's no Drupal specific problem. Here again, we can help to make the situation better on Packagist.org, by improving the search. This again would not help only Drupal but the whole PHP Community. The drupal.org search isn't perfect either.
In addition you have one place you can find a lot of PHP projects (not only Drupal ones). This is very helpful for finding projects that deal with the same problem, you want to deal with.

Increases package count, releases, and traffic for Packagist in a pretty significant way - causes a problem for other people

See above. Improvements to packagist.org will help everybody.

Pushes many projects (modules, themes, profiles, drush extensions) that aren't at all applicable to the wider PHP community

Why does a package have to support multiple frameworks? Packagist is a tool for package information and discovery. There are a lot of packages that are dependent on a Framework or CMS. Therefore you have dependencies in Composer.

Composer docs say that it may be a good idea not to do this

It doesn't. It says you could do that if you want to. From the communication with Jordi it is very clear, that Drupal's ecosystem is very welcome to packagist.org.

Requires us to build a queueing mechanism in the event we try to push data to Packagist and the request fails/packagist is down

This a solvable task and not comparable to setting up an own reliable

lack of composer.json in the repos

@timmillwood was very succesful with adding a new API endpoint to packagist. We even could initiate another one, that could allow to post package information directly to packagist.org. Again, this would be an improvement to Packagist, that would serve a broader community.
With that approach, other package systems could be easily integrated too.

If we put effort in improving infrastructure, I think it is better to do it for a broader audience.

timmillwood’s picture

there is zero doubt that packagist will be the way forward. This particular issue is to hammer out who's going to serve the json.

What does this mean? How can packagist be the way forward but not serve the json? (Assuming you mean packages.json because composer.json will be in the git repo)

I think the numbers are slightly over estimated, packagist will get pinged less if a composer.lock is present or if toran proxy is used (suggested for LSD (Large Scale Drupal) organisations).

Ultimately Drupal is a PHP project and should work like one.

webchick’s picture

Crell, if that's the case, and we're not committing some kind of horrible wider PHP community gaffe by single-handedly bloating the number of packagist projects by 25% and sends another 250K+ hits per day 10M hits/week (per Mixologic in #16) to packagist overnight, with code that's basically useless to all other PHP projects, that's great. I guess I would just caution though that Drupal modules are a lot closer to WordPress Plugins than they are Zend/Symfony/Cake add-ons, since they include UI components, markup, hooks, and various other things that make zero sense outside of Drupal, not just straight framework code like those other low-level projects. I could definitely get behind pushing all Drupal Components to Packagist automatically, which I think would be a lot closer correlation. (Again, requires #2474007: Add a "library" node type to drupal.org to clearly separate these from modules.)

It would be great to provide Jordi with all the facts of what he'd be walking into here, with a firm yes/no that this is fine to do it. If it is, then I agree using the thing everyone else uses for distributing library dependencies in Composer seems like a no-brainer to me.

I'm reading the cons column of "Pushing data directly to packagist" as well as some counter-points made in other comments, and am a bit confused.

1) We would not be pushing Packagist to anyone as a method of discovering modules. We would continue to keep project module, project browsing pages, release tarballs, etc. centralized here on Drupal.org. Packagist just becomes a standard tool for various modules' composer.jsons to get their stuff from without the need for special workarounds/hacks.

2) Module usage stats are a wash no matter where people get their modules from. They're generated from actively enabled modules on public Drupal sites with Update Manager module turned on. Update Manager doesn't care where you got the modules from, only that it can tell what release node they are.

3) Download stats are useless as they are now, since they don't capture git clones, which is where a non-trivial amount of downloads come from.. So making them slightly more useless isn't really a reason not to do it for me.

4) I agree that I don't see why we'd need a custom-engineered queueing mechanism if all the other 500+ PHP projects of the world don't need one. If anything, we should dedicate some infrastructure team time to making packagist more robust/stable for all PHP users.

kreynen’s picture

Drupal modules are a lot closer to WordPress Plugins than they are Zend/Symfony/Cake add-ons

And WordPress has http://wpackagist.org/

I think it would make more sense to simply "check" if a project's repo first has a composer.json and then, only then, do we push to packagist.

But we wouldn't push to packagist (or even list in some Drupal specific repository) before we vet the chain of dependencies for licensing and security issues... right? right?

I don't see how the DA can get behind any approach that allows developers to add dependencies in the composer.json that aren't GPLv2 compatible.

Let's say I create the D8 version of https://www.drupal.org/project/highcharts and include composer.json with a dependency on https://packagist.org/packages/kwizer15/highcharts. If @cweagans installed that for NBC, they'd owe thousands of $$ in licensing since NBC can't legally use this code under the Creative Commons Non-Commercial use license. Install a Drupal project with a AGPLv3 dependency like https://packagist.org/packages/stylecow/stylecow, you could be asked to share all the code that drives your site or SaaS regardless of whether you are distributing it or not. While those projects are using the correct licenses in their composer.json, I've found dozens of that don't.

When it comes to licensing, Packagist.org is the wild west.

cweagans’s picture

Re #12 and #14: As explained in #11, I don't think composer.json necessarily needs to be committed to the repo to do dependency resolution. Composer relies on the repository data to figure that out, not local composer.json files. If our package repository generates that from the database, then that requirement is no longer a concern for many projects. The only ones that would have to deal with it are the ones that depend on libraries from Packagist. For those projects, we could very easily get that data from the repositories into the d.o database and output it along with the module dependencies in our repository json.

Re #13: This data should be highly cacheable if it's split into many separate endpoints. I agree that we wouldn't want to generate it on every request, but I don't think it's unreasonable to generate it for the first request, cache it, and only invalidate the one or two files that change when a project has new tags or branches.

Re #15:

After speaking to Jordi (packagist / composer maintainer) this is one of the major plus points to him for us using packagist

Is he aware of the volume that we're talking about? Not just projects and releases, but bandwidth as well. It may not be obvious to somebody that's not overly familiar with Drupal.org. I agree with webchick - we need to give him data to make a decision, not a general "Hey, should we do this thing?".

As do many other PHP projects

Sure. IMO, though, they're doing it wrong. That doesn't mean we should.

I don't think this is a requirement. No other package listed on packagist has a system like this incase packagist is down.

It absolutely is. I haven't been able to find where I saw this, but I saw a video somewhere of a Github engineer that said that they have an internal queue/retry system for service hooks and web hooks. Since I couldn't find that source, I tweeted at Github for an answer (https://twitter.com/cweagans/status/632262946819731457). Given that most packages are hosted on Github (and assuming that retry system exists), they all get the benefit of that system.

In any case, we are different from other projects in that we have a security team that coordinates security releases when necessary. When security updates are available, site owners need to be able to update instantly. Packagist crawls packages once per week, so if we happen to attempt to notify Packagist of a new branch or tag while Packagist is down or otherwise unreachable, that branch or tag wouldn't be available through Composer for up to a week unless we have some way to retry the notification. That's kind of bad, especially for security releases. We need people to be able to update their packages without waiting on some other system.

Re #18: I think he meant "Composer", but it's possible he meant a self hosted Packagist instance. Also, we don't use Toran Proxy at work, and I really don't like the idea of forcing people to use commercial software to get better reliability for downloading FOSS code. That's ridiculous. You also said in #16 that it's good enough for other projects - why is Toran Proxy even a thing if there's not some issues with reliability?

Re #19:
* +1 for pushing really generalized code to Packagist. That makes a lot of sense. I don't get why we should push Drupal specific code, though, especially if we're encouraging people to generalize what can be generalized and push those pieces to Packagist.
* Point 1: The Drupal.org UI wouldn't be going anywhere - that's true. However, where do Symfony developers go to find extensions? The search box on Packagist. Ditto for Laravel and what I can only assume is all of the other frameworks. Drupal.org certainly has more specialized information, but packagist has an autofiltering list of projects that's much faster to work with. I think there's going to be a lot of frustration around trying to find Drupal packages on Packagist if we go this route, but I don't have any data to back that up.
* Point 2: Usage stats, yes. But we also have analytics data for users on Drupal.org and their behaviors, and we lose that if people are going somewhere else for modules.
* Point 3: Fair. Unfortunately, there are some orgs that don't see it that way. I worked for a client last year that required at least 25,000 downloads before they'd even consider using a module. It was really strange, but they figured that if that many people had downloaded the module, the kinks are probably ironed out of it. Unsurprisingly, they had no problem with custom code.
* Point 4: See above.

#20: Care to update the table in the issue summary with your licensing concerns? :)

cweagans’s picture

Regarding the composer.json dependency parsing at the top of my last reply: we could also alleviate some of @kreynen's licensing concerns by checking the license of dependencies specified in a project's composer.json. Packagist makes that really easy for us to do: https://packagist.org/packages/cweagans/composer-patches.json

timmillwood’s picture

@cweagans - it sounds a lot that you're trying to find issues and work around that don't really exist.

cweagans’s picture

Security and licensing issues sound like problems to me. Blowing up some other project's website sounds like a problem to me. You can be dismissive, but it's not just me that's arguing against Packagist. Sorry you feel that way.

derhasi’s picture

The licensing concern does not depend on Packagist.org being the source. Even with a custom source or an own packagist, this is still an issue. We already have that today with drush make files and modules utilizing "Composer Manager".

Putting infrastructure ressources in packagist.org instead of building a packagist clone would solve most of the issues mentioned.

cweagans’s picture

Note that I'm not actually building a packagist clone. I'm just outputting JSON, and the resources required to do it are minimal and more importantly my own to allocate where I please. If it doesn't get used, that's fine, but I want to get it working at least. Talking about hypothetical functionality is difficult at best, and I'd rather have at least a working demo when a decision is made.

In any case, it sounds like the next steps here are:

1) Get a firm go/no-go from Jordi after supplying relevant data
2) Decide which projects to consider for any of the above solutions (just the ones with composer.json? All of them?)
3) Gather input from people using Composer for Drupal development today about what their preferred process would be
4) Get a recommendation/decision from DIWG? DSWG? Both? about which direction to go

Is that a fair assessment of where things are at? Missing anything?

timmillwood’s picture

Security and licensing issues sound like problems to me.

It will be an issue no matter the result of this issue.

Blowing up some other project's website sounds like a problem to me.

Speaking to Jordi today, this doesn't seem to be an issue either.

it's not just me that's arguing against Packagist.

It sounds like it.

1) Get a firm go/no-go from Jordi after supplying relevant data

We can check again, but time and time again it seems yes

2) Decide which projects to consider for any of the above solutions (just the ones with composer.json? All of them?)

Only modules with composer.json would be accepted on packagist

3) Gather input from people using Composer for Drupal development today about what their preferred process would be

At Pfizer we are working toward building and deploying sites with Composer. Currently with packagist.drupal-composer.org, but I see that as a workaround until the modules we need are on Packagist.

cweagans’s picture

In general, I find it pretty annoying that technical reservations are met with ideological arguments. "We should use Packagist because we should use Packagist/because everyone else uses Packagist/because any other way is wrong" is not a good argument. We all have the same goals here: Drupal sites should be easy to build with Composer. Let's come up with the best way to make that happen, and not just the one that we "should" use to suit some ideological standard.

--

It will be an issue no matter the result of this issue.

That's simply not true. There's some security risk to site owners if we don't have the package repository JSON updated more or less instantly. At least one solution in the summary addresses that (including building a system to retry requests to Packagist, btw). The licensing thing can also be partially solved in the same way that we solved it for Drush Make - just have a whitelist for dependencies. However, that only works if we control the code that generates the package repository JSON.

It sounds like it.

If it was just me, we wouldn't even be discussing this - I'm at least humble enough to not argue something when I'm the only one that holds an opinion. At least Mixologic, bojanz, and myself are in favor of not going with Packagist, and given that this issue is only a few hours old, I wouldn't be so sure there's not more. I only learned about the fact that this work was happening because I happened to be in IRC when it was being discussed.

We can check again, but time and time again it seems yes

Nobody has confirmed that Jordi has seen the data that Mixologic provided. That's a shitload of bandwidth to deal with. I realize Packagist is already pretty big, but we need to be really really sure that it's not going to fall over when/if we start using Packagist in earnest.

Only modules with composer.json would be accepted on packagist

And if we go with a different solution? We can sidestep the issue of getting hundreds of maintainers to add a composer.json to their repo by committing it for them in an automated way, by integrating d.o with a self-hosted packagist instance and using the dependency and version info we already have, or by serving the json directly from d.o. Why spend the time and effort on a PR campaign for a problem easily solved by a small amount of code?

--

On a separate note, I'm sensing some hostility from you, Tim. I know it's frustrating to have people push for changing the plans that you were already excited about and in the process of implementing, but we need to make the right choice here. I think everyone in this issue agrees that managing packages through Composer is an important and generally good thing. There's also at least 2 (myself and Mixologic - maybe others) that think there are good reasons for at least exploring other approaches before committing to one that may or may not work for what we need. Anyway, that's a roundabout way to say that if I pissed you off by opening this issue, I'm sorry.

webflo’s picture

Issue summary: View changes
webflo’s picture

Issue summary: View changes
cweagans’s picture

Here's another idea: what if we submitted a PR for federation of repository metadata? That is, we host our own package metadata in whatever way we deem acceptable, and Packagist has some configuration option to say "If a package doesn't exist in my local metadata and starts with "drupal/" look over here at this other server (the Drupal.org metadata).

This way, we don't have to have any repo lines in sites' composer.json, we don't pollute the main packagist with things that aren't relevant, we can still rely on Drupal.org's infrastructure for serving that data, and we can solve the subproject issue when we are inclined to do so. This could be useful to the Wordpress packagist, too - same idea, different package repository.

Just trying to come up with other options.

cweagans’s picture

Issue summary: View changes

Updated the issue summary with all the requirements that I could find from various issues.

Mixologic’s picture

there is zero doubt that packagist will be the way forward

I'm so sorry. I really did mean to say there is zero doubt that composer will be the way forward in #16

Packagist.org is a service, composer is a technology. We should embrace open source technologies, but it does not follow that we have to embrace a service because it is provided by somebody who has written valuable open source code. We are not working with, or 'supporting' monetarily or otherwise the apache foundation, the linux foundation, the freebsd foundation or any other myriad open source technologies that we rely on, so why should this particular aspect of the open source world be any different?

Sentimentally, I agree that Jordi deserves to be compensated for the tremendously valuable tool he's created for the entire php community. Technically, Im flabbergasted that major php projects would rely on packagist.org to be a single point of failure. How robust is the security of packagists' servers? Is Jordi not only an awesome developer but is he also a security guru and infrastructure specialist ,and packagist.org is bulletproof and crashproof? Its an incredibly high value security target - its a server that tells sites what code to install. I don't want to see the day when drupal sites are running malicious code, and they blame drupal because packagist.org got hacked and the fellow who is running it (who hopefully never takes a vacation) was unable update his server fast enough in response to the latest heartbleed/poodle/shellshock type of vulnerability. Yes, I am fully cognizant that this sounds like FUD, but I also believe that these concerns are valid, particularly when looking at the alternatives of simply supplying the json ourselves.

If the entire drupal project is going to depend on an outside service to function, we ought to have service level agreements guaranteeing uptime, and we should know how much that will cost us so we can decide if paying packagist.org for reliability is worth it (or forcing companies that need that reliability to buy it ), or if asking our developers to add an additional line to their composer.json files is worth the cost.

skwashd’s picture

Disclaimer: I work with Tim, but this is posted as my personal opinion.

3) Gather input from people using Composer for Drupal development today about what their preferred process would be

I have been using composer (and composer manager) on Drupal projects for at least 2 years. I maintain the github API module and comaintain AWS SQS. I use composer for dependency management on internal projects too.

Why? It is simpler and easier than keeping track of libraries in a make file and hook_libraries_info().

Adding a composer.json to a repo is a pretty small task compared to most of the effort involved in porting a module to D8. This small effort means you're adding your module to a larger eco system.

webchick’s picture

Is Jordi not only an awesome developer but is he also a security guru and infrastructure specialist ,and packagist.org is bulletproof and crashproof?

Not saying packagist doesn't carry risks, but are you implying Drupal.org has never had security incidents, has never had performance/scalability problems, and has never had downtime? I'm highly confused by this line of logic. We don't have an SLA with OSUOSL either, nor do either they nor the DA employ security experts on staff, so...

I keep hearing how overwhelmed the D.o tech team is, so I am a bit mystified why you are trying to take on more things when there's a service that might do the job for us that seems good enough for everyone else, too.

Mixologic’s picture

Im not implying that we've never had security incidents, downtime, or performance and scalability problems, on the contrary, its because we have already had those problems, and have already taken the necessary steps to mitigate those issues, that I feel more confident in the resiliency of our infrastructure. While we don't have anybody who would claim to be a security expert on staff, we do have a security team, infrastructure working group, and many security conscious folks who have looked at our infrastructure, identified weaknesses, and helped us strengthen what we have.

As far as taking on more things, the cost/work/effort of developing a solution to serve json metadata about drupal projects is equivalent to the cost/work/effort of building a solution to send that same json metadata to somebody else's api so they can serve it. I don't expect the engineering effort to implement or maintain something like this to be significant.

The primary cost will be associated with the bandwidth and server load associated with moving to composer, just like there is a cost associated with our updates traffic now. I speculate that in less than two years, that bandwidth and load on whatever server is going to respond to that data is going to be significant and have a very real cost associated with it. If our updates traffic is a reliable indicator, the additional volume of traffic will eventually be at least double what packagist.org is responding to currently.

So, who pays for that? If its being served from drupal.org, and the Association pays for it, it adds one more piece to our value proposition pie, and it becomes another way we serve the community, and therefore another reason to support us. If packagist.org is serving it, then the only (current) way to pay for it is to encourage larger drupal shops/sites to become toran proxy customers. Either way, that cost isn't going to go away, and the community is going to need to pay for it somehow.

The risks of packagist.org are magnified to me by the fact that (as far as I can tell) it is one person behind it. It's not a company or organization that we're signing a contract with (like we have with OSUOSL, Fastly, Edgecast, ServerDensity, PagerDuty, etc). Perhaps Im completely wrong here, and packagist.org is an entire organization, but the first person tweets and blog posts (http://seld.be/notes/composer-hosting-improvements) lead me to believe otherwise.

So while there is a service that everybody else may be comfortable with, Im questioning whether its wise for the drupal project to rely on a service that is essentially one person. Perhaps the hope is that a project of our magnitude will bring in enough revenue and customers for packagist.org/toran proxy to be able to grow and become more bulletproof/redundant/secure, which would in turn help out the entire php ecosystem. At the same time, even if that were to happen, would it be wise for the entire php ecosystem to be wholly dependent upon a single business?

Much of my argument is probably muted by the fact that any d8 site out there is going to need to hit packagist.org to get the metadata for all the other packages we're already using, so perhaps Im tilting at windmills here. (though perhaps we proxy/cache that metadata as well).

webflo’s picture

Issue summary: View changes
webflo’s picture

Regarding licensing. Composer has a build in command to get a list of all installed packages and their licenses. Run composer licenses in your project.

Name: drupal/core
Version: 8.0.9999999.9999999-dev
Licenses: GPL-2.0+
Dependencies:

Name                               Version             License
behat/mink                         v1.6.1              MIT
behat/mink-browserkit-driver       v1.2.0              MIT
behat/mink-goutte-driver           dev-master cc5ce11  MIT
doctrine/annotations               v1.2.1              MIT
doctrine/cache                     v1.3.1              MIT
doctrine/collections               v1.2                MIT
doctrine/common                    v2.4.2              MIT
doctrine/inflector                 v1.0                MIT
doctrine/instantiator              1.0.4               MIT
doctrine/lexer                     v1.0.1              MIT
easyrdf/easyrdf                    0.9.0               BSD-3-Clause
...
...

or as json with composer licenses --format json

Mixologic’s picture

Also regarding licensing: composer supports both source and dist packages. We can specify the tarballs to be downloaded and installed in the composer.json file, which means we dont necessarily have to include the license files in the repos if a user builds their site with --prefer-dist, they will get the corresponding licenses. If they use --prefer-source, they would not, but they dont currently get them on a git clone either.

Crell’s picture

The single point of failure concern for Packagist is valid, and Mixologic is not the first to bring it up in the PHP world. As is the case for most Free/free infrastructure, figuring out how to solve it requires money, which is often not readily available. Toran Proxy is one model being used to try and address that.

Personally, I'd love to see us spend our time/effort/resources figuring out how we can help solve that problem. If we would be as massive a contributor to Packagist as Mixologic says we would be, let's figure out how to help Packagist scale with us (and therefore helps scale for everyone else, too). That's where we should be putting our thought process.

cweagans' suggestion of helping Packagist support "Federating" sounds promising. Would it work? Would Jordi go for it? I have no idea, but I think it is worth investigating because if it does work out that would be beneficial to the whole PHP ecosystem (think Joomla and WP hosting their own federated instances, too), which is both cheaper for us in the long run and major, major marketing karma for us with PHP developers. :-)

It also doesn't preclude "on git push notify packagist" hooks for now to just do vanilla integration with Packagist for the time being. Transitioning to Composer is, I think we all understand, going to be an incremental process, and the tarballs are not going away anytime soon. So a simple and cheap integration for now, even if not perfect, is still a step forward and should not preclude a more robust, federated, integrated, scalable, and other that other good stuff solution later.

cweagans’s picture

I spent the weekend writing a module for Drupal.org that exposes Composer repository JSON directly from Drupal.org. There are still some small issues that I'm going to resolve tonight, so I won't link to it here just yet, but I should have another update tomorrow or Wednesday with some more comprehensive information. It's very small (< 300 lines), as it relies on the project* modules for the vast majority of it's functionality.

Right now, it's able to expose package metadata for every full project node on Drupal.org without requiring the module maintainer to add a composer.json to the repo, and it's metadata is organized such that most of the output can be cached on our CDN and selectively purged when certain events happen on Drupal.org (i.e. module is published or promoted from a sandbox, new release of a project, etc), so performance isn't a concern. It doesn't currently include support for subprojects (I think that data might be in the database, though - if it is, this is very easy to do), and it doesn't list any package dependencies (for d.o projects or otherwise). Those things are doable, and there's a clear path forward on how to make it happen, but it's a bit time consuming, and I don't want to invest the time if we aren't going to go down this path. More on that later.

Right now, we can't just notify Packagist about our modules. Well, we could, but it won't work. We don't use a versioning scheme that Packagist will play nicely with. That fact alone is why Drupal Packagist had to change the version numbers in the parser they wrote. This is a hard blocker, and won't be resolved until the very bikeshed-y semver issue is ironed out for contrib stuff.

I think we can also completely rule out a self hosted Packagist or Satis instance for the same reason (semver). In addition, the amount of time it would take for Satis to build metadata for 10s of 1000s of projects makes it not really a feasible option.

Mixologic and I discussed this on IRC last night, and this is the path forward that we'd like to suggest:

  • Finish the composer metadata module so that it outputs the same (or similar) information as packagist.drupal-composer.org (estimate: 2-3 weeks of 1-2 hrs/day work on it. Faster if I can find more time to spend on it)
    • This involves extending project_dependency to parse composer.json files if they're present in a repository, which we are going to have to do anyway if we only want to notify packagist about repos that have composer.json in them.
    • This also requires locating subprojects in a repository and generating the appropriate metadata for them (just a "replaces" section in the package definition)
  • Resolve the semver for contrib issue (estimate: DEAR LORD HELP US)
    • This is going to involve quite a lot of work on many critical pieces of Drupal.org, and it may take a while.
  • Remove the version handling stuff from the composer metadata module (so that it's just using the branches and tags as they exist in each repo - right now, it's translating i.e. 8.x-1.0-beta1 to 8.1.0-beta1. I reviewed this semver issue today and that's not really in line with what the consensus there seems to be, so I can change this. It's not hard.) (estimate: 10 minutes when semver is resolved)
  • At this point, we can go one of two ways:
    • Look into building that federation capability into Packagist, and then having Packagist defer to us for any drupal/ package that isn't already on Packagist; or
    • Do a bulk push of all modules to Packagist and turn on Git notifications so that they stay updated.

I'd prefer the federation thing, because the other option has the possibility of being slow to update for security releases, whereas the metadata from Drupal.org is served directly from the d.o database, so we can purge the appropriate caches as soon as a security release is published and it'll be instantly available to anyone using composer.

Note that the distro/licensing related bullet points in the requirements section of the IS are partially solved by adding composer.json parsing capabilities to project_dependency. We can display a notice on the module page if a module depends on i.e. an AGPL licensed library saying "Installing this module may require you to make the code to your site publicly available to comply with licensing requirements". OR - we could disallow things like that entirely. Having that control makes it a much nicer experience for random people downloading stuff from Drupal.org.

Again, I'm willing to continue working on the metadata module and get that working if there is consensus on going that direction. When/if it's deployed to drupal.org, the repo URL will be "https://www.drupal.org" (which makes it seem pretty official).

I'm also willing to collaborate with the Packagist team and work on the federation PR I proposed earlier. If/when that gets into Packagist, people will no longer have to add the repo URL to their composer.json to download Drupal.org packages.

Generally speaking, I think this is a good way to make sure the data is right on our end before sending it to Packagist. Once that happens, there's no going back, so we need to make sure everything is correct before we pull the trigger. If we don't send our data directly to Packagist and opt for the federated approach instead, we can work on the Packagist PR and enable it even before the semver issue is resolved.

We should talk to Jordi about whether or not he'd be open to doing federated metadata with Packagist.org, but if so, I think this plan satisfies all of the requirements found here and in other issues. Thoughts?

Crell’s picture

Questions regarding #41:

* I'm still not sure why we think Packagist is going to be slow for security releases. For the few Packagist packages I maintain via GitHub, Packagist is aware of a new tag within seconds, generally. (I'm assuming all systems operational, which would be an assumption in whatever setup we end up with.)

* Many modules will need a composer.json file anyway to specify non-Drupal dependencies, as there's no other way to do so. Would we simply merge auto-gen information in with that? If so, what do we do in case of conflicts? (I'd expect the existing file to always win.)

* I'm unclear why we think it's a good thing to auto-generate the composer.json file for modules. I'd almost prefer to have module developers do it themselves so they get used to it, and can specify dependencies themselves, etc.

* Module Semver: Sigh. That is all.

* Federated approach: I'll ping Jordi and see if he's open to it, or if there's prior discussion.

cweagans’s picture

I'm still not sure why we think Packagist is going to be slow for security releases

It's not a matter of being slow. If Packagist is down when a new security release is published on Drupal.org, that difference in data may not be reconciled for up to a week (Packagist re-scans every project once/wk).

Would we simply merge auto-gen information in with that? If so, what do we do in case of conflicts? (I'd expect the existing file to always win.)

We'd only be getting package dependencies and suggested dependencies from the composer.json. The actual package vendor and name, we don't want changing based on what's in the repo. It should always be drupal/panels (or whatever). Perhaps we can allow overriding certain things (maybe the package description?), but then again, we're pulling most of the stuff that can change from project_dependency anyway, so the module author still has control over it.

I'm unclear why we think it's a good thing to auto-generate the composer.json file for modules.

We are not autogenerating a composer.json for modules (not with the current code anyway - if we want to do that, I think it's a separate discussion). This work doesn't affect what the Git repositories contain at all. We're generating the repository metadata based on the information we already have in the database (which, for a lot of the data, is parsed from files in the Git repo). This is not the JSON that you'd put in composer.json in a library. It's the JSON you find at https://packagist.org/packages/cweagans/composer-patches.json (as an example). This is how Composer discovers packages from a repo.

Doing it this way makes all Drupal modules installable by Composer on day one, and not just the ones that have committed composer.json to every branch in their repo. Dependencies can be more or less autogenerated for dependencies on drupal projects (i.e. panels depending on ctools - we already have that data in the drupal.org db). For external dependencies, let's use http://cgit.drupalcode.org/samlauth/tree/composer.json as an example. I shouldn't need to put all of that JSON in there, because most of it is either a) static (type, license), b) easily generated from the project name (issues, source URLs, homepage), or c) written elsewhere (module name, description). The important part of that file (for our purposes) is the "require" section, which we'd just merge into the repository metadata for the samlauth project. If samlauth.info.yml had "devel" (as an example) listed as a dependency, we'd also put drupal/devel in the repository metadata as a dependency of drupal/samlauth.

Module Semver: Sigh. That is all.

Yes. Yes, indeed.

Federated approach

Happy to participate in any conversations related to that, if needed. If it would help, I can dig into the Packagist code and do some research about what would be required for this, so that there's a better idea of any additional complexity that would come from this. If it turns out to be really simple, maybe he'd be more inclined to say yes.

Seldaek’s picture

Howdy.. first of all a disclaimer, I did not read *everything* that's above, so apologies in advance if I repeat something that's been said, but Crell asked me to join in so here are my thoughts.

Sync issues

cweagans's claims about security delays and overall sync issues are a bit exaggerated IMO. If this is so critical, make sure that drupal.org services detect connection/sync failures and retry. That should take care of any potential packagist downtime or cross-server link issues.

Federation

I am vaguely open to it, but I can't say that I gave it much thought. I think it could go two ways: either packagist reads/syncs the data from an external repo, that's been suggested in the past and would work I guess, but it's pretty much the same as you pushing data our way. The other option would be real federation where we say in /packages.json that drupal/* should be looked up at that server. That way however leads to a few downsides:

1) more network traffic to go check another repo, but only for things interested in drupal/*.
2) opening pandora's box, because once we have that feature I don't wanna know how many other projects will want to be a special case.
3) you mention packagist as a single point of failure, and I am not going to argue with that, but federation would effectively expand the attack surface and overall failure potential, as most likely once composer projects are available to drupal modules people will start relying on non-drupal code as well, so drupal projects would need both packagist *and* d.o to be up and running to update successfully.

Semver

I am not very aware of that debate, but here are my 2c as I understand it. I think you have module versions like 8.x-2.1 for a v2.1 module that is built for drupal8, right? In composer world (and npm and whatever else really..) that would be a package with v2.1 and a require on drupal v8. Now I guess the problem is you have two different v2.1 that require different drupal 7 and 8? And that is a proper mess indeed. I am not sure how to fix that except abandoning all past releases and just saying that anything with a composer.json will be composer-visible and has to specify a drupal require and follow semver. For that reason alone I would advise against auto-generating composer.json (except as a one-time assist for module maintainers if that helps move things faster) on the fly in the repository generation. I think a clean break would be the easiest really, it'll take effort and time but at least you won't end up with a mutant solution.

cweagans’s picture

cweagans's claims about security delays and overall sync issues are a bit exaggerated IMO. If this is so critical, make sure that drupal.org services detect connection/sync failures and retry. That should take care of any potential packagist downtime or cross-server link issues.

That's one of the solutions discussed above. Personally, I think that queue/retry mechanism adds more complexity than just serving the repository JSON ourselves, but that's just me. I might be slightly exaggerating the severity of that issue, however, we had a SQL injection vulnerability last year, and for sites that weren't patched within 6-7 hours, the advice was to just consider the site a lost cause and start over from scratch. If Packagist was down when we published the security release, how long would it have taken to get Composer working again? I don't know and any guess would be totally speculative, but I know I wouldn't want to be in the position of needing to find out.

Federation

That makes sense. Regarding other projects, I know at least Wordpress has their own Packagist (http://wpackagist.org/), and they'd probably want to utilize that functionality. I think this could be a really simple thing, though. The server side would need some way to say "drupal/*" = https://www.drupal.org/packages.json, and the Composer cli would need some way to keep track of that. So the client would only request from packagist on the first composer install/require/update operation. After that point, the client can directly connect to drupal.org for drupal package data. Would this be an acceptable solution?

Semver

There's a whole other issue about that. It's pretty messy. We have to figure out how to do it in a way that we don't blow away all of our existing releases, unfortunately, but the full discussion is over at https://www.drupal.org/node/1612910 if you're interested. The tl;dr version is that many parts of our infrastructure make assumptions about how things are versioned, and switching to semver breaks a lot of those assumptions, so we have to fix all that stuff first.

In general, I see the plan in #41 as a way to get from where we are now to a point where we can have the discussion about federation or pushing directly to packagist and not worry about technical blockers. Most of the todo items up there are going to have to happen anyway. The question is, do we want to serve our own Composer repo in the mean time?

kreynen’s picture

RE: #43

Perhaps we can allow overriding certain things (maybe the package description?), but then again, we're pulling most of the stuff that can change from project_dependency anyway, so the module author still has control over it.

Just want to make sure I'm following. What @cweagans is working on would generate composer.json files for all projects on Drupal based largely on information from the Project module, but "could" allow specific values from a composer.json a maintainer added directly to the repo to be merged into the version of the composer.json. Only the version coming from Drupal.org would use the drupal/[project name] vendor/name on Packagist.

Is that right?

cweagans’s picture

That's not correct. We're talking about two different pieces of JSON data. You're talking about the composer.json file that lives at the root of all Packagist libraries (think of this like a Drupal module info file). We are not generating those files. The other JSON that composer works with is the package repository, which is roughly analogous to our updates.drupal.org XML, and that is what we're generating.

If we generate that JSON, and it conflicts with something that the module maintainer puts in the actual composer.json file in their Git repo, Crell was asking how we reconcile that difference. My answer is that I don't think we have to because the only data that we'd be reading from the file in the Git repo is the "require" and "suggests" sections, which would be merged with the other data that was autogenerated (package name, description, etc).

Ideally, the only thing using the drupal/ vendor namespace would be packages from Drupal.org, but as you can see here, that's not the case.

Seldaek’s picture

Re #45, on federation, I'd like to hear what you think about my point 3 above (the single point of failure). Because I don't think drupal project will rely exclusively on drupal modules for very long once the rest of the php ecosystem is one composer require away. As for wpackagist this is not official in any way, it's a community experiment of sorts, so I don't know what wordpress would do if they ever adopted composer, I never heard anything from any core WP folks. Anyway *if* we went the federation route, we'd need to clean up everything at https://packagist.org/packages/drupal/ first IMO, and no matter if federation or not cleaning this up would probably make sense. It sucks a bit to have random packages under the drupal vendor. I don't know who of the drupal org would be a good spokesperson to contact these people, but I can hand in their emails to that person if you want, just email me privately about it please.

cweagans’s picture

@Seldaek I'm a little less concerned about the third party library part. If federation is done in such a way that downloading a package from Drupal.org doesn't hit Packagist at all (beyond the very first time as I described earlier), I'm happy. If a module depends on a third party library and Packagist happens to be down, I'm a lot less concerned about that - it's only going to be happening in one of two scenarios: 1) new module installation (in which case there's no previous version that we're upgrading, and therefore no possibility of blocking a security upgrade), or 2) a module upgrade (in which case, there's an existing version of the library on disk somewhere, either committed to the current project or in the Composer cache). If a module security upgrade is also bumping to a new major version of a library or something, well, we're doing it wrong. I don't think that'll happen very often.

We know that Drupal modules are going to be depending on libraries from Packagist. In fact, many modules are depending on Composer libraries today, and the fact that they are doing so is a good thing (I maintain a module that does just that, and it saved me a lot of code). Most operations will require both Drupal.org and Packagist, but if Packagist isn't available for whatever reason, the worst thing that happens is that a dependency isn't upgraded to the newest patch release.

If we were to push packages directly to Packagist (and all the other infrastructure stuff was taken care of, particularly semver), there would still be a dependency on Drupal.org to download things. I don't think we're going to get away from that no matter how we solve this issue.

Mixologic’s picture

Issue summary: View changes

I adjusted some things based on a re-read of concerns. One thing I feel we've left out is that there are some major blockers that we'd have to tackle first in order to be able to push directly to packagist.org (versioning, submodules, and missing composer.jsons). Providing our own metadata allows us to remove those issues as blockers to allowing us to utilize all drupal modules without requiring that a composer.json be in the repos or generating those and force comitting. Everybody that has been using composer to build drupal sites relies on all the modules being available, not just the ones that happen to have added a composer.json - so packagist.drupal-composer.org would still be a necessity until the day when every module eventually gets a composer.json. Our semver seems to sorta be approaching consensus, but If Im not mistaken even the C.X.Y.Z four digit (8.2.1.1) also still needs some adjusting so composer can use it. Submodules will likely need some metapackage or subtree split to support pulling in dependencies for embedded submodules.

I feel like we can implement this relatively easily, that these steps remove blockers to making composer and drupal all snuggly, and do not rule out eventually joining forces with packagist.org. At launch time, we *would* still have a dependency on packagist.org being up, but *only* for those drupal modules that have dependencies on some other package. There are somewhere around 600-700 modules that have a composer.json in their git tree, and many of those do not declare dependencies at all. So that dependency falls into the acceptable risk territory in my eyes. And yes, once drupal modules are one composer require away from adding libraries this number will grow - but that timeline is much, much longer than the 'how fast will people adopt drupal 8' timeline, which gives us the time to work with @Seldaek to build a robust, fault tolerant, distributed solution. Just like there is more than one DNS server to match up domain names with ip addresses, there could/should be an architecture to match up packages and versions with their corresponding git repos/distribution urls.

Seldaek’s picture

Just a quick comment on C.X.Y.Z, this should work just fine, composer uses a 4-component versioning internally and 3.4.2 for example is already normalized as 3.4.2.0. Maybe I misunderstood though.

Mixologic’s picture

The subtle difference in versioning is that composer takes Major.Minor.Patch and turns in into Major.Minor.Patch.0-stability (?), where the versioning that is being considered is to use Core.Major.Minor.Patch-stability, so it seems like the logic for what would be a breaking api change, and what would be a feature addition has the numbers in different positions. A module thats currently 8.x-2.1 would become 8.2.1.0. The 7.x version would become 7.2.1.0. Im not sure how composer handles those positions differently, and perhaps it just means that when people add version dependencies to their composer.json files, they need to specify more numbers is all.

8.3.0.0 would be a major api breaking change over 8.2.1.0 -but with normal packages 4.3.0.0 is feature compatible with 4.4.0.0 in that there ought to only be new features added.

So maybe that already works and is just a matter of how you specify your dependencies...

markcarver’s picture

Seldaek’s picture

Re #52, yes that's gonna be a bit of a problem because it means you can't just use the ^ operator for easy semver-compliant version constraints, since your notion of semver is off by one, and you have a X.major.minor.patch, but composer sees the X as major if you use ^. You can still use ~ for drupal modules then, ~8.2.0 only allows the last digit, i.e. .0 to go up but ~8.2 is fixed, so 8.3 wouldn't be installed in that case. ^8.2.0 on the other hand would allow anything up to 8.9999, which isn't good for your model of sort-of-semver.

cweagans’s picture

So what are the next steps for making a decision here?

@Mixologic - you said you were going to ask some people on IRC. Did anything come out of that?

timmillwood’s picture

I think this is going to be a hard decision to make, because whatever we go for we need to support it for the length of time people use it.

I'm sure I've made my self clear that I think Packagist and supporting the wider PHP community is the way to go.

As this overlaps infrastructure, d.o, core etc I think we need a panel of DA and core contributors to make the final choice.

joshuami’s picture

We'll be introducing this topic with a little bit of background information during the quarterly working groups meeting at DrupalCon Barcelona. I'm also forwarding this issue to the members of various working groups so they can update themselves on this issue and its importance.

I doubt we will get to a full decision in the short time we have to review priorities on September 21st, but we should at least be able to decide who needs to make the final decision and what information is needed to make a decision that will affect core, contrib, and Drupal.org infrastructure in a long term way.

The good news is that we don't have to have a decision today on how we handle Composer in the long term, so a couple months to get this correct will be well spent.

cweagans’s picture

Thanks, Josh. If there's anything I can do to facilitate the discussion, let me know. Happy to help.

timmillwood’s picture

@joshuami - Great to hear, unfortunately I'm unable to make it to Drupalcon but would love to be part of anything remotely if available. I'm happy to see that @crell shares all my views so hope his input can be made.

timmillwood’s picture

Issue summary: View changes
Seldaek’s picture

For what it's worth, I'd also be up for joining remotely, if the schedule works for me and people want me there that is :)

joshuami’s picture

There is a core conversation in Barcelona that I plan to attend. I know a couple others on the D.o team will be there. (https://events.drupal.org/barcelona2015/sessions/composer-and-drupal-8) Perhaps we can figure out a way to stream this for those that want to be involved remotely. I do not expect the decision will be made during the session, but it will be another opportunity to talk through some of the details.

@Seldaek, I think your input will be very helpful. Thanks to @cweagans and @timmillwood for the offers of help. If you can provide names of people that *need* to be in this conversation, I would appreciate it. It will help with scheduling follow up meeting for those involved with a recommended way forward. I'll start pulling together some possible times once the list of participants is finalize. We should be able to schedule that for the first week or two of October.

If all goes well, we will be able to make a decision based on that follow up meeting and start implementing the infrastructure for that on Drupal.org.

Thanks to everyone for your understanding and patience to work this out.

timmillwood’s picture

Issue summary: View changes

Please do include me on anything drupal / composer related.

If needed email or send calendar invites to tim@...
millwoodonline.co.uk or appnovation.com

timmillwood’s picture

Issue summary: View changes

Thought it was an interesting point to add that the patch in #2315545: Install composer dependencies to D8 when packaging makes d.o depend on packagist to package drupal 8 core. This could mean multiple calls per day from d.o to packagist. There is also an effort in #2315537: Install composer dependencies before running tests for a testbot that also depends packagist. This could mean that every drupal 8 core patch tested will also make a call to packagist.

Therefore if primary concerns are the supportability and stability of Packagist, then it's a little late, we're already using it.

Albert Volkman’s picture

Just wondering... would testbot not allow for composer to cache the modules? If it does, then Packagist wouldn't be getting hammered.

timmillwood’s picture

@Albert Volkman - DrupalCI runs tests in a different docker instance per patch, so there would be no composer cache.

Crell’s picture

Perhaps Drupal.org itself could run a Toran Proxy instance for its own use. We don't need to require anyone else to do so, but if we'll be hammering the same dozen or so packages hundreds of times a day that would probably help out everyone.

timmillwood’s picture

@crell - don't you think we'd be better putting time and money into packagist rather than indirectly via toran?

Mile23’s picture

Toran is a licensed product made by the Composer guy so I'm sure he'd appreciate it.

Fabianx’s picture

I missed this thread and did not read everything, yet, but talked a litte with Mixologic about it and expressed my thoughts elsewhere.

Disclaimer: I like composer and use it personally.

The time when composer / packagist was not reachable from travis (due to some internet routing error in France), was a time of loss, hair pulling and despair. I would not like drupal.org critical things to be that way, on the contrary I think it would be good if d.org mirrored used dependencies (if we are legally allowed to do so, e.g. GPLv2 compatible code).

While security issues in Symfony or any other contrib library code could happen, too, I think Drupal core or critical contrib modules are more likely to have an effect over the years.

(Off-topic: Did we discuss, yet how we handle that? If module X depends on library Y, do I then need to subscribe to the ML of library Y and all other libraries I might use as part of Drupal 8 core, or would d.org security announcements be made for code used within Drupal 8 and e.g. a contrib module falling under the security policy?)

So two things:

a) I think it would be good if drupal.org directly created the necessary things and provided its own packages.json. I also would like the default packagist to forward all requests to packages with drupal/ to our packagist. This could either be server side or a little extension, which we require or suggest in Drupal 8 - composer is flexible enough that you can 'hook' into anything, so you could indeed have drupal/ namespace map to packagist.drupal.org.

b) What happens if packagist is down when an important update needs to be downloaded? Of course we can say: We still have update.php, etc. so this might not be a concern, but currently there is no real way to detect and apply only security updates - though that could change of course with more integration.

We discussed for our needs for a client to hook into update module then have it mail the right 'composer commands' to a list, of course it would still need to be available upstream for that to work.


Overall I also don't get the need for centralization, OSS is diversity, not one centralized service that is a SPOF for everything depending on it.

- RPM, DEB, the linux kernel is mirrored world-wide.
- GIT should be mirrored, but lots of code is now centralized on GH - which as many have pointed out already is a little problematic from a radical FOSS standpoint ...

If we really wanted to help packagist, then why not mirror it completely?

E.g. packagist hosts drupal.org things, while drupal.org hosts packagist things.

And due to the mirror each user can choose their primary source or even have a fallback server. Drupal remains in control, but everyone can choose who to 'trust' more.

That would also mean that e.g. for test bots if we really run composer on the fly, then we don't possibly SPOF our testing infrastructure, keeping latencies down, etc.

chx’s picture

A few points.

  1. Last I checked composer was unbearably slow. Most especially the dependency resolution. I talked to Fabian, he said he is up to rewrite it and if the Drupal community wants to go with composer, well, then so am I (I already have one in Drupal core) but neither of us had the time to do it yet. Is there a chance of funding?
  2. Packagist currently has 69743 projects. Drupal.org has 16589, that means every 5th project on Packagist will be a Drupal project. Is this really what the Packagist people want? If so, then hosting a proxy on our infra indeed makes the most sense.
cweagans’s picture

DrupalCI runs tests in a different docker instance per patch, so there would be no composer cache.

We should absolutely make the packaging scripts cache composer metadata. It should be fairly straightforward to just store all those files in a data container that exposes the files as a volume, and then run the appropriate container with `--volumes-from`. We should do this in the packaging scripts and Drupal CI - both will need attention to cache this data appropriately.

I'd also suggest that the load that we put on Packagist through packaging and Drupal CI is an order of magnitude or three less than the load that would be placed on it by the entire Drupal community installing modules and themes all day every day. I don't have data to confirm this, but it seems to line up with the data supplied earlier by Mixologic.

don't you think we'd be better putting time and money into packagist rather than indirectly via toran?

Why should we have to do this? We've already built out the infrastructure to handle the load of the Drupal community installing modules, checking for updates, etc. I don't understand why you're pushing back so hard on just re-using what we already have, rather than spending money on doing something for a philosophical reason.

(Off-topic: Did we discuss, yet how we handle that? If module X depends on library Y, do I then need to subscribe to the ML of library Y and all other libraries I might use as part of Drupal 8 core, or would d.org security announcements be made for code used within Drupal 8 and e.g. a contrib module falling under the security policy?)

I don't think this has been discussed yet. We should probably work with any upstream library maintainers to ensure that they have a responsible disclosure process + ideally, coordinate with our security team to ensure that the release of the new library and the Core release (or whatever) can be more or less synchronized. I wonder how this is handled in other projects?

I think it would be good if drupal.org directly created the necessary things and provided its own packages.json. I also would like the default packagist to forward all requests to packages with drupal/ to our packagist. This could either be server side or a little extension, which we require or suggest in Drupal 8 - composer is flexible enough that you can 'hook' into anything, so you could indeed have drupal/ namespace map to packagist.drupal.org.

Actually, this is pretty interesting. Core already provides a Composer extension for some reason IIRC, so it wouldn't be completely out of the question to provide another one that can handle adding the Drupal.org repository. This would even alleviate @timmillwood's reservations about having to have an extra repo line in composer.json. I don't see the extra line as a problem, but I also wouldn't be opposed to having it "just work" when Drupal 8 is installed/managed via Composer.

b) What happens if packagist is down when an important update needs to be downloaded? Of course we can say: We still have update.php, etc. so this might not be a concern, but currently there is no real way to detect and apply only security updates - though that could change of course with more integration.

I think this is partially echoing my earlier concern about just blindly pushing data to Packagist. If Packagist is down when we tag a security release, we still need people to be able to install that update - even the ones that are using Composer to manage their Drupal installation. There was some discussion earlier about a retry queue sort of thing to at least partially mitigate that problem, but there was also some resistance to it (see https://www.drupal.org/node/2551607#comment-10222707 and comments around there - my full response to this issue is in about the middle of that comment)

@all - I'm going to be updating the issue summary at some point this week. I'm going to try to make it more digestible for the people making the decision, try to make everything more objective (i.e. no philosophical points - we need to be making technical decisions with objective information), as well as update for more recent discussion. I recognize, however, that I've been fairly outspoken with my opinions and that could be interpreted as bias, so when I do update the IS, I'll summarize the changes I've made and if there's anything off, I'm willing to jump on a Hangout or something to discuss in real time. I want the working groups to have as complete of a picture on this as possible.

Mixologic’s picture

Re #66:

DrupalCI runs tests in a different docker instance per patch, so there would be no composer cache.

Thats not entirely correct. It runs the tests inside the container, but it builds the codebase on the host os. The host bots are not around for very long, but could be around for many, many tests. (depending on if there is a lull in patch/branch testing), so they could either be configured to proxy through the same point that *is* around for a while, or they would at least have a composer cache on the bots. Besides, *currently* patch testing is going to go against core's composer.lock file. It's only contrib testing that would ever need to hit packagist.org to build the dependencies.

timmillwood’s picture

Why should we have to do this? We've already built out the infrastructure to handle the load of the Drupal community installing modules, checking for updates, etc. I don't understand why you're pushing back so hard on just re-using what we already have, rather than spending money on doing something for a philosophical reason.

We don't *have* to do this, but as we (and the rest of the php community) are already using Packagist it makes so much sense to use it for everything. Also in the spirt of open source we should be giving back to the wider php community and stop focusing on drupal, drupal, drupal. Drupal 8 now uses many many components from other PHP projects, we (as a community) should pay back and put our weight behind a solid solution for all. Yes, it may be very *philosophical* but it also makes sense.

(i.e. no philosophical points - we need to be making technical decisions with objective information)

No! We are an open source project, this makes philosophical points very high on the agenda. Many drupal / open source companies have philosophical points in their key values.

cweagans’s picture

We are an open source project, this makes philosophical points very high on the agenda. Many drupal / open source companies have philosophical points in their key values.

Philosophical debates will make this drag on forever. Everyone involved is well aware of the philosophical drive to push packages directly to Packagist, and in fact, I think mostly everyone (myself and possibly others excluded) is on board with pushing data to Packagist if we can ensure that we're not going to run into issues later.

However, the fact is, we're assembling two technical teams to make a technical decision about our project infrastructure. This is not the appropriate time or place for philosophy. These teams need to weigh the technical impact of each solution, and decide based on that.

Nevertheless, since you seem strongly opposed to limiting this discussion to technical issues, I'll leave a point in there with something to the effect of "Pushing to packagist = joining the wider community". As I've stated many times, I disagree with this point for a few reasons, but if keeping that point in makes you happy, then whatever. I'll leave it in because it's not worth the argument to remove it.

webflo’s picture

Issue tags: +composer
cweagans’s picture

So where did we land after the Drupalcon meeting?

timmillwood’s picture

Well...

The conclusion I got from Drupalcon was:
- Use a (CORE x 100) + MAJOR.MINOR.PATCH version numbering (eg 801.0.0, or 802.3.7), as this is three sections (like semver) and works really well with composer.
- Move @webflo's packagist.drupal-composer.org over to d.o hardware (packagist.drupalcode.org was suggested) but outputting the above numbering.

However that all seems to have fallen apart since then, which is a real shame.

cweagans’s picture

Where has it fallen apart? Kind of seems like there might be a different place where this discussion is happening and/or summarized?

Move @webflo's packagist.drupal-composer.org over to d.o hardware (packagist.drupalcode.org was suggested) but outputting the above numbering.

Is there an advantage to doing this over having Drupal.org output the metadata directly? The latter is mostly working at this point, though there are some dependent problems in project_dependency that need to be resolved to fully complete it.

timmillwood’s picture

The main advantage of moving packagist.drupal-composer.org over to d.o hardware is its 100% working and has been for months and months, its very stable and highly tested and dependable, unlike introducing new custom code.

The only reason to move it is to get d.o backing.

joshuami’s picture

Assigned: Unassigned » joshuami
Status: Active » Needs work

@timmillwood and @cweagans, it has not fallen apart, but @mixologic and I have been trying to pull together our notes to update this issue and a few others. We are close to posting something, but I don't want to put out any promises until I know what part we can handle and what parts need help.

I'm assigning this to me and changing the status to needs work. Expect an update to the issue summary by early next week.

cweagans’s picture

Fair enough. I have no objection to that. Saves work for me!

@joshuami, Awesome. I'll look forward to that.

Mixologic’s picture

Sorry, have been taking way too long to get all the information summarized.

Long story short:

Drupal.org will host a shim interface to smooth over the bumps between whatever drupalisms we have and what composer cli expects to interface with. So we can proceed with the POC you started.

We will keep packagist.drupal-composer.org (It's not in my notes that we were actually going to run that on our hardware - webflo was going to keep maintaining that for everybody that is already using that endpoint, drupal.org is going to host the next gen)

Like Tim mentioned, the composer concerned people at DC BCN had settled on 801.1.0 as the 'least bad' option which solves all of the composer issues.

However, the semver issue has gone off the rails with a whole lot of "But thats a Drupalism!" but any suggestion to do away with the drupalism is met with "but then I lose features unique to drupal" - We cant really have it both ways without a compromise.

So, I think work can proceed here, with the caveat that whatever versions we hand out, may need to change when a final decision is made as to what those will be. We'd ultimately like to avoid having three different composer compatible versioning schemes floating around in the wild.

Fabianx’s picture

Just chiming in:

+1 to 801.0.0 and 701.0.0

as the best semver compatibility option, which allows 99 releases, which should be more than enough ...

timmillwood’s picture

It's great to have a DA supported solution however I have a number of issues with the shim solution:
- its 100% drupalism
- its all custom code, we know as Drupal devs that open source code developed by hundreds is often more stable than custom development
- its untested, the packagist code (and the Drupal fork) has served millions of packages
- its currently not working, drupal-composer packagist has been running for a long time and ready to go.

You all know that pushing modules to proper packagist is my ideal solution, but we won't be able to do that for some time. In the composer BOF at Drupalcon the shim idea was hardly even mentioned, all the discussion was around working with @webflo to move drupal-composer packagist to d.o hardware.

cweagans’s picture

Honestly, if moving packagist.drupal-composer.org to the d.o infrastructure is an option, I think we should do it. I'm not 100% sure that I'll have the bandwidth to complete project_composer and all of it's dependencies in a way that will be acceptable for deployment on Drupal.org, and I don't want to be the bottleneck on this. Of course, someone else could pick it up and run with it, but if the existing solution is an option, it may be much easier to bring that into the infrastructure, change the version translation, and do whatever work is necessary to serve the package metadata with our CDN (and properly handle cache invalidation and updates to the metadata).

Mixologic’s picture

Issue summary: View changes

Deploying packagist on drupal.org's infrastructure is not exactly an easy option. We're not keen on running web applications on our infra that we haven't thoroughly vetted, and we've enumerated the myriad reasons why it will be much more work for us to do that.

I think theres a fair amount of hyperbole when talking about custom code. The only thing we need to do to support composer from drupal.org is to add database queries and output the results as json in the same format that packagist puts out. Everything else the packagist software provides is redundant with what drupal.org provides, therefore we do not want to have to aggregate data from, or combine with it. We do not want the web front end of packagist, we do not want the stats to be stored in packagist. We do not want packagist to crawl either github, drupal.org, or bitbucket looking for updates and we dont want to write any glue code to pull data from packagists database back into drupal.org's database. All of that is a lot more work than adding json endpoints.

We need to be able to support
/packages.json, - the 'all the modules we serve' endpoint
/search.json&q=%query%, - search endpoint
/downloads/, and /downloads/%packagename%, The notify and notify batch for letting us know when modules are installed.
and the provider-includes and providers-url for the individual package metadata.

None of that is insurmountable or complex, is already mostly working, and allows us to take all of the packagist.drupal-composer.org customizations (for things like modules with no composer.json, fixes for the versioning issues, as well as handling submodules) and incorporate those as well.

In the composer BOF at Drupalcon the shim idea was hardly even mentioned, all the discussion was around working with @webflo to move drupal-composer packagist to d.o hardware.

We never talked about running webflo's service on drupal.org, we talked about preserving it in order to continue supporting those sites currently using it, and simultaneously offer the shim on drupal.org.

Keep packagist.drupal-composer.org to do translating 8.x-2.3 = 8.2.3

packagist.drupalcode.org infrastructure puts out json for 8.x-2.3 = 802.3.0

is exactly what I wrote down during the bof.

Other things that came up during the BOF:

  • d.o infrastructure will investigate using a toran proxy on drupal.org if necessary.
  • Potentially double tag. for contrib. (both versions simultaneously)
  • Generate tags automatically eventually when we want to convert contrib from one semver standard to the other.
  • When full cutover happens to new semver:
    • Make changes to updates handling
    • Make changes to project module
  • Update coder module and module updater to help users transition
  • Make sure distributions are packaged and ship with a composer.lock file
  • Add another installer for anything left out (theme engines, library types etc)
cweagans’s picture

hestenet’s picture

Title: [meta] Decide on path forward for Composer support » [meta] Path forward for Composer support
Issue summary: View changes
jhodgdon’s picture

We still need to figure this out. On at least two contrib modules I maintain, I've been asked to add a composer.json file to the project. I've held off pending resolution here, but nothing is being decided... Personally I have no opinion and can add nothing to the discussion, sorry!

timmillwood’s picture

jhodgdon - there are many good reasons to add a composer.json to your projects, and it doesn't make any difference to the outcome of this issue.

bojanz’s picture

Yes, I consider composer.json files to already be community mandated.

jhodgdon’s picture

There are several problems with this, as noted above. Most serious is that you don't get the license file. Some solutions proposed above do not require each project to have its own composer.json either. So, I'm holding off until the community figures out what it is doing.

kreynen’s picture

@bojanz what do you mean by "community mandated"? Are people implying that a composer.json is a requirement? If it is a requirement (stated in policy or in use), then the libraries users linked to would all need to be compatible with both GPLv2 or GPLv3 if the parent project is going to be considered GPLv2+.

bojanz’s picture

I am implying that people are supposed to provide composer.json files that contain similar info to the .info.yml file, including the module name, description, license, project url. Example from Entity API 8.x-1.x: http://cgit.drupalcode.org/entity/tree/composer.json?h=8.x-1.x

Now that we are telling people to use Composer as a Drush Make alternative, any module not installable via Composer will get a bug report about a missing composer.json file.

Mixologic’s picture

jhodgdon - if a site builder is creating a site using composer, and they use --prefer-source, then they will not get the license file. They *will* get a license file if they choose to download a tarball using --prefer-dist.

This is no different than the current situation where if you use drush make to build a site, or drush make with git_deploy. There is no license file in the git repo.

We are currently in the process of building out a service that will provide a façade between the various special drupal ways of doing things (submodules, drupal versions to semantic versioning, , etc) to the composer way of doing things.

Mixologic’s picture

bojanz: We will be working around modules not having a composer.json file. There really isn't a technical need to force people to have one if their module does not have any external dependencies. There was some talk of "just make an issue in the queue and let it happen organically" but since the shim service would completely obviate the need for that, I cant think of why we'd add that administrative overhead onto contrib maintainers and put blockers in front of site builders that want to use Composer to build a site with a module that doesn't yet have one.

cweagans’s picture

Now that we are telling people to use Composer as a Drush Make alternative, any module not installable via Composer will get a bug report about a missing composer.json file.

Just want to clarify here: all modules, themes, and even profiles are installable with Composer right now thanks to packagist.drupal-composer.org. Every single one of them. Maintainers don't need to do anything to make this work.

The shim that Mixologic was talking about essentially moves the responsibility from packagist.drupal-composer.org to drupal.org (where it will be served directly from Drupal, so there won't be any lag time between tagging a release and it being available via composer, for example).

bojanz’s picture

Thanks for clarifying.

Mixologic’s picture

@kreynen: So Im not super clear on the licensing requirement. drupal.org will host modules that are gplv2+ (meaning they are both v2 and v3 licensed, and its up to the downloader which license works for them), and all code we distribute needs to be gplv2+ (ie. it cannot be gplv3 only).

Composer packages that come from packagist.org are not hosted or distributed by drupal.org. They could be on either bitbucket or github. Are you saying that we should not be hosting modules that have external requirements on non-gplv2+ code? because I think there are several examples of modules that exist now that ask that you download incompatible code to /libraries (https://www.drupal.org/project/wysiwyg_ckfinder and https://www.drupal.org/project/filedepot both come to mind, Im sure there are more).

One feature the shim may allow us to provide is the licensing requirements of dependencies - composer licenses shows all of the licences for a project (once it has been initialized) - this may be something we could provide on the project pages.

So, is there something special about the fact that the dependency is 'linked' in the composer.json or is it good enough that we would not be distributing packages that contain non gplv2+ code?

webchick’s picture

Yes, that's exactly what he's saying. See https://www.drupal.org/packaging-whitelist. The Legal team goes out of its way to make sure that anything that gets bundled in with Drupal's GPLv2+ code is GPLv2+ compatible, and that includes external dependencies via Drush Make (and now, Composer).

We could change our contrib licensing policy instead to "meh, whatever, get a lawyer" but it would be a big shift, and AFAIK need sign-off from both Dries and many, many copyright holders. Feature parity with Drush Make / distros would be getting Composer to cross-reference the packaging whitelist before allowing a composer.json to be distributed.

Mixologic’s picture

Im still somewhat confused.

1. With distributions, this is what we do now:

  • a. We do host/distribute packaged distributions that have whitelisted components that are GPLv2+ compatible (The tarball would come with the compatible software)
  • b. We do not host/distribute distributions that require non compatible GPLv2+ (no tarballs with incompatible software)

2. With contrib, this is what we do now:

  • a. We do host/distribute contrib modules that have instructions that require libraries to be downloaded that are not gplv2+.
  • b. We do not host/distribute contrib modules that *contain* non compatible GPLv2+ code dependencies.

3. With composer for contrib, this is what we are proposing:

  • a. We do host/distribute contrib modules that contain a pointer to libraries in the modules composer.json that can be downloaded *by the end user* that may or may not be gplv2+. In the interests of transparency we should specify the licenses of the dependencies so nobody is surprised (which is already better than we're doing now)
  • b. We do not host/distribute contrib modules that contain non compatible GPLv2+ code dependencies.

We would not be 'bundling' the dependencies for a contrib module using composer.

So, my question is 3.a somehow violate what we do? and how is that different than what we have always done with 2.a? (or are all the examples of 2.a simply anomolies that we shouldnt be hosting?)

timmillwood’s picture

How do others handle this? Many open source projects have dependencies pulled in via a packaging system. Don't they have the same issue?

dsnopek’s picture

Re #101:

Yes, that's exactly what he's saying. See https://www.drupal.org/packaging-whitelist. The Legal team goes out of its way to make sure that anything that gets bundled in with Drupal's GPLv2+ code is GPLv2+ compatible, and that includes external dependencies via Drush Make (and now, Composer).

As a shorter version of what Mixologic wrote in #102: the packaging whitelist currently only affects distributions which actually get packaged by Drupal.org. So, to be consistent with current behavior, that would only affect moving the distro packager to using composer (rather than drush make) - not modules using composer or the Drupal.org "composer facade".

I can't find an issue for moving the distro packager to use composer, but if there is one, we should make sure there is a note about the packaging whitelist on it!

geerlingguy’s picture

#95: Now that we are telling people to use Composer as a Drush Make alternative, any module not installable via Composer will get a bug report about a missing composer.json file.

That's news to me... I thought we were still pretty much at ground zero with recommended Composer-based workflows. Besides a few scattershot blog posts on Drupal Planet, I haven't seen any official stance on how Drupal and Composer will work together, especially w/r/t contrib. Which is why I've actually pulled out composer.json files that I used to have in a few of my contrib modules.

#98: Just want to clarify here: all modules, themes, and even profiles are installable with Composer right now thanks to packagist.drupal-composer.org. Every single one of them. Maintainers don't need to do anything to make this work.

This was my understanding; for now, I'm planning on sitting on my hands w/r/t Composer and Drupal contrib, because it seems there is still no solid direction. Until there is, I point people to packagist.drupal-composer.org, let them figure things out on their own, but personally still use Drush make for Drupal projects because Composer-based Drupal workflows are still not to a 'best practices' level of support yet.

kreynen’s picture

The issue is related to whether composer.json is required or optional. When packaging a distribution, the distribution's .make is processed. Module level projects can include .makes, but that's optional. Those dependencies aren't packaged in what users download. If using a module with a .make, the user has the option to pull those additional dependencies into their site regardless of license, but it's their choice. This is just way to automate the process of going out, finding the additional code, downloading it, and placing it in the correct directory... which the user can still do.

This changes when the things that is automatically downloading non-GPL dependencies that projects require itself becomes a requirement. This violates the "if you stick to code coming from Drupal.org, we work hard to make sure it is GPLv2+ or compatible" policies we've always had.

If Drupal adopts composer as a requirement and accepts that method of installing as a standard, we can't continue to claim that projects that include non-GPL dependencies are GPL. We might be able to get away with that with javascript assets, but not PHP.

As far as #103,

How do others handle this? Many open source projects have dependencies pulled in via a packaging system. Don't they have the same issue?

They have always handled it poorly. Take a few minutes to browse through popular WordPress plugins. It's easy to find GPLv3, Apache 2, and even commercially licensed code in those projects. I've always been under the impression that the effort we put into keeping everything GPLv2 compatible with a GPLv3 option was one of the things that set Drupal apart from other CMS projects. I'm operating under the assumption that we want to keep that distinction.

cweagans’s picture

Composer-based Drupal workflows are still not to a 'best practices' level of support yet

Strongly disagree here. https://github.com/drupal-composer/drupal-project is the best way to handle things in my opinion. Drush Make was great, but I use Composer for everything else. It's so much nicer to have one tool that handles all/the majority of my PHP projects. Does everything Drush Make will do except validate that a patch matches a SHA1 hash (if you're using cweagans/composer-patches that is), and it should work for both D7 and D8 at this point (though I've only used it for D8 projects).

Either way, maintainers shouldn't need composer.json in their repos.

Re the licensing stuff - Composer makes it really really easy to validate the licenses for dependencies on a package. composer licenses --format=json will do it, and it doesn't require you to do a composer install first. That will work in literally any directory that has a composer.json file in it. Since it gives you the specific license, we could even just check the name of the license, and if it's one of our approved licenses for packaging, bring it in. No need for a whitelist in that case. To avoid the case where people fork a lib on Github and just change the license so that they can bring it in on Drupal.org, we can disallow adding Composer repos in the composer.json file, so it would be locked down to only the Drupal.org repo and the official Packagist repo.

Simple :)

geerlingguy’s picture

Strongly disagree here. https://github.com/drupal-composer/drupal-project is the best way to handle things in my opinion. Drush Make was great, but I use Composer for everything else.

What I mean is that there are currently very few Drupal projects currently built using Composer, and very many built using Drush make files. And very, very, very many more that don't use either approach (but instead, someone downloads core, then downloads a few modules and tosses them in the modules directory), but that's besides the point.

Outside a few blog posts, and a few discussions in the issue queues, I've seen extremely little activity or standardization around how people use Composer and Drupal together. Composer and general PHP/other projects? Yes, quite a bit. Drupal? Very little. This issue's already been through many layers of bikeshedding, and it doesn't seem there's yet consensus on exactly where we want to point yet.

(As an aside: the way this documentation page is written seems a bit myopic... yes, we want to get off the island, yes, composer is awesome, but I think we need to warn people that the entire state of Composer usage with Drupal is still being worked on, specifically in this issue and elsewhere.)

I know you have a workflow that works, and that you've standardized on... but I think you may need to sell it more (and work out some rough edges) if you want to get this issue out of the 'Needs work' state. I'm excited for the day when I can will be able to build everything on a composer.json file, but that day is not yet here—I don't want to have to retool my build processes entirely over and over until I'm sure I have a solid setup that's built on a solid foundation (e.g. will modules someday have composer.json inside? Is packagist.drupal-composer.org ever going to be an official Drupal resource (e.g. run by DA, on our infrastructure, with account integration, etc.? Will we get licensing issues figured out?).

cweagans’s picture

What I mean is that there are currently very few projects currently built using Composer, and very many built using Drush make files

That's simply a side effect of having Drush Make available for a long time and not having Composer available during that time. The writing is on the wall, though. AFAIK, Drush is moving toward not supporting global installations (and will require you to install it via Composer on a per-project basis), though that could have changed. There are a handful of modules that depend on libs from Packagist (including one that I wrote recently), and I think that number will only grow. And probably most relevant is that the rest of the entire php community is using Composer. The fact that we aren't saying that's a best practice is at best a temporary oversight.

In other words, Drush Make was a fantastic tool for Drupal users, but it's time to finally let it die and move on to a tool that's considered a best practice in the rest of the PHP world.

Outside a few blog posts, and a few discussions in the issue queues, I've seen extremely little activity or standardization around how people use Composer and Drupal together.

That's what the drupal-composer org on Github is for: codifying and standardizing the way that Drupal sites can be built with Composer. The thing is, though, the process really isn't that different from any other Composer project. Just require something and be done with it. The only thing that could be considered nonstandard is that there's a separate package repo to use, and that requirement might go away if somebody creates a PR for Packagist that allows for federated metadata (i.e. if I ask for drupal/samlauth, and it's not available through Packagist, Packagist will delegate responsibility for it to the Drupal.org metadata).

This issue's already been through many layers of bikeshedding, and it doesn't seem there's yet consensus on exactly where we want to point yet.

That's only partially accurate. You're right: this issue has been an endless bikeshed. However, a direction has been chosen and the DA and some volunteers are actively working toward it. The tl;dr version is Drupal.org will host the metadata that's currently being provided by packagist.drupal-composer.org (though it won't be a packagist instance - it'll be served directly by a Drupal module or the metadata will be pushed out to something similar to updates.drupal.org so that it's all just static files). That's happening in project_composer, and there's also https://www.drupal.org/node/2576285. After that point (or in parallel - idk), we'll work on the more infrastructure-y things to make composer packaging and the like work properly.

I know you have a workflow that works, and that you've standardized on... but I think you may need to sell it more (and work out some rough edges) if you want to get this issue out of the 'Needs work' state.

It's not just me. We're using the scaffolding provided by the Drupal Composer org on Github with some very minor modifications. AFAIK, there are a handful of Drupal agencies using it as well. I know of at least one agency using it for very large projects both for their customers and for internal stuff. There's no rough edges from my standpoint, but if you've come across them, you should open issues on the drupal-composer/drupal-project repo on Github. Feel free to @ me and I'll help.

I'm excited for the day when I can will be able to build everything on a composer.json file, but that day is not yet here

This is not true. You can build sites with Composer today. See the readme on https://github.com/drupal-composer/drupal-project for more info.

I don't want to have to retool my build processes entirely over and over until I'm sure I have a solid setup that's built on a solid foundation

You shouldn't have to retool anything. If your CI system supports running `composer install`, you're good. The other issues literally don't matter to site builders.

will modules someday have composer.json inside?

This will either happen by some automated process (i.e. commit it automatically on behalf of module maintainers), or it won't be required at all. The only reason we'd need it is if we were pushing packages directly to packagist.org, but we can't do that right now because contrib doesn't use semver.

Either way, it doesn't impact site builders at all. Add the packagist.drupal-composer.org repo to your composer.json (which is done for you already in drupal-composer/drupal-project) and `composer require drupal/modulename`.

Is packagist.drupal-composer.org ever going to be an official Drupal resource?

For now, it sort of is. webflo has said that it's not going away any time soon. The metadata will be moved to Drupal.org when https://www.drupal.org/node/2576285 is done, and instead of packagist.drupal-composer.org, you'll just use drupal.org for your repo (or composer.drupal.org or similar - the specifics haven't been ironed out). There will be no need for account integration this way because there's not really accounts in the first place (in terms of serving the metadata). It should "just work".

Doing it this way also means that the download counts will be updated when a module is installed through composer. I know that's not a huge thing, but I think it's kind of cool.

Will we get licensing issues figured out?

Yes, we will. But that doesn't impact site builders as far as I'm concerned. Requiring GPL compatibility on Drupal.org is to protect the DA and Drupal.org. Site builders can almost completely ignore licenses unless something is AGPL. In any case, this will be figured out in the same way that the Drush packaging whitelist was figured out, but it doesn't impact the ability to build sites with Composer right now. (IANAL, this is not legal advice, YMMV and all that).

Crell’s picture

*Puts on Licensing Working Group member hat*

As both webchick and cweagans noted, the licensing enforcement question is, if anything, made easier by Composer. We would no longer need to whitelist projects, just licenses, and can scan a module's composer.json file (composer provides this tool) to see if any of the licenses its dependencies use is not one of [GPLv2+, LGPLv2+, MIT, BSD]. If one is outside of that whitelist, we flag the module (either just for review, or block it from being downloaded, or whatever we decide).

If anything, my only concern is the number of Apache2 licensed packages I know full well people are already using without realizing that it's not GPLv2-compatible who will get cranky when we finally call them on it. (To be fair, many of the package authors aren't either; I've worked with a few to change their license as a result.) That doesn't mean we shouldn't do it, mind. In fact, it means I'm looking forward to having an automated Composer tool to do it for me so I don't have to. :-)

This question is the same for both modules and distributions. And in the long run, our packaging of distributions may go away in favor of distros just being a composer.json file that you run yourself rather than downloading a tarball for. Time will tell, there.

Mile23’s picture

OK, so we have some stuff going on here.

We have a conversation about whether requiring someone to use a non-GPL-compatible library breaks our rules.

We have a conversation about what's the best-practice way to build a site.

And we have a conversation about how to resolve dependencies using Composer. This splits out into a few conversations:

Should we implement our own packagist?

Should addition of composer.json to modules be 'organic' or automatic with the Drupal-specific packagist?

So let's back up a little...

*I* want to be able to install stuff using composer. I want to be in control of which dependencies I use (in terms of licensing and other concerns). I want project maintainers to maintain their own composer.json file. I also still think it's generally a bad idea to rely on our own home-baked infrastructure for packging information, though I'm not as dead-set against it as I used to be.

With these in mind, here are the things we should do. Note that there is a lot of overlap here with the goals of people with opinions opposed to mine. :-)

  • Drupal.org/DA *should* create infrastructure to list projects with composer.json files to packagist.org, as a service similar to that available to github projects. This allows D.O to cut off the firehose whenever it wants to, and disallows people from using packages d.o has taken action against. In other words: D.O will list your package on packagist (or drupal-composer or whatever) so you shouldn't, and if you misbehave they can shut it off.
  • D.O should have a listing of Composer-based dependencies per project, similar to what you see on packagist.org. As an example, the package I use for testing stuff: https://packagist.org/packages/mile23/page-example Note that it requires Crell's api-problem package. You can click through the package name and see that crell/api-problem has an MIT license, and so therefore is GPL compatible. Exposing this information allows people to make decisions. I'm sure the drupal-composer.org people could implement this in a day. :-)
  • Similarly, the concerns about non-GPL-compatible licenses for dependencies of D.O hosted projects is pretty much solved already as noted in a few places in this issue thread. Libraries module lets us offload the responsibility for that. However, there's one concern: Should the testbot be able to load such libraries and run them? There might be a test fail for non-GPL library. This would be terribly annoying if there's no ruling for a specific license, but might be a reasonable limitation, depending. DA legal team to the rescue. :-)
  • Probably more...
cweagans’s picture

I want to be in control of which dependencies I use (in terms of licensing and other concerns).

Dependencies will be downloaded by Composer on your machine no matter what. The question is about how our packaging system works. If you want to restrict licensing for your dependencies, you can write a Composer plugin to do it. I'm not sure why you would though, other than philosophical reasons.

I want project maintainers to maintain their own composer.json file.

If we get to the point where this is necessary, sure. For now, we can automate some of this to ease the transition. Personally, I'd love to not have to maintain a composer.json for a module that doesn't absolutely require it. Note that if a module requires a composer dependency, it'll have to have a composer.json anyway, but the metadata that we serve to the Composer client won't change that much (other than the require/require-dev sections)

I also still think it's generally a bad idea to rely on our own home-baked infrastructure for packging information, though I'm not as dead-set against it as I used to be.

It's really the only feasible option right now, other than spinning up our own Packagist. Until we have real semver for contrib extensions, we can't push directly to Packagist, and there are some other concerns in the IS, namely that we cannot know for certain that a security release will show up instantly on Packagist.org (i.e. if there's a Packagist outage when the release is published). Ensuring that requires some additional complexity on Drupal.org's end to queue/retry notifications in the event the notification request doesn't complete successfully the first time.

Running our own Packagist was discussed, but there's a lot of functionality that is duplicated between what Drupal.org has right now and what Packagist provides (user accounts, download statistics, release lists, etc). Why spin up a gigantic application that we'd have to do substantial integration with when we can just output JSON from Drupal.org? This is not a complicated thing.

Drupal.org/DA *should* create infrastructure to list projects with composer.json files to packagist.org, as a service similar to that available to github projects. This allows D.O to cut off the firehose whenever it wants to, and disallows people from using packages d.o has taken action against. In other words: D.O will list your package on packagist (or drupal-composer or whatever) so you shouldn't, and if you misbehave they can shut it off.

No, we are not doing that. It's already been decided and the DA is working in another direction. We are going to serve the metadata ourselves for every extension available via Drupal.org.

D.O should have a listing of Composer-based dependencies per project, similar to what you see on packagist.org. As an example, the package I use for testing stuff: https://packagist.org/packages/mile23/page-example Note that it requires Crell's api-problem package. You can click through the package name and see that crell/api-problem has an MIT license, and so therefore is GPL compatible. Exposing this information allows people to make decisions. I'm sure the drupal-composer.org people could implement this in a day. :-)

This will partially happen. project_dependency will start pulling in information about Composer dependencies. The license issue is only related to packaging, though, and I don't think there are any plans to start packaging Composer dependencies (or anything else for that matter) with anything other than Installation Profiles.

Similarly, the concerns about non-GPL-compatible licenses for dependencies of D.O hosted projects is pretty much solved already as noted in a few places in this issue thread. Libraries module lets us offload the responsibility for that. However, there's one concern: Should the testbot be able to load such libraries and run them? There might be a test fail for non-GPL library. This would be terribly annoying if there's no ruling for a specific license, but might be a reasonable limitation, depending. DA legal team to the rescue. :-)

Again, the license concern is only related to install profile packaging. The rest doesn't matter too much. Personally, I think the testbot will need to be able to pull in *any* library that is necessary for testing (and completely ignore the license), but I'd defer to Mixologic on whether or not that will happen.

cweagans’s picture

If anything, my only concern is the number of Apache2 licensed packages I know full well people are already using without realizing that it's not GPLv2-compatible who will get cranky when we finally call them on it.

I thought there was a distinction between usage and distribution. If I pull in a bunch of libraries and use them on my site, I haven't violated the license of any of them, have I? If I distribute them in one package and claim that it's GPL2+ and some of the libraries aren't compatible, then there may be a problem.

My thoughts:

* For Drupal.org, we should really only care about the licensing when we're doing install profile packaging.
* We shouldn't package dependencies with modules (so that we avoid having sites with a gazillion separate autoloaders).
* If a module requires a package (even if it's not compatible with GPL2+), we should be able to pull it in on the testbot so that we can test the functionality of the module.

Am I off base with any of that ^ Crell/other LWG members?

Crell’s picture

AFAIK we've disallowed modules that had non-GPL-compatible dependencies in the past, even if they weren't in the repo. Since "distribution" can include "give from consultant to client", we really don't want to be in a position of creating situations where a consultant is able to build a project but not, strictly speaking, distribute it to their client. :-)

Mile23’s picture

(re: licensing and other concerns) Dependencies will be downloaded by Composer on your machine no matter what.

Well, no. :-)

If we have information about licensing or if a package has a reputation for malware or whatever, then as site builders/devops we don't ever do the installation.

This is why we have Libraries module: So d.o doesn't distribute software under the wrong license.

I just think it's important to be open about it and automate making it visible.

"Drupal.org/DA *should* create infrastructure to list projects with composer.json files to packagist.org, as a service similar to that available to github projects.[..]"

No, we are not doing that. It's already been decided and the DA is working in another direction. We are going to serve the metadata ourselves for every extension available via Drupal.org.

OK, so that means I'll just make a few packagist items in the drupal/ namespace and serve malware from them.

Personally, I think the testbot will need to be able to pull in *any* library that is necessary for testing (and completely ignore the license),

So I'll get to work writing a license that says 'this license is GPL except in the case of Drupal.org where you have to pay me a meeeeelion dollars for every testbot run.'

Only has to work once. :-)

Mixologic’s picture

AFAIK we've disallowed modules that had non-GPL-compatible dependencies in the past, even if they weren't in the repo.

Is this just policy that hasnt been enforced? Or is this misinterpretation of the policy? There are tons of examples of drupal modules that rely on code that is not GPLv2+ to be downloaded to /libraries. The CK finder module I referenced above is a perfect example.

This is the question that I keep asking that I have yet to get a satisfactory answer: Can module developers, right now, upload modules to drupal.org if they depend on non GPLv2+ compatible code to be downloaded to /libraries? And if not, why are there so many of them on drupal.org?

OK, so that means I'll just make a few packagist items in the drupal/ namespace and serve malware from them.

The drupal namespace on packagist.org is already claimed. Nobody can add more code there unless they already had something in the drupal namespace before it was claimed.

Side note: you cannot run composer licenses and get the license information from the composer.json without *first* running a composer init - i.e. you have to have downloaded all the dependencies in order to determine their licences. This will be a step in the process d.o. does so that we can capture the license data because we *do* want to "have a listing of Composer-based dependencies per project, similar to what you see on packagist.org"

Re: testbots - there isn't really a reason I can think of to not to test code because of its license. But we could definitely add a license check to validate and verify that what is being tested yields a bundle of code that is still gplv2+ (again, Im not sure if we even have to do that).

Crell’s picture

Mixologic: The enforcement of licensing rules on Drupal.org has been... not up to par for the last few years. The LWG was formed to try and fix that, although we're stalled at the moment on some logistics and a pending change of chair. (I'm as frustrated on this point as you are, believe me.) Modules that depend on GPL-incompatible code are not *supposed* to be on Drupal.org. If you find them, please advise the author that they're not allowed to do that. (Because of the way the GPL works, the resulting total work is then unlicenseable and incompatible with either the GPL, the 3rd party library, or both.)

One reason I really like the idea of forcing 3rd party libraries to go through composer.json rather than the libraries module is because catching those violations, as noted above, becomes way easier.

Mixologic’s picture

Modules that depend on GPL-incompatible code are not *supposed* to be on Drupal.org.

(emphasis mine)

I am unable to find that policy expressed anywhere (on the site or in any of the draft policies that Holly showed me). All mentions of 3rd party code refer to it being included/comitted in drupal.org's git repos, or distributed as part of a tarball for a distribution. There does not seem to be now, or in the past, any mention of "external dependencies hosted elsewhere need to be GPLv2+ compatible"

jhodgdon’s picture

See https://www.drupal.org/about/licensing#q10 -- which I think is pretty clear.

But this is off-topic... let's keep this discussion on Composer?

gisle’s picture

In #116, Mixologic wrote:

Can module developers, right now, upload modules to drupal.org if they depend on non GPLv2+ compatible code to be downloaded to /libraries? And if not, why are there so many of them on drupal.org?

I've created a new issue: #2631056: Policy on bridge modules for non-GPL PHP libraries for this in the LWG issue tracker, as it is a licensing policy question that goes beyond Composer support.

Mile23’s picture

alexpott’s picture

There are three issues that I'm not sure are addressed anywhere:

  1. Composer changes the autoloader making code available even when the module is uninstalled
  2. Drupal modules are also packages which can contain more than one module and where the package name does not have to equal the module name
  3. We allow packages to be installed all over the place - /modules, /core/modules /sites/all/modules/, sites/www.example.com/modules, /profile/CHOSEN_PROFILE and /core/profile/CHOSEN_PROFILE. Additionally people often split up modules by contrib and custom inside these folders.
Mile23’s picture

Composer changes the autoloader making code available even when the module is uninstalled.

Yes, this is just part of its build process. Without a Composer plugin to query whether the module is installed or not, this will be a side-effect of using Composer. The real problem here isn't that PSR-4 would be crowded, but that an uninstalled module might have a dependency which prevents other dependencies from being met.

Drupal modules are also packages which can contain more than one module and where the package name does not have to equal the module name.

True. #2494073: Mark modules with unmet composer dependencies uninstallable looks for *.info.yml files and then tries to find composer.json files next to them, for this reason. Any dependency glue between Drupal and Composer would have to adopt this strategy. We might also just go ahead and add a composer.json accessor similar to Extension::getExtensionPathname().

We allow packages to be installed all over the place - /modules, /core/modules /sites/all/modules/, sites/www.example.com/modules, /profile/CHOSEN_PROFILE and /core/profile/CHOSEN_PROFILE. Additionally people often split up modules by contrib and custom inside these folders.

Yes, this is similar to the last point, and is similar to the reasoning behind #2301873: Use ExtensionDiscovery to register test namespaces for running PHPUnit. It would be wise to make sure ExtensionDiscovery works properly and has a way to perform scans without a lot of dependencies. Then it can be re-used all over, even for discovering Composer dependencies, and can be the single point of success for all these different systems. (It has a bit of a problem with its cache: #2605654: Modify Drupal\Core\Extension\ExtensionDiscovery to allow for scanning multiple file systems, enabling vfsStream testing)

gonssal’s picture

My 2 cents:

  • *.drupal.org infrastructure (including an extra composer repo is common and would quickly become known as one of the first things to do when creating a new or updating an old module)
  • Require composer.json in the module for it to be composer "aware". This will have the additional benefit of purging a lot of unmantained modules from the ecosystem as composer becomes the norm.
hestenet’s picture

Issue tags: +Community Initiative
matslats’s picture

I think I'm missing something in the above.
My concern is that my distro uses the address module which has a dependency on a commerce guys library.
Anyone attempting to install my distro has to go through an undocumented process of installing composer on their machine, downloading composer_manager, initialising it, and running composer drupal-update.
What I most want is for the drupal.org package manager to include ALL distributions' dependencies.

bojanz’s picture

@matslats
Yes, D8 distributions will stay half-broken until Composer support is added to their packaging. Now that it's done for core, it shouldn't be a hard next step.

tvn’s picture

kreynen’s picture

Now that it's done for core, it shouldn't be a hard next step.

Packaging any composer dependency into a distribution that can be downloaded from Drupal.org is relatively easy, but I'm afraid making sure what we distribute from Drupal.org remains GPLv2 is going to be difficult.

Is there a specific issue for adding composer support to distribution packaging?

I really want to understand how this is going to work from a licensing perspective. Will the composer dependencies be packaged the same way 3rd party libraries are with .make? Are you expecting the LWG to vet the .json files or can I include anything from packagist? If we're not using a Composer whitelist and we are going to package Apache2, GPLv3 and unlicensed PHP files with D8 Drupal distributions, we need to update the licensing of distribution to reflect that they are not GPLv2 and users will need to vet the licensing of what they've downloaded themselves.

gisle’s picture

kreynen wrote:

Are you expecting the LWG to vet the .json files or can I include anything from packagist,

According to its mandate, it it the LWG that owns the policy on packaging. I.e. I believe it is up to the LWG to decide what those policies shall be. The LWG should not fulfill any expectations in this area, it must lead.

For instance, if the LWG decides that .json-files cannot include materials that would not have been permitted in the Git repo - then the LWG should instruct the infrastructure group that any repo containing such .json-files should not be packaged by the package script.

To me, implementing this policy look more attractive from the LWG-perspective than the current packaging whitelist. Since the .json syntax supports declaration of license and authorship for all assets included, vetting for compliance can be done with automatic tools. When a release is pushed that includes assets with no license or a non-whitelisted license, these are not packaged but produces an error message to the pusher explaining what is wrong. If done right, this would nudge people to "do the right thing" without the LWG even getting involved in vetting of the contents.

(PS: After looking carefully at composer.json, I realize that it provides something that I personally think essential for the LWG to function: a machine-readable declaration of assets and associated licenses, making automatic vetting of license compliance feasible. I realize that the tools for this do not exist yet, but as soon as we've got a revised and better Git repo policy in place, I'll volunteer time to write those tools.)

Seldaek’s picture

Re #130: This could be something to add at the composer level really, saying that you only want projects with licenses compatible with X or Y. If you'd like to contribute it there we can start a discussion on composer issue tracker when you're ready :)

gisle’s picture

Seldaek wrote:

If you'd like to contribute it there we can start a discussion on composer issue tracker when you're ready :)

Is this the composer issue tracker on GitHub (https://github.com/composer/composer/issues) ?

fgm’s picture

Issue summary: View changes
hestenet’s picture

Issue summary: View changes

To close the loop on recent updates - the Drupal.org composer repositories are now available in an alpha state. Documentation on using them can be found here: https://www.drupal.org/node/2718229

dww’s picture

RFC: Why does the "best practice" of using composer to manage a site result in so many composer.lock merge conflicts?

I (and a half dozen issues and handbook pages) seem to be missing something very important. I can't believe I'm the only one to have this problem. What's going on? ;)

Thanks!
-Derek