Allow additional external libraries with composer

I like this idea in general. First change I'd make is to convert composer.json to example.composer.json, since that shouldnt be in version control.

Definitely would love to take this approach as it would mean doing it closer to right. There was a lengthy discussion over at #1475510: Remove external dependencies from the core repo and let Composer manage the dependencies instead, but I think the two biggest issues are:

1. Having Testbot run a composer install during compilation
2. Having the d.o packager run a composer install when building Drupal core

Generally, I'm more for focusing building a new system that can handle dependencies like this easily, instead of trying to retrofit and re-invent our existing system to meet new requirements ... but it should be feasible. Given test run times, I'd be wary of anything that increases the amount of external calls and processing, though. We key on /core/vendor to filter out syntax checks, so there will be some additional impacts there too.

A change like this is not trivial from a testbot perspective ... and my available dev time is quite limited. As a result, just a heads up that implementation could potentially take a number of weeks once a firm 'go' was decided.

What's the difference to #1475510: Remove external dependencies from the core repo and let Composer manage the dependencies instead ? This issue looks like a 100% duplicate?

Yeah, totally different problem.

Seems like those in this queue might be interested in this, but as a personal project, I'm experimenting with splitting all core modules into separate repos with `git subtree split` on a regular basis, with composer.json files and perhaps running travisci tests on individual repos.
https://github.com/drupal-fracture/split-scripts
https://github.com/drupal-fracture

re: #13: Thanks for not clarifying.

1) Removing external vendor components from the core repo should be exclusively discussed in #1475510: Remove external dependencies from the core repo and let Composer manage the dependencies instead

2) Moving /core/vendor into /vendor has consequences: It essentially reverts a major part of the epic #22336: Move all core Drupal files under a /core folder to improve usability and upgrades — How will users without composer-command-line-fu install or update Drupal?

3) Why do we need to touch /core/vendor at all to allow contrib to use composer? Especially since the issue title talks about additional libraries?

1) good point. totally fair.

2 & 3) I think it's that it is misleading to ask people to edit `core/composer.json` and build non-core libs into `core/vendor/`. If composer is going to be able to properly do dependency resolution, `vendor/` needs to be in one place (as far as I can tell), and the repo root seems to be most sensible.

@sun - I was just agreeing with Kim, because he explained it in #12. He also explains the problem quite well in the issue summary.

First off, I want to say that I appreciate this issue. I think figuring out a good way for site/product/distribution builders to manage vendor libraries across core, contrib, and non-Drupal code is an excellent goal, and I appreciate the thinking that's gone into it so far.

Here's just some issues/questions to figure out how to make it work.

If you are downloading from the Drupal project page, the tarball would contain a pre-build vendor dir.

Right now, to update a site from 8.0 to 8.1 (once those releases are available) involves just:
- Download the 8.1 tarball.
- Remove your site's core/ directory and replace with the new core/ directory.
- Run update.php.

With this proposal, the new process would be:
- Download the 8.1 tarball.
- Remove your site's core/ directory and replace with the new core/ directory.
- Remove your site's vendor/ directory and replace with the new vendor/ directory.
- Run update.php.

Except the above wouldn't work if the site is running any contrib or non-Drupal code that resulted in the site's vendor/ directory needing to be different than the one in the tarball (e.g., they're using AWS module that requires a newer Guzzle). For these people, is the only solution to figure out how to use Composer? Note that the majority of Drupal sites are operated by people who are not developers or system administrators. In some cases, they may be on a shared hosting environment that either doesn't provide command line access, or the website operator has not yet had to learn how to use the command line.

There is still the likelyhood that different modules depend on different versions.

Well, even before tackling this issue, there's the corollary that I presume we would want core's composer.json to be more relaxed. For example, instead of depending on Symfony's HttpKernel 2.2.0-BETA2, as we do now, we'll want to relax it to something like ">2.2.0-BETA2, <2.4". However, this affects the Drupal core debugging process. For example, suppose I know that my local development site that I use for Drupal core development worked on 8.x commit foo. Then, I update to 8.x commit bar and something's not working. I want to use git bisect to figure out which commit broke it, except it doesn't work, because when I go back to 8.x commit foo, my site is still broken. WTF? Oh, it's because what actually broke my site is Symfony's HttpKernel 2.2.0-RC1, and when I switched back to 8.x commit foo, that didn't simultaneously revert by Symfony code to the same version that that Drupal core commit was originally known to work on. Maybe this is ok, because people will get used to checking for vendor-related bugs separately from Drupal bugs, but it will take some adjustment, because for a lot of Drupal developers, all the Symfony stuff is just under-the-hood magic, not something they're used to having to think about.

they may be on a shared hosting environment that either doesn't provide command line access

While I agree with the rest of your post, I'd like to point out that the number of hosts that do not provide command line access is rapidly approaching zero. Even GoDaddy provides command line access these days.

@effulgentsia thanks for bringing us back to the basics -- we sometimes jump ahead. That's not to say jumping ahead is bad; we just need to go back and think on how to "fix" or accommodate this stuff. I really appreciate your post. Need to think on it a bit before I comment directly

------
A bit of an aside, but I came across this Martin Fowler article recently and found this quote very relevant to all these discussions about Composer and Symfony and the future of Drupal. Thought maybe you guys would find it motivating too :)

...I see a startling parallel between DHH [Ruby on Rails creator] and Kent Beck [Extreme Programming creator]. For either of them, if you present them with a constrained world, they'll look at constraints we take for granted, consider them to be unessential, and create a world without them. I don't have that quality, I tend to try to work within the constraints gradually pushing at them, while they just stick some intellectual dynamite under them and move on.

I truly feel that to stay competitive and relevant, we have to first imagine our Drupal ecosystem without all these constraints, and then backtrack into covering ground, putting the muscle into solving problems like effulgentsia brings up. Especially now, Drupal needs all of us to start packing intellectual dynamite in our toolbelts :)

I'd like to point out that the number of hosts that do not provide command line access is rapidly approaching zero.

It's true that they are making it available, but it is not trivial to actually turn this on in a shared hosting site. My host, for example, requires a copy of my photo ID ... if this had been a requirement before I could first evaluate Drupal a software framework, I would simply have moved on to the next product in my evaluation matrix. The point is that it's an additional barrier to entry for non-developers ... which in turn affects potential overall adoption.

I've already had a robust debate with kim.pepper about this already, so he knows my opinions, but for this record:

Composer is awesome, but...

Personally, I feel that any requirement to run anything from command-line for normal operation of Drupal is a step too far. If you've never used it before, its very confusing and difficult. Adding a dependency on it raises the bar for maintaing Drupal sites. Sometimes, as developers, we forget that a great deal of our users not only have no development background, many of them struggle just to maintain Drupal via the UI. But, we accomodate them and make it possible for them to build sites that no other system can build.

Now, for a developer, running composer is easier than dropping a downloaded library in a folder, but for the average non-technical user the latter is something the ACTUALLY know how to do, and the former is gobbledegook.

Our packaging solution must be driven by the same requirements which have defined Drupal to date. That means

- Vendor libraries should have a non-command line method of updating
- Package dependency failures should be elegant and informative, not silent and fatal.

Split core and non: core/vendor & /vendor
Composer normally installs everything in /vendor, however this can be overridden in the config directive...

 "config": {
      "vendor-dir": "custom/path/for/vendor"
 }

Using this pattern, core libraries could be easily placed in core/vendor, while non-core would simply be placed in /vendor as usual, without having to tell anyone to add a directive to their composer.json.

Want to update a core library? This is not supported, and shouldn't be, that's why its core. If you really want to do this, you are probably a developer who understands composer and can work this out for yourself.

Run composer from Drupal

Any solution for using Composer for vendors should include a utility to run composer, similar to Drupal update (or part of Drupal update). This could well be contrib, as its only really required if you want to use external libraries which aren't in core. Modules which want to include a library can specify this module as a dependency.

If a module has declared its version to this composer module, then it could responsible for checking versions and elegantly bailing when there's a problem.

Libraries is heading in this direction anyway, however it doesn't seem to have composer support explicitly, so this looks like a new contrib.

I feel that any requirement to run anything from command-line for normal operation of Drupal is a step too far.

That idea is correct. We cannot require command line operations for normal operation of Drupal under any circumstances.

IMO, this is a feature request and should be moved to D9. We have http://drupal.org/project/composer already. For users on a command line, using drush dl to pull down a new module will also `composer install` the 3rd party dependencies for you. For users not on the command line, they can manually put those files into place.

#19 by effulgentsia

With this proposal, the new process would be:
- Download the 8.1 tarball.
- Remove your site's core/ directory and replace with the new core/ directory.
- Remove your site's vendor/ directory and replace with the new vendor/ directory.
- Run update.php.

By using Composer's vendor-dir configuration ala #1805316: Move composer.json to the project root, the upgrade process stays exactly as is currently documented.

#24 by xtfer

Split core and non: core/vendor & /vendor

Yup! xtfer has it right. Let's move that discussion over to #1805316: Move composer.json to the project root. That patch needs a re-roll.

#10 by jthorson

Generally, I'm more for focusing building a new system that can handle dependencies like this easily, instead of trying to retrofit and re-invent our existing system to meet new requirements... A change like this is not trivial from a testbot perspective ... and my available dev time is quite limited.

Understandable... If you show us where the Testbot and project build process take place, we could have a look at what would be required. Both systems must already have Drush available, so leveraging the Composer Drush component might possibly be an option. More investigation would obviously be required.

#10 by jthorson

We key on /core/vendor to filter out syntax checks, so there will be some additional impacts there too.

This is due to Testbot not understanding PHP 5.4 code, and is easily fixed by ignoring those files. We have issues for those too! #1867192: Testbots need to run on 5.4, 5.5, 5.6 and 7 and your #1683942: File syntax check will choke for modules which require PHP 5.4 ;-) .

#14 by patcon

https://github.com/drupal-fracture/split-scripts
https://github.com/drupal-fracture

Well done, Pat. I believe subtree split was something that sun was having a look at a while ago as well. I could be wrong though. That guy is always up to awesome stuff, as are you apparently.

#15 by sun

Moving /core/vendor into /vendor has consequences

It can still live in core/vendor: #1805316: Move composer.json to the project root

#24 by xtfer

Run composer from Drupal

cpliakas and I have been talking about something like this. I've seen crazier things than a web interface for a console application for a web interface :-) . We should get together on IRC one day for a brainstorm/idea dump one day. Seems a bit out of scope for this discussion, however.

#24 by xtfer

Libraries is heading in this direction anyway, however it doesn't seem to have composer support explicitly, so this looks like a new contrib.

This is something Libraries could handle, but figuring out what it should handle is the key. So far, I've seen a few modules that assist with the use of Composer in the contrib/module space:

1. Composer Autoload to load any Composer-generated files
2. Composer Vendor to load Composer-generated files explicitly in sites/all/vendor.
3. Beer O'Clock Bot to load Composer-generated beer... Nah, I'm joking... But seriously.

#25 by kim.pepper

While you can specify where the vendor directory is (e.g. /core/vendor) you can't have 2 vendor directories. The general idea is that you have a single directory for libraries, without duplicates.

Although the best practice is to have one vendor directory to rule them all, it is possible to have multiple vendor directories. PHP's class loader will load the first class it finds when there are multiple installs of the same namespaces. Again, not generally recommended, but it does work. Would be nice to have it fixed though, and this issue does take steps to fix that.

#21 by patcon

I truly feel that to stay competitive and relevant, we have to first imagine our Drupal ecosystem without all these constraints, and then backtrack into covering ground, putting the muscle into solving problems like effulgentsia brings up. Especially now, Drupal needs all of us to start packing intellectual dynamite in our toolbelts :)

+1... We have to step up and push technology forwards. We need to be leaders in the web world. "The drop is always moving"... Right?

Actionable items
This discussion touches on a lot of things. Here are a few issues that I think we should be looking at, along with a couple of things that we should find issues for:

1. #1805316: Move composer.json to the project root
2. #1867192: Testbots need to run on 5.4, 5.5, 5.6 and 7
3. #1683942: File syntax check will choke for modules which require PHP 5.4
4. Find out where the D.o project packager code is so we could run a composer install on it when packaging
5. Find out where the Testbot code is for downloading and building a project so we could composer install it before running tests

I think there's too many different issues being circled here.

1) Having each contrib module that uses some composed library pull dependencies down itself is a complete non-starter, as noted above. The use of such libraries, however, I expect to only increase (which is IMO a very good thing). I don't think resolving that is something we should punt to contrib; it's core's job to resolve that.

2) Having one vendor directory for core and another for contrib is also a non-starter. As noted above, that breaks things. We shouldn't try to rely on composer figuring that out, because frankly I don't think it is something that should be figured out.

Which leads me to:

A) Yes, we need to have a core-sponsored way to handle adding composer libraries that does not conflict with core's libraries.

B) No, we don't actually need to be flexible with core's dependencies. If core decides to move from Symfony 2.3.4 to 2.3.5, core will do so and the next core update will include that. Presumably that would likely coincide with a security release in Symfony, and therefore in Drupal, too. If you really want to jump ahead, well, learn how to edit your own composer.json file.

Now here's an important observation:

1) Yes, we want composer to manage dependencies, and handle autoloading of composed libraries.

2) If you use Composer on the command line to bring them down, great, it does that all for you.

3) If your most advanced tool is FTP... How exactly do you setup your manually downloaded file to, you know, actually work and get autoloaded? I don't know. That's a good question we need to figure out.

I would also call out the "must work without CLI" statement. That's been true in the past as a goal, but it's not like we've always met it. A LOT of things in Drupal you can only do if you have shell access. Like, Features, which anyone serious about Drupal is using for at least some things, if not all. (Node types are still problematic.)

But if you just want to test drive Drupal, really, "download and stick on a $5 host" is no longer a useful way to do so. With Acquia, Pantheon, Web Enabled, and similar companies all offering dead-simple often free test drive options, the market has changed. How much, and is it enough that we can expect people not on a Drupal-centric host to have shell access? I don't know, and no one else in this thread has hard data on it either that they've cited. So "the majority of people running Drupal don't have shell access" is not a statement that has any current data to back it up.

With Acquia, Pantheon, Web Enabled, and similar companies all offering dead-simple often free test drive options, the market has changed.

This options are known to those already informed about the community ... but the hobbyist does only what they have always known. Download, extract, execute. We've even made it easier for them by auto-creating the database tables, thus eliminating another manual step - let's not counter this improvement by adding another step in it's place unless it's absolutely necessary.

If you show us where the Testbot and project build process take place, we could have a look at what would be required. Both systems must already have Drush available, so leveraging the Composer Drush component might possibly be an option.

I've opened #1923582: Add ability for testbot to run 'composer install' during installation for conversation on testbot dependencies/requirements this would introduce.

Added packing info

Composer Manager is proven to work in Drupal 7 and will be ported to Drupal 8. Fracturing Drupal into distributed packages (ala https://github.com/drupal-fracture ) will likely not be able to happen by Drupal 8.0.

The composer.json file is in the Drupal root. Changing that is like changing Symfony's composer.json file, and doesn't make sense. Changing this is hacking core. Instead, you should specify composer.json file in your module, and use Composer Manager.

As a side-note, #586146: [policy, no patch] Decide if and how to implement semantic versioning for Drupal 8.x would REALLY help, because otherwise we can't automatically index correct version numbers.

Added related/followup tasks

In the mean time, I'm going to mark this as post-poned until we can properly download and install Drupal core with Composer and Composer Installers.

#586146: [policy, no patch] Decide if and how to implement semantic versioning for Drupal 8.x

I think we can get back to discussing this. Semver is a go, at least for core.

IMO, even if it doesn't make 8.0 we need to get first-class composer support in 8.1 at the latest. In core, not via contrib.

Yeah, we should keep this discussion going, IMO. I'm pretty sure the decision to use semver applies to Contrib in addition to Core.

FWIW, I'm more interested now in adding better composer support into core Drush. Drush can now be installed via composer and will likely commit fetching libraries via composer.. Then I think Drush is ready to help solve the 'how to download libs for Contrib' need for CLI users. I'm aware of Drush Composer, and some form of that makes sense in Drush core.

Drush is ready to help solve the 'how to download libs for Contrib' need for CLI users

That's what Composer manager already does for Drupal 7, piggy-backing on top of Rob Loach's Composer plugin for Drush. However for Drupal 8 you run into some pretty serious issues such as the ones below that make it really hard to get around.

• #2128353: D8 strategy for Composer Manager
• #2177729: composer_manager cannot deal with autoloading for container rebuilds

To me this issue is a lot more complex than just using Drush (which fundamentally I do agree with). I would recommend reading through some of the issues in the Composer Manager queue as a lot of thought and debate has happened which I think is relevant to Drupal 8. One issue that is not directly related but has a lot of good context is regardless is highlighted below:

• #1955386: Automatically register Composer's autoloader early in the bootstrap process to improve DX

I might have just found the solution.
(previously posted here, #2128353-16: D8 strategy for Composer Manager)

With the "replace" setting in composer.json, it should be possible to fill a second vendor/ directory that will NOT overlap with core/vendor. It simply means, we tell Composer that we have these packages already covered.

Boot sequence:

1. Bootstrap with core autoload.php for core + core/vendor, at a time when Drupal does not know yet which multisite we are on, which modules are enabled, and which further composer packages beyond core/vendor are needed.
2. Core decides which multisite we are on, and which modules to load.
3. Optionally, we could now run something equivalent to composer install/update (*, **), if dependencies have changed since last time we checked. Of course this only works if we have a web-writable vendor directory.
4. Register the additional packages - preferably to the core autoloader, not a second autoloader.
   To do this, we need a custom version of the AutoloaderInit class. Or we convince the Composer maintainers to make the generated AutoloaderInit class more flexible.
5. Request goes on, page is served.

(*) Requirements:
- Composer itself needs to be in core/vendor, so we have it available for direct PHP calls and don't need to go through the commandline.
- We need a web-writable vendor directory for packages required by modules. We already have a web-writable code generation directory, so this should not be a big deal.
- We need internet access whenever this stuff needs to be updated, and the sources (packagist, github) need to be available. This means, we need to be sure that stuff does not break if an update does not work.

(**) Composer install/update for the additional libraries would work like this:

1. Collect Composer dependencies from enabled modules. These could even go through an alter hook or similar, allowing for dynamic package requirements. (**)
2. Generate a composer.json with the additional packages. Add a "replace" setting for packages already in core/vendor. We might not even need to save this composer.json in the filesystem, we only need the data.
3. Ask Composer to download the packages, recursively, into the web-writable vendor directory, and generate the autoload stuff.

(***) The obvious thing would be that modules have composer.json files. But this might not even be necessary. For my taste, we could also have a composer section in *.info.yml. Imo, this would even be semantically preferable. A Drupal module is not a Composer package, and we should not make it look like one.

In Drupal 9 we might totally turn this around, and get rid of this custom magic stuff.

@donquixote
It would be so great if you could do some basic prototyping of your idea to ensure that it actually works. Given the amount of great packages out there, this would be a really huge win for the ecosystem.

Register the additional packages - preferably to the core autoloader, not a second autoloader.
To do this, we need a custom version of the AutoloaderInit class. Or we convince the Composer maintainers to make the generated AutoloaderInit class more flexible.

https://github.com/composer/composer/issues/2699

@dawehner: Yes, I will attempt to do this when I have time.
I am a little worried that the "replace" setting might not always be sufficient.
We need to look at ./composer.lock, which contains a list of installed packages and detailed information for each package, and some of that could put further constraints on the packages we are going to download.
But I still think it is going to be doable.

Another question: How dynamic do we want this to be?
If you enable a module via the UI, do you really want Composer download to kick in at this exact moment?
Or would we rather have Composer attempt to provide packages for all *downloaded* modules, not just the enabled ones?
What would this mean for module update, if new dependencies are introduced?

So, new idea:
- drush dl will download the module and trigger a composer update.
- If you download the module manually (zip file) and it has a missing dependency, you won't be able to install it via the UI. Instead, you will see a link to http://example.com/core/composer-update.php, which does this stuff for you.

Problem: Composer will attempt to download dependencies and register their autoloads/initialization scripts even for sub-modules that you never intend to enable. You suddenly need to be careful with experimental modules in subfolders.

A solution could be a 3-state or even 4-state module status:
1. Module is disabled, and I don't want packages for this module to be downloaded.
2. Module is disabled, and I want packages to be downloaded.
3. Module is enabled, and packages are already downloaded.
4. Module is enabled and was updated or modified, so some packages may be missing.

This sounds an awful lot like Embedded Composer:

http://srcmvn.com/blog/2013/05/23/symfony-live-2013-portland-embedded-co...
https://github.com/dflydev/dflydev-embedded-composer

It's probably worth talking to Beau about it to see if there's any already-trodden ground here we can leverage. Seems like there is. (Andreas, you know Beau from FIG. :-) )

Crell: Yes. Already mentioned this in the related discussion going on in Composer manager.
[#8477789-29]

So far I missed Beau (simensen) on IRC, and I did not have the time to study the dflydev-embedded-composer library in more detail, but I am also quite busy with client work atm.

Yes, thanks @donquixote for pointing this out. Just to be clear, if we can leverage other tools and eliminate the need for modules like Composer Manager or greatly reduce their responsibilities in favor of something more tightly integrated with core, then I will jump up and down with excitement and give digital hugs to everyone who helps make it happen.

With that being said Composer Manager has solved a lot of "what if"'s and debunked a lot of "why can't you just"'s when using Composer with Drupal in it's current state, so I want to make sure we use the lessons learned when evaluating other tools so we aren't running into the same problems that led to Composer Manager being created in the first place. After taking a look at https://github.com/dflydev/dflydev-embedded-composer it seems like a really great project, but I want to note some of the concerns here as we take a deeper look.

• Does it scale beyond one application and one extension? i.e We have Drupal core and N number of modules, and my concern is that we will have to write a bunch of code to chain together modules as if they are each an extension of the previous "app".
• Will module developers have to do anything else beyond creating a composer.json file?
• Are there any additional workflows beyond normal Drupal and Composer workflows that are required to install dependencies?
• Does it solve the component issue detailed at #2128353-25: D8 strategy for Composer Manager?
• Are we still downloading duplicate library code if the module has a dependency on a packaged included in core?

These are the initial questions I would want vetted, I am sure more will come up as we get deeper.

Somewhat related, but I also discovered that Drupal has modified packages in it's vendor directory which will prove difficult for solutions like Composer Manager and https://github.com/dflydev/dflydev-embedded-composer. For example, all of Composer's metadata files think that Symfony Translation v2.4.1 v2.3.4 is installed, when in fact only the \Symfony\Component\Translation\TranslatorInterface interface is there. See #1852106: Add the symfony translator interface or translation component as reference.

Posting here to hopefully avoid potential WTF's as solutions are experimented. Would also be helpful to know if there are any other modified packages in core/vendor.

@RobLoach - reason for postponing seems to be fixed now.

Not sure if anything still needs to be done here.

You can now use Composer manually or use @bojanz' new and improved Composer manager.

Actually, I think we need to either provide an example or documentation of how to pull in your own composer libraries. This will come up a lot for D8 devs, I'm sure, so we need to make sure that the process for doing so is either painless or well-documented (preferably both).

I agree with #48 you can (and I am) adding additional external libraries with composer.