Drupal.org Composer Logo

Drupal.org's Composer endpoints have been available in beta for some time now, and in that time we've begun to see many, many people use Composer to manage Drupal modules and themes. We first launched these repositories before DrupalCon New Orleans as an alpha release, and move into beta a few months later. After receiving your feedback and bug reports we've made updates, and are ready to call this service stable.

What is Composer?

Composer is a tool for dependency management in PHP. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.

… Composer is strongly inspired by node's npm and ruby's bundler." - Source

In a nutshell, Composer allows you to declare the dependencies of your project in a composer.json file in the root of your PHP project. Those dependencies, which you then install through Composer, can have their own composer.json files and their own dependencies—all of which will be automatically managed and installed by Composer. When you need specific control over the versions of dependencies, you can use a composer.lock file.

You can read more about Composer at GetComposer.org.

How do Drupal.org's composer repositories work?

Drupal.org offers two Composer repositories—one for Drupal 7, and one for Drupal 8. Composer requires that packages adhere to semantic versioning, which Drupal 8 core does, but Drupal 8 contrib, and Drupal 7 core and contrib, don’t. To solve this problem, we've created a Composer façade, which takes all of the metadata about projects on Drupal.org and translates them into a format Composer can understand—including translating the Drupal-specific versioning for Drupal 7 and contrib into semantic versioning.

By creating this façade, we've made sure that Drupal.org is still the canonical source for metadata about Drupal.org projects, and that we can update this translation layer as the versioning schema changes. (Learn more about the effort to move Contrib projects to semantic versioning).

In addition to providing endpoints for building projects, Drupal's automated testing suite— DrupalCI—now uses Composer to test Drupal core and contributed projects. This allows developers to test any external dependencies.

How do I use Drupal.org's Composer repositories?

To begin using Drupal.org's Composer repositories, you'll need to update your composer.json file to include the appropriate Composer repository for the version of Drupal. To use Composer with Drupal 7, use the repository url:

https://packages.drupal.org/7

. To use Composer with Drupal 8, use the repository url:

https://packages.drupal.org/8

, as in this example.

After setting up composer, simply run the command:

$ composer config repositories.drupal composer https://packages.drupal.org/8

And your project's composer.json should be updated to look like the following:

{ 
    "repositories": { 
        "drupal": {
            "type": "composer",
            "url": "https://packages.drupal.org/8" 
        }
    }
}

Once you've made that change, you should be able to use Composer for Drupal modules and themes as you would for any other PHP package, using the drupal/ namespace:

$ composer require drupal/<modulename>

There is one caveat about the pattern: there are some namespace collisions among modules, and so it is on our roadmap to update Drupal.org project pages to specify the exact namespace to use to require a given project.

To learn more about how to use Drupal.org's Composer repositories, and for some troubleshooting tips, read the Project Composer documentation.

What about licensing?

All the projects hosted on Drupal.org are licensed GPLv2 or later or have an entry in the packaging whitelist. This means that you can rely on Drupal Core and contributed modules and themes to be licensed under the GPL or compatible. And if you need to redistribute your code created with Drupal projects, it must be redistributed as GPL-2.0 or GPL-3.0, but please note that Drupal.org will only host GPL-2.0 licensed projects.

However, because Composer is a tool that can manage packages in the wider PHP ecosystem, you might find that you want to require a non-GPL package in your project. Using GPL-licensed Drupal projects with external packages that are GPL compatible is fine. Just be aware that if you redistribute that code, you will have to redistribute under a GPL license.

We cannot provide legal advice for your use of open source software. If you use Composer to install packages that are not compatible with the GPL alongside GPL-licensed projects like Drupal, you may use that software together, but per the terms of the GPL you may not copy, distribute, or modify that software.

"Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted…" GPL 2.0 Section 0.

For more questions about Drupal and the GPL you can read the Licensing FAQ.

The Composer "licenses" command

If you need to check what licenses are in use by your Composer-installed dependencies, you can use the licenses command:

$ composer licenses

This will list the licenses of all the packages your project is using. However, we recommend that all users double check licensing information as some of these packages may be dual-licensed or have other licensing concerns that may not be immediately apparent from this command.

What's next?

At this point, the Drupal.org Composer service is stable and you can use it to manage modules and themes in your production websites. That said, we do have a roadmap of additional features that we'd like to add. And your contributions are welcome!

  • As development on Drupal.org's Composer service continues, we want to focus on the following features:
  • Supporting Composer-based workflows for distributions and install profiles
  • Providing sub-tree splits of Drupal Core
  • Updating project pages to provide information about using Composer with any given Drupal.org hosted project
  • Adding features to the updates service, to collect statistics about projects installed with Composer, and to explore providing update alerts about external dependencies
  • We also hope to work with core maintainers to add the Drupal.org Composer repositories to Drupal Core's composer.json file

If you're interested in learning more about our roadmap for Composer, or contributing to this service on Drupal.org, you can learn more in the Composer plan issue.

How you can help

If you’re interested in helping to improve Drupal.org's support for Composer workflows, please take a look at the issue above, find us on irc in #drupal-infrastructure, or send us a volunteer proposal.

Thanks to our Community Initiative contributors

We'd like to thank the individuals who worked with us as part of this Community Initiative.
In particular, we'd like to thank:

We'd also like to thank Appnovation, who sponsored the initial development of Drupal.org's composer endpoints.

To these volunteers and sponsoring organizations—it is your expertise, your insight, and your affirmation of our work that make these Community Initiatives successful. Thank you!

Comments

mikeytown2’s picture

Something that would be helpful is if there was an example in the examples project https://www.drupal.org/project/examples

kilhage’s picture

I totally agree, would be very helpful

jhodgdon’s picture

I maintain or co-maintain several contributed modules on drupal.org. I have read over the documentation on using composer -- which starts here: https://www.drupal.org/docs/develop/using-composer ...

One of the pages says:

If a contributed project maintainer wishes to add a dependency on a packagist library that is not hosted on drupal.org, they can add a composer.json file to their contributed project. Most contrib developers do not need to do this as long as their drupal.org dependencies are expressed in their info.yml files.

Is that correct -- do I need to add a composer.json file to my contrib projects or not? And is it even a good idea to do so?

I have another question too: if I have a contrib project that contains several sub-modules (like a Base module and two or three add-on modules), how would I use Composer to indicate the differing dependencies? For instance, one of the sub-modules might depend on Views, whereas the Base module doesn't. It seems like there is only one composer.json file for the whole project... the documentation on drupal.org does not seem to say anything about this very common scenario.

Mixologic’s picture

So the first question of "Do I need a composer.json?"  is "it depends" - 1. If your module depends on a packagist library, then you must have a composer json. 2. If your module wants to use some of the more advanced dependency specifications like ~'s and ^'s, then you'll benefit from having a composer.json, (but that only affects other users using composer.) 3. If, during testing, you would like to be able to submit a patch that modifies your dependencies (like say you're adding a new dependency and want to test), then your module will require a composer.json (because changes to info.yml cannot be seen until they are committed, whereas the testing infra can detect changes to composer.json)

Other than that, a composer.json is entirely optional, there's no harm in having one if you dont need those features, but you are not required to have one either.

Regarding your second question:
There are two ways this can be handled, and 'best practice' has yet to emerge.

You can either have one composer.json for the entire project, or you can have one composer.json for each module that is contained in the project, and the difference is subtle.

The use case for a single composer.json for the entire project is thus:  an end user does a composer require drupal/{projectname}
and ends up with *all* dependencies that are required by that project and are able to enable any module or submodule that comes with that project. drupal commerce is a good example of this use case. You download the whole store, and can enable payment processors without additional steps.

The use case for having a separate composer.json per module within a project is if a project comes with multiple optional components and dependencies, and it doesnt make sense for the end user to have all of the code downloaded just to use a couple of parts.  In this case a single composer.json per submodule means that the end user will have to composer require drupal/{submodulename} to get all the parts they need. Projects like the cod_support project are a reflection of this use case. If you had to download *all* of the dependent modules for all the features of cod_support, you would end up with about 100 modules on your site when you might only have needed a few. 

So really its up to the maintainer to decide what style of management they would want to use, and what is going to be the best end user experience. 

 

jhodgdon’s picture

I appreciate the clear answers. It might be a good idea to get those into the Composer docs on drupal.org... :)

philsward’s picture

Can't wait to see this baked into Core and used as the defacto method for installing and updating both core and contribs in the background.  It's getting there!

joseph.olstad’s picture

Our shop found composer to be déficient for site building and site renewal.
Alors on a inventé un nouveau outil. Ça s'appelle git plus. On espère pouvoir bientôt libérer le code source.

fiammybe’s picture

@joseph.olstad: can you elaborate a bit more on where composer fell short according to your experience? We are looking into using a combination of composer and jenkins at work, and I'm interested in all information you could offer. Thanks!

joseph.olstad’s picture

There's too many barriers to adopting the composersphere .  our tool, 'git plus' is programming language agnostic and works for any project.  We're hoping to publish it on github soon hopefully.

Mixologic’s picture

Composer is not intended to be a deployment tool, it is intended to be a *dependency resolution* tool which downloads the required software in a manner that satisfies all of the dependencies of every component in the system.  If git plus comes with an SAT solver, or some other dependency management solver, I'd be impressed, but would also question why you'd rewrite what composer already does.  

Im also curious as to what these 'barriers to adopting' are. I see criticisms of composer (see NikLP's comment below), but none of them appear to be about any particular frustration/problem/issue that can be addressed. It would be much more helpful if the "Gotchas", "Idiosyncracies", and "Barriers" were articulated as concrete issues.

joseph.olstad’s picture

All dependencies for nearly all recent internet projects are in some sort of git repository.  We've just taken the logical step to resolve dependencies by creating a tool that extends git capabilities and goes beyond sub modules, does not limit you to sub modules.  We haven't completed yet all our wishlist of features however with a small amount of work it will be able to scan your drupal project (or any project) for .git folders and auto create a config.json capturing all of your dependencies in its structure.  How we're currently using it is we've got a config.json that is part of a 'build' project, each master project (for instance a distribution or client site) will have a build project that contains a config.json .  git plus will find it and you can simply run one command using 'git plus' and all the projects will be updated (or not), depending on config.

We designed it for drupal sites in mind, however it can work for any project of any language with any type of dependency and any type of sub project and any number of sub projects.  (as long as they're git, which nowadays nearly everything relevant >IS<)  It can even work with a composer site and take over that.

The author of this tool is not yet prepared to liberate it on github however I am working with him on a schedule for doing so.  He doesn't want to release something until its ready.  Right now its ready for our projects , we have a few things to do before its ready for large scale public consumption.

As for composer , it doesn't work for development, because it doesn't help us with rapidly changing sites and doesn't really keep up to constant patching and development of all our dependencies whereas 'git plus' does.  'git plus' isn't limited to just doing development, it can also be used as a 'dependency resolution tool' sort of like what drush make does, but isn't limited to drush or composer.  The only dependency of 'git plus' is a version of 'git' 1.9.x or newer (and I almost forgot, 'git plus' is written in php , so it does need php as well).  Whereas composer takes more effort to get working (phar sources and registered phars) and has a lot more moving parts and isn't git friendly.  'git plus' on the other hand just needs config.json and if it doesn't find one you'll be able to tell it to auto scan your projects and it'll guide you through making one.  Once all the features are in place you'll be able to run the tool and it'll set up the dependencies and clone them as needed if they aren't already in place.

joseph.olstad’s picture

Not only does it handle git commands to all configured git projects dependencies, in the authors words, 'it runs whatever the hell command you want' , so you can use it like a regular unix command that iterates over all your git projects (dependencies) or just specific ones as it has an --includes and --excluses flag option.

On one of our drupal sites we have 20 git projects , with one 'git plus' command we can update all of them (or again specify which ones with --excludes or --includes ).

Mixologic’s picture

It sounds like git plus is similar to fabric (http://www.fabfile.org/) does, which is the deployment tool we're using on drupal.org.  

There are definitely things composer was not designed to do, like downloading javascript assets, as its meant to be a php dependency resolution tool, and not a site building and or deployment tool.  People have extended composer to support some aspects of deployment and site building (notably https://github.com/cweagans/composer-patches and things like https://asset-packagist.org/), but its still not a deployment tool. Perhaps the issue then is people are misunderstanding the role that composer plays in an engineering workflow.

Composer is a php package manager and dependency resolver, intended to determine which php packages are required given that every package has its own constraints and requirements. This is a very, very difficult problem. You cannot look at a git repository and know what dependencies you need to download. You have to look at *all* of the git repositories, and all of their dependencies too, and compare them against each other to make sure you dont end up with conflicting version requirements.  If you are unfamiliar with how difficult of a problem this truly is, I suggest you start with https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c...

drush make is a manifest, and does not do any dependency resolution except when drush detects an unsatisfied dependency. But it doesn't do any resolution because it cannot resolve version constraint conflicts, nor can it fetch external dependencies. Drush is planning on removing drush make and also suggests using composer for drush 9: https://medium.com/@weitzman/drush-9-roadmap-9eb5cd227c50#.mg0i60xyd / https://github.com/drush-ops/drush/issues/2528 

So if you are going to write a tool  to help solve the more holistic problem of 'maintaining multiple sites and deployments', more than likely that tool should rely on composer internally to perform the functions of dependency resolution. 

NikLP’s picture

I have to say, I've tried using composer a couple of times and (purposely) threw the baby & the bath water out of the window in horror. It's full of gotchas and idiosyncrasies and we *do not* need more of that in the Drupalsphere. If we're going to all adopt this as we did drush, we need a foolproof way to use it, and at the moment that necessitates a "newbies/idiots guide to composer", IMHO.

bgrobertson’s picture

I wrote this quick-start guide mainly as a reminder for myself on how to start a new drupal project with composer.

How to Initialize a Drupal 8 Site with Composer

Googling "newbies guide to composer" came back with this helpful resource as well:

A Beginner’s Guide To Composer

Mixologic’s picture

You might want to avoid using '~8.0' in your example for how to include a module. While that will work for a module that has an 8.x-8.0 branch, most of them do not, and it seems as though users would potentially get confused that they need 8 in their require statement (for d8) when they do not.

bgrobertson’s picture

Good point, thanks for the catch! I've updated it.

ressa’s picture

... on a recent project. So I used good old drush make in stead. It is much more intuitive and understandable -- you get what you see.

bgrobertson’s picture

Of course if you are more comfortable with drush make, then use that. The benefit for me of using composer is that it helps us manage our Drupal project dependencies in the same way that we manage our other PHP framework project dependencies. We aren't primarily a Drupal shop, so it helps maintain consistency between projects for the devs who aren't as familiar with Drupal.

Mixologic’s picture

Good ol' drush make is going away : https://github.com/drush-ops/drush/issues/2528, so, if there's specific issue you encounter along the way that would help us improve composer's documenation, it would be very helpful to know what those issues are. (and beneficial to all other community members)

cgmonroe’s picture

So, where do you report issues for the new endpoint?

I'm working on converting to from packagist.drupal-composer.org to the new endpoint.  However, I'm finding that many distributions, e.g. eck 1.0.0-alpha3, do not get installed as a distribution.  They get cloned from the git repo instead.  Even with prefer-dist set.

This causes problem with checking modules into the project level git (because it contains a .git directory) and the info.yml files do not have the version info so they don't get monitored by the normal available updates process.

Didn't have these problems with packagist.drupal-composer.org... sigh

--
A vast majority of stuff done in Drupal is easy...
it's finding the trick that makes it easy that takes Sooo much time.

Mixologic’s picture

You can report issues in the project_composer issue queue: https://www.drupal.org/project/project_composer

Open an issue there, and please provide a composer -vvv require drupal/eck:1.0.0-alpha3 (or whatever your constraint actually is)

It does download dist files fine from what I can tell, so this is likely something with your conversion process that we've failed to properly document.

cgmonroe’s picture

Your example above got me pointed in the right direction.

It seems that it was a combo of an old version of composer and using the right process cleanly go from a .git dev version to a dist version.

With composer 1.2.1, I had to do:

composer remove drupal/eck

composer require drupal/eck:1.0.0-alpha3   (Version needed).

If you do, composer require --prefer-dist drupal/eck ... you do not get the expected latest alpha because composer seems to be using cached information and returning to the dev version.

After I updated to composer 1.3.2, doing a composer require drupal/eck got the latest alpha version without having to include the version.  (Have --prefer-dist in the composer.json file as well).

Bottom line:  Keep your composer up to date with composer self-update!

--
A vast majority of stuff done in Drupal is easy...
it's finding the trick that makes it easy that takes Sooo much time.