As part of the pending release of Drupal 8 I am starting to think about the changing landscape in the contrib space. Specifically I am looking at Composer, how both core and contrib can be deployed using it. One of the small steps towards this is for all contrib modules to have a composer.json file in the root of the module directory.

We already have a documentation page talking about this and a pending patch for the example module.

An example of the minimum each module needs in their composer.json is:

{
    "name": "drupal/example",
    "description": "This is an example composer.json for example module.",
    "type": "drupal-module",
    "license": "GPL-2.0+"
}

The name uses "drupal" as the vendor namespace, followed by the module name after the slash. The description provides more information on the module. The type tells composer it is specifically a Drupal module (there are also types for themes, profiles and more), this allows developers to put "drupal-module" packages into the modules directory rather than the vendor directory on composer install. Finally we add Drupal's license of GPL 2.0+.

Adding a composer.json to each module will allow developers to include the modules via composer and eventually it will allow us to make all modules available on Packagist.

This issue is really all about drumming up support for composer in contrib and formulating some sort of PR (ideally coming from the Drupal Association) targeting module maintainers to add a composer.json.

Comments

timmillwood created an issue. See original summary.

timmillwood’s picture

Title: [PR effort] Contrib maintains should add composer.json » [PR effort] Contrib maintainers should add composer.json
hron84’s picture

I agree with the idea but not with the content of the example composer.json. I think the vendor ID should be not 'drupal' but the nickname of the developer (nickname = d.o. username) to make forked modules distinguishable.

derhasi’s picture

As result of #2401519: [policy] Decide on Composer Package Names we have composer package naming convention. That requires the drupal vendor for all projects hosted on drupal.org.

dddave’s picture

Project: Drupal.org webmasters » Drupal.org content
timmillwood’s picture

I just blogged about this: http://tim.millwoodonline.co.uk/post/126088861940/composer-in-contrib-st...

Feel free to do the same!

DamienMcKenna’s picture

@hron84: Projects downloaded from d.o are supposed to be the authoritative editions of those modules, not an individual developer's personal fork - you don't download projects from drupal.org/users/hron84/[projectname], you go to drupal.org/project/[projectname]. Furthermore, projects have multiple maintainers, so you couldn't tie it to one person's name. Therefore we should specifically not follow a github-like convention here, it should absolutely be what was decided in #2401519: [policy] Decide on Composer Package Names -"drupal/[projectname]".

greg.1.anderson’s picture

The Drupal Composer project is working really well, and provides uniform access to drupal.org via a scripted mirrored repository that correctly adds composer.json metadata to projects. I am concerned that replacing this working process with a manual one might introduce more problems (wrong content in composer.json files) than it solves.

I think that the first step should be to get infrastructure in place to use any composer.json file in drupal.org, and to solve the problem of how to get these into Packagist without requiring individual module maintainers to manually register their projects. Until we have solved these problems, we're better off with the external automated solution, which provides uniform coverage of the contrib space.

timmillwood’s picture

@greg.1.anderson Re:#8 - I worry that enterprise users might not want to depend on drupal-composer.org for their site builds and deployments. I would rather see this kind of stuff managed on drupal.org and packagist.

What you considered to be the first step is a lot of what I was thinking for step two. Mainly because it requires a lot more effort, but also becuase to push things from d.o to packagist we need modules to already have a composer.json. eg:
- PR for packagist to add a create endpoint for the API (https://github.com/composer/packagist/issues/553))
- Patch d.o to create or update package on packagist when project release is done.

This is really just a PR effort to get the word out about contrib modules having composer.json files.

Note: individual module maintainers cannot manually register their projects on packagist, unless they are already a maintainer or are added as a maintainer for the drupal namespace.

joshi.rohit100’s picture

I would rather see this kind of stuff managed on drupal.org and packagist.

I am having a very good feeling if it happens :)

derhasi’s picture

@greg.1.anderson, I second this. We cannot rely on maintainers pushing their projects to packagist.org on their own. And we cannot really work with a packagist not providing all Projects from drupal.org.

So for making the Packagist approach work, the Drupal Infrastructure needs to have full control over all "drupal/..."-Packages and we must make sure that no one can manually create drupal/*-Packages on packagist.org. That could get a hard one.

timmillwood’s picture

@derhasi - Currently only people who have packages on packagist under the drupal namespace can add new ones, so it's already locked down. We can defiantly add something to d.o to push modules to packagist using the API key for one of the maintainers, such as Dries or a DA account.

Yes, it's a lot of work and yes it could be hard, but I really believe it's better than asking large corporations, some deploying new drupal sites every day, to depend on drupal-composer.org.

but... this issue is about the first, very small step, making module maintainers aware they should add a composer.json to their module. Once modules have a composer.json in their d.o repo then, and only then, can we push them to packagist.

DamienMcKenna’s picture

Can those who know the steps necessary please work out a plan and necessary infrastructure / d.o Customizations issues so that we have someone to point people to as the long-term goal? Thanks.

timmillwood’s picture

@DamienMcKenna - Good thinking, I will open up a d.o customisation issue now, I will then blog about and push that topic on monday.

timmillwood’s picture

greg.1.anderson’s picture

We are in strong agreement on the end goals; however, I think that we should start with infrastructure first, and reach out to individuals after it is in place. Drupal Composer is going to be way more reliable than the "herding cats" approach; if it is enterprise customers that this is for, then starting with infrastructure is even more important. There is little value to a system that allows you to use Composer to manage a smattering of modules, particularly if a portion of those contain errors. At a minimum, then, we should have enough infrastructure to report syntax errors if someone commits a bad composer.json file. I will reiterate that I think that individual module maintainers should only be doing this if they wish to populate the 'require' or 'autoload' section of the composer.json file, or any other non-default values.

Also, Pantheon is supporting packagist.drupal-composer.org, currently only at the level of paying the monthly hosting fees, but there are a number of people who would step up if something bad happened. Drupal.org, which is backed by a nonprofit organization dedicated to its promulgation is even better, of course, but I don't think that we should be concerned that packagist.drupal-composer.org is likely to be less stable than, say, packagist.org.

johnv’s picture

When a module is downloaded from d.o., a LICENSE.txt file is added to the zip-file, although this is not supplied by the module maintainters. Can we use the same approach for composer.json?
Some modules (like the D8-version of devel module) already provide susch a file. That file should not be overridden ofcourse.

greg.1.anderson’s picture

In order to add a project to Packagist, you must provide the URL of a git repository that contains a composer.json file at the root of the project. So, the eventual goal is to have a composer.json file checked in to the repository of every project.

timmillwood’s picture

Pantheon is supporting packagist.drupal-composer.org

This is awesome to hear and should reasure many enterprise clients, however maybe needs a PR effort of its own.

I still think packagist is the source where all modules need to be, the canonical place for everything PHP.

Crell’s picture

We should be pushing modules through the Packagist, not our own fork of it. There's no added value to running our own other than saying "we're different and special and egotistical". :-)

A Drupal specific site was fine as an experiment, but if D.o is going to support anything officially and automatically it needs to be Packagist. I've already spoken to jordi and he's confident that the site could easily handle the traffic, and the API for updating things is really simple.

Step one then is to simply add that auto-push so that Drupal modules that have composer.json files automatically get created in/updated on Packagist.org, just like a GitHub repo you opt-in does. The opt in for us is simply if (file_exists('composer.json')). It sounds like we have a consensus on semver for contrib (CORE.Major.Minor.Patch), which should work with Packagist. Let's stop talking and someone write that patch for D.o. :-)

Once that's in place... we can just let The Market(tm) push projects toward creating a proper composer.json, and/or filing patches to add them to existing modules. We don't need to do anything else fancy. The Console generator scaffolding tool already creates a composer.json file that covers the basics.

So let's just build the Packagist integration and move on. No need to keep talking about it.

jhodgdon’s picture

What about #17 - couldn't the Drupal project packaging script, which already adds LICENSE.txt, also add a composer.json file if the project doesn't have one in its 8.x packages? Odds are that many contrib maintainers will not do this themselves.

Crell’s picture

Re #17, see #18: Packagist needs the composer.json file in the repo, NOT in a packaged tarball. And having d.o force-add files to a repository directly seems like asking for trouble.

webchick’s picture

We force-committed when we did the Git migration, so it's not unprecedented.

jhedstrom’s picture

Step one then is to simply add that auto-push so that Drupal modules that have composer.json files automatically get created in/updated on Packagist.org, just like a GitHub repo you opt-in does

I think this is the first step, regardless if we eventually assist semi-neglected or abandoned modules in providing a composer.json file. Actively maintained modules that already have, or soon get, a composer.json file will immediately benefit.

timmillwood’s picture

The main reason I personally am against automating adding composer.json files to git repos is resouces, whos going to build that? we'd be better focusing efforts on #2547617: Notify Packagist about repository update, patching packagist and patching d.o.

kreynen’s picture

I asked a similar question on @timmillwood's blog, but didn't get a response so I'll ask again here. Is the plan to allow contrib maintainers to declare the dependencies for their project in the composer.json?

While Packagist claims to be "The PHP Package Repository", it isn't limited to PHP... or to compatible licenses... or to even having the correct licensing information in the composer.json. Packagist is full of packages we haven't allowed to be distributed with the GPLv2 licensed code users download from Drupal.org like https://packagist.org/packages/fortawesome/font-awesome, https://packagist.org/packages/sunnysideup/flowplayer, https://packagist.org/packages/ohvitorino/openlayers3, etc.

Unless the composer.json file is added programmatically for only the Drupal project itself, I don't see the point of continuing the LWG (and possibly even the Security Team) since we'd be opening things up for developers to make it easy for users to pull in code additional code from any random repository when using Composer to install a module.

If the composer.json is pointed at the just the repo, we also need to revisit #1663716: Drupal.org packaging script overwrite LICENSE.txt file. There might be a better issue where the need for the LICENSE.txt in the git repo is discussed, but we've been ignoring the fact that since the CVS -> Git migration we've been distributing code without a license. Efforts like https://github.com/drupalprojects should have made the need for the LICENSE.txt in the repo painfully obvious. As these projects fork, they get further and further from a version that includes the license. I'm fairly sure the larger open source community would lose their minds if GitHub announced they were only going to included licensing information in their compressed downloads, but for some reason Drupal is willing to overlook the lack of the license in git.

timmillwood’s picture

@kreynen - Like in Drupal 7 a module can declare dependencies in it's composer.json file. Take my module "CouchDB Statistics" as example (http://cgit.drupalcode.org/couchdb_statistics/tree/composer.json) is requires "doctrine/couchdb" in it's composer.json.

I'm not sure I really understand the issue?

License.txt won't get added to packages downloaded from drupal.org via composer because they'll be downloaded directly from git and therefore not go through the packager. Maybe we need a separate issue for this. There is a similar issue relating to the version number not getting added to info.yml. However these have always been an issue if modules were downloaded via git (or previously CVS) as I have been doing for years.

jhodgdon’s picture

Right. I think the point in #26 (and definitely it should be a separate issue) is that going forward, if people are using Composer and/or Git more to download Drupal and contrib stuff, we should make sure that LICENSE.txt is part of our Git repository, and not rely on people getting it as part of the tarball. The GNU Public License requires that the license be distributed with the source. Not having it in Git seems like a potential violation of that, but... perhaps the Licensing Working Group could figure that out.

greg.1.anderson’s picture

#26: Yes, the plan is to expect contrib maintainers to declare the non-module dependencies of their projects in their composer.json file.

While I do understand your point, the situation does exist today where some modules require external, non-GPL-licensed software in order to run. For example, jQuery is distributed under the MIT license. Utilizing Composer to automatically pull in these modules makes it much more convenient to create mix-licensed projects, but it does not fundamentally change the fact that some folks are building Drupal sites today using non-GPL OSS software.

It is a deep philosophical question how the Drupal security team's charter would change in light of common use of external software, but I imagine that things would not be much different. The security team is responsible for responding / coordinating security issues in code hosted on drupal.org. If there is a vulnerability discovered through an external library, then the modules using that library would be noticed, and would have the usual consideration to either fix the problem, which might involve fixing the external library, if possible, or stop using it. If neither of these things are possible, then the module is de-listed. This is a separate issue, though.

Regarding the LICENSE.txt file, I do agree with you that it would be far preferable to always include this file in the repository of every project on drupal.org, to clarify that the software hosted here, at least, is GPL. However, this is also a separate issue to what is being discussed here.

kreynen’s picture

some folks are building Drupal sites today using non-GPL OSS software.

True, but that's not being distributed from Drupal.org.

From a licensing perspective, my understanding has been that our goal was that everything committed to Drupal.org was GPLv2+ or a compatible license like MIT and everything downloaded was GPLv2 only (which lax licenses like MIT and BSD allow). Someone could go off on GitHub and create a GPLv3 fork that included code licensed as AGPLv3 and Apache2, but if users only downloaded from Drupal.org they could safely assume everything was GPLv2. I thought that is why we bothered vetting the URLs included in the .make files for distributions. Developers can add anything they was to their .make locally, but the whitelist ensures what gets packaged for download from Drupal.org is GPLv2 compatible. While there are still issues, that is the goal.

Beyond distributions, most libraries a module depends on required users to go to a URL, download the library, and place the expanded download or some part of it on their server. This made it very obvious when code was and wasn't being vetted by Drupal.org's security reviews and made it the user's decision to create a combination of code that couldn't be considered GPLv2.

While I agree with everyone that the LICENSE.txt issue needs a separate issue, I think it's a requirement for anything beyond individual developers adding their own composer.json to move forward or you are exposing the DA to risk.

With some changes to the process and new tools, the LWG and Drupal.org Whitelist maintainers MIGHT be able to vet the licensing on dependencies added to the composer.json files... as well as the dependencies of those dependencies, but I don't see how the security team could do that... and I doubt adding a notice that notifies users that any project that including a composer.json with dependencies isn't being vetted by Drupal's security is going to go over well either.

I'm not sure what the answers are. I'm just voicing my concerns. I really don't know why who is going to try to review anything for security or licensing issues on Drupal.org if Composer becomes a recommended way to install Drupal projects since they aren't doing any of that on Packagist... which could be one reason to maintain a Packagist fork. There are also efforts like https://security.sensiolabs.org/.

The open source landscape is very different today than when Druapl.org's original CVS/Git Repository Usage policy was drafted. The LWG has been trying to take some small steps towards relaxing our GPLv2 on everything policies and allow non-code assets like fonts and images that use licenses that aren't strictly GPL compatible, but accepted as GPL friendly. Embracing Composer at this level seems less like a next logical step for Drupal.org and more like a leap off a cliff.

greg.1.anderson’s picture

I'm not really familiar with the process of vetting URLs contained in .make files. Is there a document that you could refer us to that describes what is being done in this area today? An external library in a .make file is very similar to a requirement listed in a composer.json file; perhaps the existing process might be compatible? It *does* sound labor-intensive, regardless of whether the reference is in a .make file or a composer.json file, so I do see your concern.

jhedstrom’s picture

kreynen’s picture

#31: and https://www.drupal.org/project/drupalorg_drush

Because the whitelist is really a collection of regex for the URLs of repos vetted for licensing issues, it's possible we could build on that... but the check would need to be on the commit vs. the make/packaging.

greg.1.anderson’s picture

Okay; I presume that since the whitelist is full of regular expressions, that the drupal.org packaging tool probably rejects makefiles that contain external references that are not whitelisted. This makes sense, since the process of creating a distribution involves copying all of the software included in the build onto drupal.org.

Adding a composer.json with external references is slightly different than this situation, since in the case of Composer, the external libraries are not being distributed from Drupal.org -- the end user must download them via `composer install`. If someone were to write an equivalent tool, that built distributions from composer.json files, it would be possible to install the code via `composer install --prefer-source` and then examine the composer.lock file to find the repositories where everything was pulled from. At this point, you could reject repositories that are not listed in the existing whitelist.

So, my take on this is that switching to composer.json does not substantially change our current situation today vis-a-vis external library usage. This issue is only about adding a composer.json file, though; using it to create distributions on Drupal.org is a separate discussion.

Crell’s picture

Moving to composer-based module installation is inevitable. It's just a matter of timeline and if we fight it or try to get out in front of it. I'd rather get out in front of it.

It sounds like there's 2 licensing-related concerns here:

1) If modules are being installed via composer, they bypass the packaging system so we need to start adding LICENSE.txt to the repo directly. That sounds like a task for the LWG to take on to coordinate with infra.

2) The composer.json file is way harder for the LWG to audit to make sure no one is depending on a GPLv3 or Apache library, and use of 3rd party libs will, presumably, become more common not less, so this creates a lot more work for LWG.

However! It may make it easier to automate, since composer.json files can specify their license. Composer can also report the licenses used by all projects in a given install. (Go into /core of a D8 checkout and run composer license.) That means if a module uses composer.json to specify its dependencies it should be super-simple to scan it and say "wait, there's an Apache2 project in here, team back..." So in a way, we can leverage composer usage to make it easier, not harder, to enforce license dependencies. It likely involves some d.o work to run some kind of auditing tool, but that's a solvable problem.

So I think we need 2 spinoff issues: One to start requiring LICENSE.txt files in the repo, and one to write a composer-based license audit script for d.o. We can still soft-encourage developers to provide a composer.json file to make their module installable via composer in the meantime. The more do, the easier it becomes to audit dependencies. :-)

Mixologic’s picture

We should be pushing modules through the Packagist, not our own fork of it. There's no added value to running our own other than saying "we're different and special and egotistical"

The drupal.org infrastructure team is responsible for maintaining services critical to the drupal project. That means that we provide reliable, fast, geographically distributed, redundant services for the community, and we maintain the communication channels surrounding the uptime/downtime/maintenance or otherwise. We've got agreements in place with service providers due to our non-profit status that allow us major cost savings for the considerable amount of resources that our services take up.

From what I can tell, packagist.org is *not* offering that level of service, on the contrary it seems to be suggested that if you need something 'Fast and Reliable' that you should pay for Toran proxy. It feels a bit like we're forcing people into a situation where paying for Toran proxy may be a necessary expense in order to support enterprise grade drupal.

At the same time, we aren't going to get away from a dependency on packagist.org, as we're already going to be using it to resolve all of the non-drupal packages.

An additional consideration is that packagist.org (or any packagist installation for that matter) offers two main features: its a metadata repository that composer communicates with, and its a website for package discovery. We already have a website for package(module) discovery: drupal.org.

Putting another place on the web out there where developers do module discovery has the potential to obscure/obfuscate the data and stats that we get from our website analytics and usage stats, including geographic distribution data like which ip addresses hit us/abuse us. Additionally the packagist module discovery experience isn't particularly stellar. If I'm a drupal developer and want a mailchimp module, a search for mailchimp turns up 93 packages on packagist currently. Assuming that I just want drupal modules because the laravel, concrete, zend, and every other frameworks implementation is useless to me, how would I even do that?

As far as serving metadata to composer itself, drupal.org could setup satis to serve as the metadata repository for composer to consume for drupal core and all of the drupal modules on d.o: https://getcomposer.org/doc/articles/handling-private-packages-with-sati.... The only drawback I can see with having a different location for metadata than packagist.org is that devs will have to add a 'repository' section to their root composer.json. Which they will probably have to do anyway so they can support their own custom modules.

If drupal.org were running its own satis install, we could generate composer.json files for ones that didnt have it relatively easily, unless we explicitly do not want that.

Mixologic’s picture

Sorry to derail the thread with what should be its own issue as well. But on the actual subject of the issue, what do we do about the modules within modules pattern that we support now?

For example, the search_api_elasticsearch module has a composer.json in its root:

http://cgit.drupalcode.org/search_api_elasticsearch/tree/composer.json

but it also contains a submodule with its own composer.json:

http://cgit.drupalcode.org/search_api_elasticsearch/tree/modules/elastic...

How does this change things?

greg.1.anderson’s picture

#37 brings up a good point that is relevant here. Composer does not support the multiple-modules per project that Drupal uses. To be fair, this is only partially supported by drupal.org; you cannot map from a module to a project, and dependencies are by module, not by project, making it difficult to automatically find the right project for an arbitrary module -- the site builder must pre-select the projects (supported by drupal.org / drush make), and then Drupal handles resolving dependencies between the available modules as they are enabled.

Composer can resolve dependencies at project selection time, but each project is its own atomic unit; a project cannot contain multiple sub-project inside of itself, each with their own dependencies. The way Composer handles that is by having you define each sub-project as a separate project in its own right. Once you do that, then you can have each project manage its own dependencies via the normal `require` mechanism.

Drupal-composer.org does not attempt to reconcile this difference. Any composer.json file inside of submodules in a project is ignored. In order for the submodule's dependencies to be correctly resolved (at `composer install` time), they must be `require`d from the top-level project's composer.json file. This only causes submodule's dependencies to be downloaded; at pm-enable time, runtime dependencies are still handled by Drupal, as usual, from the module's .info or .info.yml file.

Crell’s picture

Mixologic: I think we've been around and around this issue in previous threads, in particular #2477789: Use composer to build sites.

In short, my stance can be summed up as this: We can find all the reasons why we can't use Packagist, fight the trends in the PHP community, continue to stand-apart from the rest of the PHP world, and spend some non-trivial amount of money building and maintaining our own Packagist clone...

... or we can simply accept that Packagist/composer has won in the PHP OSS world, and fighting it is a battle we will, inevitably, lose, and instead spend the same amount of money making Packagist work better for us. We've already had an influence on it; Drupal's desire for an "owned" namespace is one of the reasons Packagist now has that feature (not the only, but it helped) and Tim is working on adding a create-project API call PR.

Could we run a Drupal-specific Toran proxy perhaps? I don't know, maybe. Could we provide some stipend to Packagist to help with uptime/stability in exchange for goodwill or marketing? I don't know, maybe. Could we add features to Packagist to limit a search to certain frameworks? Probably, and when Laravel made their own non-Packagist.org Laravel-specific front-end they got a lot of flack for NOT contributing back. (Or not; we still maintain Drupal.org as a place to develop and find modules. What do we care if it's possible to locate them through Packagist, too?)

We're an Open Source project. Instead of figuring out all the reasons we can't work with other OSS projects, let's figure out ways that we can work with other OSS projects.

derhasi’s picture

Drupal-composer.org does not attempt to reconcile this difference. Any composer.json file inside of submodules in a project is ignored. I

Actually Drupal Packagist plans to provide metapackages for those use cases. To make that approach work with drupal.org and Packagist, maybe some changes would be needed to the Packagist API besides the "create" callback.

But in general that should not be a problem, because in Composer dependencies are managed on a "Package" base, not on a module base. If a maintainer wants the module to be installed separately, the module should have its own project. Dependency of modules within Drupal should be not a concern of the composer integration.

Instead of putting resources in a separate Packagist, I think it's worthwile using those for improving packagist.org.

cweagans’s picture

Re #9: We're using drupal-composer.org at NBC. Not sure why it would be a problem, tbh.

Re #39: I don't think anyone is arguing against supporting composer-based workflows for Drupal modules. The main argument is about whether or not to push to the official packagist.

and spend some non-trivial amount of money building and maintaining our own Packagist clone...

I don't think we need to build a packagist clone. We just need a very small module that outputs JSON data. That's it. Nothing special, and certainly not expensive. Hell, I volunteered in IRC today to write it for free, and by doing so, we get all the benefits of our existing D.o infrastructure.

We also avoid some of the problems. What happens if Packagist is down when we're trying to push a new release? Does that release just not get pushed? Do we have to add a way to manually reconcile the data with packagist with the data in the d.o database? Is there just a queue? Historically, we've avoided adding hard dependencies like that to Drupal.org.

I get that integrating with the wider PHP community is important, but I don't see how doing it this way is anything like that. If we push, i.e. Panels, to the official packagist, who's going to use it? Drupal sites. Nobody else. It doesn't make sense to try to force something like that into the wider community if they'll literally never be able to use it.

Mixologic also pointed me at the official documentation - https://getcomposer.org/doc/05-repositories.md#hosting-your-own

If you have a project which has its own ecosystem, and the packages aren't really reusable by the greater PHP community, you might want to keep them separate to packagist. An example of this would be wordpress plugins.

Wordpress plugins are analogous to Drupal modules, no? We're a PHP project, but the vast majority of projects on Drupal.org are only useful to Drupal users. It's possible that will change in the future, but for right now, that's the reality.

Technical issues aside, I find it hard to see how "working with other OSS projects" can mean "foisting 10s of thousands of modules on an OSS project". That sounds like dumping our problems on somebody else, to be frank. According to the Packagist statistics (https://packagist.org/statistics), we'd be increasing the total number of packages by 25% overnight, and doing at least that to the releases. Traffic is the big thing, though. I know you mentioned in #20 that you talked to somebody and they said it'd be fine, but it's possible that they weren't aware of what they were actually committing to. Mixologic noted that we have 250,000 transient drupal sites being created and destroyed every week, not to mention the traffic from existing sites that need to update or install packages. That's a lot of responsibility to drop on a third party. That's not to say that we couldn't help them out either in terms of manpower or finances, but when we can write a simple module to do it for free as an experiment, why not give it a go?

We've also only been talking about modules here, from what I can tell. Do we really want to push themes and distros and drush extensions and the like to Packagist? Maybe? But those packages are going to be even less less useful to the PHP community.

In any case, I think we have some things that we have to figure out on our end before we just blindly push all of our modules to packagist anyway, because once that happens, we're kind of stuck. Want to change how subprojects are handled? Tough. If we served the JSON ourselves, we could label it as "experimental" or "beta" or whatever until we have those things ironed out.

timmillwood’s picture

What happens if Packagist is down when we're trying to push a new release?

If you're managing the whole site with composer (and not just contrib modules) then you'll still be stuck because of all the dependencies drupal/core has which are distributed on packagist.

Also Re: #41. This issue we are just talking about modules adding a composer.json. We don't need a big discussion about if they go to packagist or somewhere else. All that is needed is for all module maintainers to start adding a composer.json to their D8 modules.

#2547617: Notify Packagist about repository update might be a better place to discuss if we push to packagist or not, because there I have a patch which *does* push to packagist. Yesterday the packagist maintainer deployed my patch to add a create endpoint to the API, so we can now create packages via the API from drupal.org.

cweagans’s picture

If you're managing the whole site with composer (and not just contrib modules) then you'll still be stuck because of all the dependencies drupal/core has which are distributed on packagist.

Actually, I was more talking about "push" in the context of Drupal.org sending information to Packagist on a node save. That is, as a project maintainer, if I create a new release on Drupal.org (which would send some information to Packagist) and packagist is down, what happens?

This issue we are just talking about modules adding a composer.json. We don't need a big discussion about if they go to packagist or somewhere else.

Yeah, this is probably not the right issue, however, if we *don't* go with packagist and just serve the JSON directly from Drupal.org, I don't think we have to parse through all the composer.json files in the project repositories. We already have that information in the database, so if we go this other direction, we can avoid the effort of encouraging people to commit composer.json to their d.o repos.

timmillwood’s picture

if I create a new release on Drupal.org (which would send some information to Packagist) and packagist is down, what happens?

As you'll see in #2547617: Notify Packagist about repository update I am looking to push to Packagist on git commit as Packagist cares about tags and branches, not d.o release nodes.

cweagans’s picture

This issue may not be necessary depending on the approach chosen in #2551607: [meta] Path forward for Composer support

webchick’s picture

Status: Active » Postponed
Crell’s picture

Symfony, Zend, Cake, and many other project-specific packages live on Packagist. Most of the major frameworks now distribute themselves and their add-ons via Packagist.

This is also about enabling Drupal site owners/builders to setup and manage their Drupal site entirely through Composer/Packagist. That means even if we weren't using Packagist.org, we would still need to have composer.json files in the repo so that Composer could handle dependencies properly.

cweagans’s picture

Please see most recent responses in #2551607.

timmillwood’s picture

Status: Postponed » Needs review

I'm not sure this should be postponed, whatever direction we go, having modules listed on packagist or not, modules will still need composer.json files in their git repo. Getting module maintainers to add them is defiantly the most feasible and scalable option.

cweagans’s picture

It's not a requirement to do that if we use the composer parser from drupal packagist or serve the json directly from drupal.org or if we write our own integration with a self hosted packagist instance. Literally the only reason that this needs to move forward is if we're going to publish directly to packagist.

I won't change the status back because that's unproductive, but I don't see any reason to push this until after a decision is made in the other issue.

timmillwood’s picture

Ok, if we don't do a workaround or "hack" and do things like most others in the PHP world, we'll need all projects that want to be made available to composer to have a composer.json file in the repo.

gisle’s picture

timmillwood wrote:

Ok, if we don't do a workaround or "hack" and do things like most others in the PHP world, we'll need all projects that want to be made available to composer to have a composer.json file in the repo.

Insisting that any other solution than the one you're trying to sell is a "hack", and that "most others in the PHP world" favor your solution does not impress me much. Instead I've noticed that some PHP projects with plugin architectures similar to Drupal (i.e. WordPress and Joomla!) does not impose such a requirement on maintainers, and wonder what makes Drupal so different as to require this.

IMHO, using Composer for dependency management is one thing, requiring all maintainers (including those with no dependencies) to add composer.json is another, and exactly how we make use of Composer is a third.

Insisting that all contrib maintainers add composer.json to their repos is just one of the possible outcomes of discussions taking place elsewhere (including #2551607: [meta] Path forward for Composer support).

So far, I've noted some notable objections to your suggestions has been dismissed with vigorous hand-waving ("doesn't seem to be an issue"), and condescending remarks ("it sounds a lot that you're trying to find issues and work around that don't really exist"). That don't impress me much either.

I won't change the status back because that's unproductive, but I don't see any reason to push this until after a decision is made in those other discussions.

Crell’s picture

Status: Needs review » Fixed

I think we can close this issue at this point, as we now have our own Composer endpoint that handles generating composer.json files as needed, plus many people have added them, plus Drupal Console's scaffolding tools create one.

PMs, reopen if you disagree.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.