Proposal: Composer Support in Core initiative

Will moving the vendor directory up a level become part of the default packaging or is that too big a change?

What about making it so that installing modules or libraries with Composer doesn't then leave core's composer.json file in a hacked state?

This is a great proposal. I love it.

Hello,

Great initiative.

To download translations with Composer I had made a Composer plugin (based on Drupal Scaffold), if that can help: https://github.com/FlorentTorregrosa/drupal-l10n

It is not manageable directly with Composer, you need to had a repository in the repository section of composer.json.

I have not yet added it to Packagist because I wait for the issue https://github.com/drupal-composer/drupal-scaffold/issues/64 to land.

So this is basically the 'turn Drupal into drupal-project' initiative? Great!

Re API changes: And doing this correctly might even need to introduce BC breaking changes. :\

That can't be possible, because there is currently no Composer-based API, or defined behavior for what should happen when you use Composer with Drupal. (Which is why we're in the state we're in.)

Concrete things to do:

• #2867757: Improve instructions for updating composer.json and /vendor
• #2876669: Fix dependency version requirement declarations in components
• #2874198: Create and run dependency regression tests for core

Are there any specific workflows or behaviors that can be described in concrete terms in the IS?

'First-class citizen' is evocative, but not descriptive.

Here are some questions to answer:

Should we be able to generate a filesystem similar to the current tarball distribution using the tools created for this issue? (I say yes)

Should we have a way to hide all executables from docroot? (yes)

Does that mean using assetic or maybe extending our cache system to put files in public places? (maybe)

How will we test outcomes? (add continuous delivery steps to testbot triggers)

[ add yours ]

Over in #2940731-32: Automatic Updates Initiative overview and roadmap we learn of a doc with some specs for improving composer: https://docs.google.com/document/d/1QoKtjuufURIRwO8tcQ9EPUKyvIQmn55Vy4kC...

That google doc was just some of the brainstorms that Bojan and myself came up with at drupalcon in about 30 min. Theres more to be done, and more thats needed, and we likely didnt catch all of the things. For example, core should not ship with a .lock file, etc.

Allow contributed projects to be downloaded via Composer (currently this requires manually copy/pasting JSON configuration into your code base)

Can somebody clarify what this means? AFAIK, this is already done, is packages.drupal.org, and does not require any json configuration. (it used to, a long time ago, but that has long ago been added to core)

Can somebody clarify what this means?

I think it means people don't know you can type composer require drupal/contrib-module and not have to edit any files.

people don't know you can type composer require

before you can do that you have to convert your existing install to one of "composer projects" which all contains lots of useless packages (which require to be removed manually)

Imagine tons of sites was started by drush make or just downloaded zip tarball at times of 8.0 when composer integration was weak...

In russian support chat I see few times a week a question about how to upgrade project which was created without "drupal-composer/drupal-project" or how to deal with shared hosting which does not have composer.

Lots of shared hostings does not give you ssh-shell access and dictate you to keep all files in one "web-root" so each time upgrade of d8 becomes nightmare - you delete vendor/lock then manually compare new core shipped composer.json with your local one and only after this "black magic" you trying to rebuild your whole site

This issue should start care about shared hostings! Good hosting is one which works stable and not the one which allows you to have shell access or dev tools.

One goal of this initiative is to end up where any new drupal site would never have to go through any sort of 'conversion' process. They would be set up properly from the beginning.

Part of that, which should probably be reflected in the issue summary is a migration path to help those users move to the new, usable layout that doesnt have upstream composer.lock conflicts etc.

There are some crude migration tools that could probably use some exercising to potentially leverage:
https://www.drupal.org/project/composerize and https://github.com/grasmash/composerize-drupal

Lots of shared hostings does not give you ssh-shell access

This shouldn't be a problem. Manage composer local. Deploy artifacts.

dictate you to keep all files in one "web-root"

This is what I could see being the problem. But I remember my shared hosting always having "public_html" and then things outside of it. While I don't remember public_html being configurable, that could be a barrier to document "how to customize the web root directory"

We should have as a goal the ability to generate the site the way the tarball is now. Essentially the reverse of drupal-project: We want repo-core to be more like drupal-project, but with a different build to make the tarball style.

Then d.o can perform that build to make the tarball.

One big big thing here: i have a simple site using drupal-project, and EVERY time i need to upgrade drupal i found lots of issues for this, each time have to dig on composer and everywhere to find how to upgrade. So, we need a simple way to do this, and a way that works.

@Mile23 re #9

Re API changes: And doing this correctly might even need to introduce BC breaking changes. :\

That can't be possible, because there is currently no Composer-based API, or defined behavior for what should happen when you use Composer with Drupal. (Which is why we're in the state we're in.)

Agree there is no existing API for composer, but there are lots of other APIs and features that users currently rely on:

• List of available updates /admin/reports/updates
• Install module /admin/modules/install
• Update module /admin/reports/updates/update
• Equivalent drush commands: pm-updatestatus, pm-download, pm-updatecode, pm-releases
• Structure of git repository - wrapping everything in a top-level web directory would have implications and could create problems for every clone (e.g. conflicts or breaks to scripts)
• Structure of webroot - wrapping everything in a top-level web directory would have implications and could create problems for every site (e.g. need to update webserver config, and maybe can't if using web hosting; breaks to scripts)
• Server requirements, in particular memory, where my experience is that composer can use 5 times as much memory than it takes to run a low-traffic site

So I think there are some key questions

• How much will users who don't need composer be forced to change? - ideally none.
• How much will users who want to use composer have to change compared with non-composer? - ideally as little as possible. What I mean by that is that if we can avoid the need for users to learn how to use composer and instead present them with familiar web and drush interfaces then adoption is much easier.

@webchick Great initiative. Please can I add the following ideas:

Scope

1. The obvious goal is to manage PHP dependencies of modules.
2. I propose that a second important goal for this initiative is to integrate with drush. In particular the last symfony version upgrade caused many difficulties #2874827: Drush 8.x doesn't install Drupal 8.4.x and Drush master doesn't install Drupal 8.3.x, and as I understand the drush maintainers have made clear that a site-local drush is essential to avoid dependency conflicts. There are no particular technical difficulties - I'm just mentioning it to request that the initiative explicitly cover it.
3. Thirdly, some modules are currently using composer to manage JS dependencies. This has some pitfalls: JS modules don't typically provide composer.json, and even if they do Drupal+composer requires a Drupal-specific line ("type": "drupal-library") in the composer file. The initiative could provide guidance on whether or not this is recommended and how best to proceed.

Obstacles

@cilefen has already linked #2845379: Provide optional composer integration but don't force users to understand how to use composer which is helpful as a source of potential obstacles to composer. In addition to things you have already mentioned in the IS:

• Using composer directly is complicated. Much better not to expose it and instead provide familiar UI and drush commands. This is the same sense that Drupal uses symfony and twig, but someone can build a site without realising that.
• Composer uses lots and lots of memory, which can easily be more than is available on the hosting for the site. Can we optimise it and push fixes upstream? Any other options?
• We need to be careful not to lose function. E.g. as far as I know Drupal+composer doesn't provide a way to install only security updates.

The obvious goal is to manage PHP dependencies of modules.

I think that's not correct. I think what has to happen is that we determine some workflows for tasks like installing, updating, etc. The thing is, we already have those (modulo #2867757: Improve instructions for updating composer.json and /vendor), so we should support them to the extent we can.

Ostensibly, we also have workflows that we want but can't do without a lot of hassle. That's why we have drupal-project, which deals with some of the problems this represents.

So we need *those* workflows as requirements and test cases, and they should be folded in to core's behavior.

That's why I posted #10.

I propose that a second important goal for this initiative is to integrate with drush.

You'll need to talk to the Drush people about that.

But ultimately: Composer allows you to gather dependencies. It can be overloaded to do other things, but that's it's job. Therefore this isssue seems to be about assembling a codebase and dependencies.

For building a site, we have #2940733: Site Builder Tool/Project Browser initiative and for managing sites we still have drush and drupal/console.

Thirdly, some modules are currently using composer to manage JS dependencies. This has some pitfalls:

Agreed, there are some pitfalls, but not the ones you mention. :-) Using https://asset-packagist.org/ to manage JS dependencies is not a bad idea, if you're already using Composer and don't want to force a node.js installation. The downside is that you better need one of the packages they have, because I don't see a way to submit your own. Regardless, though, JS and other assets are really the domain of #2940733: Site Builder Tool/Project Browser initiative. It makes much more sense to have a Drupal-specific tool that can, per module, pull the asset package while also compiling your LESS or whatever and putting it all where it belongs.

@Mile23 I agree that in #10 you asked some valid questions. We all come from different backgrounds and use cases, so naturally I will raise different questions, which is presumably the point of the discussion.

I agree that there are various workflows already. These are clearly a good reference, but are not necessarily sufficiently robust/clean/easy-to-follow to be the end product of this issue. I have re-read the IS, and I still tend to think the obvious goal is to manage PHP dependencies of modules. Drupal has a system for upgrading and installing that works, but lacks the ability to manage PHP dependencies. The goal is to add that ability without losing/breaking anything else. Anyway we are potentially just getting tied up in different ways of expressing ourselves.

#10:

'First-class citizen' is evocative, but not descriptive.

I interpret it as meaning things like this:

• Single well-defined, well-documented procedure to follow.
• Uses standard existing interfaces: GUI, drush etc. Required level of expertise comparable to rest of Drupal - not expertise in external tool such as composer.
• Robust with clear error-reporting.
• Suitable for full spectrum of users (novice->export, tiny (shared hosting)->huge site, etc.), matches existing set of available function.
• Seamless migration procedure, without requiring memory upgrades / webserver config changes/ etc.
• Required code is integrated into Drupal.org system with defined maintainers, working tests, etc.

You'll need to talk to the Drush people about that.

I am asking that this initiative take the drush issue into scope - I don't think it makes sense for me to tackle it independently. I am not expecting that it would need any changes to drush - it's just a matter of factoring a drush option into the composer files that are generated.

Agreed, there are some pitfalls, but not the ones you mention. :-)

Well the ones I mentioned are the ones that I have personal experience of in my modules and one or two others I've seen:-) You might have different experience, and you might even believe you have a solution to the pitfalls I found - in which case good. Nevertheless I feel these are worthwhile points to raise so that this initiative can define and document the correct way to do it - or as you say perhaps defer it to another initiative. Much of the problem we have now is lack of clarity and guidance on the right way to do things for those outside the "inner circle" of composer experts.

@AdamPS, yes, I apologize. I committed the smarm fallacy. I was trying to steer the scope here... This initiative seems to be the 'make Drupal core more like drupal-project' initiative. That will eventually allow for creating new better tools to address other problems like the ones you mention.

Updating IS to reflect #14, #20.

I'm not folding in #11 because that document deals with a lot of specifics and this issue is general. Leaving the 'Needs issue summary update' tag, however since we need some of that info to migrate here.

Hello

My question is is there a deadline for this project ? when we can have the Phase 1,2,3 done ? before that, we still have to struggling with the composer command which always doesn't work.

I'm unclear on point 1 in Phase 1. Is the intention to have the Composery stuff happen without the person running any Composer commands or any command line interaction? Or does that mean add the scaffolding bits into core so when they do run commands it "just works" ?

if I may interpret: it's sound like "yes: we're going to use composer to do Composery stuff"

OK, here is a radical proposal and it might be totally flawed, but I feel it is worth asking the community to consider it because it seems potentially to solve the problems here in a much simpler way.

Principle goal

As I see it from reading the IS: allow users to install modules that have PHP dependencies as easily as they currently install modules.

Existing proposal

Lots of changes!

1. In addition to the current way of installing modules (drush/GUI etc), add a second one (composer). Same for Core and scaffold.
2. Provide a migration from the existing to the new.
3. Provide an interface to the new as usable as the existing ("distant future").
4. Move to semantic versioning.
5. Address composer scalability issues, or increase Drupal memory requirement.
6. Potential complications with web root moving into sub-directory.

Alternative proposal

I have this roughly working on live sites, and it is order of one hundred lines of new change, with exactly the same interface - end users are not aware of any difference. No doubt there are complications, subtle cases that need the community to help figure out.

• Install modules exactly as now. Use composer only to manage external dependencies.
• Drush hook post-download and post-upgrade. Similar code in GUI. In each case run new code that scans all modules for external dependencies, combines with Core dependencies and builds a composer.json.
• If that's different from the previous version then call composer to update dependencies.

Composer config is simpler without all the modules so less memory needed. Sites happily using the existing proposal can continue. The alternative is for sites/users struggling with composer or preferring to minimise change from existing processes.

Feedback please

Please try to have an attitude of brain-storming "it could work if we fix X" rather than just rejecting it. Thanks for reading.

@AdamPS
No need to reject it, we built it and used it in production for many months: https://www.drupal.org/project/composer_manager
This approach is older than the approach of using Composer to manage entire modules.

However, there are no upsides to it. You are still using Composer, just letting it do only half the work, while the other half is clumsily done on the Drupal side (for example, half-hearted conflict resolution, since Composer's resolver is now only half involved). All CLI and performance downsides remain.

Hello,

For custom modules (and for contrib modules with dependencies not on packagist.org such as JS dependencies), the usage of the https://github.com/wikimedia/composer-merge-plugin can help.

And maybe, it can provide a standard approach to the problem.

There is an issue on drupal-composer/drupal-project to have it the project template: https://github.com/drupal-composer/drupal-project/issues/25

Also for distributions and Composer, what is the standard approach?

To have distributions modules into separated repositories and when building a distribution it will be in your_project/www/modules/contrib?
Or to have distributions embedding custom modules your_project/www/profiles/contrib/your_profile/modules/?

Can composer be included in Drupal core (with the composer.phar file included) and replace the current mechanisms of updating drupal via web interface?

If that is feasible, instead of having "two ways" of updating that may not be compatible, everything would be using the same method.

@webchick I would greatly value your input here. In #29 you say GUI/drush integration is "distant future ATM" but if so I wonder whether this initiative would achieve its goal of removing the "composer problem".

Below I ask the question whether a variation solution (already implemented but no longer used) could be more effective for those struggling with composer (as it works with the existing GUI/drush/etc) and less implementation effort (does not need semantic versioning, migration job, etc.) Do you welcome the discussion or would you prefer me to shut up:-)

Thanks @bojanz, so in an attempt to pull together recent comments there are two main classes of solution and it seems it will help to have names for them.

1) "Composer Lite" = CL
Manage modules using existing methods and just use composer for "external" dependencies. Examples are #30, #31 "Composer Manager", and presumably #32 "Composer Merge Plugin".

2) "Composer Full" = CF
Use composer for everything including Drupal core and modules. Examples are the current documented options A, B.

I realise that the community of composer fans find CF suits their needs and will feel that I am heading backwards here. However I would like to raise the possibility that CL might actually better meet the needs of the site builder who knows nothing of composer but understands the existing processes and would prefer not to learn something new. The issue summary suggests that these users are the main target here - presumably the composer fans are already using composer and are content and I am not proposing anything to prevent these users continuing with CF.

CL advantages for users struggling with composer

1. Keep existing interfaces - no need to retrain users or for developers to alter automatic scripts/hooks. (Implementation uses drush hooks as per composer manager, plus also hooks from GUI; automatically updates composer file and runs composer if needed.)
2. No need for a migration procedure to "turn on composer".
3. Works with existing version numbering (not dependent on a task to move to semantic versioning).
4. Keep existing options such as "only apply security updates".
5. Does not require moving web root into sub-directory.
6. Reduced memory requirement and faster execution as composer has fewer targets to process - and do not need to run composer at all except where module add/alter external dependencies. (Decreases the need to consider performance improvements to composer.)
7. Future possibility to use composer merge plugin to allow modules to define composer wrappers for non-php dependencies as per #32 (the required composer statements are only allowed at the top-level composer).
8. Increased stability as tends to preserve the tested versions of dependencies from core composer.lock whereas as I understand it with CF any site could by bad luck pick up a new release of an external dependency that has a compatibility problem with Drupal.

CL challenges/disadvantages

1. bojanz points out the problem of conflict resolution, which I interpret as meaning different modules requiring the same target dependency with different version constraints. Work is needed, possibly using composer merge plugin could help?
2. What else?

Comments please.

Bojan, Webflo, Mile23, and myself have been working on a proposal to tackle this initiative, and have been vetting this proposal with various key members of the community (core committers/contributors, drush maintainers, etc) to make sure we're on the right track.

Before I present our proposal, it would probably be a good exercise to define what it is we're trying to achieve.

The ultimate goal that is to eliminate the issues people face when using composer. So we should clarify who "those struggling with composer" are, and why they are struggling.

They fall into several categories:

1. There are users who are unfamiliar with the command line, and learning a tool like composer implies learning a whole stack of skills they currently lack. And composer doesn't make things easy by emitting cryptic error messages and using domain specific language that is obtuse.
2. There are users who lack the resources to execute composer, memory wise, server wise or environment wise.
3. There are users who started their drupal site in a recommended manner (drupal tarball, using composer create project drupal/drupal, or with a git clone), and then run into utterly broken, painful situations where there is no officially sanctioned way to have a composer managed site, and requires a difficult migration.

We should definitely try, on an ongoing basis, to solve all of the above scenarios, however we believe that in order to make progress, we should focus on #3 first, which is what the issue summary indicates:

This initiative aims to introduce improvements to core's Composer support to allow a single best-practice method for managing installations and updates out of the box

In order to solve #1, we need to build an abstraction/gui/UI around composer, which will probably require solving #2 to make possible. Solving #2 likely requires some upstream changes to composer and packagist, which may mean we're stuck with it being a memory hog. #3 is not dependent upon the other two, and is definitely a requirement for #1.

Therefore the scope of this initiative is to focus solely on making is so that there is an official method to supporting composer in core so that we do not continue with the current situation which is "you should use composer to manage your site, but there isn't an officially sanctioned way to do that yet, and some official ways that will prevent you from doing it properly".

Proposal forthcoming.

Just saw this tweet about Contao CMS, and one of the replies also mentions Bolt CMS installing extensions through the UI with composer:

https://twitter.com/seldaek/status/1004656842449063936

At @ContaoKonferenz, pretty impressive how they managed to integrate Composer fully into their admin UI for plugin installation/updates, together with "cloud dependency resolving" to make it work on restricted shared hosting. Other CMSs should have a look.
- Jordi Boggiano (@seldaek)

The immediate problem is:
If a user

1. Starts a drupal project with the current drupal tarball OR
2. Starts a project with `composer create project drupal/drupal` OR
3. Starts a project from a git clone or copy of the git repo

And then attempts to use composer to manage their site, they will invariably be set on a path of misery, confusion, and frustration because:

1. If the user adds a module with composer, their composer.json and composer.lock will have changed from what is shipped with core. When they need to upgrade core, following the documented manual upgrade will result in conflicting changes in the composer.json and composer.lock file. We’ve always said, ‘don't hack core’ but using composer in the scenario forces you to do that. And they’ll have to manually resolve conflicts in multiple json files. Not fun by any stretch of the imagination.
2. There is no way to use composer to upgrade drupal core in this scenario, so they have to upgrade manually.
3. The user’s most likely to follow this path are the ones who have been doing drupal this way all along, and finally stumble across a module they need that requires that they use composer to install it, and wander into a situation that sets them up for failure. This is mostly due to the fact that we’re not providing an official solution.

The root cause underlying many of the issues we face is the result of using Drupal core’s repository as *both* where we do our development work *and* as a ‘product’ that we ship.

We propose that we can separate those two conflicting concerns by doing the following:

Conceptually separate Drupal, the product, from Drupal’s git repository, and provide a mechanism that creates a composer ready drupal installation

1. Start thinking of the Drupal git repository as a ‘monorepo/manyrepos’ situation, where the git repository is where core development takes place.
   • Git clones of Drupal core are for development purposes only.
   • Directory layout would stay the same, and be capable of behaving as a Drupal site for the purposes of development
   • composer.lock file in the root of the git repository would no longer conflict with existing sites, and is to ensure that developers are all working with the same set of dependencies.
2. Provide, in core, composer project designed to allow users to “kickstart” a Drupal project.
   • Similar to the current drupal-composer/drupal-project template, but even simpler.
     • Example: https://github.com/webflo/drupal-kickstart
     • Also follows Symfony’s pattern: https://github.com/symfony/skeleton
   • Would exist on packagist.org as a starter project by using the existing subtree split tool
   • In order to start a Drupal project, the end user either starts with a `composer create project drupal/kickstart my_project` OR they download a tarball that had been constructed using the kickstart
   • The kickstart plugin would depend on a scaffolding plugin that deploys the scaffold files from core into a sane directory layout for the product.
     • Example: https://github.com/drupal-composer/drupal-scaffold
   • Using a new composer namespace allows us to break from a giant chunk of bloated metadata that is the current drupal/drupal and avoids conflicts with accidental attempts to upgrade from a current drupal/drupal site to the new kickstarter.
3. Introduce build testing to prove that we can create products out of core’s repository.
4. Alter drupal.org's packaging of drupal core to generate our current tarballs using this method so that composer users and non-composer users would still all have the same starting point
5. Provide an upgrade path for any existing sites to turn them into a composer ready site, so that the site behaves as if it were built with composer in the first place
6. Update all documentation to instruct users to *only* start with composer or with the composer created tarball, and provide separate instructions for developing for drupal core.
   • Also integrate the existing composer documentation into the various locations that are more user focused so that the options for extending a site are under the documentation for adding a new module, and instructions for upgrading a module are solely under upgrading a module, and not under any 'composer specific' documentation on drupal.org

There are several additional advantages that we gain by splitting these things conceptually:

1. The core git repository can have development resources for drupal core development, and not have to treat those resources as API's for contrib, including their location (phpunit.xml.dist/eslintrc/stylelint.json/etc/etc)
2. The plugins and kickstarts would be in the core repo, meaning we wouldnt need to enact duplicate policies and access management for additional repositories that are not drupal core. All of our policies surrounding contribution would stay intact.
3. For users with modest needs, that do not need to rely on a module with composer dependencies, this change would be transparent to them and they would still be able to manage a site "the old way", but their site would be 'composer ready' if they did end up needing to switch to managing it with composer.
4. When Drupal the 'product' becomes something 'composed' of various components inside of the core repo, we would have the opportunity to offer additional 'kickstarts' that might even allow for minimal versions of drupal, or pre-extended versions, like distributions, or simply variable filestructure layouts supporting both docroot/webroot, and one directory type hosting scenarios. We could have starting points with 'experimental modules' and those without, for example.
5. The scaffolding plugin could be reused by distributions and other staring points, reducing maintenance for distro maintainers.
6. Some of composer's performance penalties are a result of how we have structured our cascading composer.json's. We may be able to get great performance wins by doing some metadata restructuring.

Once we've stablized this workflow, there are several other things we can continue to incrementally improve:

• Improve the export quality of our components.
• Add composer.json files to all modules in core, and have them be requirements of drupal/core, and offered by the composer facade.
• Figure out how multisite drupal can be managed with composer - what are the limitations, what's possible etc.
• Figure out how to separate php code from assets in modules and in core so that code can be vendored, and assets under the docroot.
• Make patching a site a default feature of the kickstart plugin

Identify coordinators(s) for this initiative, ideally with dedicated time.

Myself, Bojan, Mile23, and Webflo have all stepped forward to get this over the finish line, and several others have expressed interest in helping this come to fruition.

I would also like to return to the ideas that have been expressed in the comments. There is a *lot* of things that are difficult about constructing a drupal website, and they are all things that should be addressed, eventually, but I think if we lay this foundation first, and tackle all of this in chunks, we'll be much better off than if we try and solve all the problems at once. So, hopefully this makes sense the amount of descoping we've attempted to do to get the greatest value out of minimal changes.

So, hopefully we can keep discussing this and get to where we can :

Get framework management + release management sign-off on idea

Thanks

I also saw the "cloud dependency resolving" tweet as well, and have reached out to the contao folks to find out what they did. (its also something we've considered attempting as well)

#37 Sounds like an awesome idea. Splitting the development from the product has been a big pain point with distributions, and this approach would help solve that problem.

I was waiting for somebody to agree with the scope boundaries, or suggest we update the summary. I was planning on doing so, but I wasnt sure when to do so.

When I re-read the phase 1 of the original scope, it's pretty much the same, with the Addition of "We can do all of that, If, we treat core's git repo as "not the product anymore", with the promise of a product built out of disassembled and re-assembled parts of core.", and with a couple of things descoped (patching/updates).

I can transfer much of #37 to the Issue Summary if that addition seems reasonable..

There is a lot of good in there.

Based on the tweet mentioned in #36 I spun up a Bolt CMS demo and made a short screencast:
http://www.yoroy.com/elders/Bolt%20CMS%20composer%20installs%20from%20UI...

At the end there’s a link to list the dependencies used for a given extension but that timed out. Maybe because this is on their demo server

It’s an open source project, and symfony based so potentially some interesting code to inspect as well.

I just read #37 and it's so great to finally read something that explains to me what I didn't (don't) get about composer. I think I also conceptually understand the proposed solution and it sounds great :)

As initiative leads you should all feel totally free to make the issue summary yours! Exciting :)

My background is as a site builder, and that taking into account my experiences plus colleagues and clients, then I potentially fit into all three of your categories of target users on #35. I suspect that there are 1000s of similar users out there but most of them don't comment on issues on Drupal.org.

Thanks so much to the team that are taking this initiative on. @Mixologic I also agree that there is a lot of good in your proposal. As I understand it, you are proposing to solve #3 but not initially #1 or #2. Of course, I understand, it is easier to take things step by step. However that does mean that the proposal stops short of providing a working solution for some/many sites. Would you agree therefore that the existing install mechanisms need to keep working as they do now for the needs of the remainder?

In order to solve #1, we need to build an abstraction/gui/UI around composer, which will probably require solving #2 to make possible

Surely #2 (memory hog) is exactly as much of a problem whether using composer directly or via an integration (#1). Hence #1 is not dependent on #2?? The lack of a drush wrapper makes matters problematic for site-builders looking after many small sites who use the power of drush to run a command on many servers/sites in one go. I guess what I'm wondering is whether the drush wrapper might be really easy to write and whether you would consider expanding the scope to include that? For example "drush pm-upgrade" could detect if composer is being used, and if so convert to calling "composer upgrade".

Solving #2 likely requires some upstream changes to composer and packagist, which may mean we're stuck with it being a memory hog.

Potentially the memory hog would mean the proposed new upgrade/install would fail on some/many existing websites. If someone on your team is smart enough to improve the memory hog problem then that would be amazing and I think it would be worth sharing it by whatever means possible. Maybe in the short term, Drupal could ship with a customised composer and in the long term, hopefully the upstream projects would accept patches?

Add composer.json files to all modules in core

As far as I can see, each composer.json added increases the memory composer requires, probably worse than linearly, so this might not be a productive avenue unless there have been substantial performance fixes.

===

I think that to "solve the composer problem" it is useful to understand what is lost in the process. For example at the moment a site-maintainer it is convenient to run drush @SITES pm-update --security-only. Trying to move into the composer world, the experience fails in many ways

1. Without drush, you have to log into each server one by one, change to the correct directory and call composer
2. composer does not support --security-only
3. The servers don't have enough memory to run composer and the directory structure is wrong
4. composer by default installs dev dependencies that should generally not be present outside of development and it's easy to forget to pass --no-dev (i.e. we need a setting "default to --no-dev for this site" - this could be handled within the drush wrapper)
5. No easy way to ensure each site is running with the same stable version of dependencies (maybe need https://github.com/webflo/drupal-core-strict?)

It would certainly be great to understand which of these your proposal would fix.

However that does mean that the proposal stops short of providing a working solution for some/many sites

This is definitely true, but the point Im trying to make here is that this proposal isn't the end of the work we're planning on doing to fix composer for everybody, just that we must do this particular work first as it is foundational, and that if we try and solve all the problems at once, and get into the varied intracacies of all the different aspects then we'll forever be stuck in an unfocused quagmire of all the intersecting concerns of building, extending, (auto) updating, and even testing a site.

Drush wrappers

As far as a drush wrapper goes, drush, moving forward, only supports sites built with composer: http://docs.drush.org/en/master/install/ (see note at the top of the screen).

So commands like pm-update are *already* going to rely on running composer under the hood to accomplish updating the sites. Composer is not at all meant to be a replacement for drush, and drush is going to continue to be valuable running on top of composer. composer just replaces the downloading and dependency management aspect of pm-update/pm-download.

Security updates with composer

As far as --security-only not being supported by composer, that is true, composer itself does not have a concept of security built into it, however we have added security information to the package metadata for coverage information, (https://www.drupal.org/project/project_composer/issues/2863103) and we have the data that we generate for the updates.xml data, and we could potentially add that metadata into the composer.json data that packages.drupal.org provides and have an additional plugin that understands that data, such that drush would be able to determine which packages are in need of a security update/are marked as insecure. But drush should already still be able to do --security-only updates, because its using the xml data to determine *which* updates to run, and then composer to actually run those updates, so this may already work.

Server memory issues

So the issue with composer using a lot of memory comes down to the volume of metadata it needs to process in order to determine a workable set of dependencies to install on your site. Right now, to install one module, composer is going to download about 110MB of metadata. When composer loads that metadata, json_decode uses up about 384MB of memory to load *just the packages that come with drupal, out of the box*. Then, the number of versions of every package in all of that metadata is the second factor when it builds the ruleset for the solver. Because drupal and symfony and a lot of the other packages we rely on have a *lot* of releases over time, the ruleset gets very large, so it adds another 120MB or so of memory. This is why any suggestion that a 'composer lite' that is "just the external packages" isnt really a workable solution because even that is already well beyond the 64mb of memory that is recommended to run drupal 8. There are things we can do to reduce this workload, but, like I said, they may require upstream fixes (because its not composer, its the metadata itself that composer relys on that comes from packagist.org). Hence why other projects are building remote 'composer as a service' which we may still consider. Its also why some companies have emerged offering the same sort of thing to the drupal community (composy.io).

This is why even if we did something like include composer with core, and have the update manager (gui) or drush hide the gory details of composer, you'll still run into memory and resource issues by trying to run composer on a production server/site if you're still expecting that 64mb is enough to run that site. Hence why I said earlier we cant solve #1 (make things easier) without addressing #2 (make it work in low resources). It is also why attempts at a "composer lite" are pretty much non-starters, because even if its *just* external packages, composer will still need a considerable amount of resources to run.

Locked Dependencies

No easy way to ensure each site is running with the same stable version of dependencies

I had this in the other document that I've been working on but it didnt make the transfer of knowledge from google docs to #37, but yes, that is pretty much the plan is to create an additional plugin that can be subtree split from core that contains pretty much the same type of data as https://github.com/webflo/drupal-core-strict - allowing sites to have a stable set of deps. I know that drupalCI will definitely need this for testing contrib modules, so its pretty much required that we do something like this.

I went ahead and updated the issue summary.

@Mixologic Many thanks for the detailed explanation in #47 and updated IS, that's much clearer.

So it is agreed that phase 1 (this initiative) is an important step, but not a working solution for some/many sites. Therefore those other sites will need to continue using other existing install methods - please can we update the IS to explicitly state that existing install methods will still work after phase 1?

As far as a drush wrapper goes, drush, moving forward, only supports sites built with composer: http://docs.drush.org/en/master/install/ (see note at the top of the screen). So commands like pm-update are *already* going to rely on running composer under the hood to accomplish updating the sites.

Yes I am aware that drush 9 does not include pm-update. However drush 8 does and I believe many people are still using pm-update from there. I can't find the link right now, but I recall some statistics that around 50% of installs are still from tarball. Unpacking tarballs by hand is horrible and a properly secured GUI doesn't have permission to write to code files so I guess the majority of tarballs installs are using drush. Surely pm-update will be needed until the later phases of this initiative are complete and the composer alternative is ready for all users??

attempts at a "composer lite" are pretty much non-starters

Interesting stats, thanks so "composer lite" would need about 500MB if I follow you. However I found that "composer full" can need well over 1GB. So for anyone who has the 500MB spare and needs a solution with drush integration working now, then "composer lite" is a good answer (i.e. "cheat" and just "composer require" a couple of extra dependencies into the set downloaded by core). I'm cheering on all efforts to get a proper solution working for all users, but as a small-site builder you have to take the most practical alternative.

please can we update the IS to explicitly state that existing install methods will still work after phase 1?

Part of the point of the initiative is that some existing install methods are a road to pain.

Also, there are a lot of existing install methods, so it's unclear what behavior to promise.

So if you download a tarball to install a new Drupal site, then that should work as before. The idea is that the tarball will work the same way, but will have been build using a Composer project. The caveat will be that you really should add new modules via Composer, tho you can get away with not doing that if the extensions don't have dependencies.

If you're cloning the repo, then bets are off, officially, but it might still work depending on use-case.

But I think you mean to talk about maintenance and updating. That's where the rubber hits the road. For that you'll need to composer-ize.

There's this part in the IS:

Provide an upgrade path for any existing sites to turn them into a composer ready site, so that the site behaves as if it were built with composer in the first place

That would be a native build tool which would composer-ize your site and get you on the path. There are currently some existing tools to do this, such as Composerize and drupal/console. We've talked about adding one to core with tests so we know it works.

Updated IS with this info.

@Mile32 Let me see if I can clarify. There are various reasons why a site cannot (or wants not to) use composer even after this initiative, which as @Mixologic states in #47 is just a step, phase 1. The main cases are outlined by @Mixologic in #35: if composer requires more memory than available or admins need drush/GUI rather than typing composer commands. These sites might have no modules with external dependencies so don't need composer anyway.

In order not to break these sites, they will need to continue to not use composer and instead use the existing tarball installation, e.g.

• Install new drupal site from tarball via GUI or drush
• Download module tarball via GUI or drush
• Upgrade module tarball via GUI or drush
• Upgrade drupal core tarball via drush

Hopefully what I've said is obvious and not controversial. Please can we add a back-compatibility statement like this to the IS?

Yes. Tarballs are go for core. See this in the IS:

In order to start a Drupal project, the end user either starts with a composer create project drupal/kickstart my_project OR they download a tarball that had been constructed using the kickstart

And from #52, using tarball extensions isn't the best way but it might work:

So if you download a tarball to install a new Drupal site, then that should work as before. The idea is that the tarball will work the same way, but will have been build using a Composer project. The caveat will be that you really should add new modules via Composer, tho you can get away with not doing that if the extensions don't have dependencies.

So you *should* use Composer, and if you don't you're not guaranteed. But in the mean time we'll make super awesome tools to help.

*If* the following statement is true:

These sites might have no modules with external dependencies so don't need composer anyway

Then all four of the use cases you listed *currently* work, and will continue to work in the future. This initiative has no impact on the existing tools for these users. But those users neither have a problem, nor need a solution until they *do* need a module with external dependencies.

*If* a site builder/maintainer needs to use a site that has a module with an external composer dependency, then the following happens:

Install new drupal site from tarball via GUI or drush

After this initiative their site will be 'composer ready' and will be able to switch over to using composer on the command line, or using drush 9. Currently there is a very difficult path from starting this way to being able to use composer. This is the thing we're focusing on and this is what we're fixing.

Download module tarball via GUI or drush
Upgrade module tarball via GUI or drush
Upgrade drupal core tarball via drush

Assuming you mean drush 8, then these activities do not work now, and will continue to be broken after this initiative. We will never be able to support this on drush 8, and we will not be able to support the existing update manager until we change the way the update manager works to either use composer as part of the application (requiring upstream and performance fixes) or modify the GUI to leverage something like contao has done (as mentioned in #36) The Contao folks put out a blog post detailing their efforts in this regard: https://medium.com/@yanick.witschi/composer-cloud-resolver-e64254f5728e

Fixing composer's performance issues, or providing a composer resolution service is desirable, and we want to do one of those things, but again, it is out of scope for this initiative. It will be in a future initiative.

So, to reiterate. There currently is no support for GUI/drush 8 to install a module that has composer dependencies, nor will there be support for it after this initiative is done. That does not mean we don't think it is something that needs to be fixed. It means we can't do it yet, and that the work we want to do for this initiative is a pre-requisite.

@Mixologic #55 is exactly what I was hoping for, thanks for the very clear explanation.

modify the GUI to leverage something like contao has done (as mentioned in #36) The Contao folks put out a blog post detailing their efforts in this regard: https://medium.com/@yanick.witschi/composer-cloud-resolver-e64254f5728e

It would be great for Drupal if we could do the same.

This is why any suggestion that a 'composer lite' that is "just the external packages" isnt really a workable solution because even that is already well beyond the 64mb of memory that is recommended to run drupal 8. There are things we can do to reduce this workload, but, like I said, they may require upstream fixes (because its not composer, its the metadata itself that composer relys on that comes from packagist.org). Hence why other projects are building remote 'composer as a service' which we may still consider. Its also why some companies have emerged offering the same sort of thing to the drupal community (composy.io).

I was assuming we'd defer the topic of low-memory Composer dependency resolution for the auto-updates work, but I guess I should share my thoughts on it here and now.

First, I'm reluctant to rely on services implementing Composer-Dependencies-as-a-Service (CDaaS). If we consider the worst (but still common) case of a major security release, we'd be under pressure to deliver an enormous number of updated builds in a short period of time with little means to cache (and hit) existing computed results. Also, because a site would need to fully trust the service designing the build, there would be limited options for the community to help scale it out. Compare this to the Linux world of package mirrors, which don't need to be trusted because the packages and metadata are signed.

There are ways to mitigate these issues, including having platforms/hosts run their own CDaaS or designing Drupal.org to freshen previous builds and have them ready to go (cached) before release announcements. However, I still think we'd get DDoSed (in a non-attack way) following security releases unless we spend lots of time and money avoiding it. We'd also always need to worry if we'd be surprised by the load caused by the next release; even auto-scaling cloud services will buckle under heavy, surprise loads.

Second, I think there's a better way, which is to iteratively freshen an inactive partition (really, just a folder) or ostree-style vendor directory before promoting it to be active once completely freshened. This wouldn't rely on any real-time CDaaS, and it would support untrusted parties distributing updates. It also avoids the problems associated with updating a web application in-place -- particularly when we're not sure when the active request will timeout or be otherwise killed (e.g. timeouts in PHP or even the client). I produced a proof-of-concept of this approach.

Finally, we also need to consider sensitive deployments, like intranets and sites deployed with ingress/egress controls. While we could tell such users that they simply have to run the full Composer build themselves (or deploy their own internal CDaaS), it would be nice if tarball installations could auto-update with nothing more than access to an HTTP server hosting signed, rsync-able data.

We should be wary -- but not dismissive -- of adopting solutions that are only proven for smaller projects.

We currently have 18 sites (adding about 2 per month to that) each with 4 or more environments, all run with a composer based platform, workflow, and build process.

Here are the packages we make use of:

       "composer/installers": "^1.2",
        "davidbarratt/custom-installer": "1.0.*@alpha",
        "cweagans/composer-patches": "^1.6",
        "webflo/drupal-finder": "^1.0.0",
        "webmozart/path-util": "^2.3",
        "jkribeiro/drupal-composer-paranoia-acquia": "~1",

Plus a custom "platform" package which includes drupal/core and a bunch of contrib/custom modules, and a "profile" package, which is just a custom Drupal install profile.

It'd be great if we didn't have to depend on the version of core hosted on github, and the other composer packages needed to "build" Drupal.

Two issues we hit all the time are:

1. Drupal contrib modules are not using semantic versioning, therefore we can't just depend on a stable minor release we either need to pin to specific release, or get all the updates.
2. The second is not really Drupal specific, but it's that requiring a branch in Composer in unreliable. Therefore requiring the latest dev branch of a contrib (or custom) module may not always get the latest release.

I recently noticed that our Official drupal user guide instructs users on how to start a project with composer, using the exact methods that will get the users into trouble: https://www.drupal.org/docs/user_guide/en/install-composer.html

This trouble leads to these kinds of messes that people have to do in order to update their site:
https://drupal.stackexchange.com/questions/167922/what-is-the-correct-co...

It'd be great if we didn't have to depend on the version of core hosted on github, and the other composer packages needed to "build" Drupal.

One thing to note is that those are considered currently "official" - in that the drupal infra team maintains the subtree splits and puts them on github, specifically so that packagist.org can find them. Once this change is approved and we have a starterkit that defines packages.drupal.org as a resource, then we can put the subtree splits back on drupal.org, but we currently lack a mechanism for hosting things that are not "projects" on d.o.

Updating the tags so that the release/framework managers can weigh in and unblock this from moving forward.

Regarding the User Guide, we have an issue to fix that temporarily at least, which is linked as Related to this one (as of today). If you have any suggestions for what we should put into the User Guide for the moment to avoid these issues, and/or what would go in there eventually, that would be helpful. The issue is:
#2980636: Fix Composer instructions for downloading Drupal core in User Guide, or add warning
(I just started following this issue today and haven't read it all yet, so apologies if this has already been discussed...)

Is there anything that would need to be done in 8.6 to help this move forward? We were wondering with @webchick how to support you best :)

AFAIK, this proposal is still waiting on acceptance in the form of "Needs release manager review" and "Needs framework manager review". I've talked with Alex Pott and effulgentsia, and am going to go ahead and start creating a meta and some child issues, but it would be nice to move forward and be sure that everybody is on board with the proposal.

Some of the longer term things here give me a bit of a headache (not the fault of the plan, just it's a tricky thing to resolve), but the immediate plan looks great to me.

. So if this initiative doesn't have any aims in that regard

If the question is "does this initiative hope to do anything by 8.6", then actually, yes - there are three concrete things we _could_ do now that would make a big difference:

1. Add the kickstart component
2. Add the scaffolding component
3. Add the core-strict component

I feel like this will be doable because these tools are already well used / defacto standard components that exist in the community, and we'd be moving them under the core development process, and paring them down to the bare necessities.

This would go a long way to clearing the existing minefield, with some updated docs to tell people to avoid stepping on the mines that remain.

For those interested, I've started a core meta issue to get a start on a few issues #2982674: [meta] Composer Initiative Phase 1: Add composer build support to core

I suspect it's probably past-time that this idea reaches RTBC, per the Ideas Queue process. Especially considering that the target is to get these changes in for 8.7 and work is under way in that direction?

Would any followers like to be brave and RTBC the issue? 😊

I re-read the entire summary. It's a solid and needed plan. I've been following some/many of the sub issues. The reality of core development has finally changed in a way that makes semver for contrib possible (and necessary). The way we partially "handle" composer support now is a confusing and painful mess. This plan is a clear way out of the mess, a path to an improved experience for everyone, in a way that (hopefully) minimizes impact on existing sites. Our tools (and the team supporting them) can and will be able to implement this. Onward! :)

Thanks,
-Derek

Aww, @dww beat me to it. +1 RTBC from me.

I reread the issue summary too and it all makes wonderful, elegant sense to me. This plan is totally doable, and a good chunk of it can likely land for 8.7. There are already issues open, and patches to review.

Full steam ahead.

+1 from me, thanks for the thorough approach and effort here

Phase 1 looks great to me and happy to sign off on it. Phase 2 does not really look like something that could be signed off on yet.

One thing I'd point out is if we update instructions to only use composer, then that should probably include what to do about multisite. Note there's a long discussion on #2306013: Multisite is a valued feature that will not be deprecated. about multisite.

Bumped multisite up to phase 1 in the sense that we shouldn't have polished documentation for a composer workflow that completely ignores it. Back to CNR, but if you're happy please move back to RTBC. Untagging for release manager review.

Bumped multisite up to phase 1 in the sense that we shouldn't have polished documentation for a composer workflow that completely ignores it.

I think it's fair to say that, if using multisite functionality, you cannot have specific Composer dependencies per site. Not everyone who uses multisite uses it to maintain independent dependencies between sites (like a SaaS implementation.)

Multisite might have a few phases:

1) You can specify multisite as long as all the vendor/ dependencies for all sites resolve against each other. If different sites have different dependencies then this could require adding composer.json per site in some way, and might also run into trouble because Composer and the installer plugin don't know what to do if site A needs <2.0 of a module while site B needs >=2.0.

2) Somewhat terrifying vendor-directory-per-site architecture.

There was also talk of doing multiple live codebases in the auto-update issues. Whatever solution is developed there might help us with multisite, because that might give us some tools or maybe even a useful replacement that meets the needs of multisite.

All the changes that happen in phase 1 would be transparent to a multisite user, and as such would not have any impact on either the documentation or the code surrounding the multisite feature. Phase 1 does not address the challenges of using composer in conjunction with multisite, because as far as I know, you simply cannot build a multisite in drupal 8 and use composer to manage your site. This has never been architected or addressed, and I believe it is out of scope to attempt to put that into phase 1.

That being said we need to clearly define what the multisite feature *really is* beyond a loosely related collection of use-cases, and determine how to approach building, extending, updating a 'multisite'.

#2306013: Multisite is a valued feature that will not be deprecated. is full of excellent real world use cases that can help us understand the end user needs, and also allow us to determine what it is we will, and will not be able to support.

Lets not derail this issue with 'how to fix' multisite - lets open another and start brainstorming: #3004496: Document best practices for multisite and composer

If there is some interim documentation on multisite that we should update to say "you cannot currently use composer to manage a multisite", then we should definitely make sure that its sorted out.

We're not saying phase 1 has to solve how to fix multisite

We're just saying we need to document what will be supported with composer and what will not.

And we're hoping that will just be a cookbook of 'composer recipes'.

Example 1:
We can support installing theme x only in sites/y/themes/contrib (via composer installers)
But we can't support installing theme x only in sites/y/themes/contrib and sites/z/themes/contrib
We can however advise people to add a symlink in that scenario.

Example 2:
We can't support two different versions of the same module in composer multisite.

Example 3:
We can't support two different major version of a vendor library in composer multisite (but we can't support that in a single site either).

So we're not expecting a fix for 'multisite'. We're just expecting clear guidelines around what will and won't be supported.

Does anyone disagree with some documentation being bumped up to phase 1? If not, let's put this back on RTBC per #77.

Intro/Motivation

Current state

I have been working on a Composer plug-in to automate the conversion of existing “drupal/drupal” Drupal 8 sites to “drupal-composer/drupal-project". Features will include: Creating the custom “composer.json” file for your current site with the new project template. Modify all necessary files & folders to the new project format including Drupal Scaffolding files. And finally, perform the update to the latest version. I would hope to have it tested and ready for deployment in the next couple of weeks.

Proposed Solution

As mentioned in the initial post these are the basic problem scenarios:

1. Starts a drupal project with the current drupal tarball OR
2. Starts a project with `composer create project drupal/drupal` OR
3. Starts a project from a git clone or copy of the git repo

I believe for all 3 they should be implemented to morph to the template of “drupal-composer/drupal-project". This template can be seen in the following link:
https://github.com/drupal-composer/drupal-project

I believe for Scenarios 1 & 3 can be solved with the composer plugin “grasmash/composerize-drupal" which can be seen in the following link:

https://github.com/grasmash/composerize-drupal

The most difficult is scenario #2. This is the one I’m tackling with this new composer plugin that I’m working on.

Basically, the final plugin will work as follows:

1. Add it as a dependency to your composer.json
2. Run a Composer command provided by this new plugin
3. IT will auto-magically convert your project to be fully managed by composer & with the latest and greatest template from the “drupal-composer/drupal-project".

Its a work in progress…

The initial release of this new plugin will have all the features mentioned above. I would like peoples opinions & feedback as to any improvements that can be made for this plugin. I could also do Separate composer commands that will solve #1 & #3 but I believe that these are currently easily solved with the existing composer plugins.

Stuff I’m contemplating..

I think one of the main issues is to make sure unified semantic versioning for all Custom, contrib modules & themes. Some Contrib modules do not implement this. My solution is to have a default version ID created and added to the modules info.yml file and used in the main composer.json file. Maybe this should also be done to existing custom modules or theme? whats your opinion.

Looking forward to receiving your comments/suggestions…

I'm happy to announce the deployment of the composer plug-in I mentioned previously... You can find about it in the following blog post:

Having trouble updating your Drupal 8 Website? Then this Guide is for you…

The original Repo is in the following link:

https://github.com/WebKings-ca/gocomposer

Enjoy!

After spending several hours (days) dealing with composer installation problems, I figured out that composer does two things when it installs a Drupal module:
1. It installs libraries and the module by downloading and unpacking them in vendor/ and web/modules/contrib/
2. It generates (modifies) vendor/composer/autoload_*.php files

Installing module and library files (#1) is easy to do manually. It's pretty much the same as D7 when I unpacked modules in modules/ and libraries in libraries/
There is the issue of finding out where to get the 3rd party libraries from. Perhaps composer can be used in drupal.org to automatically get all non-dev dependencies for each module (except drupal core or other drupal modules), and make them available for download in the module's page. Or provide a link to the repository where the library is located. Many Java libraries typically deal with this problem by including 3rd party dependencies in their distribution. Why can't Drupal modules do something similar?

Generating the autoloader files (#2) is not so easy to do manually, but it seems that it could be done easily by a lightweight tool that is part of Drupal. For example, each module could have its own autoloader files (packaged in its tarfile by drupal.org), and the tool could merge them together. Hopefully composer could be used to generate these modular autoloader files for each contrib module in drupal.org. Drupal could merge them automatically from /update.php, or whenever I delete the global autoloader files. Only modules that use 3rd party libraries need to modify the autoloader files. All other module code uses PSR-4 loading conventions.

In summary, I propose that some partial composing is done separately for each module by the packaging process of drupal.org (flattening out dependencies and modular autoloader files), so that the installation of modules and their dependencies can be easy to do manually or by Drupal itself.

If I can install any module manually with the above method, then I would not need to use Composer at all. I may want to use it for convenience, but by not using it:
1. I would be happy that I can install Drupal and modules the way I did so in D7.
2. I would not have to deal with the numerous Composer problems that I've encountered.

And some additional notes from a user's perspective:

The initial proposal is quite lengthy, technical, and intimidating for me to follow in all its details, which obscure the user requirements.

The user problem mentioned is not the only one, and currently none of the scenarios mentioned in this issue follows the official installation instructions in the user guide. I would add:

When I (a user):
- Start a Drupal project with the current drupal tarball, as documented.
- Install a drupal website and happily do some work, even write some custom modules for it.
- Need to install a contrib module that mentions composer as the only installation option.

Then I end up miserable, confused and frustrated, because:
- I think I would need to start over again reinstalling my site (knowing from previous D7 experience that changing the directory structure of where modules are can lead to trouble).
- Composer requires installing additional swap space on a VPS with <= 2GB RAM, and in some virtual environments it runs intolerably slow (~ 15-25 minutes per project/module). See:
https://discuss.linuxcontainers.org/t/php-composer-is-excessively-slow-i...
- The Composer instructions contain things like 8.x-dev, --stability --dev, --no-dev, that make me feel uneasy about what exactly I'm installing. See:
https://github.com/drupal-composer/drupal-project/issues/443
https://github.com/drupal-composer/drupal-project/issues/444
https://www.drupal.org/project/user_guide/issues/3022675

In addition:
- Composer seems to be a development tool, not for end-users (non developers). It is similar to maven for Java (which compiles projects and manages dependencies), or a C/C++ linker ("composes" various libraries into a single application), but the user who installs Drupal may not be a developer and even if he is, he may not be familiar with Composer.
- Drupal 8 has been released for years. Why aren't these installation problems fixed yet? Please make it a priority. It doesn't seem that hard.
- Nextcloud uses Composer and Symfony, but their installation instructions mention nothing about Composer. Nextcloud can install plugins from the GUI. How do they do it?

Hi @alex.a:

It's on the roadmap, so yeah, it's already a priority. There are lots of edge cases and considerations which is why this issue is intimidating to follow and is in fact hard to do. But maybe we've missed simpler solution. Definitely make a proof of concept if you think it would work.

I apologize for the condescending comment "it doesn't seem that hard".
As for the technicalities here, I also looked at the road map and at the high level strategic initiatives, and I still see that they are too technical, instead of being user-centric. Why is "Composer Support in Core" a high level strategic initiative? What user problem does it solve? The user requirement is "Simplify Installation and Update of Modules", or "User-Friendly Installer". Composer is a developer tool and the support is already there for developers who know how to use it.

After reading what @alex.a said there are some good points there, perhaps we need to put a much greater focus on #2940733: Site Builder Tool/Project Browser initiative which is already linked in the issue summary.

I'd be a little concerned though if we started trying to build functionality in core to mimic parts of Composer as that would not only just be redundant code duplication but also potentially cause long term compatibility issues where our "mini Composer" becomes out of sync with normal Composer and needs constant attention to address issues. Having said that @alex.a did point to some issues that people have with running Composer so that also puts a roadblock in the way for just dropping in Composer and assuming it'll work for everyone.

With that in mind, perhaps we could target the Composer problems head on and try to resolve those through direct contributions to Composer and/or by modifying Drupal to mitigate potential issues.

Let's say for example we implement the Project Browser, if we can find a way to run Composer in the background in a way which doesn't cause issues for some users then we would be in a situation where every site is using the same standard approach through Composer, but the site maintainer has the option of either just using Composer and never having to touch the GUIs, or they can use all GUIs and never have to touch Composer (or never even have to know that it exists). Essentially it gives people options while allowing for an easy to maintain approach.

In my mind I'm picturing a project browser and background installer/updater which works somewhat like the iOS App Store where when you install something or update something it just runs in the background and keeps you informed of progress in real time, working even if you navigate away to another page or just close the Drupal site all together. To implement that though would also mean we'd have to improve the existing batch processing to not require a user to be present during the process. Doing that would possibly be a benefit though as it may elevate some of the frustrations people seem to have with Composer taking a long time in some cases since the user wouldn't feel such an obligation to sit there and watch everything happen and might feel more free to perform other tasks while waiting, which could give the illusion of time passing and things taking less time. Having said that though in my experience of installing modules using Composer it tends to be quick because Composer simply installs that single modules and any dependencies it has (which is usually none or only a couple), so even when installing modules using a Composer backed GUI I suspect in most cases things will happen fairly quickly.

Finally I want to address the reality of the situation, yes Composer may have some issues for some users, personally I've never had any issues with it, but the reality is that it's here and here to stay, with Composer adoption growing among developers. So we can't ignore that fact, we can't simply carry on not fully supporting Composer, because a time will come (if it hasn't already) where people will look at Drupal and one of their considerations in using it will be how well does it support Composer, and so the harsh reality is that we either start fully supporting Composer or we stunt Drupal's future growth and potentially cause developers to move away to different frameworks.

@alex.a I agree that Using composer to manage a Drupal 8 site might be technically intimidating for new users looking to adopt Drupal 8 as there website building platform.. On the other hand using composer will make a huge technical advantage for any Drupal 8 site due to the features that composer brings from managing all dependencies to easily install library packages with the right versions. In my opinion, adopting a fully composer managed solution is so beneficial feature wise and thinking long term for future releases of Drupal 8 & 9.On a side note: The plug-in which I released main aim was to ease the adoption of fully composer managed Drupal 8 sites. I believe in having this unified template, the process for installing a contrib module and it's dependencies would be a simple composer "require" command. A GUI implementation of this seems to me straight forward.

As far as the plug-in obscuring user requirements, the plug-in has been made to accomodate all user scenarios in its workflow by prompting for additional information from the user if a specific scenario has been detected. The main purpose of me making this plug-in is these 2 reasons:

1) Adopting a unified template from the latest recommend install profile. Easing Core updates, adding security features and prep your existing site for future releases.

2) Making this plug-in really easy to use and able to handle almost any scenario. This can be seen by the fact you just have to require it and run the command 'composer gocomposer' and it'll do the rest, prompting for additional info if needed.

In any case, I'm posting another link to a new revision of the blog I posted earlier with the ability for anyone to post comments:

Having trouble updating your Drupal 8 Website? Then this Guide is for you…

Looking forward to your comments/suggestions..

Just to point out: We're working on making Drupal into a Composer-manageable codebase so that we can do two things:

1) Make a tarball-equivalent codebase that won't break when you start using Composer to manage it: #2982680: Add composer-ready project templates to Drupal core

2) Allow the development of other tools which aren't Composer but use it to build your codebase for you. This includes the automatic updates initiative which is making a lot of progress: #2940731: Automatic Updates Initiative overview and roadmap And of course the eventual build tools: #2940733: Site Builder Tool/Project Browser initiative

The current blocker for much of this work is in porting drupal-composer/drupal-scaffold to the core codebase, which features some tricky testing issues: #2982684: Add a composer scaffolding plugin to core

Once that's in, we can add the 'legacy' template: #2982680: Add composer-ready project templates to Drupal core This will allow users to actually use the composer.json that's shipped with their core tarball, as well as allow drupal.org to package the tarball using composer create-project. This is the BC layer for the whole Composer initiative. One of the test requirements is that it accurately reproduce the current Drupal tarball download.

From that, all the rest of the sub-issues on #2982674: [meta] Composer Initiative Phase 1: Add composer build support to core flow forth.

Anyone wishing to participate in the work should go ahead and add some expertise to code reviews so we can move things along.

@cilefen The proof of concept of what I suggested is already there in the Ludwig module. With some more work, Ludwig can be used to install any module, not just modules with ludwig.json. I wrote a ludwig generator here:

https://www.drupal.org/sandbox/alexa/3025040

By the way, drupal modules are released with composer.json but no composer.lock file. This is goes against Composer's own Basic Usage instructions: https://getcomposer.org/doc/01-basic-usage.md#commit-your-composer-lock-...
The composer docs say that a development team should use the same composer.lock file, so they use exactly the same versions of libraries.
Drupal essentially says to its contrib module users: Don't worry: Use whatever versions of libraries are out there at the time. It doesn't matter if the module developer used different ones to test.

@alex.a Your comment about the composer.lock file is not correct. You SHOULD commit your composer.lock file, but only for top-level projects. For libraries like drupal modules, it does not make any sense to do so, because that file is not taken into account. What you should do is use version constraints in the composer.json file. See also https://getcomposer.org/doc/articles/versions.md#next-significant-releas....

In fact libraries should be tested with a "normal" composer install and with a --prefer-lowest composer install to test against both extremes of all dependent packages to be safe.

Most libraries will lock a package/library to a major version with e.g. "^3.0" as the version constraint.

In fact libraries should be tested with a "normal" composer install and with a --prefer-lowest composer install to test against both extremes of all dependent packages to be safe.

Drupal core dev is working on this. #2976407: Use drupalci.yml for composer dependency min/max testing - DO NOT COMMIT It might well be that we can offer a Composer min/max test for contrib at some point. It's also entirely possible for contrib to test this on travis-ci or using a similar patch to drupalci.yml.

Drupal essentially says to its contrib module users: Don't worry: Use whatever versions of libraries are out there at the time. It doesn't matter if the module developer used different ones to test.

There is a project called drupal-core-strict which forces very narrow constraints on your project. @webflo made it. It's a composer package with the same constraints as the composer.lock file for that version of Drupal core: https://packagist.org/packages/webflo/drupal-core-strict

If you're concerned that your project has too wide a range of dependencies, then just add that package and always get the core lock file ones, even if you say composer update.

This is an ideas issue, so it works a little differently from a regular core issue.

The main core meta related to this issue is here: #2982674: [meta] Composer Initiative Phase 1: Add composer build support to core

Set to NR in #77 over concerns about multisite. We have this issue discussing it: #3004496: Document best practices for multisite and composer

Setting this back to NR.

That is... NR. :-)

Sorry. Lastest Firefox version is going nuts on me and posting these things just as I scroll through. I am making no effort to post, and in fact avoiding it since I don't know enough to contribute. Came back in Chrome and my system is still posting things automatically. If someone has authority to delete posts please do so. Otherwise ignore. Again I'm sorry, I don't know what's going on with my computer to make this happen. Maybe something in the latest Windows update ... I've never had this happen before.

... Mary and Joseph. I just posted my apology and saw Webchick's response and the system posted another one under my name. IF anyone has a hint what's happening just email me at my link and I'll fix it.

Check and see if you have lastpass autofill or some other extension like that.

Hestenet sent me a solution which involves disabling a setting in LastPass. Did that and drupal.org looks like it's working properly.

Sorry about the problems and distractions.

...

Know I'm coming a bit late into this process, but would a similar paradigm to follow be cPanel/Plesk as it relates to Web hosting and server configuration?

To be specific, server config is rather specialized knowledge, and online documentation can be a bit challenging to grok for non-technical users, to put it mildly. That's where visual tools like cPanel (although that UX could stand some major improvements) come in. A lot of common tasks can be accomplished through a GUI, and the resulting output is to a standard .htaccess file that (theoretically) any Apache server can use.

Would something similar be workable for Composer management in Drupal 8/9? If a power user needs more or is comfortable, they can bypass the GUI entirely, but for those who aren't it could lower the technical barrier significantly.

@karolus this issue is different than that, focusing on making Drupal work in a more normal Composer way. What you're talking about could be more along the lines of https://www.drupal.org/project/ideas/issues/2940733 which would require and build upon the work in this issue. If you're interested in a Composer GUI https://glamanate.com/blog/conductor-composer-ui might interest you.

@karolus Having a web ui/gui as the mechanism that non-technical users interface with to extend their drupal site already exists, except it doesnt work with composer. Most of what we've been doing here in the composer initiative has been improving core to where it has an officially supported way to use composer, for technical users.

But the future goal would be to do exacty what you're suggesting, essentially taking the web interface we have and making it work with composer.

There are a lot of roadblocks that we'll have to clear to make that possible, primarily in the architecture of composer itself to improve its performance characteristics so that it can be used from a web interface.

Here is very interesting Composer 2.0 RC1 performance test by Gabor Hojtsy.

Composer 1 has a huge memory needs. Therefore it is not possible for hosting companies to even consider adding composer to shared hosting servers at the moment.

However, if Composer 2.0 will have such friendly memory use, I can see it coming to shared hosting servers soon.

That would be cool.

Yes @devad, it does looks very promising. Adding a few related Drupal issues.

I just want to add an update that my (shared) hosting company has upgraded CLOUDLINUX 7.9 to cPanel v94.0.4 which has global Composer 2.0.6 support enabled by default for all cPanel users.

The previous cPanel version had Composer 1.8.x which was not resources friendly enough to be used nicely at my share hosting, so they were kind and following my support request they have updated cPanel version to newer.

All I need now is a common terminal access to my account (I am using PuTTY for that) and I can run all "composer ..." commands directly on server. So, there is no explicit need for local server installation any more. I can composer install and manage and update a Drupal website directly on shared hosting server.

I suppose that Composer 2.x will become a standard option in the offers of shared hosting providers very quickly.

On my shared hosting using Cpanel (inmotionhosting) they had a default 1.x version of composer. However, I was able to use their terminal tool to change into my public_html and install my own 2.x version of composer. Using the composer self-update command I can now run 2.0.12 of Composer and use it to install and update Drupal files in the public_html directory. This puts the resource constraint problems of Composer 1.x versions behind us as far as I can tell. Now, I'm testing ways to "composerize" my Drupal 9.1.6 ... OH HELL'S Bells they've released 9.1.7 ... so I can reasonably run composer update on my shared server. The github version of composerize doesn't work with Composer 2 yet and there are several posts recommending that you just composerize manually ... which I am working on on a local test site, if they could ever keep the current version of Drupal stable long enough for me to get to it.

Developer Workflows.
more safely, you would run composer on your local setup machine and push the code up to your server once the build is complete and you can test out which ever things (including core) that may have been updated -- depending on what composer command you ran to trigger core update, how versions of things are set in composer (if your are using ^ (above) or are version locked..)

composer 1 vs 2 -- depends mostly on what Composer Plugins you maybe using and if they (the plugins) have been upgraded for composer 2 as well. keep in mind: Composer has it's own 'contrib community'

As there's only one child left