Use Composer to build sites without forcing users to learn Composer

In last comments of #1543858: Add a startup configuration for the built-in PHP server that supports clean URLs raised idea of "starter" script like SF does, so split looks reasonable
But we can't make it before 9.x

Same time composer scrips & commands are competitor to #2242947: Integrate Symfony Console component to natively support command line operations

Personally I prefer to build tarball with build-in composer script and reuse it for d.o packaging

btw web, docroot and other names leads to bikeshed

@Mile23 because the same script could be re-used later in projects and custom CI system

+1 to the sentiments in this issue, thank you @Mile23 for putting it together! I will attempt to contribute more than just positive vibes, too, but for a start, here are my positive vibes :)

🌟 🌟 🌟

Well that was quick, I have a question.

@Mile23 Could you please explain the distinction between #2845379 and this new issue thread? I'm not new to a d.o issues per se, but I am new to Drupal core discussion threads I guess.

Thanks!

> Re-organize Drupal core to be similar to the outcome of these instructions.

This seems to only address core. What happens to our hypothetical user who downloads the core tarball when they find they want to install, say, Address module?

@Mile23 oh okay, I'd pictured those things ending up in the same issue, but I think I get it (enough, for now). Thanks!

@joachim Not a "real" answer, but I wonder if contrib module development follows/is influenced by core principles/methods/standards?

It seems to me that we already have a solution to both desired goals.

1) Use Composer to build best-practices-type sites with isolated docroot and so forth. This has all the advantages you can learn about from talking to the drupal-project maintainers.

User who want to use composer are already doing so. Certainly there is a desire to make it easier to use composer, and lots of other issues talk about that.

2) Leave Drupal like it is so that it can be used on cheap shared hosting by 'non-experts' who can't reconfigure a web server doc root (either due to lack of skill or because their hosting forces them, or whatever else). This has the advantage of leaving Drupal as an entry-level site building tool for people who want to host their own site but maybe aren't technical experts.

Users who don't want to use composer are already using the tarball.

====

I agree with one sentiment of your issue: we should maintain support of the tarball until composer is a valid alternative for all users and in line with the Drupal core deprecation policy. I am about to raise another issue along these lines.

However I suggest that there is not necessarily a need to change the directory structure of core in GIT. The webserver site directory structure might end up with either of the two ways you suggest (with top-level split www/vendor OR vendor inside webroot). However core is just one part of the whole, and it has a consistent internal structure either way. I think the vendor directory isn't part of core but gets created by composer: either a site local composer, or the Drupal.org packaging composer that makes the tarball.

I hope that when you see my issue, you might find it addresses your problem/motivation but requiring less big changes.

I didn't create a new issue and instead commented on existing ones #2845379: Provide optional composer integration but don't force users to understand how to use composer and #2477789: Use composer to build sites. It seems like there are far too many composer meta/plan issues already so I wonder if it would be better to close this new one and keep the discussion in the existing long-established issues.

+1 to @joachim

Is there any idea of how to tarball contrib projects that require external dependencies? How to resolve version conflicts then? Etc.

I don't care which issue we "use" to address this tricky situation, but, a reason I like this thread is that there isn't any contentiousness "baggage" that tends to put people on defense, and, this issue seems a lot more neutral than the others, which (in a good way) focus on composer or non-composer tools/methods/integrations. In contrast this issue seems to me to be about making sure that all the composer and non-composer "things" don't obstruct each other.

(As an aside, I don't care what the directory structure ends up being, and I'm comfortable deferring to the people who've been working on this, as long as we reach the aforementioned goal -- my two cents.)

EDIT: "there isn't any contentiousness 'baggage' that tends to put people on defense, and, this issue seems a lot more neutral than the others, which (in a good way) focus on composer or non-composer tools/methods/integrations" -->> Like this "taking a side" statement right in the #2845379 description:

This should be seen as an opposition to #2477789: Use composer to build sites becoming a requirement unless a GUI is provided.

(To be clear, I'm firmly on the side of "don't force users to understand how to use composer"!! As someone on that "side," I don't want the "solution" to be something that was done to accommodate "us," I want it to be a communal "for everybody" thing -- this issue (2908394) is much more in that vein, IMO.)

@pingwin4eg I'll reiterate my earlier thought: I wonder if contrib module development follows/is influenced by core principles/methods/standards.

Regardless, probably is something to add to the issue description ("remaining tasks"), @Mile23?

@alisonjo2786 My view is that this issue is confusing/helpful because it proposes a radical reorganisation of core that would likely cause back-compatibility problems. This could be exactly at odds with the message I am trying to put forward that we need to maintain compatibility of existing install methods.

@pingwin4eg From a back-compatibility perspective, I would say that users interested in "projects that require external dependencies" should use composer. Users who need to stick with tarballs get the same experience as now: they can't use such projects unless they fetch the dependencies themselves.

Breaking compatibility in minors is no-go, I have a lot of d8 sites that does not need composer and they build with `drush make`

No idea when customers will decide to upgrade but when it will happen, I bet it would be hard to explain "migration to composer" instead of upgrade drupal

> They learn enough Composer to type composer require drupal/address, because that's how that contrib module works.

But isn't this issue (and related ones) about finding a way to avoid this?

> In the case of the Address module, no, we can't avoid it. That's a requirement of the Address module, and not core.

Right, but this issue surely should be about finding a way for site builders to avoid having to use composer for both core and contrib.

Otherwise, this issue title is no good:

> Use Composer to build sites without forcing users to learn Composer

'Building a site' involves installing contrib module. If the plan fails as soon as you hit a contrib module that has some composer dependencies, then it's not a good enough plan.

@joachim Your logic is sound in theory, but this issue is really only about core things -- requiring contrib modules to adhere to that kind of rule would have to be a separate issue (idk anything about how contrib module governance works -- tho I think there's less governance than there used to be). Your best bet for that is to target the specific contrib module issue queue.

Seems @Mile23 (issue reporter) is also stated for core only, so I renamed the issue.

> Use Composer to setup Drupal Core without forcing users to learn Composer

Who uses up only core to build a site? I don't see any value in doing this.

> If you want to not have to think about Composer in that scenario, then please add a set of steps you want to follow to accomplish it. In

Surely it's about the steps we *don't* want users to have to take.

So, when I install drupal/address, I will not:

- have to run composer
- have to suffer cryptic messages from composer about version numbers that make no sense
- have to delete my composer.lock vendor folder and reinstall all dependencies because composer has got itself stuck in a hole it can't get out of

As for my own expectations, I honestly don't know what's possible in this area. I'm just saying that solving this problem only for core is not really solving it at all.

Given that the Address module supports Ludwig, I'd like to see some mention on the Address module's project page that it can be used instead of Composer for this specific use case. Ludwig has the potential to really solve a lot of these problems for beginners and I'd love to see it promoted more heavily.

Ok, I'll take a stab:

"When I install drupal/address, I will take the following steps: 1) ______ 2) _______ etc..."

What I'm hearing from folks in various discussions (and my own personal preference) is:

"When I install any contributed module, I should be able to take exactly the same steps. Whether the maintainer chose to manage their dependencies in .info files or Composer is an implementation detail based on their own personal preferences, and should be utterly irrelevant to me as a site builder."

The most common ways of installing contrib modules are (I believe, in this order):

1. Downloading the tarball and sticking it in /modules.
2. drush dl/en modulename
3. Using the Update Manager UI

Therefore, it seems to have and eat cake, we need to:

1. Absolutely no idea. Maybe the modules submission form does some composer magic in the background?
2. Make Drush smart about Composer (maybe it already is) so it runs the right commands in the background for you.
3. Integrate Composer into the Update Manager UI (I know we have [at least] an issue for this somewhere), probably also turning it into a nice graphical module browser akin to WordPress's to encourage more people to use it, since I'm pretty sure #1 is unsolvable.

@Miles23: Yes, it's a workaround, one that solves the use case of "as a site builder I would like to install the Address module without running the Composer command".

In short, I think the "how do we use composer without using composer" is the wrong question.

The better question IMHO would be "how do we support 3rd party libraries without using Composer?"

Reminder: running composer on the server side to dynamically a) opens up security problems because we would require codebase files to be writable, b) has massive system requirements that will fail on many hosting platforms.

@webchick: Ludwig helps solve scenario #1, it just needs improved documentation and support from other module maintainers.

1. Absolutely no idea. Maybe the modules submission form does some composer magic in the background?
2. Make Drush smart about Composer (maybe it already is) so it runs the right commands in the background for you.
3. Integrate Composer into the Update Manager UI (I know we have [at least] an issue for this somewhere), probably also turning it into a nice graphical module browser akin to WordPress's to encourage more people to use it, since I'm pretty sure #1 is unsolvable.

^^ 100% agree that these are explore-worthy options. [three minutes later] Ugh also nervous about @DamienMcKenna's points in #32.

@Mile23 what's wrong with Ludwig as a way to handle contrib modules that need composer? Then we "just" have to worry about core, and we can set aside "what about contrib modules that need composer," right? (@DamienMcKenna said that the address module supports using Ludwig instead of composer.)

Score? Manuscript?

So getting back to the user story: 'Exactly the same steps' now means using some kind of builder app to generate a Drupal codebase. You spec the site you want through clicky-click on an app and a tarball comes to your download folder. You put it on your server and life is good. At update time you tell the app to look at the composer.json file it made during install, and badabing it informs you that there are updates. Click OK and you get a new tarball, or a new composer file and then we figure out the instructions for updating custom code.

An alternative to this would be to have some service on drupal.org that handles all the dependency resolution (probably using composer itself), grabs all the libraries required and packages them in a Phar file along with an autoloader for them. Drupal's own autoloader would need to be instructed to also look for an autoloader.php or something in that Phar file.

This is how it could work for the three scenarios above:

1. Downloading the tarball and sticking it in /modules.

When the user tries to install the new module they just downloaded, Drupal looks for a composer.json file and checks it against the list of libraries already available in /vendor and inside that Phar file, and, if it doesn't find something, it redirects the user to a screen where they can click a button which invokes the drupal.org service described above and gives them a new Phar archive that they need to upload manually on their site.

2. drush dl/en modulename

Drush could have the same dependency resolution and Phar archive building code that runs on d.o, or it can simply call the "official" service as well and get the Phar automatically

3. Using the Update Manager UI

This would be very similar to 1. but even more automated. Since the Update Manager can download and extract entire modules into the codebase, surely it can download an additional Phar file and put it in its place.

A quick search on the interwebs found this little utility https://github.com/box-project/box2 that we could use for building/signing the Phar files.

I may have gone a bit too far on the implementation details above, but the gist of my proposal is actually really simple:

For scenarios 1. and 3., where the user does not use any command line tool, we move the dependency resolution somewhere else (e.g. on drupal.org)
For scenario 2., Drush can be improved to manage everything on its own.

@Mile23 would those two thingies cover whatever update management (incl dependency changes) and uninstallation stuff? Those are the only other stuffs coming to mind for me so far. Thanks for the blog post rec.

FWIW there was some conversation on one or both of these threads about adding some basic drush go core:
EDIT: It was this one! #2906637: [META] Drush and core compatibility is fragile
https://www.drupal.org/node/2907224
https://www.drupal.org/node/2905741

Naming: I like Score, via @John Pitcairn (#36).

Or Comdruposersh 😁

Couple other related issues, re: drush 8 + Drupal 8.4.x incompatibility bc Symfony version, drush + core integration.

Back to Ludwig..

I opened #2909495: Put libraries in /vendor instead of inside the module's codebase which would (in theory) resolve the problem with duplicates.

BTW Ludwig would be no less open to dependency conflicts than Composer itself, there's still the potential for the respective dependency definitions to conflict between multiple modules, using Composer directly doesn't remove this possibility.

Leave Drupal like it is so that it can be used on cheap shared hosting by 'non-experts' who can't reconfigure a web server doc root

Does somebody have some data on how many (and which) shared hosting providers make it difficult to have vendor (and other directories) outside the doc root?

I feel the security benefit could easily outweigh the usability problems (most computer users can handle directories, right?), and the problems may be overstated - without data, we are just speculating.

For everyone following this issue, the idea from #37/#38 seems to be feasible: #2910136: Experiment: package PHP libraries in a single Phar file

Based on 3 from #30 - I'd like to offer a tactical solution which doesn't involve a big shift to the current UX.

Current state:
/admin/modules/update -- checks for updates on D.O and shows status latest versions available
/admin/modules/install -- install new module / theme

Proposed state:
/admin/modules/update --

Connect to an endpoint on D.O (as proposed in #37 & #38) and post your composer.json, composer.lock & site uuid, D.O would then run composer update and the result would sit at the endpoint /[hash of composer.json + site uuid] and be encrypted with the site uuid as the composer.json file can contain sensitive information.
The update page would wait till this was available to start processing the updates available and the endpoint would have some way of knowing if this hash had already been posted to stop rebuilding. This might even need to be queued for processing, so will pass back a status that it's processing and check for the result endpoint.
Once a valid result is available the update module would update the site composer.lock ad display the updates available.
An extra local task would be available to run composer install via update.php before autoloader was required on the local site which could be batched as composer install is idempotent. After install it would take you to update.php to complete any update hooks.
update.php should also be able to recover from a bad filesystem system state.

/admin/modules/install --

This would be a way to update the composer.json file a bit like composer require but wouldn't actually run the update just edit composer.json and either use default version "^1.0.0" or possibly ">0" (need to work out safest approach here). Then it would submit the lastest version to the update endpoint for processing. Install done the same way as updating.

/admin/modules/remove --
Possibly a similar endpoint to remove package from composer.json, updates processed the same as install.

/admin/modules/composer -- [optional submodule ?]
Endpoint to make edits to composer.json packages and version constraints. updates processed the same as install.

Still some specifics to work out but at least there's not a big UX change.
And the update module is optional so can be disabled for site which don't require this functionality or even only enabled to run updates then disabled.

RE: issues with running composer on server -- https://www.drupal.org/node/2538090#comment-11931781
This is no different to the current state which allows downloading directly from D.O to the server.
It should be able to be disabled in cases where you never want this functionality.
Composer update / require and dependency resolution is very memory intensive task which would only be run by the D.O endpoint not on the site itself.
Composer install is basically like the current install process of fetching the packages and resolving file system differences with the lock file, which is why this process can be run whenever or if failed for any reason can run again and fix issues.
To help the install you could have a way to download the cache from the resulting build on D.O so it's a single compressed archive, which is used by the installer.

Maybe the version of last successful update of composer.json & composer.lock could also be stored in the DB so the site could recover from these being corrupted or deleted, in this case the database with it's known site uuid could effectively be rebuilt from a default D8 codebase. Could even be handy if you use git ever needed to revert an old db backup and match up it's codebase.

Major benefit of #46 is later on if you decide to start using composer via CLI there's no changes required.

or someone builds site with composer CLI then it's maintained via UI by the client for example..

Would love to start PoC'ing the proposed endpoint in #46 with something like https://github.com/openfaas/faas
The hope would be that you can run you own instance so you didn't have to rely on D.O, and potentially be upstreamed to packagist service..

However, would it make more sense to align with the existing infra for drupalci?

Keen to talk more about this a BADCamp if anyone is interested..

Some progress on the PoC -- https://github.com/thom8/faas-composer

So you can run curl -d "$(cat composer.json)" http://localhost:8080/function/composer_update -o composer.lock to remotely resolve composer dependencies then composer install to install the updates.

Whoa! I don't understand how your code works, but that curl command looks awesome. Generating the .lock isn't that far from generating a .phar. To install a new package, it'd just need to take both the composer.json and composer.lock (to know what's already installed, right?) and a "require" param to add a new package. (Actually, for the update shouldn't it need to take the current composer.lock too?)

@dsnopek Yeah, it could potentially be used to generate a .phar.
Not that familiar with the phar approach but would this still work with xdebug and other profiling tools?

In terms of a "require" based on #46 an update will still work if you manually edit the json file to include a new dependency.
The lock file is not necessarily required when doing a full update, but would be needed if you allow the ability to update a subset of packages.

And just to be clear the idea is you would deploy this function to a remote cluster via swarm / k8s rather than running it locally.
Here's a quick video of how that could work -- https://www.youtube.com/watch?v=0DbrLsUvaso

See #37 and #2910136: Experiment: package PHP libraries in a single Phar file for why I'm talking about .phar files and requiring individual packages. It'd be super cool if you could adapt your code to do what is proposed in #2910136!

Happy to help, if you get the ball rolling so to speak..

I'm using bash as the processor but this could just as easily be PHP -- https://github.com/thom8/faas-composer/blob/master/faas-composer.yml#L10 you'd just need to capture the input from STDIN.

Also depending on the complexity you might need to look into async functions - https://github.com/openfaas/faas/blob/master/guide/asynchronous.md

Thanks @Mile23

For me I'm scratching my own itch and I've got quite a few other use cases for openfaas but it would be nice to be able to contribute at the same time :)

Just want to point out that there's some official motion from the DA, so before we do too much work we should check in with that project.

@Mile23 Which project?

FYI - Nils a co-founder of packagist happens to be in San Francisco during badcamp and is planning to come along to the Contribution Lounge on Wednesday -- https://2017.badcamp.net/event/sprint/contribution-lounge

Just want to point out that there's some official motion from the DA ...

Do you have more info about this @Mile23?

Hold on for a sec. The gist of this issue as I understand it is to achieve composer's dependency resolution through drupal module focused user interfaces. Right?

So, you don't need to do that with cloud infrastructure. Couldn't we just bake composer's dependency resolver into the "more powerful solution" being discussed in #2912406: [META] Replace update_manager with a more powerful solution? Composer itself seems highly...composable. That is, the code providing its functionality looks nicely decoupled from the CLI UI we're mostly familiar with; see e.g. its DependencyResolver namespace.) The license is conveniently compatible, too.

So what you'd end up with is the user selecting the modules they want using this proposed update_manager replacement, and it re-evaluates and downloads dependencies.

I'm a little late to the party admittedly as I've been focusing mostly on the first identified priority in #2912406: [META] Replace update_manager with a more powerful solution, to update core.

@mbaynton I was thinking a similar thing, but with drush in mind ([META] Replace update_manager... > comment #11) -- idk if it makes any sense, buuuuuut there it is :)

@mbaynton whilst it's technically possible to run composer via a web UI the practically has a few serious issues which would need to be overcome.
@mixologic sums up the issues pretty well here -- https://www.drupal.org/node/2845379#comment-11980878

I've actually built a PoC of composer running via a web UI and even for a basic site you need in excess of 1GB of mem to handle the dependency resolution.

https://github.com/thom8/faas-composer is a way for a site to leverage a service for the dependency resolution, so the memory is not required locally, also being centralised with big caches available the update process can be quite fast.

I guess it a way it's actually quite similar to the existing process which pings D.O to check for updates.

Thanks for identifying the memory challenges you've run into and @mixologic's comment, @8thom. Of course, the actual problem being solved (dependency resolution) doesn't fundamentally need amounts of memory anywhere near that, but if that's how much Composer uses, I agree it's a serious issue. I'd love to find time to do some PoCing of my own, mostly trying to split the dependency resolution step and the various resulting download/install steps out to separate requests in a batch framework, and seeing what the memory profile looks like.

Go @mbaynton Go! :D

@mbayton here's my PoC -- https://github.com/thom8/drupal-project you should be able to vagrant up that project.

Working but needs ~2GB mem -- https://github.com/thom8/drupal-project/blob/source/.beetbox/config.yml#L2

@mbaynton @8thom see also: #2910136: Experiment: package PHP libraries in a single Phar file

It occurs to me that as Composer increasingly comes into favor as the way to build sites, contrib modules may be inadvertently being given more of an opportunity than necessary to affect a site's overall security.

Consider the update to Guzzle for httpoxy. Drupal can update core/composer.json all it wants, but if any contrib module on the site has done "guzzlehttp/guzzle": "~6.1", that site's not going to be seeing the security fix. The version constraints are incompatible, and the process that rebuilds that site is destined to fail.

If we're proposing to abstract away composer and make it a magic black box suitable for novice users, that magic black box needs to reliably build secure sites, yeah?

You could maybe say for dependencies that employ semantic versioning, contrib module authors are not to constrain any dependency beyond major version. That's heavy-handed.

Is this a problem others have already thought about, or have thoughts about? I might suggest a preliminary, tiny improvement would be to add to the contrib module author documentation on adding a composer.json some guidelines about selecting version constraints that are as relaxed as possible.

So with a core auto update system that ignores composer dependencies you either update and break a module with a hard dependency on an older version or remove/disable the module otherwise put the site into maintenance mode until the module maintainer pushes a compatible update which probably requires some other change considering the hard dependency.

At the end of the day we need to decide if we as a project want to leverage dependency management or revert to the model of update and hope that there's no conflicts...

There's a CMS Contao which has built an auto updates backed by composer via the web UI, which integrates with a remote dependency resolution service. - https://update.contao.org/en/

From my point of view I don't see any other option if the goal is to continue to have proper support for composer.

It's still not a prefect system as there's sometimes packages that remain unmaintained like - https://github.com/acquia/acquia-sdk-php which still has a dependency on guzzle/service -- https://github.com/acquia/acquia-sdk-php/blob/master/composer.json#L17 which has now been abandoned.. - https://packagist.org/packages/guzzle/service

but apparently it doesn't need updating -- https://github.com/acquia/acquia-sdk-php/issues/5#issuecomment-30457355

But yes, having composer.json allows the project to fall behind on security benefits for specific versions of dependencies. But: This has always been true of any contrib that required you to add something to libraries/, so it's really same problem, different day, but with a better build tool. It falls into the maintainer-duties slot more than the composer-is-a-problem slot.

This isn't quite true. Drupal Core never used sites/all/libraries. I think the point mbaynton was making was that if a module maintainer changed a dependency version requirement that it wouldn't kill the ability to build the whole site/core. In the past, you'd only be talking about one contrib module possibly causing a conflict with another contrib module that could temporarily be disabled and still allow the site to be built/deployed.

It's scenarios like this that constantly have me questioning the inevitable march to composer as a requirement, great as it is.

For #72 we have issue #667058: Add a libraries folder with a README.txt in it to DRUPAL_ROOT

So far this issue has given us some good ideas about how to work around requiring users to interact directly with Composer. But, all the solutions seem to suffer from a couple drawbacks:

1. Risk of site breakage. Automatic updates, a priority for Dries (yay!), would be just too scary to be practical I think, because the overall composed codebase for a given site could easily be pretty unique to that site, and even from update to update. When time to update your site comes around, Composer's dependency resolver will see some newer versions of this, maybe a modified composer.json in that module, and so may make very different decisions from site to site and build to build about the overall codebase used. Did anyone ever test that packagist packages A and B really do integrate as expected with version Y of dependency C? Unlikely, there's so many permutations to test. But our magic box Composer-driven tool for novice users made it for you! Go run your site with it now! Anything less isn't automatic.

   We should not forget that Composer was intended as a developer tool, where dependency re-evaluation would occur infrequently and the results would be subjected to a project's full test suite and prerelease cycle before creating a release.

2. Risk of inability to patch security vulnerabilities. After a security announcement/release of a 3rd-party dependency, any old module can hold back the site from receiving it with an ill-authored version constraint. Now, it's true that if all modules were actively maintained, or all dependencies ever used semver and all version constraints in all module's composer.jsons were judiciously selected, or maybe if all module maintainers were privy to security vulnerabilities in 3rd party dependencies before they were announced, this wouldn't be a problem. But since that's not how it works, there is a risk of sites not getting security updates in a timely fashion or ever.

There's also the well-known problem of two modules potentially just being incompatible because of conflicting version constraints on the same library, and the significant technical challenge of Composer's dependency resolver blowing Drupal's sys rq's out of the water memory wise, requiring clever workarounds like @8thom's Composer-as-a-service PoC.

I do have a proposal that addresses these issues, but it's maybe too soon for that. First, is there agreement that all these things are real drawbacks to the road we are going down? If so, I would like to update the issue summary to highlight them.

There's also the well-known problem of two modules potentially just being
incompatible because of conflicting version constraints on the same library

(Y'all know where I stand, but) Is this problem different from the kind of version requirements we've all dealt with for contrib modules over the years anyway? I totally might be missing something, it just seems to me like the same concept, not a new composer-related risk.

@alisonjo2786, no you're right, the theoretical problem is there already. And for 3rd-party dependencies, @Mile23 is right as well that

This has always been true of any contrib that required you to add something to libraries/, so it's really same problem.

Where I see Composer as germane to the discussion is not that Composer is creating a new class of theoretical problem, but that it encourages more dependencies and makes it easy to have bigger dependency relationships. So the already-existing problem is more likely to actually bite you in one form or another. With more external libraries and dependencies (which Composer is really just enabling in practice), the risk of version conflict complications increases while we're simultaneously pushing for automatic updates, which decreases what should be considered an acceptable tolerance for those complications. There's a mismatch there.

If we can agree there's a problem already even, I feel like we should be highlighting it more and talking about possible mitigations.

It would be nice if composer had a "noconflict" option that could be invoked to load two library versions side-by-side a la something like https://stackoverflow.com/questions/5792574/loading-multiple-versions-of-the-same-class in the event of an unavoidable conflict and rewrite the namespaces/classes according to version, and the modules with no conflict would call their respective version. But I can't imagine how that would work with modules having to parse the json and identify which to use. If that could be solved, the issues in #74 would be gone and all we're talking about is hiding composer's details from the user, which is the topic. It seems the same problems keep bringing up questions about composer and the ability to hide away its details, thus getting in the way of this thread's goal.

So I am in agreement with mbaynton, and I'd like to hear the proposal/idea he hinted at, since as I've alluded a change of the parameters we're looking at could remove some of our arguments and substantially change our whole outlook on which solution would be best for not forcing users to learn composer.

"encourages more dependencies" (@mbaynton) -- yep that makes sense, like, in The Old World, a module will depend on other modules that all exist within the Drupal universe, whereas in The New World, a module might depend on other modules and/or one of several PHP packages, b/c it can (and like, legit reasons -- some of which are prob part of why core devs bothered with all the work to support composer + Drupal). And then like, more dependencies = higher risk of version conflicts, *and* the messiness of dependencies built and maintained outside the Drupal universe. Again, not saying it's bad that there's composer integration, just saying, these are things.

• Are there mitigations or other strategies we could look at from Node or Laravel or other dependency management heavy Things? (methods for having multiple versions like @DigitalFrontiersMedia suggested, or whatever)
• The other sorta example I can think of is how you can have multiple jQuery versions in a Drupal site -- it's not the same, but, #brainstorming...

But, as I say all that... This branch of the conversation prob belongs over on this issue about improving Drupal + composer integration:
#2477789: Use composer to build sites

@alisonjo2785, thanks (I think?) for linking to #2477789: Use composer to build sites. On the one hand, lots of existing discussion I was unaware of. On the other hand, the entire issue produced no deliverables, and did take at least an hour for me to digest :).

I'll throw a dart at a board in order to select which one of the four or so very related issues I should post my proposal to now, because after all that talk it turns out nobody's suggested my idea yet. It _is_ very much inspired by "strategies we could look at from...other dependency management heavy Things", just not things in the PHP world.

It would be nice if composer had a "noconflict" option that could be invoked to load two library versions side-by-side a la something like https://stackoverflow.com/questions/5792574/loading-multiple-versions-of... in the event of an unavoidable conflict and rewrite the namespaces/classes according to version, and the modules with no conflict would call their respective version.

IIRC that would kinda destroy PSR-4, never mind the headaches of trying to manage the class usage in code.

Yes the numerous issue threads have been super fun! Thanks for slogging through and everything, @mbaynton; here's hoping for, yknow, outcomes ;-)

I agree that namespace rewriting is probably not so hot. It would work for "most" code, but it's not too difficult to write PHP code that'll run until you change it's namespace, allowing even for a very intelligent parser-based rewriter that updates everything. Which means there's packages out there that would break. Other code might run but behave unexpectedly; e.g. if static class properties are used - now you have 2 of them.

But besides that, what I really dislike about namespace rewriting is that it exacerbates the security problem: you don't have just one guzzlehttp/guzzle (<- or insert your favorite package here) to update per site anymore, but many, and the way you do it requires a complex namespace rewriting step.

Tellin' ya, a new repository like I describe over in the other issue is the way to go here ;)

By no means am I arguing in favor of namespace rewriting. I was just trying to get some additional thought going since other systems have contended with similar situations like multiple versions of jQuery, etc. However, I will point out that the security concern is perhaps a moot point since as noted previously, if Core had a security update that also included a new version of a library that was in conflict with a contrib module requirement and you were being held back from updating Core, you're just exchanging one security vulnerability for another.

I *do* hope the community can come up with some solutions. Will check out your idea in #2477789: Use composer to build sites

The no conflict approach has its own drawbacks, this is how npm manages its dependencies which has led to the famous memes like -- https://img.devrant.com/devrant/rant/r_760537_vKvzh.jpg

I've updated a list of issues for "Composer-as-a-service" https://github.com/thom8/faas-composer/issues - and I'm working on a doc to clarify the problems and propose some solutions here.

At the end of the day, I'd like to see a solution where ppl have the choice between composer CLI and a UI method similar to how the update manager currently works.

Created a basic overview of related problems and a potential solution with a composer service.

https://github.com/thom8/faas-composer/wiki/Use-case-for-Drupal.org

If anyone is keen to help out I've created a bunch of issues in the project itself - https://github.com/thom8/faas-composer/issues

Also, caught up with Nils a packagist/composer co-founder and are hoping to add support for this in the next version of composer as it isn't a Drupal only problem..

An update on my investigations in this area:

1. I've started an experimental composer repository that aims to house LTS versions of Drupal & related packages, like I've been talking about in the related issues. https://iglue.io/.(not sure if I have an installable set of packages loaded at the moment; been tinkering all weekend.)
2. I've been studying the sat solver at composer's heart and why it uses so many resources. When using a repository like iglue, only the latest version of each package needs to be considered, and this will allow for the number of clauses in the binary algebra equation to be linear with the number of require relationships.
   It should be easy to annotate my repository as such and prune the set of packages before invoking the solver, maybe with a plugin or maybe as a new Composer feature. I expect this will result in arrival at installation solutions well within the confines of web script sys requirements.
3. But this trick only works if you don't need any external packages at all from packages.drupal.org or packagist. While I aspire for that to be the case for a meaningful proportion of simple sites, I certainly don't want to limit the install/update technology to it -- this would clip Drupal 8's wings and practically put it back on the island. So here's what I propose: since solving the dependencies equation will be so fast when only iglue packages are considered, we first try to find an installable set of packages in this way. If the user hasn't requested some strange library to be part of their site, a solution is found & we're done. If however no install solution is found and we are running in a web UI, then we add packagist and the rest of the repositories from the site's composer.json to the pool and try solving again, but using @8thom's Composer-as-a-Service.

Awesome work, thank you so much @mbaynton for your efforts so far in building a Composer and Drupal LTS-model, iglue looks really promising.

I tried the Composer install command, but got the message below, perhaps I need to wait a bit before the packages are loaded and ready?

[InvalidArgumentException]
Could not find package drupal/drupal with stability dev.

@ressa yeah, been tinkering, like I said ;). I'm experimenting with not clobbering packagist namespaces. The command that works at the moment is

composer create-project --repository=https://iglue.io/1-unstable -s dev drupal@iglue/drupal my_new_drupal

Ah, much better, thanks! The installation went through without problems, only thing different to the normal Composer-based installation was that it asked Do you want to remove the existing VCS (.git, .svn..) history? [Y,n]? at the very end.

My hypothesis about how to reduce composer's memory utilization to something that can be done within a web UI looks like it's correct! The basics, again, are to put one version of each of the packages needed to install Drupal (and eventually some number of the most popular modules) into a new composer repository. Besides making composer runs fit into Drupal 8's system requirements, I believe managing the code in such a repository with a stability-first rather than new-features-first philosophy aligns well with the needs of less technical savvy users that would be likely to be using an in-browser installer/updater. So, I've been slowly adding all the packages needed to install Drupal Core into my iglue composer repository, and completed that task last week.

Now, by replacing packages.drupal.org with https://iglue.io/1-unstable/ and disabling packagist in the root composer.json of a fresh drupal 8 site, I get around 37.1 MB memory utilization (max resident set size as measured by the time utility on linux) when running composer update. This compares to around 338.3 MB when running composer update against the standard packages.drupal.org + packagist.org repositories.

Reproducing these results requires Composer PR 6843, which Nils has declined for Composer 1.x but suggested more openness to in 2.x. Likewise, it sounds like fully realizing @8thom's idea of handling sites whose dependencies can't be satisfied through packages available in a nice curated repository like this by performing the resource-intensive part on a remote server farm is going to require refactoring that we won't see before Composer 2.x. So, I think what might be the best path forward on this issue, for better or worse, is to make it depend on Composer 2.x, and participate in / have Drupal's needs inform the capabilities of this next major Composer release.