Improve multisite compatibility with composer

Just to point out that both Simpletest and functional PHPUnit-based tests make use of multisite features in order to run tests.

So short of refactoring all that away in preference for something else, all the multisite stuff that tests touch are solid base use-cases and need-very-much features.

Sometimes they upgrade a module on just one of their sites to 'test out the upgrade', so that one site has a newer version of the module than the other sites.

While this works for modules, it doesn't work for core, and updating one multisite can be done with a separate check out. Given the necessity to run database updates immediately after updating code (especially core minor updates) I'm not sure how larger multisites handle this at all.

Sometimes they share database tables.

The last site I worked on that did this was on Drupal 7 iirc, and it caused a lot of trouble, while we still technically support it, it would be good to know what the up-to-date use cases are and what exactly is being shared.

I'm not sure how larger multisites handle this at all.

They can do it in parallel; I have a job that basically kicks off drush updb on each site in parallel after the updated code is deployed. Assuming you don't have thousands of sites on one small database cluster, the use case is handled well. Any errors bubble up in the build as a failure, then you can fix those individual sites that had update issues as needed (this rarely happens, though, at least in my situation).

Re: Shared tables. I'm not sure why we still support this... it was originally meant for environments where a customer has one database (e.g. on shared hosting or something) and they want to install a bunch of sites into that one database. I haven't seen that usage pattern almost anywhere recently, though. Most shared hosting providers capable of hosting a Drupal site nowadays also have the ability for users to click a button and add a new database as needed, with a lower limit of 5 databases usually. (I could be proven wrong here though, but that's how it is for most shared hosting plans I encounter nowadays.)

• Sometimes they use multisite to save on per-codebase hosting expenses either via upfront provider costs or because of difficulties provisioning and managing hosting in-house.

I think that the testing framework probably relies on the shared database table stuff at the very least.

The testing framework uses different prefixes, but it doesn't share some of the tables. We are save here.

Like @geerlingguy just said, you can solve problems with multisite with automation. You can though also replicate multisites using automation in the first place. I am wondering whether we could introduce in a minor release some way of tracking that a site is executed in multi-site mode. I would expect its a really small number.

Maybe its used to share users between sites? or maybe used for sharing content? Taxonomies? dunno.

Historically I've used it to share users indeed, but for example for that usecase, there are much better single sign on solutions out there for example. Also for sharing content between sites there are various tools out there already.

In feels like in general with multisite, no matter which usecase you have, there is a better solution/alternative out there.

Users use multisite to reduce resource usage: Running multiple sites that all share a single opcode cache is memory efficient

This is precisely why I use it, and pretty much the only reason many of the sites I run are Drupal. If I couldn't run a dozen or so sites in the memory footprint of a small 512 MB VPS (owing to the fact that they all run in shared opcache space so it's highly memory efficient), there would be at least a few dozen fewer Drupal 7 sites in the world, and about 5 fewer Drupal 8 sites in the world ;-)

No way I can pay for the memory to host all the small business, personal and non-profit sites I currently run on Drupal if each codebase needed its own memory allocation.

I definitely agree that automation solves the problem, but at an ongoing monetary cost. Enterprise level, it hardly matters, but for thousands of smaller sites, .edu sites (many unis host hundreds or thousands of sites in one multisite install), etc. the cost of hosting would go through the roof based on memory requirements alone.

Shared tables should be considered unsupported at this point - it's writing to the database without going through core APIs (for the site that doesn't trigger the database write). We should open an issue to document this somehow. Prefixing itself is a bit wider. Similarly core updates aren't written to be re-entrant, so a schema change to a shared table would be made twice. I would not rule out a site using shared tables, but there can't be many at this point, and if similar functionality is needed (like sessions or something) it could be achieved by replacing the storage entirely to use a shared database.

Users use multisite to reduce resource usage: Running multiple sites that all share a single opcode cache is memory efficient

opcache takes symlinks into account (whereas apc did not), this means that if 100 symlinks point to the same directory, opcache will cache a single directory, not 100.

It should therefore be possible to build a 'multisite' deployment where everything except the /sites directory for each site is a symlink, and get the same memory benefit.

Should this be a target issue for Drupal 9? Also, should it live under the 'Drupal core ideas' project for now? It doesn't strike me ATM that we know exactly what we want to do with multisite just yet.

Based on the very interesting debates in #2306013: Multisite is a valued feature that will not be deprecated. and other related issues I kept a few highlights worth sharing here. They're primarily outlining the cons to keeping multisite (as we know it) which is where progress needs to be made the most.

#2306013-18: Multisite is a valued feature that will not be deprecated.

For me the composer issue is really the biggest concern.

#2306013-23: Multisite is a valued feature that will not be deprecated.

We need real life cases where people are using multisite in 7.x (or ideally 8.x), then document what the better alternatives to those are.

#2306013-38: Multisite is a valued feature that will not be deprecated.

One thing that needs to be addressed in the new 'off the island' world of drupal and composer is that modules are no longer something you simply download and place into a folder. You composer require a module in order to download it to a single folder, and its dependencies get placed in /vendor. As more and more modules utilize the approach of having a dependency on a composer package, this is going to move towards becoming a defacto requirement of site building. In composer's data model you are running one application off of one codebase, and in the multisite model of drupal you are running multiple applications off the same codebase. So we have a collision here about to handle this sort of build.

#2306013-43: Multisite is a valued feature that will not be deprecated.

it would be sensible to deprecate (8.x)/remove (9.x) the ability to have per-site modules and themes. That doesn't work with Composer as #38 anyway explains and in the world we live in that means it doesn't work (TM).

#2306013-57: Multisite is a valued feature that will not be deprecated.

Maybe Composer (or PHP itself) could introduce a feature where multiple PHP projects would use a shared pool of php vendor files.
This would make Drupal multisite obsolete, if the only purpose is to limit memory usage.

#2845379-24: Provide optional composer integration but don't force users to understand how to use composer

There are many modules that rely on third party dependencies, and that number will only grow as more and more essential functionality becomes based off of php libraries that contain solutions that the drupal community should not be re-inventing. Right now, it covers things like 3rd party integration API's, and we are gradually moving to a point where you will not be able to build an effective, functional drupal site without using modules that rely on composer. There are already some modules that are commonly used that fall into this category, such as the address, devel, backup_migrate, colorbox. Right now about 10% of all d8 modules have a 3rd party dependency.

#2996988: Consider adding "Domain Access" (or equivalent) to core (IS)

Look at what typo3 or eZ offer natively, it's valuable to compare with competitors. We have some catching up to do... Natively or via module.

#2958021-80: Proposal: Composer Support in Core initiative

as far as I know, you simply cannot build a multisite in drupal 8 and use composer to manage your site. This has never been architected or addressed, and I believe it is out of scope to attempt to put that into phase 1.

That being said we need to clearly define what the multisite feature *really is* beyond a loosely related collection of use-cases, and determine how to approach building, extending, updating a 'multisite'.

---

I keep on reading Drupal is no longer friendly for hobbysts, EDU and shared hosts on the one hand, and on the other hand those thinking that Drupal should catch up with the rest of the PHP community and fully embrace Composer everywhere at the risk of making radical decisions, like removing multisites. It looks like with such a wide spectrum nobody will be happy as we look at different personas with different needs and a different skill set.

To measure success with this plan, IMHO we should make sure to prevent the bike shed and antagonistic arguments like those often made in #2306013: Multisite is a valued feature that will not be deprecated. where multisite users felt betrayed or feared that something was going to be taken away from them.

On composer, these are the limitations compared to tarballs:

Because you can only have a single version of a dependency, and it will always be installed to the same location (as configured in composer.json installer-paths). Given contrib modules are installed by composer, this means you can only have a single version of a contrib module.

Site-specific modules and themes could be put in sites/SITENAME directories via configuring composer.json to do so, that's an extra manual step but not a massive one in the scheme of things. You could also commit site-specific modules directly.

The thing that composer doesn't allow, but tarballs do, for multisite, is the following then:

1. Install a contrib module for sites/sitename1, sites/sitename2 but not sites/sitename3

2. Install a contrib module globally as 8.x-3.x, but have sites/sitename4 use the 8.x-4.x version.

So really you're OK as long as the modules and themes you're using are either in a 1-all or 1-1 ratio to sites, what composer doesn't support is 1-some.

Those two use-cases are really quite minor though, and the solution to it would for people to have two multisites instead of one to handle those cases.

The other consideration is that the 'optimized' composer classloader (and composer in general) does not have any concept of installed/uninstalled, and therefore trying to use it on a multisite would mean every single class in the installation being available to every other site, but this is already a problem for single-site installs too.

Some really good reading here, and thanks @anavarre for the summary.

Perhaps we could do the following:

1. Deprecate shared-tables in 9.x for removal in later versions (maybe 10.x, maybe 11.x), at that point we're giving people years to stop using it. Document that you shouldn't use it, the problems it can cause, and good alternative solutions of varying difficulties for common use cases (such as shared sessions and shared users).
2. Next, continue with the Composer Initiative but just make people aware of the limitations currently available for multi-site, mainly that per-site modules and themes don't really work as expected.
3. Then, discuss these challenges with the wider Composer community and try to find a solution to allows Drupal multi-site to work properly with Composer, maybe developing something like the concept of Composer sub-sites which inherit the dependencies of the parent Composer site but ignore any sibling sites.

To me that seems like the only way forward at this point, we can't not use Composer because of multi-site, but we also can't not have multi-site because of Composer.

to answer *why* do people need it, and *how* do they use it?

i am site manager of one institute, with over 20 sites, running with multisite feature in drupal8 and 20 users with high level permission and without shared database ( we really wanted shared user but it seems it so tricky in drupal8 and there are better solution like LDAP )

why we don't use one site for all 20 departments?
1- most important thing is stability:
it happens lot of times that a misconfiguration or bad setting in blocks, path, menus or even bad content damage one site while other sites works perfectly, specially main site.
2- i am not sure if i can separate departments in one site and handle access for 20 users with core features (access to menus, taxonomies and ....). when we decide to go with drupal8, domain access had alpha release and what if there is no release of domain access for next drupal version?
3- every department is kind of independent, they really want a subdomain of main institute domain, and IMHO dept1.domain.com is more cleaner and feels more independent than domain.com/dept1 ( also when we compare dept1.domain.com/section1/page1 and domain.com/dept1/section1/page1) and i think it's tricky to have one instance of drupal site and address all pages of different departments with different subdomain.

why we don't have 20 different sites for all 20 departments?
- this sites have very similar functionality and modules are same, even theme is same for all sites. during one year it happens lot of times that i have to change theme a little bit to adjust things, it means i have to change code for all 20 sites individually.
also for core and modules updates, multisite solution is much simpler that separate sites.

- at last one day if there is not a perfect solution like multisite and i am forced to select one solution between 20 different sites or one site with 20 different section separated with built-in feature or module like domain access, i think i go with 20 different sites because more important thing to every organization is stability of site, even though this makes my life miserable with lots of updates and ...

> 1. Document what multisite is, and the use cases people use it for. i.e *why* do people need it, and *how* do they use it?

I often use multisite during development. It's really quick to spin up a new subsite to work on a specific issue.

It's possible to override installer paths on an individual package level. This allows you to place some modules or libraries into a profile's modules/ folder, eg:

        "installer-paths": {
            "docroot/core": [
                "type:drupal-core"
            ],
            "docroot/profiles/custom/endeavor/libraries/{$name}": [
                "**COMMENT** these must go before the path for the drupal-library type",
                "bower-asset/cropper"
            ],
            "docroot/profiles/custom/endeavor/modules/contrib/{$name}": [
                "**COMMENT** these must go before the path for the drupal-module type",
                "drupal/ctools",

(Note the brokenness of using a format that doesn't allow comments... :( )

That's pretty fiddly though, as you have to specify each module. Also, I'm having to do that in the top-level composer.json. It would be nice to be able to do it in the profile's composer.json, and rely on merging but: 1) installer-paths that override single packages MUST go above the package type path declaration, and 2) Composer merge-plugin doesn't document the order in which the data is merged, so that all seems a bit brittle to me.

> Because you can only have a single version of a dependency, and it will always be installed to the same location (as configured in composer.json installer-paths). Given contrib modules are installed by composer, this means you can only have a single version of a contrib module.

Just thinking aloud, probably totally wacky: write a Composer plugin that minimally bootstraps Drupal, figures out which profile you currently mean (by a command line parameter? by your current folder location like Drush does?), and then does the same sort of work as Composer merge-plugin to merge that profile's composer.json into the root, AND add into the Composer data the location overrides for that profile's modules. You'd then need to run 'composer install' once for each profile.

Hmm but at runtime we'd still need multiple autoload folders for the different combinations of Composer packages... which means we'd need to load the Composer autoloader for core packages, figure out the profile, then load the profile's Composer autoloader...

I've used multisite on every single Drupal site I ever work on, simply because of the hostname configuration files. There was only one project I ever worked on that has per-hostname modules or themes, and even then in hindsight they should have been shared and just kept in separate sites/all/modules/[project] directories. I don't think we need to work out a way of shoe-horning composer to let sites have multiple copies of the same files in the same repository, I think we could just drop that piece of functionality while still allowing the per-hostname configurations.

Just thinking aloud, probably totally wacky: write a Composer plugin that minimally bootstraps Drupal, figures out which profile you currently mean (by a command line parameter? by your current folder location like Drush does?)

No, that's pretty close to the solution to allowing different dependencies based on different sub-sites. That is, we'd have multiple vendor directories and there'd be some very early boot phase that figures out which one to load based on request. Except that might be hard if we're using Symfony\Component\HttpFoundation\Request (since we can't autoload it yet). That's solvable (I mean, there was a time before Symfony), but who wants to? Anyway...

So in this scenario, the Composer install phase would be a plugin that knows how multisite works, can determine where to merge a composer.json file from, per sub-site, and then individually runs composer install per sub-site to generate a vendor directory. This means a few things:

1. Your lock file is meaningless.
2. You can't just say composer require drupal/address at the top level, even if you know that all sites need it. You'll have to edit a composer.json file per sub-site and then re-install.
3. Some of the benefit of caching multisites' opcode and so forth will go away, because now each site has, for instance, all the symfony classes in its own almost-copy of the vendor directory in a different path.
4. The contentious question of where to put all these vendor directories and their security holes comes to the front.

That's a complex spec to implement before 8.7.0 is released. :-)

Hmm but at runtime we'd still need multiple autoload folders for the different combinations of Composer packages... which means we'd need to load the Composer autoloader for core packages, figure out the profile, then load the profile's Composer autoloader...

Zactly. Just to use the plugin, we'd need some parts of core and core's dependencies. It might do us good to refactor to allow determining profile and/or subsite without any dependencies on the rest of core, but again: Who wants to?

So now, in the interest of staying on topic: This conversation is off topic. :-) What are the use-cases?

> So now, in the interest of staying on topic: This conversation is off topic. :-) What are the use-cases?

The summary says:

> 2. Brainstorm potential solutions and look for a path forward that allows a multisite user to be able to use any of the modules that have 3rd party composer dependencies, and allow them to update and mange their site with composer.

I thought this was brainstorming :) Should we stick to use cases for now?

>1. Document what multisite is, and the use cases people use it for. i.e *why* do people need it, and *how* do they use it?

How embarrassing, this issue starts to look like window dressing! How to inform people that don't want to listen?
Documented here: https://www.drupal.org/docs/7/multisite/multi-site-sharing-the-same-code...
Additional use cases provided in #2306013: Multisite is a valued feature that will not be deprecated.

I've been avoiding this issue for far too long, especially since none of the other Aegir maintainers have chimed in yet. Apologies for not getting to this sooner.

But first, a little bit of background...

Aegir is the proper way to manage multi-site set-ups

If you're using multi-site, and not Aegir, I can only think of four (4) reasons why. (If I had 9 more, this could be a Netflix show.)

1. You don't know about it. See Wikipedia and the Web site.
2. You tried it years ago, before it was ready for mass consumption, and it didn't work well. This is no longer the case, and hasn't been true for years.
3. You're planning to use it, but haven't got around to setting it up yet or you'd like help with it (e.g. where to start, need a new feature, etc.). Ask us; we do consulting, set-ups and support.
4. You prefer to spend countless hours doing devops manually on the command line or in a self-maintained or proprietary CI/CD system, instead of taking advantage of automated best practices wrapped in a completely open-source well-maintained customizable user-friendly web application, where you can upgrade 200 of your sites at once with a single button click, rollback on any failures, clone sites trivially, HTTPS-enable each site with a checkbox, control it via Web services and take advantage of a healthy contrib ecosystem (just to name a handful of highlights).

Basically, not using Aegir for multi-site is nuts.*

Now that that's out of the way...

Composer in Aegir

Aegir core has native support for Composer. After a new platform (Aegir-speak for a codebase) is provisioned, a build happens if necessary. (Technically, it's running composer create-project --no-dev --no-interaction --no-progress.) So any sites that are deployed to the platform (i.e. have their site directory created or moved to sites/) can take advantage of it, even though they can still keep whatever they want in their site-specific folders (which are properly maintained).

Additional functionality to support Composer on platform creation is handled by Aegir Deploy (which hasn't made it to Aegir core yet). This allows for, what we believe, to be all of the options necessary for the base case: All sites are using the exact same code.

Narrowing focus

As was alluded to earlier, the base case for this issue is: All sites are using the same code, and there are no modules, themes, or libraries in site-specific folders. For situations where that's not the case, we (the Aegir maintainers) haven't figured out a good way to handle this with Composer yet either, and would love to be part of that discussion.

But let's not have that discussion here now. Let's restrict our focus to the base case for now, and leave the site-specific asset handling/overrides for a future phase of this endeavour. Make it a follow-up issue.

Shared tables

I agree 100% on not supporting this. Aegir doesn't support this, and since I've been around, nobody's asked for it. I would actually create a Drupal core issue to officially un-support it, but do we really need to? Do we have support for it anywhere? If we do, let's create the ticket. Otherwise, let's simply drop it from the scope here. As everyone else here has said already, it's asking for trouble, and there are far better solutions for that (which we don't need to worry about here).

Moving forward

I believe #17 is definitely the way to go. Does anyone have a problem with that plan? If not, let's update the IS, and move forward with it. We can handle discussion like #20 and #22 in the follow-up issue.

I'm assuming by now that we've heard enough use cases. But please add more if we're missing anything.

* Not to be taken seriously.

I've got nothing against you or Aegir, but did you check with the entire community before declaring everyone not using Aegir "nuts"? It comes off a bit aggressive. There are reasons you haven't thought of not to use it.

That's wasn't meant to be taken seriously. I'll edit to make it explicit. (Thanks for the tip.)

Got it!

I don't think Aegir adds anything here, the main issues I see with Multi-site under Drupal 8 are things that seem solvable, I just don't have the capacity to fix it myself. We are running 50 websites using Drupal 8 and Multi-site here at UCSF.

The key issues we run into are as follows:

1. The Auto-loader will choke ( fatal error ) when trying to load the same module/code/plugin in two different places. The Auto-loader should only try to load the files "closest" to the Sites folder. It should know that if its a multi-site environment and the module ABC lives inside /sites/modules/custom/ABC then it SHOULD NOT load the module named ABC inside the modules/contrib/ABC folder.
2. I should be able to define explicitly in Composer where I want some modules to exist, reading comments above this SEEMS possible, but maybe is not as easy as it should be.
3. I don't see a need for Shared Tables or anything like that, all those things can be overcome using decoupling and REST services, which seems like the better way to go these days anyhow.
4. Lastly I think getting Sub-profiles into CORE would go along way to making Multi-site useful again. The tickets on over #500 comments already and it seems like there isn't a single source driving this to ground, would love to see this finally finished.
   https://www.drupal.org/project/drupal/issues/1356276

Didn't read everything, but we already agreed a while ago to drop support for sharing tables in 9.x, there should be an issue somewhere.

Looking for the issue... here we go: #2551549: Deprecate per-table prefixing. Committed and documented. End of (that part of the) discussion :)

Agree with Shared Tables being off topic. It is not something that belongs in this discussion. You don't need multisite for Shared Tables, you need multiple sites.

For the same reason #2551549: Deprecate per-table prefixing should be reverted. Per table prefixing can be used for Shared Tables, but there are other uses as well. Per table prefixes allows you to integrate with data in other databases. It is very silly to have the requirement for data that you want to integrate with to be in the same database. DBA'ers will have a laugh at you and probably say no to that.

And if the above is not bad enough, you are taking something away. That is not how D9 is marketed. No need to give them a reason not to go to D9.

Per table prefixes allows you to integrate with data in other databases. It is very silly to have the requirement for data that you want to integrate with to be in the same database. DBA'ers will have a laugh at you and probably say no to that.

I think you misunderstand what per-table prefixes/shared tables are.

It's perfectly fine to have multiple database connections configured to different databases, nobody wants to take that way. That's how migrate works, among many other things.

Shared tables is about e.g. having different drupal sites share some of the tables, for example use the same user tables so users can log in on all "connected" sites. That concept is flawed and conflicts with a lot of things like caching and invalidation, dynamic table management and more. As others commented, it just doesn't work in D8, never did, so we shouldn't offer it. It will not be reverted.

Despite removing that, if you have a specific use case, you could still offer a specific API/service that would connect to the same database/table and share data through that, it's just no longer available transparently/magically for anything that uses the database API.

That said, I do agree on it not being related to multisite, it equally works/doesn't work wheter or not your multiple sites have the same code base or not.

I think you misunderstand proper release management. It should be unacceptable to remove functionality without reason. Another reason it should be reverted. It is not about multiple connections to possible different datasets or even database servers. For which you need to write code. This about removing existing functionality just because you can. Leaving those that use it without alternative. That is not what deprecation is about. First you provide a better alternative and once most have adapted you can deprecate the old way.

Forcing current users into coding solutions maybe acceptable for you as developer. It is unacceptable for those who were promised an easy upgrade to D9.

In addition you seem to misunderstand what I said about per table prefix and shared tables. Just because a dog is an animal, it doesn't mean an animal is a dog. You needs to use per table prefix to to have shared tables. Shared tables is a working but risky solution. Agree on advising more stable solutions. As mentioned shared tables is not the only use of per table prefix. My current understanding is that combining data from different connections require coding. It is not transparent as it is with per table prefix.

Looks like this needs an issue re-scope. Suggest to remove the offending and condescending introduction and point to the available documentation. Its scope should be the identification of the shortcomings of managing your codebase with composer for building a site with Drupal. #16 by catch is a good start although I disagree on his classification of them. So, after identification they should be classified. The assumption should be that the question is what needs to be done to have the benefit of composer without loosing current capabilities.

Suggest to have an unrelated issue for improving Shared Tables and considering the poor understanding mention it is not related to managing the codebase.

The scope here is collecting use-cases to try and support with Composer.

Thanks @colan for #25... I'd point out that #22 is a rough sketch for something I hope we never have to do.

Added #3022403: Warn users about per-table prefix in multisite during requirements hook so that we can signal people that per-table prefixes are deprecated.

The scope here is collecting use-cases to try and support with Composer.

Read again:

Lets use this issue to do 2 things.

1. Document what multisite is, and the use cases people use it for. i.e *why* do people need it, and *how* do they use it?
2. Brainstorm potential solutions

Which is why I requested a re-scope. The scope should be how to fix composer to support managing the codebase for Drupal sites.

The requirements are clear already:
- Needs a GUI in addition to cmdline
- composer.json needs to be maintainable via composer subcommands
- Needs configuration option for --no-dev
- Needs to support: composer require vendor/package --dry-run
- Needs command like composer outdated, but then respecting version constraints
- Need to be able to specify a destination for the dependency (both direct and indirect)
- Need to be able to specify the scope of dependency
- Needs to resolve dependencies per site
- Needs to autoload per site

> Your lock file is meaningless.

> - Needs to autoload per site

Sudden thought: we use the sites/* folders to segregate our settings, file, and code between different multisites. So why not put a composer.lock and vendor/ folder in there as well?

Then, each lock file and vendor folder is segregated. Composer sees each one as a completely separate thing.

This wouldn't be simple -- we'd need various Composer plugins to do things such as:

1. add command-line options to Composer to tell it which multisite you're currently doing things to, or figure it out from your current location
2. put composer.lock somewhere other than it usually does
3. put packages somewhere else if they're not core

This would be all rather complex, but it's reasonably sane and logical at least.

That is indeed what comes closest to the way composer works currently. It fails to deliver on the concept DRY which is implemented with multisite. You don't want to repeat yourself if you don't have to. As mentioned the most common use case is the same code for multiple sites. When more than 80% of the code is the same you don't want that duplicated. That might be possible to implement with the current merge plugin.

On the other hand composer is already familiar with multiple requirements (require, require-dev). Unfortunately, that is of little use for Drupal, only developers need the require-dev. Which is why there needs to be a configuration option to negate the default --dev for all those sites that don't want that crap on their live servers. If require-dev complements require already, it can't be that difficult to have domain requires to specify the exceptions for particular sites.

So, the question is: do we want to make composer work for drupal or do the site builders need to work for composer. In my opinion it should be making composer workable for drupal. It is drupal that provides more than 95% of desired functionality. Otherwise they wouldn't have chosen drupal to build their sites. Currently composer creates more problems that it solves for site builders.

The most inferior solution would probably be to extend Ludwig with the capacity to produce a ludwig.json for those not providing that service. That would allow site builders not having the problems composer claims to solve to stay away from it. They can keep the workflow they are familiar with.

Comment made in error.

We're now about 40 comments in, so attempting an issue summary update (please feel free to tweak; not sure I captured all the nuances here). Looks like we're quite covered in terms of goal #1 and outlining use cases for multisite that Composer builds need to support. Ideally, these would be captured somewhere less ephemeral than a random issue, however, so documented at https://www.drupal.org/docs/8/multisite/use-cases-for-drupal-multisite for posterity. (Feel free to edit/add to this.) As this is a pretty critical feature for many sites/use cases, I've escalated the issue priority to major.

As far as shared tables use cases go, I agree there are better workarounds, but here are some common reasons I've used this in the past (D6 and earlier):

1. Shared users + sessions tables: allows for a really cheap single-sign on (as others have mentioned, there are more robust options available in contrib)
2. Shared roles + permissions tables: ensures access control is consistent across sub-sites, at least where they're all using the same modules (nowadays roles/perms are in config, so you'd sync that vs. sharing the tables)
3. Shared taxonomy / menu / etc. tables; ensures consistency in navigation, tagging, etc. (could be accomplished with something like Migrate or Feeds module these days)

Basically, anything where there's a crap-ton of initial set up / ongoing tweaking involved, and you want to enforce consistency across all of your sites. Works best in a model where the sub-sites only vary by something very small, like theme or site name.

But anyway, it seems this is rather moot since we're not planning on supporting that capability into the future. (Leaving the multisite feature in tact, though.)

Do we want to close this issue and move onto a new one for solution-eering? Or keep the discussion going here?

Still TBD. The trickiest use case to support with Composer seems to be a multisite where there are variations of modules/themes on a per-site basis. (For example, someband.com and someotherband.com offer the same features [photo gallery, discography, tour], but have vastly different designs, and/or someotherband.com has an integration module with some one-off analytics application that no other sites want/need.)

Next, continue with the Composer Initiative but just make people aware of the limitations currently available for multi-site, mainly that per-site modules and themes don't really work as expected.

The reason I think we don't need to prioritize this stuff right away and continue with #17 (without worrying about it too much) is that you could simply take all modules and themes that all of the sites need, and place them in the shared codebase at the root that's currently supported by Composer. Sites that don't need other sites' extensions simply won't enable them. While it's true that you won't be able to put these things in site-specific directories, you generally shouldn't have to. This should suffice for now, as an initial phase.

In any case, I'd like to see us move forward with solutioneering, yes. (And thanks for the IS update!)

The one use case I can think of that that approach breaks, which has come up only once in my own experience with multisite, is where someotherband.com is using that crazy one-off analytics package, and it requires a different version of, say, the jQuery Update module, compared to all the other sites. The sites/foo location will override sites/all and allows for this use case, which of course everything being in sites/all would not. It's not a super recommended use case, since it can bring all kinds of hard-to-debug problems, and leave you potentially open to security vulnerabilities if you're stuck on an older version of a library, but it does come up. It is possible to document this limitation, however.

If the module in sites/foo with a different version does not have differing dependencies to the one in sites/all, I think that would work. You just can't install it to sites/foo with Composer, but it could be placed there by other means.

Thanks for taking the time to summaries the use cases @webchick

When thinking about shared tables and their use cases a thought came to mind that I wanted to share to at the very least plant the seed of a possible idea to explore.

With the initiatives to improve the configuration management and synchronisation features as well as the new Workspaces module, maybe there's some kind of scope there to use that functionality to provide some or all of the benefits of shared tables. Not so much around session management, I think you'd still want some kind of more advanced single-sign-on functionality, but for entities (such as Users, Taxonomy Terms and Vocabularies, even Nodes) and other configuration looking at those existing tools and concepts as possible solutions could be an interesting path to explore.

Just a thought.

Removing the ending period so it looks better when using automatic linking - [#nid]. (Sorry about the "useless" title change.)

Some of the main reasons in using a multi-site are:
1. Scalability - PHP opcache is reused on all multisite instances, and it will multiply if we go with per-site-sintallations. I've had cases of sites with 300+ sites running in production on a multi-site platform. This monstrosity needed somthing in the range of 128-256 MB of opcache. If this was site per installation it would have needed 5-10 GB RAM per webserver just for the the PHP's opcache. This is translating directly to monthly costs for operations and running the platform.

2. Consistency - you know the limitations before making it - all code is the same. You might change configs between multi-site instances and that is it. You could have a module enabled or not, but never 2 different versions of the same module.

3. Ease of new site provisioning - new database, new site declaration (sites.php) + v-host and you are ready.

I have a virtual machine which has 2GB of memory.
It contains 35 Drupal 7 sites sharing same codebase in a multisite setup.
The sites are not related to each other, some are light with few modules and some are huge eCommerce sites.
Some sites have their own modules in the /sites/all/sitename/modules folder, but some sites does not have any "own" modules. Maintaining many years these kind of setups have been really easy.

I don't want to even guess how much memory I would need more to run these same 35 sites in a separate Drupal installations (non multisite) on a same server. Would even 32GB be enough?
Also that would be a catastrophe to me to update all these sites cores separately.

So for me, and for many others, the multisite feature is a must.

If I understood correctly, D8 and composer can work currently with D8 multi-site only so that all sites share all modules? For me, this is not a problem, I think I could live with a D8 multi-site where all the sites shares exactly same modules. But this is mainly because I am alone hosting these and don't care if all the modules are visible to each sites. I understand that usually
separate sites needs their own versions.

I have just started to install a D8 multisite setup, and got a little scared about this problem with composer and multisite, but I will try to find out how it works.

Instead of thinking how to get rid of multisite in the next Drupal versions, why not to think how to get rid of this awful composer? When composer is a must, it removes the possibility to host Drupal sites on a hosting companies which does not provide shell access. It's a small con compared to other CMS. But I understand Drupal is for pros.

(Those Aegir advertises do not belong here?)

If I understood correctly, D8 and composer can work currently with D8 multi-site only so that all sites share all modules?

Consider that incorrect. It is composer that limits the management of packages to one destination. It also resolves to only one version of a package. It is these limitations that makes you choose all/modules as the destination for those having external requirements and optional for others that fit there naturally. You can still manage individual site/modules in the traditional Drupal way. That includes different versions or exceptional modules for particular sites. As long as they don't require composer due to their dependencies.

For downloading you need to move away from Drush or create the missing commands. You can use Composer or Drupal Console for downloading.

For downloading you need to move away from Drush or create the missing commands. You can use Composer or Drupal Console for downloading.

So with D8 multisite and composer, if I want to download a site specific module to /sites/site1/modules/ -folder, I can't use Drush anymore but just copy it manually there?

With D7 I just go to that specific site folder like /sites/site1/ and type "Drush dl modulename" and it downloads the module to /sites/site1/modules folder. So you mean this is not possible with D8 and composer combination anymore?

> So you mean this is not possible with D8 and composer combination anymore?

That's correct, it's no longer possible:

- Drush no longer does any downloading of modules.
- Composer wants to put all modules in modules/contrib.

There is a way of telling Composer to put specific modules in other places, but it's fiddly -- see one of my earlier comments.

Just throwing my use case into the mix:

• We run 50+ ecommerce sites in a single Drupal multisite.
• We've created an internal ecommerce distribution offered by the company.
• We manage the configuration of each site for our customers.
• No customer administration role has access even close to uid 1.
• No customer administration role can see the modules list.
• We've been doing this for nearly 10 years of active-every-month development, and have only needed to make one customer-specific module. Because no customer administrator can ever see the module list, this unique module is stored in the primary (D7) sites/all/modules, and only distribution administrators can see it. The focus of our distribution is to create features generic-enough for our entire customer base (i.e., as you'd expect in a distribution) and, if a specific customer needs it, we'll "just enable it".

Over the past six months (and for many months in the coming future), we've been working on the move to Drupal 8/9. Using composer (and drupal-composer/drupal-project) has been working just fine for us because of how we've structured things above already (i.e., all contrib and custom modules available for all sites, and distribution administrators turn on the ones we need as needed). We don't "need" the concept of sites/SITE/modules in our distribution and, if we did, we'd use it only for custom code that wouldn't be handled by composer anyways.

It's not only that Composer cannot install site-specific extensions, there is also #2985199: Extensions in Multisite Directories Not Registered When Rebuilding Cache.

@jukka792 Actually, you can still use an old version of Drush 8 to do your download. You do need Drush 9 for other commands on your D8 site.

Indeed, it is possible to configure composer to put things where you want. It still doesn't provide the flexibility of Drupal itself and the configuration is very clumsy. It requires editing json before you require the module. Also, there is only one destination for a module.

I have an idea which covers the problems here and more -- needs minimal composer support although it is possible it's totally scriptable.

First of all, I believe Drupal now recovers fully on cache rebuild if you move your module to a different directory (check this).

Let's presume composer is somehow magically capable of installing to modules/contrib/foo-1.10 instead of modules/contrib/foo (or foo-8.x-dev-defa123 you get what I mean). Now add a mechanism via Settings and a conflict resolution model during project scanning instead of the current blind override -- ie if foo.info.yml is found at foo-1.10 and foo-1.9 as well and said settings contain foo-1.9 then that one wins. Additionally we could have a list of potential projects also via Settings where the UI doesn't show other projects and module installer throws an exception if said Settings exist and project is not listed.

This would allow keeping multiple versions across projects with fairly minimal work in codebase and in maintenance both. For a project with multiple versions, change the package version in central settings.php after an update and keep the local version in the local settings.php.

I hope the "magical composer capability" can be achieved by a script listening on pre-package-install /update. https://stackoverflow.com/a/27293479/308851

> Let's presume composer is somehow magically capable of installing to modules/contrib/foo-1.10 instead of modules/contrib/foo (or foo-8.x-dev-defa123 you get what I mean).

That sounds pretty simple and probably doable.

One potential problem is if we have:

- module foo-1.6 requires composer package (ie not a module) bar 1.0
- module foo-2.0 requires composer package bar 2.0

I have a rather wackier idea than @Charlie ChX Negyesi's. Composer currently has a notion of two different requirement trees: require normal and require --dev. Could we hack into that concept and create further trees, one for each site?

There are apparently rampant rumours circling around that we're getting rid of multisite in Drupal 8/9 despite #2306013: Multisite is a valued feature that will not be deprecated. was closed 2 years ago as "works as designed" (meaning we're NOT doing that), and despite the issue summary here further enforces that it's a critical feature of Drupal.

Trying a title change here in case that is possibly contributing to the misunderstanding.

There were some use cases already written up in https://www.drupal.org/docs/8/multisite/use-cases-for-drupal-multisite based on this and the previous thread. Is this sufficient for what you need, or does it need additional detail?

Given how late in the game we are with Drupal 8, the only sane answer is we need to support all of these, at least until Drupal 10; like all other critical features in core, we need to provide feature parity/BC.

If one or more of these individual use cases can't be supported for various challenging technical reasons, then we should drill in on that, which AFAICS is what #55/#56 are attempting to do.

In case further anecdotal data / user interviews is needed, some of the replies in https://twitter.com/webchick/status/1161379581208612864 are detailing what specifically they benefit from / use out of multisite. (And also from others what they don't like about it. ;))

Historically, we've used Multisite in our hosting platform in what you might call a "reverse" multisite configuration, in order to provide control over when each individual site in the multi-site was updated.

The web hosts would have multiple versions of Drupal and a set of contributed modules configured in a directory like this:

/www/drupal-7/drupal-7.65/drupal/

This would be a docroot for sites using that release. Inside of the sites directory, we'd have simlinks to each site on that release, IE:

/www/drupal-7/drupal-7.65/drupal/sites/example.com (symlink) -> /www/example.com/drupal/example.com/

We'd control the management of what release a site was on and the creation of the sym-links automatically through our provisioning system. That'd allow us to push out a new release of Drupal to our web hosts and then update the release for each site in the infrastructure individually, either manually or scripted.

To the comments in #55 / #56 - The old inheritance model of putting per-site modules in that site's module directory makes a lot more sense from a ease of understanding perspective. Having multiple versions of a module piled up in /modules/contrib/ with some reference somewhere else to control which version is used for each site seems far more difficult to manage and understand at-a-glance than figuring out how to put modules in sites/example.com/modules/contrib, sites/example2.com/modules/contrib. If we could use something like require-example.com in composer to control locations of specific modules that would seem preferable. I've used the installer-paths option on one multisite to install a module for a specific sub-site, IE:

        "installer-paths": {
            "docroot/core": [
                "type:drupal-core"
            ],
            "docroot/sites/example.com/modules/contrib/{$name}": [
                "drupal/example_module"
            ],

Fair point. Do you have an example of an existing document that's more "speccy" that you'd like those use cases reformulated as?

And can we turn your question around? What of the documented use cases do you feel are "rarely" used and you propose we should be able to drop? We can then try and validate those assumptions with actual multisite users, vs. arguing amongst ourselves. :)

One way to help validate assumptions might be to put out a survey via DA channels about multisite usage, if that's where the stalemate is.

Is there some way of adjusting usage statistics gathering to identify if there's more than one site present in /sites/*?

OooooOoh. That's a GREAT question. Though it sounds like we'd ideally need a bit more detail than that as well, if possible, to dig in on specific sub-features. (per-site modules/themes; shared tables; etc.)

Shared tables has been deprecated and it's impossible to use for anything meaningful in D8 (like user tables), the interactions between entity, cache and other systems are way too complex for that to work: https://www.drupal.org/node/2768219. You can't just update the user table and expect another site to pick up that change, it's going to be a mess, with queries against it reflecting the change but then actually loading entities would load them from cache, saving wouldn't invalidate cache tags for rendered elements and so on and so on.

Some people aren't happy about it as discussed here, but there really is no point in keeping it, it just doesn't work.

It has technically also nothing to do with multisite, in that doesn't matter if multiple sites sharing tables are actually multisite or separate installations. I think I've mentioned there before as well ;)

@Berdir: Oh, right, I forgot that we already deprecated shared tables in D8. (I knew they were unpopular with various core devs for various valid reasons, but couldn't remember formally making that decision.) But yes, there are documented workarounds, and while it's a feature that some multisite installs use, it isn't part of the multisite architecture itself, nor what I believe most site owners mean when they say "I rely on multisite."

@Mixologic: Versus unfortunately my "spidey sense" is that #68.2 is much more commonly something people rely on, though #68.3 does indeed "feel" more "edge casey," and even when you did it successfully in D7 and below, you often opened yourself up to weird side-effects. So my utterly non-scientifically backed opinion is that we need per-multisite modules/themes but we do not natively need to support multiple versions of the same extension in the same site, at least as long as there's some other workaround people can do (symlinks or whatever).

Do you want to work together to put some kind of quantifiable data together on this? (I'm guessing survey is our only option, since it sounds like hacking at the usage stats telemetry for it is out.)

Pinged the other thread with this question, too, to help point multisite users over here. #2306013-95: Multisite is a valued feature that will not be deprecated.

And re-pinged the Twitter thread of DooOOOOOom. :) https://twitter.com/webchick/status/1161714301549174785

That's about all I have at my disposal for data gathering, unfortunately. :(

So my utterly non-scientifically backed opinion is that we need per-multisite modules/themes but we do not natively need to support multiple versions of the same extension in the same site

The issue is: #68.2 and #68.3 are vanishingly similar, from a complexity standpoint.

The #2 scenario: If we have two subsites and they want two different modules, one per subsite. And each of these Drupal modules has a Composer dependency that would ordinarily conflict with the other. We would expect that they wouldn't conflict, because they're never used at the same time per request. But Composer doesn't know how to set that up, and core doesn't know how to autoload differently per site.

This is just about the same as the #3 scenario in terms of making it work.

That's why I'm generally +1 on #68 (ie, please for the love of all that is good, can we not need to support that), but thumbs way down on this description: "Supporting #2 is difficult, and really isn't very different from. #1 other than the extensions shown in the admin Interface"

It's very, very different.

I think he means the similarity from an end-user perspective.

Like, one of the largest multisites I ever worked on was something like 2,000+ artist websites all run off the same code base. Each one had its own theme in its own sites directory. When you went to configure the theme at admin/themes, you just saw the default themes Drupal ships with + the base theme we were using + "Artist Name Theme." So, like 6 options.

Versus if all of these were in sites/all/themes, I'd have 2,000+ options on the themes page and would need to scroll past all of them to find "Artist Name Theme." Which is super not helpful. :\ (And woe be to you if you accidentally set Death Metal Band's theme on Super Happy Pop Artist Band's site, lol.)

That said, these themes were all obviously custom themes, and in that case I don't think there's a problem with putting them in sites/XXX/themes, because there's no composer.json to trip them up? (Unless they were using composer.json to track dependencies for the theme, maybe, hmmm.)

Anyway. That's an example use case currently solved by per-site modules/themes directories. And there might be other ways to tackle that, e.g. via Drupal UI configuration (some means of setting hidden = true on everything but this one, for example).

Even though #68.2 is something I have used and seen often, I guess sharing all modules and simply not enabling them on certain sites could be good enough. Although adding a module specifically for one site reduces risks of sites enabling the modules when they shouldn't. Most of the times it would be custom modules that were not managed by composer though.

I have not seen different versions of the same module in subsites yet as #68.3 mentions. This seems very dangerous imho, I would hate to debug issues coming from this in complex multisites.

I could live with only supporting #68.1.

Re: #73

Right, but what I'm saying is that for the purposes of using Composer to manage that, we have to assume that:

(Unless they were using composer.json to track dependencies for the theme, maybe, hmmm.)

...is true, and that all the dependencies conflict with each other x 2000. :-)

That's a great use case, BTW.

Use-case to throw into the mix: farmOS is a Drupal distribution for farm record keeping. I host it for farmers, and use multisite to share the common distro code, but occasionally put custom modules for individual farms into sites/[farm-url]/modules.

So for me, the requirements described by #68.2 above apply.

Personally, a solution along the lines of #36 seems to make some sense, simply because I consider everything in sites/*/modules to be "downstream" from the distro/core considerations.

Eg: I might move a site to a different server, so it's handy to be able to just move the sites/*/modules directory, and not have to touch anything in parent directories (like an "upstream" composer.json).

I haven't given any thought to how that would work in practice, though - so I'm not sure of the feasibility. Just my 2 cents as a distro maintainer. :-)

I think the greatest chance where you'd run into some of the edge cases is a situation where you're not running a 'distribution' per-se (whether custom or contrib, like farmOS, Lightning, etc.), but you're running one codebase + module set on a variety of wildly different sites.

This is what I currently do for a set of Drupal 7 sites—each with a custom theme, and a few with custom modules—on a shared Drupal codebase.

Over time I've found that it's more painful this way than having separate codebases, especially with Drupal 8 and the composer issues with theme dependencies (so helpfully highlighted by @webchick above), but it _is_ nice to be able to run 5 moderate traffic D7 sites on one $5/month 1 GB VPS without worrying about PHP memory issues (I could not run 5 separate codebases on the same server).

So the 133% price reduction ($5/month instead of $25/month for these 5 sites) is worth the pain for me, since it's an upfront/one-time cost of getting all the custom stuff to play nice in the shared codebase.

core doesn't know how to autoload differently per site.

hmm, really? This does not work in D8?

I am using multisite with D8 and composer, and also with D7 and composer, and actually I do not have any issues.

In D8 we have a simple setup, using the exact same modules on every website, maybe that is why we do not have issues.

In D7 we went even over the edge with multisite setup. We are using a set of custom modules on a set of websites, like
set1 (module A + module B) is visible only to site1 and site2
set2 (module C + module D) is visible only to site3 and site4
and there is only 1 instance of every module.

The usecase is: multiple countries with multiple websites per country. The websites should not see modules from other countries.

The trick is we install the custom modules outside the web folder and use symlinks from sites/x. The drupal-custom-module composer package is not 1 module but a set of modules, therefore we only need 2 symlinks per site (1 link to country specific modules, 1 link to site specific modules and theme).

And on top we use openshift (similar software as kubernetes), we can scale up as needed. It is not true that multisite does not scale.

Not sure if you needed more convincing or more use cases for multisite, just wanted to say, for me the multisite is good as it is..

@Mile23: core doesn't know how to autoload differently per site.

@Pasqualle: hmm, really? This does not work in D8?

Core can autoload different *modules* per subsite, just like always.

Core can't autoload different *Composer-only dependencies* per subsite, which is a big technical problem that we'd have to solve for some of the use cases.

This is mainly for D8, but your D7 situation looks like one that would be reasonably common in D8, too.

It seems like I will need an example to understand the problem.

As I understand there is only 1 composer.json file even in multisite setup, and almost every package is installed in the /vendor folder. If site1 needs for example the "sebastian/diff" package, then the autoloader can find it in the vendor folder, if site2 needs a "symfony/dotenv" package, then the autoloader can find it also in the vendor folder.

Well, without getting too technical in this use-case issue....

Let's say siteA needs moduleA which needs the external Composer-based package mile23/composer-package, but only version 1.x and not 2.x.

siteB needs moduleB which needs mile23/composer-package 2.x but not 1.x.

Now we have a conflict. We can't reconcile both of these at the same time because they have mutually-exclusive Composer requirements. So they can't both live in the same vendor directory. Composer won't let us even start.

However, as Drupal people, we'd expect this to not be a conflict, because in Drupal-land these two sets of dependencies are independent of each other at runtime. We expect that we'd be able to fix it so that sometimes we'd use mile23/composer-package 1.0 and sometimes 2.0, depending on which subsite should handle the request.

Any solution to fix that issue would be a big ol' pain, and probably require things like refactoring the kernel. So the question is: What use cases can we support without, for instance, refactoring the kernel, vs otherwise.

Thanks for the explanation, this is quite clear this option does not work with composer.

But if you try the example without saying siteA and siteB, single site setup, then the example still does not work.

Multisite Drupal installation is a 1 composer project, and must be without version conflicts, otherwise composer will not able to install. There could be only 1 instance of a package.

I understand why Drupal developers might expect it to work. But as I see no package manager can support such setup, and we should not even try to support it.

(edit) double comment removed..

That is correct, it is no different from a single site. Composer imposes that limitation. Same thing for multiple composer.json in drupal multisite or multiple single sites. Assuming you use a virtual package based on the shared composer.lock to prevent having different versions of a dependency. That is the same logic used by webflo/drupal-core-strict, only the scope is different (sites/all instead of core).

So I've been following this discussion for a while, and I'd like to reference back to what @Mixologic said in #68

I think the main issue, with regards to composer is how we handle extensions.

1. All extensions shared between all sites on the multisite. (sites/all/modules)
2. Some sites have different extensions specific to the that site only (sites/example.com/modules)
3. Some sites have *different versions* of the *same extension*

Supporting #1 is essentially already supported.
Supporting #2 is difficult, and really isn't very different from. #1 other than the extensions shown in the admin Interface
Supporting #3 is extremely difficult, and would potentially subvert some of the nicer features that multisite like having a shared opcaches.

It seems that every use case for #3 is attempting to use multisite as a form of deployment mechanism, which isn't really what it was meant to do, but how it was utilized.

A lot of the discussion in recent comments seems to be around scenario #3 above, or more specifically what happens when Drupal extensions installed across a multisite require different versions of the same Composer package.

When it comes down to it the problem that is being discussed is simply this: how do we handle Drupal extensions which have dependencies on the same Composer package but requiring that package at different versions.

Now, correct me if I'm wrong, but that problem is not technically restricted to just multisite, you might come across that same issue in a single site setup, it's just that chances are if you were going to experience that issue it would be with a multisite setup.

That being said if we can agree that the problem is not necessarily specific to multisite, but more of a general Composer support problem, maybe that reduces some blockers here. Maybe we can simple note that as an unlikely edge case which isn't currently supported in a multisite Composer setup, and discuss that specific problem in a different issue.

A final thought, it's possible that to achieve full Composer support for multisite, we might have to open issues on the Composer side and start contributing code changes there. I'm not sure if we will actually need to do that or not, but it's worth remembering that we don't have to solve this problem entirely within Drupal itself, and for some of the harder blockers, contributing changes to Composer itself might be the easiest approach.

Re #86

The difference is subtle, but on a single site setup, composer has access to the versions of all dependencies, and if a conflict arises, it is able to report to the user that there is a conflict, and it will not continue. In this situation the end user would have to decide how to proceed, but a conflict is detectd.

In a hypothetical multisite situation where composer is only concerning itself with the dependency map of a particular site, but sharing its /vendor directory between all sites, there is a possibility for an *undetected* conflict between the different sites because it cant account for all the needs of all the sites dependencies. This could cause composer to say there is no conflict, and keep flip flopping between two different versions of a dependency, breaking one site in order to make another site work.

In a multisite setup, is there a reason Composer then couldn't be aware of every site.

What I mean by this is say for example we implement something like the approach described in #62 of:

If we could use something like require-example.com in composer to control locations of specific modules that would seem preferable. I've used the installer-paths option on one multisite to install a module for a specific sub-site, IE:

        "installer-paths": {
            "docroot/core": [
                "type:drupal-core"
            ],
            "docroot/sites/example.com/modules/contrib/{$name}": [
                "drupal/example_module"
            ],

The root composer.json is aware of every extension installed in every site, so that would avoid that subtle issue @Mixologic described above. We could then add support for the --sitename flag which could be used when installing a extension to place it in the correct location in the file system.

I can though see a couple of limitations here, one which I think we might be able to solve:

1. If a extension is installed on one particular site, and it needs installed on another site, we could build a Composer plugin which if the user tries to install that extension again for a different site, the plugin informs them it is already installed for a specific site and gives them the option to make that extension available to all sites, if the user chooses that option the plugin would move the extension to the relevant "all" folder and clear the Drupal Cache.
2. If we do the above, the only remaining edge case I can see would be where the user wants the same extension on more than one site without it being available to every site, or multiple different versions of the same extension in different sites, but that almost seems like an even less likely edge case which we might be able to just accept as a limitation and one it doesn't seem like we'll be able to solve.

I think this might be a better solution than the previous one involving the --sitename flag, because it preserves the benefits of having extensions physically located under the site they are being used on.

To add a bit from my experience of supporting multiple codebases that did have multiple versions of modules in different site folders, I've found that to be *extremely* problematic with apcu. We've ultimately ended up disallowing that approach as a part of our build process (at least not allowing a site to keep an alternate version of a module that is in /modules - we don't have a good way to restrict sites that have competing versions of something in /site-a/modules and /site-b/modules that is *not* in /modules). So for the scenarios presented in #68 I have already been working toward a situation where 1 & 2 are fine but 3 is deprecated for my users, and I think that's somewhat optimal.

Please pardon me for getting a little technical here.

Here is a sort of back-of-napkin diagram for some of these use-cases.

This is not a solution, but an estimation of how much work would have to happen at minimum to support #68.2/#68.3.

Let's tackle #68.3 first (and I'd argue, also #68.2, but maybe that's just me). I call it:

The Isolated Multisite Front Controller Model

(AKA: The Lovecraftian Horror Of Process-Isolated Subsites)

From the perspective of how a request is handled:

• Our HTTP server gets a request.
• We boot up a miniature kernel, called something like MultisiteKernel. (This means refactoring DrupalKernel to separate concerns.)
• This determines whether this request is a multisite request, or whether it should map to sites/default/.
• If it’s not a multisite request, rock on with DrupalKernel much like we do now.
• If it's a multisite request, we make a new process and send the request to a secondary front controller within it.
• This second process autoloads everything from its sites/example.com/autoload.php, and uses DrupalKernel to handle the request. This is why it's in an isolated process.

Based on this request handling process, we have to retool Composer to work like this:

• Each subsite has a composer.json file. (If there isn’t one, use the sites/default autoloader.)
• A Composer plugin knows how to read in sites.php.
• This plugin visits all the directories in sites.php and performs composer install.
• Each subsite will have its own vendor/ directory.
• Each composer.json file also merges the dependencies from core/composer.json and the root composer.json, enforced by the plugin I suppose.
• See #62 for some clues on how we might structure installation of mutually-exclusive Drupal extensions. We’d have to roll this as our own Composer plugin, however, because it’s very likely we’d need to do things the composer-installer project does not want to.

Now we essentially have a series of almost completely independent Composer-based projects in subsite folders, that share the filesystem of the root Drupal. They can have mutually exclusive dependencies as needed. A thin front controller figures out which one to boot at request time.

Using this technique, we will have lost the benefits of opcaching, memory footprint, and disk space as mentioned in #86, for sites which have mutually-exclusive dependencies.

Scaling back, we could then also design #68.2, which I will call:

The Multisite Requires All Dependencies To Resolve Model

For this model, we have one vendor directory as we do now. From the perspective of a request to our HTTP server, it works the same as it does now.

However, we must tool Composer in this way:

• Our Composer plugin can understand sites.php.
• Our plugin merges all composer.json files present in all sites.php subsites, much like the wikimedia merge plugin.
• See #62 for some clues on how we might structure installation of Drupal extensions that we only want per subsite. Again, we’d have to roll this as our own Composer plugin, because it’s very likely we’d need to do things the composer-installer project does not want to.
• Any non-reconcilable dependencies are now your problem.

In this version, we keep the opcache, memory, and disk space benefits, while allowing Composer dependencies to be declared per-subsite. This means you can have a custom extension in sites/example.com/modules/bespoke_module and if it needs a Composer-based package you'd declare that in sites/example.com/composer.json. However it also means that all subsites use the same classloader, so a subsite which doesn't require a Composer package could (and maybe in some cases, will) still use it anyway. (For instance, one subsite wants PHP's yaml extension, and another wants symfony/yaml. Symfony probably wins that one.)

Clever readers will notice that the work for the All Dependencies Must Resolve model is to some degree a subset of the work for the Isolated Front Controller model. However, if we allow users to declare their dependencies per-subsite, then switching to the Isolated Front Controller model later is tricky because we lose the memory and disk space benefits.

It’s also possible that there is a better solution than any of this. :-)

I wouldn't want to have to implement something like #68.3, but if I were to do so, I'd do something like this:

• Every multisite gets a composer.json at sites/*/composer.json
• Each multisite's composer.json includes the composer.json from the project root as a path repository. Maybe this config could even be injected on-the-fly by a custom Composer plugin to avoid having the boilerplate actually in the multisite composer.json
• Whenever composer update runs on the composer.json file at the project root, then a Composer script or plugin also runs composer update on each of the multisite composer.json files. However, we will patch out the install operation, so the composer.lock file is generated, but nothing else.
• Now we have a pile of composer.lock files, which we will post-process. We will read in the composer.lock from the project root, but not modify it. For each multisite composer.lock file, we will read it in and then **remove** any entry that also exists in the project root's composer.lock file. Once done, we will rewrite the reduced composer.lock file.
• At this point, we can just run composer install on each multisite directory. The result is that we will have one vendor/autoload.php file at the project root, and one more vendor/autoload.php file in each multisite directory.

While this would be a bit of a pain to implement, it has loads of advantages. It maintains the benefits of opcaching and disk footprint, and each multisite may have different versions of the same modules if they really want to. What's more, the two autoload files can be merged together freely (which is to say that any ONE multisite autoloader may be mixed in with the root autoloader), so there is no need to make a special reduced Kernel, and there is no need to alter the way multisite bootstrapping works. The only thing that needs to happen at request time is to include sites/*/vendor/autoload.php if it exists as soon as the specific multisite is identified during bootstrap.

The main downsides of this plan is that it needs special knowledge about the composer.lock internal format. There probably are not Composer APIs to help out here, but I did not investigate.

This idea is unproven, but I'm reasonably confident that it would work.

As an added reference, I will point out that the first steps described in #90 are very similar to what is already being done by the Composer Test Scenarios Composer plugin.

The main downsides of this plan is that it needs special knowledge about the composer.lock internal format. There probably are not Composer APIs to help out here, but I did not investigate.

If we do go with the approach in #90 and if the above is true then of course we also have the option of contributing those missing API endpoints directly to Composer, it would take more time but would probably be better in the long run.

Another reference: the code that generates the webflo/drupal-core-strict metapackage relies on special knowledge of the composer.lock format.

In related autoloader changing #3020296: Remove Symfony's classloader as it does not exist in Symfony 4

I'm curious, does anyone know if Composer 2.0 opens up any new/improved ways to better support multisite installations?

I've had a read through and I don't see anything new that would help.

I skimmed through all the comments and what I would like to ask is how multisite feature affects exporting of config files. Suppose we have a /config/sync folder outside of docroot and also a sites/default/files in .gitignore. Naturally files/ therefore config_/sync will not be committed. So, Drupal should change the location of the default /config/sync folder I think.

This was discussed before, the default location was chosen to be in files to avoid having yet another installation step. The composer template could possibly provide a default config setting that changes that, but it's really not related to this issue, that problem is not related to composer and is exactly the same issue for a single site installation.