Change how modules and submodule dependencies are handled to have more consistent namepsaces [DRAFT].

I forget why our Composer needs to care about submodules.

#2: so one is able to e.g. do composer require drupal/admin_toolbar_tools, without having to know that one would actually need to do composer require drupal/admin_toolbar.

Or: drupal/metatag_views vs drupal/metatag.

* Submodules can have dependencies that the entire project does not have. -> optional dependencies if you are using an optional submodule. Example: the drupal/cloud pro

+

3. need to know which projects have dependencies in their info.yml files that are *not* namespaced to the project.

This is the case that just came up while working on https://www.drupal.org/project/automatic_updates + https://www.drupal.org/project/project_browser.

The case of admin_toolbar_tools being a submodule of admin_toolbar is fine, because it only has a dependency on the "root" module.

However, the case of metatag_views being a submodule of metatag is not fine, because it has one additional dependency:

dependencies:
  - metatag:metatag
  - drupal:views

… which is not reflected in the generated metapackage:

$ composer show -a drupal/metatag_views
name     : drupal/metatag_views
descrip. : Provides views integration for metatags.
keywords : 
versions : 2.0.x-dev, 1.x-dev, 1.22.0, 1.21.0, 1.20.0, 1.19.0, 1.18.0, 1.17.0, 1.16.0, 1.15.0, 1.14.0, 1.13.0, 1.12.0, 1.11.0, 1.10.0, 1.9.0, 1.8.0, 1.7.0, 1.6.0, 1.5.0, 1.4.0, 1.3.0, 1.2.0, 1.1.0, dev-2.0.x, dev-1.x
type     : metapackage
license  : GNU General Public License v2.0 or later (GPL-2.0-or-later) (OSI approved) https://spdx.org/licenses/GPL-2.0-or-later.html#licenseText
homepage : https://www.drupal.org/project/metatag
source   : []  
dist     : []  
names    : drupal/metatag_views

support
source : https://git.drupalcode.org/project/metatag

requires
drupal/metatag *
drupal/core ^9.4 || ^10

IOW: I would expect drupal/views * in the requires list too.

AFAICT #3064900: Always or only respect disambiguated submodule names is tangentially related.

But more importantly, #3099771-2: Don't expose submodules in Drupal's packagist at all by @mixologic seemed to suggest #4 was already supported, but this issue summary makes clear it's not yet supported:

In any case, we wont be removing the 'submodules become metapackages, that contain their optional dependencies + a dependency on the parent package' anytime soon.

(I agree with that quote, just saying that the submodule's dependencies are currently missing.)

Are we saying that since composer require drupal/metatag_views is currently working, nobody can register a metatag_views (project/module machine name) on drupal.org nor have it working with composer? This is scary, as current maintainers could add a plethora of submodules kind of unoticed ..?

I'm trying to fix a bunch of projects that got the double names here: https://www.drupal.org/project/infrastructure/issues/3371966#comment-151...

I'm thinking there could be a way in project_composer modules to handle this.

The user story I am thinking of:

1. Module with a submodule get released under a single project.
2. Module grows, the maintainers want to move the submodule to it's own project.
3. The maintainers want to make the new module package name drupal/$project_name, so they must remove the association between the original project_name component_name and project_namespace.

Perhaps we need a little UI for listing and updating a Project nodes Package Maps?

The request

I'm going to politely repeat my request from ~4 years ago in #3099771: Don't expose submodules in Drupal's packagist at all as I've been bitten by this in two different ways in the past few weeks, one of which I shared on Slack

Please treat the Drupal.org project equivalent as packages and stop shipping submodules as separate metapackages.

Previous Feedback

I'll start by addressing the feedback provided on the linked issue from back in the day, which I didn't do at the time because the issue was closed.

2. There is no requirement to namespace your dependencies in your info.yml file. Drupal core itself does nothing with the project name metadata, and in fact, just strips it off and discards it. This is just a suggestion, but a huge number of modules and projects do not have this information.

I feel this has now been in the coding standards for many years. I think it exactly solves the problem we have: we need a dependency, we need to know what project to pull it from. If our impediment here is "this is not something that Drupal enforces" then I don't think it's a big ask of Drupal core to start enforcing this in a major version (if we had done that 4 years ago we could now reap the benefits, but even if it takes til the Drupal 11 release, we can benefit in the future).

Having that format enforced can have advantages for static analysis tools too and if we're unwilling to at some point turn this suggestion into a requirement, and Drupal core does not use it, then we should probably instead stop suggesting it.

1. There is no such thing as a submodule. There are only guesses and heuristics that we can derive based on location of the filesystem or an exact match between project machine name and a module name that lives inside the project.

At runtime for Drupal I think you're right. At packaging we would have all the info to make this decision, but the only issue would be that a random dependency might hit multiple modules if multiple projects contain the same module. This could be solved by solving 2 (which is why this feedback is out of order).

3. Composer has no concept of an *optional* dependency linked to an *optional* feature (aka submodules). When you have a submodule that has a 3rd party dependency, whether thats a composer package or another drupal module, those dependencies are *different* than the "primary" module. Example: https://www.drupal.org/project/flexiform has a 'submodule' called flexiform_rules -> that has a dependency on the rules module. we definitely do not want to have to download every optional potential dependency that a project may have - we would only want that if the user explicitly asks for flexiform_rules.

I think this is debatable. Outside of Drupal, if I download something through composer it'll come with everything I need, even if I don't need all of it. If this would truly be a problem for the module then moving it to a new project might also be a good answer. Disk space is cheap? :D

4. The issue with open social is that fact that distributions themselves are not supported with composer drupal.org, and we havent yet created a space for 'distributions 2.0' that allows for proper namespacing, doesnt have things like whole copies of core within them etc. Basically, hosting a module on drupal.org that has a dependency on a module that ships with a distribution is currently undefined behavior.

That's fair, that's one of the two issues I now ran into again which I'll describe below. However, the problem would also go away if we treat a project as a single versioned package of code and stop trying to process what's inside of it. This problem might be alleviated by recipes, but it's going to be a few years before major distributions are going to be converted.

The problems I ran into

I'll describe the two issues that cause me to revisit this issue, the second one I ran into today which is caused by Open Social being a distribution, the first one prompted the Slack post 2 weeks ago.

In both cases the top of the project includes a composer.json file that already describes what it wants and ideally that would not be altered.

A "submodule" depending on a parent module

• We have a module social_pwa which is a Drupal.org project of the same name. This project has a submodule activity_send_push. The module contains its own composer.json file that works just fine.
• The module does something silly and enables the submodule as a dependency of the top-level module (it's stupid, but as far as my Drupal experience goes, not actually illegal)
• Drupal's Packagist now decides to add a dependency of drupal/activity_send_push: * to the composer.json.
• drupal/activity_send_push because it's a submodule has a dependency on a specific version of drupal/social_pwa.

Composer will now disallow updating the module from 1.x to 2.x with any of its update commands. The cause for this is that Composer 2's optimizations will exclude the 2.x version of activity_send_push/social_pwa because of this cyclical dependency.

This results in the lovely error of:

  Problem 1
    - Root composer.json requires drupal/social_pwa ^2.0, found drupal/social_pwa[dev-2.0.x, 2.0.0, 2.0.x-dev] but these were not loaded, likely because it conflicts with another require.

If the submodule is a dependency of the parent, why is it a submodule?

Historical reasons; a developer made an incorrect decision; the submodule implements an optional plugin for another system that it interacts with and we ship it as default implementation, but there might be scenarios where a user wants to use only the submodule and provides their own overarching system (the submodule does not depend on the parent).

However as mentioned in the reply to my previously created issue There is no such thing as a submodule. So we really should be free to organize our code and dependencies as we wish, without this breaking composer installability.

A module depending on a module in a distribution

It was already mentioned that there's currently no support for distributions. It was also mentioned that the project:module syntax is ignored. However for the generation of the composer.json the format does matter.

In social_geolocation 2.4.0 I worked on Drupal 10 compatibility and as part of cleanup moved to the new project:module syntax. I suddenly found myself unable to install that version because it depended on social_event, social_profile and social_group which are modules in the Open Social distribution. This is covered by the goalgorilla/open_social dependency in the composer.json but listing this separately will cause composer to complain about missing packages (or worse, install unexpected projects that override module names).

Changing the names back to module (removing the project: prefix) caused the Packagist to ignore those dependencies which solved the issue. This can be illustrated by looking at the composer info for the two versions.

$ composer info -a drupal/social_geolocation 2.4.0
name     : drupal/social_geolocation
descrip. : Provides geolocation functionality for groups, events and users in the Open Social distribution.
keywords : 
versions : 2.4.0
type     : drupal-module
license  : GNU General Public License v2.0 or later (GPL-2.0+) (OSI approved) https://spdx.org/licenses/GPL-2.0+.html#licenseText
homepage : http://drupal.org/project/social_geolocation
source   : [git] https://git.drupalcode.org/project/social_geolocation.git 2.4.0
dist     : [zip] https://ftp.drupal.org/files/projects/social_geolocation-2.4.0.zip 2.4.0
names    : drupal/social_geolocation

support
source : https://git.drupalcode.org/project/social_geolocation

requires
commerceguys/addressing ^1.0.7
drupal/address ^1.0
drupal/core ^9 || ^10
drupal/geolocation ^3.10
drupal/geolocation_address *
drupal/geolocation_leaflet *
drupal/search_api_location ^1.0
drupal/social_event *
drupal/social_group *
drupal/social_profile *
goalgorilla/open_social ^11.10.2 || ^12

requires (dev)
drupal/geolocation_leaflet *
drupal/search_api_location *
drupal/social_event *
drupal/social_search *
roave/security-advisories dev-master

$ composer info -a drupal/social_geolocation 2.4.1
name     : drupal/social_geolocation
descrip. : Provides geolocation functionality for groups, events and users in the Open Social distribution.
keywords : 
versions : * 2.4.1
type     : drupal-module
license  : GNU General Public License v2.0 or later (GPL-2.0+) (OSI approved) https://spdx.org/licenses/GPL-2.0+.html#licenseText
homepage : http://drupal.org/project/social_geolocation
source   : [git] https://git.drupalcode.org/project/social_geolocation.git 2.4.1
dist     : [zip] https://ftp.drupal.org/files/projects/social_geolocation-2.4.1.zip 2.4.1
path     : /Users/alexander/Projects/cablecar/html/modules/contrib/social_geolocation
names    : drupal/social_geolocation

support
source : https://git.drupalcode.org/project/social_geolocation
error : Invalid dependency: "social_profile" is an unknown drupal 8 package name

requires
commerceguys/addressing ^1.0.7
drupal/address ^1.0
drupal/core ^9 || ^10
drupal/geolocation ^3.10
drupal/geolocation_address *
drupal/geolocation_leaflet *
drupal/search_api_location ^1.0
goalgorilla/open_social ^11.10.2 || ^12

requires (dev)
drupal/geolocation_leaflet *
drupal/search_api_location *
drupal/social_event *
drupal/social_search *
roave/security-advisories dev-master

The project browser initiative

I'm not sure what the plans for the project browser are (I couldn't find anything concrete), but I think they'll always default to installing the top level project, since that's what has the metadata and a logo, etc.

So this might already be a good counter for the argument of having to promote all dependencies to the top level. In the flexiform instance, if the project itself does not include the rules dependency, then the flexiform_rules module might be installed through composer, but there's no control for that project over which version of rules it's compatible with. (Though I can see this is already a problem in the current implementation).

Cross posting a possible solution from another issue discussing the same problem.

After doing some research into the composer schema, would it not be better to use "abandoned": "drupal/[submodule]" in submodules, suggesting the main module? This would allow a user to find submodules through composer, but also guide them to installing the package correctly. And when a submodule is moved out, it can become a 'real' package and override the abandoned definition.

And yes, I understand that the project isn't really abandoned, but it's functionally equivalent (and the only option in the schema that achieves the same outcome).

I agree that we can't have it both ways. A submodule either needs to be treated like another package or not be treated like a separate package at all. Currently it is being treated somewhere in-between

Current Situation

composer require 'drupal/projectA_submoduleG' Results in The parent module being included using the parent module's composer.json. The submoduleG's composer.json is ignored. So the parent module must include all dependencies of both the parent and any optional submodules or there will be mayhem. A lack of efficiency is the result.

Solution One

Composer has no full awareness of the submodules. composer require 'drupal/projectA_submoduleG' would result in: Could not find drupal/projectA_submoduleG' did you mean drupal/projectA'

Solution Two

Treat submodules like distinct packages. composer require 'drupal/projectA_submoduleG' would result in: that parent module being downloaded and would utilize the composer.json within projectA_submoduleG and would also use the composer.json in projectA because it is a dependency of projectA_submoduleG.