Drupal 9 and 10's Drupal\Component composer.json files are totally out of date

I was working on automating it 4 years ago, but no one seemed excited to try and bring it to DrupalCI: #2876669: Fix dependency version requirement declarations in components

My little project might be an easier port now that we're normalized on Gitlab CI. https://github.com/paul-m/drupal_component_tester

Happily boggled that travis-ci is keeping test results from 4 years ago... https://travis-ci.org/github/paul-m/drupal_component_tester/builds/54437...

Also, it might be that a component is totally OK with using an older version of PHP, having no code specific to a newer version. Obviously keeping track of all that is an extra layer of annoying, but it could be somewhat automated as above.

It seems like we could generate some dependencies similar to the way way we generate the Composer metapackages. This would be easier if there were a policy that said: "All components inherit environmental dependencies from core, always." :-)

Adding related issue.

Related: #3103918: [policy + docs] Decide on backwards compatibility policy for Composer plugins in Drupal 8

Assuming that the constraints of all components should match whatever core's doing at the time, here's a handy patch to discuss.

Basically this patch emulates the template generators and merges core's dependencies where they exist.

It does this on composer update.

It assumes that if we're in a dev branch, then the components also should have minimum-stability: dev.

We'll see some fails for Render and Utility, because it's a pain to set up a path repo to a dependency when the dependency depends on YOU, within a non-versioned branch. #2958358: Remove Drupal\Component\Utility's dependency on Drupal\Component\Render short-circuits that issue if we fix it.

If this is the right direction, we can devote more time to it.

I recall someone from commiters used to create scripts to bump versions, it could be used too

This looks like a good start.

Though rather than decoding the composer.json ourselves, couldn't we get the data from generateMetapackages()'s $event parameter?

generateMetapackages() will need a rename as well, as it'll be doing more. Or, better still, we could add a 2nd post-update-cmd event.

I am wondering if it's possible to run a shell `composer` command in the Component folder and issue a 'composer require' in there, with parameters based on the $event.

Thanks, folks.

The scope here is: The components' composer.json files need to be maintainable, and they need to have the same dependencies as core where there's overlap.

@joachim: "Though rather than decoding the composer.json ourselves, couldn't we get the data from generateMetapackages()'s $event parameter?"

That's a good thought, but I'd rather not refactor PackageGenerator for this, because I'd want to it similar to ComponentGenerator. Once we have the general shape set up for maintainers' needs, we can optimize, or use a follow-up.

The other reason is that we analyze both drupal/drupal and drupal/core. We get the lock file for drupal/drupal, and the package file for drupal/core.

So this is a demo of the fact that we can automate this. Now we have to say what the BC policy is for components. #3103918: [policy + docs] Decide on backwards compatibility policy for Composer plugins in Drupal 8 does not address Components, and it might be a shame to rescope after it's essentially done.

Woop found another one: #3179197: Drupal components depend on ^8.8 even in their Drupal 9 version

Also in #3179197-15: Drupal components depend on ^8.8 even in their Drupal 9 version @xjm links to the release script. In particular this bit looks like it's supported here: https://github.com/xjm/drupal_core_release/blob/main/tag.sh#L76

# Update the version strings in the metapackages
echo "Updating metapackage versions to ${v} and tagging."

# Update the path repository versions in the lock file
COMPOSER_ROOT_VERSION="$v" composer update drupal/core*

CS issues addressed.

Some stuff from #11 addressed.

Here's what it looks like when you run it:

% git checkout 9.5.x core/lib/Drupal/Component 
Updated 25 paths from 771d32c33d
% COMPOSER_ROOT_VERSION=9.5.x-dev composer update 'drupal/core*'
> Drupal\Composer\Composer::ensureComposerVersion
Loading composer repositories with package information
Updating dependencies
Nothing to modify in lock file
Installing dependencies from lock file (including require-dev)
Nothing to install, update or remove
Package doctrine/reflection is abandoned, you should avoid using it. Use roave/better-reflection instead.
Package phpunit/php-token-stream is abandoned, you should avoid using it. No replacement was suggested.
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
Hardening vendor directory with .htaccess and web.config files.
72 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Cleaning installed packages.
> Drupal\Composer\Composer::generateMetapackages
> Drupal\Composer\Composer::generateComponentPackages
Updated component file core/lib/Drupal/Component/Transliteration/composer.json.
Updated component file core/lib/Drupal/Component/Datetime/composer.json.
Updated component file core/lib/Drupal/Component/Bridge/composer.json.
Updated component file core/lib/Drupal/Component/EventDispatcher/composer.json.
Updated component file core/lib/Drupal/Component/Serialization/composer.json.
Updated component file core/lib/Drupal/Component/Graph/composer.json.
Updated component file core/lib/Drupal/Component/Discovery/composer.json.
Updated component file core/lib/Drupal/Component/Plugin/composer.json.
Updated component file core/lib/Drupal/Component/Render/composer.json.
Updated component file core/lib/Drupal/Component/FrontMatter/composer.json.
Updated component file core/lib/Drupal/Component/Annotation/composer.json.
Updated component file core/lib/Drupal/Component/Diff/composer.json.
Updated component file core/lib/Drupal/Component/FileSystem/composer.json.
Updated component file core/lib/Drupal/Component/Assertion/composer.json.
Updated component file core/lib/Drupal/Component/ClassFinder/composer.json.
Updated component file core/lib/Drupal/Component/FileCache/composer.json.
Updated component file core/lib/Drupal/Component/Version/composer.json.
Updated component file core/lib/Drupal/Component/Gettext/composer.json.
Updated component file core/lib/Drupal/Component/ProxyBuilder/composer.json.
Updated component file core/lib/Drupal/Component/PhpStorage/composer.json.
Updated component file core/lib/Drupal/Component/Uuid/composer.json.
Updated component file core/lib/Drupal/Component/DependencyInjection/composer.json.
Updated component file core/lib/Drupal/Component/FileSecurity/composer.json.
Updated component file core/lib/Drupal/Component/HttpFoundation/composer.json.
Updated component file core/lib/Drupal/Component/Utility/composer.json.
If you make a patch, ensure that the files above are included.

Adding 'needs change record,' if for no other reason than the composer.json files generated here tell the user that they're automatically generated, and we need a document to reference.

FILE: ...s/Drupal/BuildTests/Composer/Component/ComponentValidateTest.php
----------------------------------------------------------------------
FOUND 1 ERROR AFFECTING 1 LINE
----------------------------------------------------------------------
 24 | ERROR | [x] The static declaration must come after the
    |       |     visibility declaration

No one but myself to blame, I suppose. :-) One line interdiff omitted.

I really need to learn to do the MR style contribution with d.o.

Updated IS with proposed change record info.

This script will apply the following changes to components' composer.json file during composer update:

• Adds a extra._readme section which explains that the file was generated.
• Always adds a PHP constraint.
• Checks drupal/drupal's composer.lock file for any specific constraint.
• Checks drupal/core's composer.json file for any loose constraint, preferring it over the lock file constraint.
• For stable releases, a core version constraint of ^major.minor is used, such as ^9.5 for other component dependencies.
• For non-stable releases, the core version constraint tracks the tag release name used in \Drupal::VERSION, following the example of the metapackage generator.
• And... Because only root packages honor minimum-stability, this key is not added to components. The exception is any component which requires another component, in a non-stable release. This allows us to test that the components can require each other in non-stable releases.

We also have two tests.

One crawls through the components and does composer install for each one, in order to make sure it works.

The other ensures that the version set in \Drupal::VERSION is the same version that ends up in the component dependencies. Similar to the metapackage generator, this supports the tagged deploy script at https://github.com/xjm/drupal_core_release/blob/main/tag.sh

To see it in action, check out the issue fork and issue these commands:

git checkout 9.5.x core/lib/Drupal/Component
COMPOSER_ROOT_VERSION=9.5.x-dev composer update 'drupal/core*'

Still to do:

• Decide this is the thing to do.
• Write the change record.
• Put the link to the change record in the extra._readme section.

One more usage of finder but still as -dev dependency

Hiding patches as MR used, overall looks great 👍

Adding as @xjm.is the best person to deal with tagging process

Hmm. I was curious if this would break when we don't install dev, after the comment above.

So it seems that we can break the process with this:

rm -rf vendor/
composer install
COMPOSER_ROOT_VERSION=9.5.x-dev composer update --no-dev 'drupal/core*'

Note the --no-dev.

So this exists because drupal/drupal says this:

    "autoload": {
        "psr-4": {
            "Drupal\\Core\\Composer\\": "core/lib/Drupal/Core/Composer"
        }
    },
    "autoload-dev": {
        "psr-4": {
            "Drupal\\Composer\\": "composer"
        }
    },

And during the no-dev update, we see errors like this:

Class Drupal\Composer\Composer is not autoloadable, can not call post-update-cmd script
Class Drupal\Composer\Composer is not autoloadable, can not call post-update-cmd script

This comes from #3076234: Relocate Scaffold plugin outside of 'core' directory to try and isolate the scaffold plugin.

What's odd is if I merge the autoload-dev into autoload and regenerate the autoloader, I can use the component generator with the proper outcome, even though symfony/finder is not present. Nothing else depends on it.

Anyway, overall, we might decide this matters or we might decide it does not. If it matters, we can merge autoload-dev into autoload, or some other solution, even in a follow-up.

Adding needs followup for @xjm's comment above and #20 as well.

Adding a whole component just for one regex seems like a lot, so maybe that's a followup too, to find other tricky semver use-cases and consolidate them.

Follow-up #1: #3280399: Mark drupal/core-bridge 9.5.x as abandoned

It turns out that the Laminas deprecation mentioned by @xjm supports a class which is deprecated for removal in D10 anyway, and it's the only class in the drupal/core-bridge component.

Follow-up #2: #3280402: Consolidate SemVer-Related Functions To One Place

Updated the MR so that if a component has a require that's not in the drupal/drupal lock file, it gives you a handy red line of text about it.

Like this:

cd core/lib/Drupal/Component/Utility
composer require abc/api-problem
[ stuff happens ]
cd [back to drupal root]
% COMPOSER_ROOT_VERSION=9.5.x-dev composer update 'drupal/core*'
[ more stuff happens ]
> Drupal\Composer\Composer::generateMetapackages
> Drupal\Composer\Composer::generateComponentPackages
drupal/core-utility requires packages not present in drupal/drupal: abc/api-problem
Updated component file core/lib/Drupal/Component/Utility/composer.json.
If you make a patch, ensure that the files above are included.

Followup #3: #3280415: Metapackage Generator Breaks Under Composer --no-dev

In Slack, @xjm mentions the MR doesn't apply to 10.x, so it needs a reroll for that, probably as a patch.

Still to do: Add CR, link to CR from generated composer.json files. I don't think we need a doc page for that at this point.

Draft change record: https://www.drupal.org/node/3293830

Patches for 10.0.x and 10.1.x.

Nice work and long overdue IMHO.

The only nit I can find is (on all patches/MR):

diff --git a/core/lib/Drupal/Component/DependencyInjection/composer.json b/core/lib/Drupal/Component/DependencyInjection/composer.json
+    "support": {
+        "issues": "https://www.drupal.org/project/issues/drupal",
+        "irc": "irc://irc.freenode.net/drupal-contribute",
+        "source": "https://www.drupal.org/project/drupal/git-instructions"
+    },

diff --git a/core/lib/Drupal/Component/Gettext/composer.json b/core/lib/Drupal/Component/Gettext/composer.json
+    "support": {
+        "issues": "https://www.drupal.org/project/issues/drupal",
+        "irc": "irc://irc.freenode.net/drupal-contribute",
+        "source": "https://www.drupal.org/project/drupal/git-instructions"
+    },

diff --git a/core/lib/Drupal/Component/Transliteration/composer.json b/core/lib/Drupal/Component/Transliteration/composer.json
+    "support": {
+        "issues": "https://www.drupal.org/project/issues/drupal",
+        "irc": "irc://irc.freenode.net/drupal-contribute",
+        "source": "https://www.drupal.org/project/drupal/git-instructions"
+    },

diff --git a/core/lib/Drupal/Component/Uuid/composer.json b/core/lib/Drupal/Component/Uuid/composer.json
+    "support": {
+        "issues": "https://www.drupal.org/project/issues/drupal",
+        "irc": "irc://irc.freenode.net/drupal-contribute",
+        "source": "https://www.drupal.org/project/drupal/git-instructions"
+    },

I think the age of IRC has passed and the line "irc": "irc://irc.freenode.net/drupal-contribute", can be removed from all four occurrences.

Removes IRC links. Probably fair. :-)

Patches attached.

Thanks @Mile23, and bonus kudos for noticing that drupal/core-bridge is heading The Way Of The Dodo.

Come to think if it, that (the marking abandoned and removing of drupal/core-bridge) might be worthy of its own, separate CR?

I didn't notice, the script did. :-) That CR is here: https://www.drupal.org/node/3258656 but it's not pertinent to this.

Outwitted by scripts and already existing documentation, story of my life... ;)

Great work @Mile23!

Added (a few?) suggestions on type-hinting, line-length and a (feeble) attempt at DRY-ness.
Besides all these nits, the functionality looks fine to me.

Here I have tried to resolved the threads of MR, please review.

Thanks @ravi.shankar, but the changes should also be made in the 10.0.x and 10.1.x patches

Updated the MR branch with current 9.5.x, generated patches for 10.0.x and 10.1.x.

Let's see what Core Committers and Framework managers think.

I created a related issue that is targetting 9.4.x-dev specifically, because this bug prevents me to update some sites to Drupal 9.4.4 now.
#3301254: Not be able to update to Drupal 9.4.4 because some Drupal 9 components require Drupal 8 components

Untagging for needs framework manager review because I just reviewed it. I feel a bit like we've created lots of problems for ourselves without any benefit for ourselves (at least not yet) by having components set up the way they are in the first place, but this patch overall improves things.

Committed/pushed the respective patches to 10.1.x and 10.0.x, but this needs a re-roll for 9.5.x

9.5 looks ready

Thank you, committed/pushed the 9.5 MR too.