Problem/Motivation

In composer/composer#8882, @Seldaek says:

Now obviously I can't really prove this right now, but technically this is not a bug, prefer-stable is just that, a preference. It still came up with a valid solution and that's what matters most.

Despite the disclaimer here, it is clear that the sentiment of the Composer project that prefer-stable is only a preference, and cannot be relied upon to produce a stable release. Even if one of the release candidates is stable, Composer's solution might include a non-stable version of that release; if all of the other constraints have been correctly satisfied, then the result is considered to be correct.

This means that it is unreliable to rely on "prefer-stable" as a substitute for "minimum stability: stable".

Proposed resolution

Drupal should change the drupal/recommended-project and drupal/legacy-project to use "minimum stability: stable" instead of "minimum stability: dev", so that all new sites created from the templates will use the safer setting.

The "stable" setting should, of course, only be used with stable releases of the template projects. When using drupal/core-recommended, it is necessary to use a minimum stability setting that matches the stability of the released version, i.e. beta for beta releases, alpha for alpha releases, and so on.

Note that nothing can be done for existing composer-managed sites, other than provide documentation and encourage site owners to manually adjust their top-level composer.json files, as there is no update process that pulls new changes from the template projects. Template projects, as their name implies, are just templates used to create new projects; site owners assume responsibility for their top-level composer.json files.

Existing sites that are not yet Composer-managed, on the other hand, do get a new copy of their top-level composer.json file when they run drush pm-update, or update via autoupdates, so the sooner we adjust this preference, the more sites we will protect once they finally do move over to being Composer-managed.

Site owners can still use unstable modules and other unstable dependencies in their projects with the "minimum stability: stable" setting. The only requirement to do this is that all unstable requirements must be explicitly declared in the project's top-level composer.json file using version constraints that allow unstable versions.

Remaining tasks

Review and merge

Follow-on tasks

  • Update drupal.org documentation with instructions on how to require modules that do not yet have a stable release.

User interface changes

None

API changes

None

Data model changes

None

Release notes snippet

In Drupal 8.8 though 9.0, Drupal's Composer project templates declared a minimum stability of dev with an additional prefer-stable setting, which allowed any development versions of other packages such as modules to be installed with the template.

However, this occasionally produced unexpected results, such as upgrading to newer unstable releases rather than remaining on a current stable release (which could be dangerous for a production website). Therefore, starting with 9.1.0-alpha1, the minimum stability will match the stability of core itself. For example, the 9.1.0-alpha1 templates will have alpha as the minimum stability, and 9.1.0 will have stable.

The core change will not affect existing projects using the previous templates, so we strongly recommend that existing Composer sites also change their own templates to raise the minimum stability. In most cases stable is recommended for production websites. For more information on updating your templates, as well as instructions for allowing individual pre-release modules to be used with your stable templates, review the change record: Drupal Project templates no longer use minimum-stability "dev".

Comments

greg.1.anderson created an issue. See original summary.

geerlingguy’s picture

I think it would be good to have ‘stable’ as a default even if it causes some people to bump into issues installing dev/alpha release modules. Maybe it would help push module authors to release stable versions of modules (even with known issues) rather than publish an alpha for 6 years :-)

alexpott’s picture

Priority: Normal » Critical

So after yet more fun in #3134648: [backport, needs scheduling] Don't pin the composer/installers version in drupal/core-recommended I think this is a critical issue. I think the default expectation for core's composer project templates should be to work with stable code. Yes our eco-system is littered with beta and rc modules but this is not good practice and we're not encouraging proper releases with this default.

Even more we're getting into situations when due this we're allowing composer update to update stable releases to unstable dev releases and that's bad.

alexpott’s picture

One idea here would be to read the root composer.json and put a warning on system/reports if the minimum stability is not stable. Otherwise it's going to be hard to convert the world. Another idea is to put a check into a post-update-cmd or drupal-scaffold - I dunno.

alexpott’s picture

I've been playing around with the Drupal 9 branch and the minimum stability flag on Drupal 9 and doing that gives me some thoughts.

I think the root composer.json should have the minimum-stability and prefer-stable flags removed. We should only include dev dependencies here and if we choose to depend on a dev version of something it should be explicit. This composer.json is for core development only.

I think the core/composer.json should have the minimum-stability and prefer-stable flags removed. This is not a root composer.json and these flags have no meaning.

The two project templates in composer/Template/LegacyProject and composer/Template/RecommendedProject are more interesting. If you do composer create-project drupal/recommended-project:9.0.x-dev my-project you get a project built on 9.0.x as expected. But it you do composer update you see these "interesting" set of updates

  - Updating drupal/core-composer-scaffold (9.0.x-dev 03a666b) to drupal/core-composer-scaffold (9.0.0-beta2)
  - Updating drupal/core-project-message (9.0.x-dev 571ad1a) to drupal/core-project-message (9.0.0-beta2)
  - Updating drupal/core (9.0.x-dev 3b298d8) to drupal/core (9.1.x-dev 17c7d96)
  - Updating drupal/core-recommended (9.0.x-dev e7f8caf) to drupal/core-recommended (9.1.x-dev 47e3ca1)
  - Updating drupal/coder (8.3.8) to drupal/coder (8.3.9)
  - Updating composer/composer (1.10.5) to composer/composer (1.10.6)
  - Updating drupal/core-dev (9.0.x-dev 34ab35e) to drupal/core-dev (9.0.0-beta2)

So drupal/core-dev, drupal/core-composer-scaffold, drupal/core-project-message effectively get a downgrade! drupal/core and drupal/core-recommended are upgraded! Fun!

If at this point you remove the minimum-stability and prefer-stable flags you get

Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for drupal/core-composer-scaffold ^9 -> satisfiable by drupal/core-composer-scaffold[9.0.x-dev].
    - Removal request for drupal/core-composer-scaffold == 9.0.9999999.9999999-dev
  Problem 2
    - Installation request for drupal/core-project-message ^9 -> satisfiable by drupal/core-project-message[9.0.x-dev].
    - Removal request for drupal/core-project-message == 9.0.9999999.9999999-dev
  Problem 3
    - Installation request for drupal/core-recommended ^9 -> satisfiable by drupal/core-recommended[9.0.x-dev].
    - Removal request for drupal/core-recommended == 9.0.9999999.9999999-dev
  Problem 4
    - Installation request for drupal/core-dev ^9 -> satisfiable by drupal/core-dev[9.0.x-dev].
    - Removal request for drupal/core-dev == 9.0.9999999.9999999-dev

So... if you then restart the whole process and do composer create-project drupal/recommended-project:9.0.0-beta2 my-project you get the beta2 as expected.
Running composer update in the beta2 project does (amongest many other updates:

  - Updating drupal/core (9.0.0-beta2) to drupal/core (9.1.x-dev 17c7d96)
  - Updating drupal/core-recommended (9.0.0-beta2) to drupal/core-recommended (9.1.x-dev 47e3ca1)

if you change the minimum-stabiilty flag to "beta" then it results in the expected no updates.

I think I think (definitely not 100% sure) that we should only really support the templates from release candidate on. So potentially I think the flags in these composer.json files should be:

    "minimum-stability": "rc",
    "prefer-stable": true,

But I think making that change will break tests.

alexpott’s picture

Note other projects which build out a recommended project - like symfony - eg https://symfony.com/doc/current/setup.html - the composer.json created does not have minimum-stability set... so maybe

    "minimum-stability": "rc",
    "prefer-stable": true,

is wrong. And we should remove them too - and only support creating stable projects. Which does make sense when written like that.

alexpott’s picture

Status: Active » Needs review
StatusFileSize
new2.87 KB

Here's a patch the removes all the prefer-stable and minimum-stability to fallback to the composer default of stable.

greg.1.anderson’s picture

I agree with #5 vis-a-vis how to handle the drupal/drupal and drupal/core composer.json files.

It is a known / intended situation that "composer create-project" followed by "composer update" is not idempotent. However, I agree that it is a little odd for "composer update" to downgrade you immediately after running "create-project".

In order to decide what to do here, we need to acknowledge that a template project isn't the same as a dependency, and does not act in the same way. This has pretty much been established. We also need to define what a template project is, and how it is supposed to behave. I don't think that this has really been done. In the community project, drupal-composer/drupal-project, there are no versions. The only version that matters is HEAD of the branch (e.g. 8.x). This template is designed to give you the starting point for a stable project based on whatever state the template happens to be in at the time. It's not really an option for Drupal to not have any tags on our template projects, but I think that we can still follow the example of the community project a little bit.

I propose that we define the template projects being starting state for a stable project, regardless of the source (tag / branch) of the template project. If you start with a template project from a dev branch or an RC tag, what you are getting is a template project for a stable project, based on the contents of the template project that will be in the next stable release.

This isn't what we do today, though. If you start with a dev release of a template project, you get drupal/core-dev, and if you start with a stable release, you do not.

I'm not sure what is best, but I tend to think that we should always ship with minimum stability: stable unless we also require the dev version of drupal/core-recommended in dev releases of the template project.

To summarize, this leaves us with two choices:

1. Remove drupal/core-dev from dev releases of the template projects, and make minimum stability stable everywhere.

2. In dev versions of the template files, keep drupal/core-dev, keep minimum stability dev, and require drupal/core-recommended at its dev version. In stable releases, make minimum stability stable.

The advantage of option 1 over option 2 is that option 1 keeps the contents of the template files in stable releases the same as it is in all of the tests that are run on dev branches and every patch. If we make things vary between dev and stable releases, then we do not really have good test coverage for the stable versions of the template projects.

Option 1 is essentially #7, so +1 on #7. I would also go ahead and remove require-dev from the templates at the same time for consistency.

Status: Needs review » Needs work

The last submitted patch, 7: 3135247-7.patch, failed testing. View results

longwave’s picture

> I propose that we define the template projects being starting state for a stable project, regardless of the source (tag / branch) of the template project.

I agree with this. I think most people who are starting from a template are looking to build based on stable software and don't intend to run -dev releases or even betas/release candidates. Anyone who does want to try out less stable versions is welcome to do so, they just have to manually opt in to this by editing their composer.json after creating the project.

+1 also to the /composer.json and core/composer.json changes.

I did see that in composer/Template/README.txt we state:

Use Composer to create a new project using the desired starter template:

composer -n create-project -s dev drupal/recommended-project my-project

Should we be removing -s dev from here?

alexpott’s picture

So it's no surprise that Drupal\BuildTests\Composer\Template\ComposerProjectTemplatesTest is failing at that calls out to packagist. See #3123933: Determine whether ComposerProjectTemplatesTest is testing the internet, and if it is, avoid that for more.

greg.1.anderson’s picture

#10: Yes, I think that the -s dev became unnecessary once we had the 8.8.0 stable tag.

alexpott’s picture

Status: Needs work » Needs review
StatusFileSize
new838 bytes
new3.69 KB

@longwave good point. Fixing the ComposerIntegrationTest too. I'm not sure how to fix ComposerProjectTemplatesTest yet.

greg.1.anderson’s picture

@alexpott I will fix ComposerProjectTemplatesTest in #3123933.

Status: Needs review » Needs work

The last submitted patch, 13: 3135247-11.patch, failed testing. View results

geerlingguy’s picture

I just wanted to note that I thought I had worked around this by running composer update "drupal/*", but apparently some other changes from 8.9.x leaked into the update still; with a fresh checkout of the codebase I was getting the error:

RuntimeException: Zend Diactoros must be installed to use the DiactorosFactory.

This looks like it's coming from the change in issue #3104015: Replace ZendFramework/* dependencies with their Laminas equivalents. So I'm going to just switch to 'stable' instead of 'prefer-stable' because I can't afford to blow up my site like this :-/

(Downstream issue in my code repo here). I'll try updating to Drupal 8.8.5 again soon, and we'll see how that goes.

xjm’s picture

It's worth seeing if we can change this during beta. In addition to the changes to our metapackages themselves, we'll also need to communicate about the risks of minimum-stability: dev to site owners who used the templates from 8.8, including instructions for them on how to use that with any pre-release dependencies or modules they have with (e.g.):
composer require drupal/module_name@alpha

We might want to spawn a series of followup/related issues for all that, not sure.

geerlingguy’s picture

I just tried updating my Drupal 8.8.4 codebase to 8.8.5 again today, after setting minimum-stability to 'stable', and again when I try to composer install on the live site, I get the error:

In DiactorosFactory.php line 37:
                                                                 
  Zend Diactoros must be installed to use the DiactorosFactory. 

And that error comes up whether I try loading any page, or running any Drush command (including just bare drush).

geerlingguy’s picture

It seems like somehow the changes from #3104015: Replace ZendFramework/* dependencies with their Laminas equivalents have leaked their way into my 8.8.5 code by running a composer update :-/

geerlingguy’s picture

Apparently the issue was I had a vendor directory that still had laminas/laminas-diactoros in it. I wiped out my vendor directory and then ran composer update again, and all is working now. Sorry for the noise, but blogged about this particular composer gotcha for the benefit of anyone else who may run into a similar problem! Watch out if composer update keeps replacing a dependency.

alexpott’s picture

So I've had a look at making this change for Thunder - which includes 13 less than stable projects. As stability flags can only be set on the root composer json it means that we need to add all the unstable projects to the project template like so https://github.com/thunder/thunder-project/pull/14/files. This would be the same for any project that wishes to change their minimum-stability setting. Obviously Thunder doesn't really want to add all of the distribution dependencies to the project template. That kinda defeats the purpose of separating the project and distribution composer.json files. Our plan would be to change the minimum stability to alpha and then to work with the contribute projects to get stable releases.

Core's templates now don't rely on any unstable packages (yay!) so as @xjm has pointed out the big problem to be solved here is one of communication. If we make the change suddenly:

composer create-project drupal/recommend-project
composer require drupal/access_unpublished

will not work.

composer create-project drupal/recommend-project
composer require drupal/access_unpublished:1.0.x@alpha

will though.

alexpott’s picture

Status: Needs work » Needs review
StatusFileSize
new12.87 KB
new1.11 KB

The recent changes to core/tests/Drupal/BuildTests/Composer/Template/ComposerProjectTemplatesTest.php are not making this any easier. Locally the test fails for me with

Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - The requested package drupal/core-composer-scaffold could not be found in any version, there may be a typo in the package name.
  Problem 2
    - The requested package drupal/core-project-message could not be found in any version, there may be a typo in the package name.
  Problem 3
    - The requested package drupal/core-recommended could not be found in any version, there may be a typo in the package name.
  Problem 4
    - The requested package drupal/core-vendor-hardening could not be found in any version, there may be a typo in the package name.
  Problem 5
    - The requested package drupal/core-dev could not be found in any version, there may be a typo in the package name.

Potential causes:
 - A typo in the package name
 - The package is not available in a stable-enough version according to your minimum-stability setting
   see <https://getcomposer.org/doc/04-schema.md#minimum-stability> for more details.
 - It's a private package and you forgot to add a custom repository to find it

In order to get composer create-project to obey the minimum-stability flag you pass in you can't provide a version so I've removed that but still no dos.

Status: Needs review » Needs work

The last submitted patch, 22: 3135247-22.patch, failed testing. View results

mmjvb’s picture

In order to get composer create-project to obey the minimum-stability flag you pass in you can't provide a version so I've removed that but still no dos.

What do you mean here? There is no such thing as passing in the minimum-stability flag. The option stability on the create-project refers to the project package. Only needed when letting composer decide the package version. So, not applicable when specifying an exact version. The minimum-stability for the project you are creating doesn't even need to be in the composer.json as it is the default to go for Stable.

The create-project has two steps, first step downloads the project package, the second step executes an install. The first step uses the --stability to download the correct version of the project package assuming you let composer determine which one to choose. The second step does an install based on the configuration provided in the downloaded composer.json.

longwave’s picture

#22 also contains a bunch of changes to the installer that are seemingly not for this issue.

alexpott’s picture

Status: Needs work » Needs review

@mmjvb I was trying to point out that the command in the test in HEAD is odd because it provides both a version and a stability argument. You're totally correct though that we can't change the minimum stability in HEAD. We need to do this in the code that creates the subtree split - so I think that's https://bitbucket.org/drupalorg-infrastructure/subtree-splitter/src/master/ ... or maybe we need to adjust the tools that update things when we tag a release. Yeah I think that's it - looking at https://git.drupalcode.org/project/drupal/commit/dc8689f - this commit will need to change minimum-stability accordingly.

As far as I can see there won't be a code change to core if we make this change. It's going to be all tooling and communications.

greg.1.anderson’s picture

Assigned: Unassigned » greg.1.anderson

I'm going to work on testing #22 now. I'll remove the installer-related diffs and post a new patch when I'm done.

greg.1.anderson’s picture

StatusFileSize
new6.47 KB
new9.82 KB

So, this is a tricky one; it is almost untestable. Almost.

As #24 pointed out, the flags passed in to the create-project command only apply to the selection of the project template itself. The minimum stability used during the composer install step are taken from the composer.json contained inside the selected project template. If we want to use a minimum-stability of stable in the project templates, then we'd have to name all of the unstable versions we're willing to accept in the root-level composer.json file (which is to say, the project template's composer.json file). Rather than doing that, I instead made the create-project's composer install work by converting the core version from the SUT (e.g. 9.1.x-dev) into a "simulated stable" version, e.g. 9.1.99, and passing that "phantom" version as COMPOSER_ROOT_VERSION. This has the effect of making all of the path repositories to appear to be stable when Composer resolves them. Nifty. Unfortunately, we still run up against the problem that dev versions of drupal/core-recommended always require a dev version of drupal/core. When we run our tests, our drupal/core-recommended metaproject is always a dev version.

To get around this, I fix up the drupal/core-recommended metaproject (our test copy of it, that is) to require a stable version of drupal/core rather than an unstable version. Since I know that drupal/core exists at our phantom version (since it's a path repository in our SUT), I just use that in place of the dev version that's in the metapackage to begin with, i.e. drupal/core-recommended ends up requiring drupal/core:^9.1.99 (or similar, for other branches).

This actually works quite well, and the test passes. The big question, though, is: is this a valid test? The answer is ALMOST.

First, let's consider the 8.8.x branch (assuming for the purpose of illustration that this patch is backported and already committed there). The test works as described above, with drupal/core-recommended requiring drupal/core:^8.8.99. Of course, once the patch being tested is committed to the 8.8.x-dev branch, there is no version 8.8.99 of drupal/core, and drupal/core-recommended:8.8.x-dev dependes on drupal/core:8.8.x-dev. However, there is no way to get drupal/core-recommended:8.8.x-dev from drupal/recommended-project or drupal/legacy-project, as both of these templates require a stable version of drupal/recommended-project. So, in production, when you composer create-project drupal/recommended-project, you get drupal/core-recommended version 8.8.5, and it in turn requires drupal/core 8.8.5. This works fine with minimum-stability stable, the test passes (as previously mentioned), and we are guarded against any unstable dependencies slipping into the build. In this scenario, this is a great test. The 8.9.x branch works the same way; you just get 8.8.5 stable when you start with drupal/recommended-project:^8.9.x-dev, is all.

What about the other branches, though? 9.0.x & c. do not have any stable releases yet. If we committed this patch, then composer create-project drupal/recommended-project:8.9.x-dev would depend on drupal/core-recommended:^9, as it currently does. However, if we bump minimum stability up to "stable", this becomes a problem, because there are no versions of drupal/core-recommended that match ^9 stable. Until the 9.0.0 stable release comes out, then, we need to either retain "minimum stability dev" in the template projects (and remember to change them back to stable again just before the stable release), or suffer a period of time where the metapackages don't work. (We could provide instructions on how you could potentially use composer create-project with --no-install and then bump the minimum stability back to "dev" with a "composer config" command. That would allow an explicit "composer install" to work.)

I think that moving the default minimum stability back to "stable" is important enough that we should decide which compromise we want to make on the 9.x branches. That's a decision that will have to be made before we move forward here, though.

Note: I removed the parts of #22 that should not have been included, but for clarity I left that removal out of the interdiff.

alexpott’s picture

@greg.1.anderson nice work and sleuthing.

I think though that what's in the template should reflect the \Drupal::version. So if we're on 9.1.x then minimum stability should be "dev", if we're on 9.0.0-beta3 it should be "beta" and if we're on 9.0.0 it should be "stable". So whatever process makes all the changes as per https://git.drupalcode.org/project/drupal/commit/dc8689f should be responsible for this. Also it'd probably be good to bring the part of that process that updates composer.json's into core so it can be tested and supported.

greg.1.anderson’s picture

Status: Needs review » Needs work

#29: Yeah, I agree.

Bringing it back into core is a bit tricky, because "beta" / "stable" does not exist until after the tag is created in git.

I think our strategy should be to keep "minimum stability: dev" in git, per what is currently committed without this patch. Then, we need to make a script in core that converts the minimum stability in the template projects to "beta" or "stable". We run it in the tests, and then test it with the test in this patch, so we would end up with two versions of this test. I'm not sure if it is sufficient to test with just "dev" and "stable", or if we need to test any of "alpha" / "beta" / "rc". In any event, though, the subtree splitter should call the same script we use in the tests to adjust minimum stability whenever it is processing a tag.

Does that sound about right?

alexpott’s picture

@greg.1.anderson yep it does sound about right. I'm very much in favour of adding the "change composer.jsons in core for a release script" to core.

geerlingguy’s picture

+1 to @greg.1.anderson's idea.

greg.1.anderson’s picture

Assigned: greg.1.anderson » Unassigned

Meant to un-assign after #28. Although I still intend to come back to this in the future, I don't want to dissuade anyone from making a patch if they feel so inclined.

greg.1.anderson’s picture

I created drupalorg-infrastructure/subtree-splitter#4 to show what would be necessary on the infrastructure side to go with this patch. Due to the way the existing subtree splitter is organized, my idea to use the same script in the tests as in the infrastructure code to set the minimum stability is not really feasible, since the infrastructure fixup happens on a project-by-project basis during the subtree split.

The modification is relatively trivial, though, so perhaps it would still be sufficient for the test and the tool to merely be in alignment, even if they are not running exactly the same code.

I'll update the tests here per #30 shortly.

alexpott’s picture

@greg.1.anderson but if we did this as part of the release tagging process then the subtree splitter would not have to do a thing. Right? And then the release tags have the correct composer.json files and not only the subtree splits. I think this should be do in core.

I've practiced what we're saying on a client project that uses the drupal/core-recommended package. Removing the minimum-stability flag was painless. Composer had already done the right thing and added stability flags to the dev dependencies, eg. "drupal/encrypt": "^3.0@RC",, so changing the flag broke nothing. Obviously there's a little bit of luck that none of my dependencies dependencies where not stable.

On the subject of dependencies dependencies, for example the Thunder project is declared stable whilst it relies on alpha code, it's been said that it is an opinion that Thunder is not actually stable. This opinion does not really translate to the composer world. You are only as stable as your dependencies. At this point you have two options. Either to set your minimum-stability to alpha (Thunders are all alpha or above) or manually include each dependency with the correct stability flag.

greg.1.anderson’s picture

OK #35 sounds like a good idea. Back to #30 then.

greg.1.anderson’s picture

This test is difficult to modify.

greg.1.anderson’s picture

Issue summary: View changes
Status: Needs work » Needs review
StatusFileSize
new12.2 KB
new15.41 KB

Here's an updated version of the tests that keep the dev branch on a dev version of Composer, with minimum-stability also set to dev, and that also provides a handy Composer script to set the Drupal version, e.g. to be run by a Release Manager in preparation for a release. The set-drupal-version script also sets the minimum-stability setting of the template projects to match the Drupal version (e.g. "stable" for stable releases, "alpha" for alpha releases, and so on). The template project tests use the set-drupal-version script, so it is known in advance whether it is possible to set a stable Drupal version and subsequently use composer create-project to successfully make a new Composer-based site from the templates.

The problem I obliquely referred to in #37 is that Composer will no allow a path repository to refer to a branch-style (i.e. "dev") version; path repositories must refer to specific tag-style (i.e. stable, rc, beta, etc.) version. I would have to switch this test to use VCS repositories to test dev versions. That would be a lot slower, so I left it at testing only the (simulated) stable release version.

Status: Needs review » Needs work

The last submitted patch, 38: 3135247-38.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

greg.1.anderson’s picture

Status: Needs work » Needs review
StatusFileSize
new1.64 KB
new15.19 KB

Minor test failure corrections.

greg.1.anderson’s picture

StatusFileSize
new1.31 KB
new14.76 KB

Make 'Composer::getDrupalRoot' protected to remove temptation.

cilefen’s picture

mile23’s picture

Issue tags: +Needs reroll
ridhimaabrol24’s picture

StatusFileSize
new14.77 KB

Rerolling patch for the latest head! Please review!

greg.1.anderson’s picture

I am still +1 on this patch. I think we need feedback from a Release Manager to move forward.

xjm’s picture

It still needs a release note, which would help move it forward.

+++ b/core/tests/Drupal/BuildTests/Composer/Template/ComposerProjectTemplatesTest.php
@@ -267,7 +282,10 @@ protected function makeVendorPackage($repository_path) {
+      // Also skip the projects that are symlinked in vendor. These are in our
+      // metapackage. They will be represented as path repositories in the test

Nit: Metapackages should probably be plural here?

xjm’s picture

Also a CR.

tedbow’s picture

Status: Needs review » Needs work

Nits and questions. Looking good!

  1. +++ b/composer/Composer.php
    @@ -21,10 +24,97 @@ class Composer {
    +    if (empty($args)) {
    +      throw new Exception('Must provide Semver version as an argument.');
    +    }
    +    $version = array_shift($args);
    

    Should we also just to be careful check that there is only 1 argument passed.

  2. +++ b/composer/Composer.php
    @@ -21,10 +24,97 @@ class Composer {
    +  /**
    +   * As part of the Composer Project Template test, set the version of Drupal.
    

    Since this is called from \Drupal\Composer\Composer::setDrupalVersionScript() it is not just for the test. It is for the release process too, correct?

  3. +++ b/composer/Composer.php
    @@ -21,10 +24,97 @@ class Composer {
    +   * @param string $version
    +   *   Semver version to set Drupal's version to.
    +   */
    ...
    +    // Update the template project stability to match the version we set.
    +    return static::setTemplateProjectStability($root, $version);
    

    Need a @return here.

  4. +++ b/composer/Composer.php
    @@ -21,10 +24,97 @@ class Composer {
    +    // satisfies '*' throws an exception if $version is not valid.
    +    Semver::satisfies($version, '*');
    

    This comment doesn't seem clear that it is referring to the Semver::satisfies() method. At first I read it just as the comment line needed to be capitalized. We could fix the comment to be more clear but since \Composer\Semver\Semver::satisfies() actually relies on \Composer\Semver\VersionParser::normalize() to throw the exception for an invalid version why don't we call it directly:
    (new VersionParser())->normalize($version);
    Then we don't have to explain why we are using '*' for the constraint.

  5. +++ b/composer/Composer.php
    @@ -49,4 +139,19 @@ public static function drupalVersionBranch() {
    diff --git a/composer/Template/README.txt b/composer/Template/README.txt
    
    diff --git a/composer/Template/README.txt b/composer/Template/README.txt
    index 47c95f1c5c..31c316c19b 100644
    
    index 47c95f1c5c..31c316c19b 100644
    --- a/composer/Template/README.txt
    
    --- a/composer/Template/README.txt
    +++ b/composer/Template/README.txt
    

    Should we make a follow-up to update this readme to include a note about requiring non-stable modules and themes? Or is this just standard composer workflow so no need to have our own documentation?

  6. +++ b/core/tests/Drupal/BuildTests/Framework/BuildTestBase.php
    @@ -7,6 +7,7 @@
    +use Drupal\Composer\Composer;
    

    Unneeded use statement.

  7. +++ b/core/tests/Drupal/BuildTests/Framework/BuildTestBase.php
    @@ -162,6 +163,11 @@ protected function setUp() {
    +    // Preserve the test site if environment variable is set.
    +    if (getenv('DRUPAL_BUILD_TEST_PRESERVE_TEST_SITE')) {
    +      $this->destroyBuild = FALSE;
    +    }
    

    It this just to make the test run faster? Is this an out of scope change?

    Also since this is in a test base class is there any concern with this change in non-core tests?

    Just checking

greg.1.anderson’s picture

Issue summary: View changes
Status: Needs work » Needs review
StatusFileSize
new14.16 KB
new2.81 KB

1. Added an argument count.

2. Correct; adjusted the comment for clarity.

3. Added as recommended.

4. Improved as suggested.

5. Yes, this is a standard Composer workflow; however, I think that it is perhaps not well-understood, so some documentation on how to add pre-release modules would be beneficial. I don't think that these README files are the place to do it, though; perhaps someplace in the instructions on how to add modules on drupal.org would be better.

6. & 7.: Removed unrelated (but useful) feature.

I added a release not snippet to the issue summary. I will add a change record next.

greg.1.anderson’s picture

StatusFileSize
new1.03 KB
new14.18 KB
new1.25 KB

Whoops, forgot to test a final simplification and missed a typo. Here's a correction.

Also attached is sample output from running composer set-drupal-version 9.1.0-alpha1.

greg.1.anderson’s picture

Issue summary: View changes
jwilson3’s picture

I don't understand the purpose of the introduced change on #50.
use Composer\Semver\VersionParser; is already there, so why add use Composer\Semver\VersionParser as SemverParser;? If there is a sound reason for this, it should be at the very least documented somehow — ie, in a code comment — why it is needed.

jwilson3’s picture

Status: Needs review » Needs work

Greg said it was an accident and he'd fix it (via Slack)

greg.1.anderson’s picture

Status: Needs work » Needs review
StatusFileSize
new14.13 KB
new1.03 KB

Whoops ^ 2, I made a typo and thought the problem was caused by two VersionParsers, but they are the same VersionParser. Fixed in this patch.

tedbow’s picture

Status: Needs review » Needs work

All the points in my review in #48 were addressed by @greg.1.anderson in #49 thanks!

a couple more small things

  1. +++ b/composer/Composer.php
    @@ -5,7 +5,10 @@
    +use Composer\Semver\Semver;
    

    Not used anymore

  2. +++ b/composer/Composer.php
    @@ -21,10 +24,104 @@ class Composer {
    +    $args = $event->getArguments();
    +    if (empty($args) || (count($args) > 1)) {
    

    won't mention but since there is at least 1 changed needed.

    Is args guaranteed to an array? If so this could just be
    if (count($args) !== 1)
    but since \Composer\EventDispatcher\Event::getArguments() doesn't use a return type declaration the current code is probably safer.

    so I guess no change

  3. +++ b/composer/Composer.php
    @@ -21,10 +24,104 @@ class Composer {
    +  protected static function getDrupalRoot() {
    ...
    +  public static function getDrupalVersionScript(Event $event) {
    ...
    +  public static function setDrupalVersionScript(Event $event) {
    

    Speaking of return types. Should we add return type declarations to all the new functions add in this class?

greg.1.anderson’s picture

Status: Needs work » Needs review
StatusFileSize
new14.1 KB
new419 bytes

1. Removed.

2. Yeah I can think of several variants that might be better than what's in the patch right now, but since it's unclear to me whether any are necessarily improvements, I left this alone.

3. I wasn't sure where to start and where to stop in terms of type hinting. There's no need to differentiate between new and existing methods in the \Drupal\Composer\Composer class, as this class is never overridden, so we *could* add typehints to all methods. Since this is a critical issue, though, and typehinting is not critical, I thought it was better to leave well enough alone for now, and handle typehints as follow-on work so that we can get this committed for 9.1.0-alpha1 (:crossed-fingers:)

tedbow’s picture

#56 seems good to me as far as not changing anything for 2,3
1 fixed.

Looks good to me

greg.1.anderson’s picture

Has change record and release note snippet; forgot to update tags earlier.

tedbow’s picture

Status: Needs review » Needs work

hmm so I ran the script.
`composer set-drupal-version 9.0.3-alpha1`
It works as expected but I am wondering if we should actually the change the message

Updated Drupal version to 9.0.3-alpha1 (alpha release)

if I was Drupal user and just looked at scripts in my composer.json and saw set-drupal-version this could be confusing. What does it do? Is how I update Drupal?

Should we add something in the conformation message that says this is only for the release process?
Or even a prompt that says something like “this is a script intended for drupal.org release process. are you are sure…..?”
Or even a required --set-release-version option that you would have to actually read the function docs to know about. So nobody would run the accidentally.
changes to composer/Template won’t be bad for site but changing \Drupal::VERSION🙀

@greg.1.anderson suggested maybe we don't need this under scripts at all since the test calls it directly the release process could use another method to call it.

greg.1.anderson’s picture

Status: Needs work » Needs review
StatusFileSize
new13.41 KB
new1.69 KB

Yeah, the Composer script and the method \Drupal\Composer\setDrupalVersionScript() are only called by the release process. Since end users shouldn't call this, and it could be confusing to have a prominent top-level script that (almost) no one is supposed to use, we should just take it out.

During the release process:

php -r 'include "vendor/autoload.php"; \Drupal\Composer\Composer::setDrupalVersion(".", "9.1.0-alpha1");'

or something similar will do the trick.

The change record and release note snippet did not mention this method anyway, so I did not change them.

tedbow’s picture

Status: Needs review » Reviewed & tested by the community
  1. #60 looks a good solution the problem in #59.

    I tried
    php -r 'include "vendor/autoload.php"; \Drupal\Composer\Composer::setDrupalVersion(".", "9.1.0-alpha1");'
    and this produced the same changes as the composer script.

  2. There is still the Needs documentation updates tag

    Should make an issue in https://www.drupal.org/project/user_guide to add a explanation in 3.5. Using Composer to Download and Update Files
    ?

  3. We still haveNeeds release manager review but perhaps @xjm or @catch will be the committer to look at this?
  4. I think the patch looks good. RTBC 🎉
Mixologic’s picture

if I was Drupal user and just looked at scripts in my composer.json and saw set-drupal-version this could be confusing. What does it do? Is how I update Drupal?

Should we add something in the conformation message that says this is only for the release process?

Worth noting that scripts in /composer.json are *only for core development* and should only be seen by users getting drupal via a git clone. All other methods at this point will not have that script.

Normal drupal users should never see that...

Mixologic’s picture

So I think #56 is okay, and @tedbow mentioned in slack

yeah I guess I am fine 56 also in that case

(because we already warn people that the git clone is not the product: https://git.drupalcode.org/project/drupal/-/blob/9.0.x/composer.json#L61-72)

greg.1.anderson’s picture

The php -r thing is a little icky, but it only has to go in the release script one time. I think we could go with either #56 or #60, at the release manager's discretion. composer set-drupal-version is a little easier to read. :)

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
  1. +++ b/composer.json
    @@ -97,6 +97,7 @@
    +        "drupal-version": "Drupal\\Composer\\Composer::getDrupalVersionScript",
    

    This is an interesting command to add given composer already has composer show. Why have we added it?

  2. +++ b/composer/Composer.php
    @@ -21,10 +23,86 @@ class Composer {
    -    $generator->generate($event->getIO(), getcwd());
    +    $generator->generate($event->getIO(), static::getDrupalRoot());
    ...
    +  /**
    +   * Get the root path of this Drupal codebase.
    +   *
    +   * @return string
    +   *   The full path to the root of this Drupal codebase.
    +   */
    +  protected static function getDrupalRoot() {
    +    return realpath(dirname(__DIR__));
    +  }
    

    While this change looks sensible - is it actually required? The reason I ask is that DRUPAL_ROOT and discovering using dirname() breaks if the composer directory is a symlink. There are issues about this for core.

  3. +++ b/composer/Composer.php
    @@ -21,10 +23,86 @@ class Composer {
    +  public static function setDrupalVersion($root, $version) {
    ...
    +  protected static function setTemplateProjectStability($root, $version) {
    

    I think given the fact this code is new we should be using scalar typehints and return typehints.

  4. +++ b/composer/Composer.php
    @@ -21,10 +23,86 @@ class Composer {
    +    $drupal_static_source = preg_replace('#const VERSION = [^;]*#', "const VERSION = '$version'", $drupal_static_source);
    +    file_put_contents($drupal_static_path, $drupal_static_source);
    

    There doesn't appear to be a test that tests this has actually occurred explicitly.

  5. +++ b/composer/Composer.php
    @@ -21,10 +23,86 @@ class Composer {
    +    // Update the template project stability to match the version we set.
    +    return static::setTemplateProjectStability($root, $version);
    

    It's pretty surprising this returns the stability. It seems a shortcut to help tests but I think the test could call VersionParser::parseStability($version); on the version they've just passed in. Also in the test that we have so far stability is always going to be stable right?

  6. +++ b/composer/Composer.php
    @@ -21,10 +23,86 @@ class Composer {
    +      exec("composer --working-dir=$dir config minimum-stability $stability", $output, $status);
    

    There doesn't appear to be a test that tests this has actually occurred explicitly.

  7. +++ b/core/tests/Drupal/BuildTests/Composer/Template/ComposerProjectTemplatesTest.php
    @@ -126,14 +121,24 @@ public function testTemplateCreateProject($project, $package_dir, $docroot_dir)
    +    // Set the Drupal version and minimum stability of the template projects
    +    $stability = Composer::setDrupalVersion($this->getWorkspaceDirectory(), $simulated_stable_version);
    

    I think we should test the explicit affects that we think this command should have here.

greg.1.anderson’s picture

Status: Needs work » Needs review
StatusFileSize
new13.56 KB
new14.54 KB

1. `composer show` reports the version installed as Composer recognizes it (based on tags), whereas `composer drupal-version` reads the version string in the php code. The distinction between these is typically irrelevant, except during the release process. Per #59, we do not need to add scripts only needed during the release process to the root-level composer.json, where they are available to all core developers. (Although not to sites installed via a download archive or composer create-project.)

During the release process, the following will run the equivalent to the `composer drupal-version` script:

php -r 'include "include "core/lib/Drupal.php"; print \Drupal::VERSION;'

I removed the Composer script.

2. dirname(__DIR__) is better than getcwd() insofar as it eliminates the requirement to have your working directory at the project root. However, for the sake of being conservative, I went back to getcwd() in case symlinks are in use.

3. It's not possible to override the Composer class (it's not even packaged with Drupal, as mentioned above), so I went ahead and added scalar typehints and return types to all of the methods in the class, for consistency.

4. I changed the test to assert that the value of Drupal::VERSION (as evaluated by PHP) matches expectations.

5. Not sure why it was like this. I agree to your point that the function is basically telling the caller about the parameters it just used, which is not very useful. I took it out. Since we are not taking the time to run nearly-identical `composer create-project` commands to test --stability=alpha et.al., I just removed the --stability option from the test as well.

6. I added an assertion to the create-project test to read in the installed composer.json file and assert that it contains "minimum-stability": "stable"

7. Handled by #4.

greg.1.anderson’s picture

StatusFileSize
new8.34 KB
new14.54 KB

I created the wrong patch file in #66. Here are the corrected versions.

Mixologic’s picture

Issue summary: View changes
StatusFileSize
new43.23 KB

I updated the CR with some edits,

And I also noticed this:

+++ b/composer/Composer.php
@@ -21,17 +23,71 @@ class Composer {
+      exec("composer --working-dir=$dir config minimum-stability $stability", $output, $status);

Is there any chance / does it have much impact if 'composer' isn't in $PATH?

should this use /vendor/bin/composer? Or am I being paranoid?

I think we should open up a couple of follow ups for this as well:

1. Drupal.org has composer suggestions on the release node pages. These should be updated to reflect these suggested stability flags:

2. It would be nice to provide the end users with a message that informs them about the stability flag options when they run into trouble requiring a package:

Example: With minimum-stablility:"stable" When users run 'composer require drupal/webform:^6.0' (which only has an alpha release so far) They see:

Composer 1

./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - The requested package drupal/webform ^6.0 is satisfiable by drupal/webform[6.x-dev, 6.0.0-alpha1, 6.0.0-alpha2, 6.0.0-alpha3, 6.0.0-alpha4, 6.0.0-alpha5, 6.0.0-alpha6, 6.0.0-alpha7, 6.0.0-alpha8, 6.0.0-alpha9, 6.0.0-alpha10, 6.0.0-alpha11, 6.0.0-alpha12, 6.0.0-alpha13, 6.0.0-alpha14, 6.0.0-alpha15, 6.0.0-alpha16, 6.0.0-alpha17, 6.0.0-alpha18, 6.0.0-alpha19, 6.0.0-alpha20] but these conflict with your requirements or minimum-stability.


Installation failed, reverting ./composer.json to its original content.

Composer 2

./composer.json has been updated
Running composer update drupal/webform
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires drupal/webform ^6.0, found drupal/webform[dev-6.x, 6.0.0-alpha1, ..., 6.x-dev (alias of dev-6.x)] but it does not match your minimum-stability.

You are using a snapshot build of Composer 2, which may be the cause of the problem. Run `composer self-update --stable` and then try again. In case it solves the problem, please report an issue mentioning Composer 2.

Installation failed, reverting ./composer.json and ./composer.lock to their original content.

It might be nice if we could somehow add a message that says "You can override your minimum-stability for this package by using a stability flag like so: composer require drupal/webform:^6.0@alpha, via a plugin or other mechanism.

Mixologic’s picture

Oher than my one question above and the request for follow ups, this is RTBC to me, but I'll wait until I hear further responses.

Mixologic’s picture

Status: Needs review » Reviewed & tested by the community

Actually, Friday is the alpha1 deadline, so, yes, this is good to go afaict. If there *is* an issue with paths, its going to be insanely rare, like, a release manager is running their script inside of a stripped down docker container kinda thing, which, we can always work around.

alexpott’s picture

I agree with #70 - the code looks good, the change record is good, and the release note looks good too. Obviously we now need to make some noise so that contrib author's know that stable releases will make things easier for their users.

greg.1.anderson’s picture

@alexpott: Anything further needed before this can be committed? Release manager +1, perhaps?

I can spend some time evangelizing / documenting vis-a-vis stable releases once I'm back from vacation. That's follow-on work for this issue, though, right?

catch’s picture

This looks encouraging to me, although didn't do a line-by-line review yet, and the alpha would be the best release to give it a first run on.

catch’s picture

Removed this from the issue summary since it was outdated:

A new Composer script, set-drupal-version is provided for use by Release Managers to prepare Drupal for a release. This script also updates the minimum stability settings of the template projects as needed.

Usage:

$ composer set-drupal-version 9.0.0

This is the recommended way instead - just tried it out on a throwaway clone:

php -r 'include "vendor/autoload.php"; \Drupal\Composer\Composer::setDrupalVersion(".", "9.1.0-alpha1");' 

We need to update the following documentation to include this information after commit:

https://www.drupal.org/core/maintainers/create-core-release
https://www.drupal.org/core/maintainers/create-core-security-release

And also: https://github.com/xjm/drupal_core_release/

Tagging 'needs follow-up' for the d.o changes too.

catch’s picture

Added a PR for the release tagging script: https://github.com/xjm/drupal_core_release/pull/13

  • catch committed d72c583 on 9.1.x
    Issue #3135247 by greg.1.anderson, alexpott, ridhimaabrol24, Mixologic,...
catch’s picture

Status: Reviewed & tested by the community » Fixed
core/tests/Drupal/BuildTests/Composer/Template/ComposerProjectTemplatesTest.php:128:11 - Unknown word (fufill)

Fixed typo on commit.

core/tests/Drupal/BuildTests/Composer/Template/ComposerProjectTemplatesTest.php:312:42 - Unknown word (symlinked)

Added to dictionary on commit.

Committed d72c583 and pushed to 9.1.x. Thanks!

Still needs the documentation updates and follow-ups created - I've run out of time for this afternoon so hoping someone else can take those!

greg.1.anderson’s picture

I have a super-busy day, but will try to get to it. Would be happy if someone else manages to do it first, though! :)

drumm’s picture

xjm’s picture

Issue tags: -beta target
xjm’s picture

Issue tags: +9.1.0 release notes
xjm’s picture

Issue summary: View changes

Did some more work on the release note. It's longish, but I think it needs to be to help people understand whether they're affected.

balsama’s picture

I think some mention of how to override the minimum stability for specific packages would be useful in the release notes.

E.g.:

Projects that depend on one or more unstable dependencies can still set their minimum stability to stable by using stability flags: e.g.: package@alpha.

Mixologic’s picture

Is the release note the place for that or is it sufficient to put it all in the change record?

balsama’s picture

Good point @Mixologic. I concur that the CR is sufficient. Thanks.

Mixologic’s picture

9.1.0-alpha1 was tagged, packaged, and split : Looks good: https://github.com/drupal/recommended-project/blob/9.1.0-alpha1/composer...

xjm’s picture

We still don't quite have the secret sauce for the pre-release milestones' templates: #3179061: Further discuss use of "prefer-stable" and "minimum-stability" in core

I don't think that's a regression necessarily as it likely happened with earlier alphas too; just a "remaining not-quite-fixed" part of the issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.