Following #3135247: Composer's "prefer-stable" setting cannot be relied on to produce a stable release, we still don't quite have the right tools to make pre-release milestones of our Composer templates behave correctly.
composer create-project drupal/recommended-project:9.1.0-alpha1 d9composercd d9composercomposer updateExpected result: Nothing happens, because everything's up to date.
Actual result: Site is downgraded to 9.0.7.
@Mixologic and @greg.1.anderson are working on a couple suggestions.
Comments
Comment #4
xjmComment #5
xjmSo we might need the templates to specify their own branch (via @Mixologic):
Comment #6
mmjvb commentedSuggest to remove minimum-stability and use stability tag on drupal/core* instead. As documented : https://igor.io/2013/02/07/composer-stability-flags.html
Nobody wants alpha/beta/RC releases of extensions unless specified with stability tag in the main composer.json.
Comment #7
alexpottI think should remove the prefer-stable flag from all non-stable releases - in fact is this flag really necessary. I don't think we can use stability flags for drupal/core because if we did component-ise Drupal core then for pre-release templates all components with there stability flags would need to be in the root composer.
Comment #8
mmjvb commentedMaybe keeping them in the same repo should be reconsidered. Same thing goes for other drupal/core-* packages.
The alternative would be to allow for their own release. That is, not tying it to the release of core. As that is the real mistake that needs fixing!
Comment #9
greg.1.anderson commented#6 is not an option; minimum-stability is a root-only option.
#7: Removing prefer-stable works just fine with the composer.json structure in #5. Seems like a good idea to me; this directive is kind of unpredictable. We'll reduce our support request load if we convince folks to stop using it.
#8: The Monorepo vs micro-repos debate is out of scope for this issue. There are pros and cons to both approaches, but switching to micro-repos would have no affect on Composer's version solver as it relates to this issue.
Comment #10
mmjvb commentedIndeed a root-only option. And for drupal projects it should not be used! Not using it would result in only allowing stable releases, unless using stability tags as documented in root composer.json. That means the project templates would require to use the stability flags for the packages already there and adding one for drupal/core because drupal/core-recommended requires drupal/core. drupal/core with stability tag would be required in the root to allow alpha, beta, RC of drupal/core.
Would consider it a bad decision to use anything but stable releases in drupal projects. Please don't repeat that mistake. You would need to add those to root composer.json with stability tags to allow them.
Comment #11
xjm@mmjvb, the minimum stability of stable core releases is stable as of #3135247: Composer's "prefer-stable" setting cannot be relied on to produce a stable release. The "alpha" in my pasted code is only there because I was testing it with an alpha. Betas will set it to beta, etc.
We discussed whether to remove
prefer-stablebut I don't think it's a good idea. That makes it possible to randomly get dev versions of other packages in your project when you're using an alpha or a beta.So far I think the simple fix of 9.1 declaring a dependency on 9.1+ is the best idea, because really, Drupal 9.1 components are not compatible with
drupal/core-recommended9.0. 9.1 has new APIs. Therefore, changing the templates to use the current branch as the minimum branch makes a ton of sense to me.Comment #12
mmjvb commentedTo prove my point:
Obvioulsy removed minimum-stability and prefer-stable. Bit hard to show thing not there anymore!
Comment #13
greg.1.anderson commentedThis isn't an issue if you also remove minimum stability (let it default to "stable"). I misunderstood the suggestion in #6. #12 works well. I think that's the best option.
Comment #14
MixologicI definitely agree with #11, we *do* need to declare a sort of forwards compatibility, and that clears out any downgrading possibiilities, plus is likely more accurate.
I prefer #6/#12 as well, because if we keep minimum-stability:alpha / prefer-stable:true we end up with the weird situation we're in now where a composer update on an alpha *downgrades* you.
If we get rid of prefer stable, and keep just the minimum-stability:alpha, then we get into weird scenarios where *other* alpha versions of things could be resolved into our dependency tree (contrib modules etc).
The only other thing that concerns me is this scenario:
What if somebody starts on an alpha/beta/rc for testing purposes, then *continues to use it*.
If their composer.json has drupal/core:@alpha, and 9.2.0-alpha1 comes out, will their 9.1.0 stable site end up upgrading to those alpha versions?
I think that same thing would happen with the other scenarios as well, and I think the thing we can do to somewhat reign it in is thus:
So, what this means is that *if* a user starts on an alpha/beta/rc from a template project, presumably to test things out, then there would be a hard stop for them to where they would have to actually use `composer require` to update to 9.2.0.
This balances the need for people to test out core, but also sort of constrains that testing environment such that if people *do* end up going from there into a production site, they dont get a weird surprise later and get updated to 9.2.0-alpha1 etc... At the cost of those folks would have to have a more complex upgrade from 9.1.* -> 9.2.0 when its released.
And more complex in this case is really just a long `composer require` that has all the packages in it that are bound with the ~'s.
Now, this strategy still doesnt address #7, which is a future time when we have all the components outside of drupal/core as separate dependencies from the monorepo, at which time we may need to add *all* of them to the templates to make it so that you get proper versions of them when we're doing pre-releases. Granted, that world is a long ways off (I think) so maybe we kick that can down the road a bit and keep letting core "replace" them for now.
(although even if they have to all be in the templates as drupal/core-util:@alpha etc, they'd all be scripted to be added, so it might not be any maintenance burden - it would just make that oddball scenario where users start with an pre-release, and turn it into a production site)
Comment #15
mmjvb commentedWith proper specifications the require could be as simple as:
As long as the intended use is properly communicated and documented it is irrelevant what it is. Currently its use is to follow the development cycle of Drupal starting at the current stage with an alpha release. When starting at alpha is too soon for you, decide on beta or RC. It doesn't stop at current minor, it stays within a major.
The suggestion above introduces a stop at minor (~9.1.0). My suggestion is the same but assumes the dependencies are fixed:
- composer/installers should be a dependency of core/drupal
- drupal/core should be a dependency of drupal/core-composer-scaffold and drupal/core-project-message
Obviously, you would need to change the version constraints when using the project for a different purpose. When you stop following the development cycle you need to remove the stability tags (@alpha)
Comment #16
xjm@mmjvb, I don't understand what point you're proving.
Found #3179197: Drupal components depend on ^8.8 even in their Drupal 9 version which is also relevant to @Mixologic's "set the branch minimum to the minor" idea.
Comment #17
xjmThat seems like bad UX and counterintuitive to me. I should be able to start with 9.1.0-alpha1, and upgrade to alpha2, beta1, beta2, rc1, and 9.1.0 and 9.2.0 with no drama.
Comment #18
Mixologic#17 sort of gets into the question of what are the alphas/betas/rc's for" - how should people be using them? And equally important, what should they not be doing or expecting to do with them?
I think the answer to that question is already in the release notes of every prerelease we put out:
So I feel like we're telling people, now, that if they do use a prerelease in production, that they should expect drama.
So its really a matter of which drama should we expose them to:
1. A simpler upgrade path between minors, but the possibility of jumping to the next prerelease when you're on a stable version.
OR
2. A complex upgrade command line that is documented to ensure that your local root composer.json no longer has the artifacts of having started a real site on a prerelease version of core.
Jumping from a stable to a prerelease on update when you dont want to is presumably more drama than hitting an upgrade wall when you've used a prerelease->stable and six months later when the new stable comes out, and having to find the docs for how to proceed, because you're doing something that we dont support in the first place. (starting a production site on a prerelease)
Comment #19
mmjvb commented@xjm The point I was proving was a response to #6, claimed not to be working, after proving that to be wrong, stated to be misunderstood.
See #6.1, #13
re #11.1 I remember how difficult it was to get that done. It is not me that doesn't understand what is going on. There is no difference, you shouldn't use minimum-stability & prefer-stable in your projects! The misunderstanding here is the scope of these settings. The scope is for anything in your project. Use stability tags to limit the scope to only packages that you allow non-stable releases for. Allowing an RC release for Drupal doesn't mean you want to allow RC releases for any of the packages either dependencies or those added.
Funny enough, composer does it correctly, right out of the box! It is the Drupal community that fails to understand!
re 16.2 Indeed, not the only mistake. You can save a lot of work by doing things right the first time. Making the mistakes is not the problem, fighting the solutions is.
Comment #20
mmjvb commentedre #17 It doesn't feel counterintuitive nor bad UX to me.
For site builders using the template to start a project, it might not be the most optimal way to start, but given the development speed of Drupal it could be justified to start with non-stable. They might not realize that they need to do something to stick with the minor.
For develpers interested in what is going on under the hood, they should be aware of the consequences. And probably expect them to be happening without action on their side. Once discovered they are constrained to the minor, they know how to turn that around so they continue following development when the next minor comes along. Site builders need to be told how to get to the next minor and suggest that they should consider to switch to stable.
So, is protecting the site builder from accidental upgrade justifying the additional action to get to the next minor.
Maybe not with alpha and beta, but what about an RC?
Would expect a developer to start a new project rather than upgrade an existing. Obviously, there will be developers wanting to test the upgrade.
Comment #21
MixologicOr maybe we leverage drupal/core-message some more and look at the projects's composer.json and see if they are on a stable version of core, and provide some warnings like "It looks like this was started as an alpha/beta/rc project - it may accidentally upgrade to nextminor-alpha/beta/rc when it comes out. To fix this, you should take the following steps: "..."
This strategy could probably also be leveraged with detecting if people's template has minimum-stability:dev and prefer-stable:true.
"We have changed the recommended settings for minimum-stability:dev and prefer-stable:true" You can go to URL with instructions to find out more and why you might want to adjust them in your root composer.json.
Comment #22
mmjvb commenteddrupal/core-message could also be the place to communicate the commands to switch from sticking within minor to stick with non-stable or the other way around.
Stick within minor: composer require drupal/core:~9.1.0@alpha --no-update
Stick with non-stable: composer require drupal/core:^9@alpha --no-update
Followed by `composer update --lock`
Again this assumes dependencies are fixed. drupal/core-* need to depend on drupal/core:self.version
This also allows for upgrading: `composer update drupal/core -W`
Comment #23
catch@mixologic:
I think this depends on the definition of 'in production'.
What we mainly want to discourage is people expecting to updating their database through pre-release versions since it's the most likely way to run into errors, and also hard to support people out of the mess. So in this sense, in production means you have a canonical database.
However, if you're in development and need a new feature that's only in a pre-release, then it's reasonable to start an new project on that, when you know you'll be on a stable version by the time you actually launch in weeks or months. A lot of projects rely on config export/import and drush si until relatively late in the development process - you can run into issues with config updates in that case, but those a much more predictable/fixable (and generally, rarer) than content update issues.
Comment #24
ressa@mmjvb:
I agree. From a site builders point of view, keeping it simple, while allowing Composer experts to use alpha, beta, and RC-releases, and how to deal with any problems down the line seems to me the best user experience.
Projects started with the default, recommended way of
composer create-project drupal/recommended-projectshould be catered for, to allow to flow smoothly through updates, with minimal need for interaction.@Mixologic:
That would surely be very helpful information for those who can interpret it, who are likely well versed in Composer usage.
Comment #25
mmjvb commented@ressa Expect site builders to stay away from dev and alpha. As of beta, database updates are to be provided. As of RC they are expected to work. So, it should be save to upgrade from an RC project to Stable for use in "production". Upgrading from beta's might be troublesome. Upgrading from alpha's asking for problems.
So, when starting the project with an RC, you upgrade to Stable. After that would you expect the next upgrade with the RC of the next minor? Would you mind changing requirement to next minor? Would you change it to next RC regardless minor?
Comment #26
xjmFWIW, core has a beta testing program, wherein participants are asked for every minor to update to the beta and report issues they encounter. So yes, a site using a beta or RC now is predictive that in the future it might do likewise. Also, it's worth noting that for minor release alphas the notion that upgrade paths are optional is academic, as update hooks are required on commit for everything to the dev branch, and any issues committed during alpha will be backports of commits to the dev branch. It's only alphas of major releases where the notion that an upgrade path is not guaranteed will actually come into play.
In general,
composer updateshould never downgrade core, even if the site has installed a pre-release milestone. Downgrading Drupal code is asking for data integrity problems, because our database update API only goes one direction.Comment #27
mmjvb commentedUnfortunately, you can't prevent downgrading drupal/core by composer update. Which is why I recommend usage of --dry-run and advise against not whitelisting.
Comment #28
phenaproximaI discussed this with @greg.1.anderson and @Mixologic to try and get a handle on it. Here are some not particularly organized thoughts.
It doesn’t seem like there’s going to be one configuration for our templates that satisfies every possible use case. So changing the templates (via some committer script) when releases are tagged seems like a reasonable thing to do.
From what I can understand, there are basically two paths we need to support here — let’s call them the safe path, and the dangerous path. The safe path sticks only with stable, tagged releases of core. It never allows pre-releases. Sites on the safe path should be able to start on 9.1.0, say, and advance smoothly through 9.1.N, 9.2.0, 9.2.N, all the way through the Drupal 9 line, without drama. It could probably continue to use prefer-stable.
The dangerous path does allow pre-releases. Therefore, I'd imagine it probably shouldn’t use prefer-stable. It allows one to start on 9.1.0-alpha1 and update to 9.1.0, 9.1.N, and so forth — but NOT to 9.2.0-alpha1. Since the dangerous path is meant to enable pre-release testing, and as @Mixologic pointed out in #18, we are explicit that pre-releases aren’t meant to be used in production, I think it’s reasonable to expect a little more care and attention from site owners who are on the dangerous path.
When you create a project (
composer create-project drupal/recommended-project), the way you invoke the command determines whether you’re on the safe or dangerous path. If you runcomposer create-project drupal/recommended-project, with no other qualifiers, you should be automatically put on the safe path. If you explicitly use a pre-release tag (composer create-project drupal/recommended-project:9.1.0-alpha1), you are putting yourself on the dangerous path.It needs to be possible to switch between the safe path and the dangerous path at any time, back and forth, to enable pre-release testing on projects that otherwise prefer to stay on the safe path. That is something that can be done with a Composer command, or series of commands. The exact commands need to be documented and available on d.o, so that developers can just copy-paste-run them without needing to understand anything about Composer, and jump between the safe and dangerous paths.
The only other real difference in between the two paths, as I see it, is that if you’re on the dangerous path, we should leverage the message plugin to inform you, after every update, that you are on the dangerous path, and tell you how to get off of it if you want to (i.e., link to the documentation).
Comment #29
xjmThis issue has gone into some best practice discissions from the initial minimal scope of preventing dowgrades for release milestones. So I'm moving the immediate, data loss critical into its own issue #3182959: Prevent pre-release milestones from downgrading to earlier releases. with a known fix, and discussion can continue here for the rest.
Comment #30
xjmComment #31
xjm#3186364: Allow pre-release dependencies in Drupal pre-release milestones has also been affected by this. We tried to follow the instructions we provide for sites for the minimum stability of three Symfony dependencies (just for pre-release milestones of core), but were unable to resolve a test failure related to the minimum stability.
Comment #33
mile23Is this still an issue?
I can't repro:
Comment #35
greg boggsThe instructions for updating Drupal core do not work once you install an unstable contributed module, and this seems like the best issue to post. One way to work around this is to allow dev stability and then add @stable to drupal core. But, it would be ideal to be able to install top modules and still have minimum stability set to stable.
To replicate the issue:
Now, once a new release of core comes out, type:
Expected result: Download the latest stable Drupal core.
Actual result: