Problem/Motivation
See #2942096: [policy, no patch] Remove old update hooks prior to each major version (Drupal 10 and later) for background discussion and reasoning. See #3290810: Remove updates added prior to 9.4.0 (9.4.4 for ckeditor) and add 9.4.0 database dumps for the 9.4/9.5 example for how the actual process happens.
Very short tl;dr:
1. When updates exist in both 10 and 11 major branches, this means they can run going from 10.0 to 10.5, 10.0 to 11.0, 10.0 to 11.5 , 10.5 to 11.5 etc. The more potential release versions there are to upgrade to and from, the more potential interactions and dependencies there are between different upgrade paths - either core updates added years apart from each other, or core + contrib updates running at the same time. We do not have an upgrade test framework that can handle every possibility of core version updates (this would only be possibly with build tests), and even if we did, there is still contrib updates to add on top.
2. To reduce (but not eliminate) this risk, for every major release since 8.x (and also before), we've remove updates added to the previous major, so that you had to update to at least 8.8.x before 9.x, and at least 9.4.x before 10.x
3. Because 10.x is going to be maintained for two years or more into the 11.x cycle, we will need to either continue to backport updates from 11.x to 10.x, so that they run on either version, or only commit them to 11.x if they're not required for backwards/forwards compatibility. Either way this increases the likelihood of things going wrong.
Steps to reproduce
Proposed resolution
For 9.x/10.x, we originally removed updates to 9.3. We then went back and removed 9.3-9.4 updates in a later issue/commit. This was partly because by then we knew we'd be releasing Drupal 10.0 in December with 9.5.
The trickiest update currently in 10.x (that I can think of) is #3365945: Errors: The following table(s) do not have a primary key: forum_index and would be included in 10.2 update removals. However we also have #2885413: Timestamp field items are affected by 2038 bug which is even more complex of an upgrade path, and is almost certainly not going to land in 10.2.x.
I think we should consider deciding now that we're going to remove 10.3.x updates from 11.x, and then do the update removals, recreate the database dumps etc. once when 10.3 approaches alpha. This will reduce doing work twice, and we can set the expectation that all sites will need to get onto 10.3 before moving to 11.x, this would be whether or not we release in the June, September, or December windows.
By getting all sites onto 10.3 before they do the major upgrade, it also helps to get sites updated to the latest contrib versions as well (indirectly), and there will be much better 11.x compatibility amongst contrib modules when 10.3 is out.
This is stricter/more pre-emptive than previous major releases, but given we have a likely, tricky, update path to add and also the longer 10.x cycle, should save everyone lots of pain later hopefully.
Comments
Comment #2
quietone commentedComment #3
catchMoving to needs review.
Assuming we go ahead with this plan, we should open a postponed issue and remove the old database updates as close as possible to the 10.3 beta, to minimize the chances of having to regenerate database dumps multiple times since that's still time consuming and error prone. On the other hand if we're able to get #3403649: Rework database update tests so we don't have to ship database dumps in git done it might then be easier to remove updates on a rolling basis.
Comment #4
smustgrave commentedI like this part.
Comment #5
smustgrave commentedBeen about a week but no one has openly objected. Want me to open the followup mentioned in #3?
Comment #6
catchThis could use a +1 from a release manager who's not me still I think, tagging for that.
Comment #7
gábor hojtsyNow that Drupal 10 will be supported until the Drupal 12 is released, would it not be in line with that intent to keep the update functions instead? I understand that removing the upgrade functions is standard practice, but https://www.drupal.org/blog/drupal-10-will-be-supported-until-the-releas... would be counter to that?
Comment #8
catchThere are three reasons to remove older update functions:
1. To encourage sites to update contributed modules to 11.x-compatible versions before updating core to 11.x. i.e. sites on 10.3.x with updated contrib modules are much more likely to update to 11.0.x successfully than sites on 10.0.0 trying to jump all the way to 11.x and updating their contrib modules at the same time.
Sites on 10.0 may not be able to update contrib modules to 11.x compatible versions at all (if they rely on APIs added in 10.1 or 10.2), and they also may struggle to composer update contrib + core all the way from 10.0 to 11.0, because updating lots of composer dependencies at the same time gets very, very complicated.
2. To minimize the number of (core and contrib) update functions that can potentially have to run in the same update, for example if a site tried to update all the way from 10.0.0 to 11.6.0 (which would represent 4-5 years of core development) there could be tens of updates that need to run, config that gets updated multiple times for different changes etc. The more updates can potentially run, the more likely it is that we end up with dependency issues where update X has to run before update Y but after update Z and similar.
3. To reduce the matrix of update paths that we don't have test coverage for. e.g. ideally we'd have test coverage for 10.3-11.0, 10.4-11.0, 10.5-11.0, 10.3-11.1, 10.4-11.1, 10.5-11.1, 11.0-11.1 etc. but we don't have anything like that and can't add it with the current update test infrastructure.
I think that #1 is about the same with 10-11 as it was for 9-10. But #2 and #3 will actually be worse with two active branches, since we'll have more total minor releases to juggle, more updates paths to potentially backport between major versions (or not), and more potential paths to for people to take between different combinations of versions.
Or tl;dr, we've said that 10.x will be supported until Drupal 12.0, but we didn't say that Drupal 10.0 will be supported until 12.0.
Comment #9
smustgrave commentedNot a release manager but believe this can be back in RTBC.
Comment #10
catchfwiw I discussed #7 briefly with Gábor in slack and we realised that removing updates from 10.0-10.3 from 11.x doesn't preclude updating from Drupal >= 10.3 to 12.0.
That will depend on the functions added from 10.4/11.0 onwards and when those get removed. e.g. there could be a situation where you can update from 10.6 and 11.4 to 12.0, but not from 10.5 and 11.3. Or it could be that you can update only from 11.4+ to 12.0 - it all depends what gets backported to Drupal 10 minor releases, and what the cut-off point is. This doesn't automatically mean we'll support a Drupal 10-12 update path, but it adds an extra thing to think about in #2942096: [policy, no patch] Remove old update hooks prior to each major version (Drupal 10 and later) and is unaffected by this issue.
What this issue will do is prevent direct updates from Drupal 9.5-10.2 to Drupal 11.
Comment #11
larowlan+1 from me. We're telling people they can jump from LTS to LTS so we'd expect people to be on 10.4 and stay there until 11.4 (assuming that's the LTS) comes out. This is like how Ubuntu users go from 20.4 to 22.4 and skip the non LTS versions in the middle
Comment #12
quietone commentedI agree with this as well. I must admit I am fond of the reduced workload part regarding the dumps.
Comment #13
catchI've updated #2942096: [policy, no patch] Remove old update hooks prior to each major version (Drupal 10 and later) quoting/linking some of the discussion here. Now that this has had an additional +1 from another release manager, going to mark it fixed. Have also opened #3416928: [Meta] Remove updates added up until 10.3.0 from Drupal 11 for the implementation issue.
Comment #15
quietone commentedChanging to latest version when this was closed.
Comment #16
quietone commented