Problem/Motivation

Splitting this out from #2608062: [policy, no patch] Pre-requisites for opening Drupal 9 since it's a fairly large change from previous major release cycles.

A couple of weeks ago I did some serious triage of the Drupal 9 queue. Between moving some longstanding feature addition/removals to the ideas queue, marking a few things duplicate, and moving a lot of issues that were abandoned in 2013 and 2014 in 8.x code freezes and betas back to 8.x due to eligibility for minor releases, we're now down to a grand total of around a dozen issues.

What's remaining falls into the following categories:

1. Removing backwards compatibility layers that have been added to 8.x

2. A small number of minor impact, medium-to-large disruption @internal API changes which don't seem worth doing in a minor release (for example #2346093: Remove views module weight or #2799911: Make the Connection parameter the first parameter in Select Query Builder.

3. Actual public API breaks, which are either hard to do in minor releases or haven't been triaged back to 8.x yet.

Between our reasonably loose bc policy for minor releases, our slowly improving mechanisms for providing bc for various kinds of changes, and the ability to provide completely new functionality in minor releases via experimental modules, it should be possible to do the vast majority of change we used to do in major releases in minor releases.

#2608062: [policy, no patch] Pre-requisites for opening Drupal 9 outlines several problems we've had with previous major releases, and proposes doing more work before opening the branch to mitigate them.

There are other issues once major versions are released though:

1. Divergence from the previous major branch means that contrib modules have to go through multi-week or multi-month porting cycles before they're stable.

2. While new sites start getting built soon after release to at least some extent, existing sites often wait years to upgrade - as evidenced by the still fairly sticky 10,000 5.x sites, 100,000 6.x sites and 1,000,000 7.x sites reporting back to Drupal.org. The migration path to 8.x isn't the only part of this, otherwise those 5.x/6.x sites would have migrated to 7.x by now.

3. This then results in some sites never upgrading. Either they move away from Drupal, or they skip a major release or three.

4. 7.x spent about a year after release to get to a working upgrade path with no critical bugs. 8.x doesn't have stable upgrade paths from 6.x or 7.x yet either. So even if people wanted to migrate, they can't without a lot of extra work at the moment.

Proposed resolution

Attempt to offer a continuous upgrade path between major versions. See https://speakerdeck.com/nicolasgrekas/how-symfony-3-dot-0-moves-forward-... for how Symfony does it, although they don't have some problems that we do, like needing to worry about content entities etc.

This means:

  1. 8.LAST.x comes out at the same time (or within a month) of 9.0.x
  2. Relative to 8.LAST.x, 9.0.x only:
    • Removes deprecations
    • Makes @internal API changes that are technically eligible for a minor but we'd prefer not to do in a minor.
    • There may be cases where there's a necessary API change which absolutely cannot be made without a bc break between majors (i.e. via creating a parallel API and deprecating the old one). Should such a case come up, this would need to be discussed case by case as to whether it could be included.
  3. We add a feature to support a more flexible core compatibility, so that contrib and custom modules can work with both 8.x and 9.x without branching.
  4. 9.1.x starts the process again until we reach 10.0.x

The benefits of this are:

1. New sites can immediately use 9.0.x - 99.9% of modules should 'just work'
2. Existing sites should be able to update from 8.LAST.x to 9.0.x with minimally more work than between minor releases. There might be exceptions if they're using a deprecated module (like blog from 7.x-8.x) but the removed module in contrib would at least run on the new version.
3. Due to this, 6.x and 7.x sites should be able to migrate to 8.x towards the end of the 8.x cycle, in the knowledge that they'll be able to quickly hop from there to 9.x, rather than immediately be on another obsolescent release with another chasm of an upgrade path.
4. Contrib maintainers can do a small port every six months, and never a massive port if they don't want to.
5. Once we get to the 9.x-10.x transition, apart from any remaining sticky 6.x or 7.x sites, the bulk of Drupal sites will be able to keep up with each major release. This could allow us to do scheduled major releases as well as minor ones (Symfony just started doing this with a two year cycle).

The drawbacks are:
1. Some things will be hard or impossible to provide bc layers for - however we've not found those yet. Even in those cases, it should usually be possible to build a completely parallel system and leave the old one in place.
2. Major versions are no longer 'big bang' releases - this is also a benefit though.

Remaining tasks

User interface changes

API changes

Data model changes

Comments

catch created an issue. See original summary.

Crell’s picture

As a general approach this makes sense to me, for the reasons given. However, PHP is a fast enough moving ecosystem, as is the web itself, that I am 100% certain we will still want/need to have "tectonic shift" changes in the future. There's also definitely places where we have, and will, done something badly that we need to fix at an API level. We absolutely do *not* want to do 15 of them at the same time again :-), but we should make sure we leave ourselves the option to still make 1-2 big-break changes in those cases where it's appropriate and needed.

catch’s picture

but we should make sure we leave ourselves the option to still make 1-2 big-break changes in those cases where it's appropriate and needed.

So two things with this:

1. If we can do them by building a new system from scratch parallel to the existing one, then it's not inconsistent with a 'continuous update policy'. Wouldn't be easy but in many cases would be possible.

2. If #1 fails then there might be a need for a hard break - but we might decide to schedule those for every other release or similar (and work on them in feature branches before opening the main branch at all). I really want to see cases where #1 fails though, which means trying it first.

Crell’s picture

I'm open to planned alternating "big break / small break" release, which would make Drupal 9 a Symfony-3 like release and Drupal 10 (Drupal X?) a "break a few more things but not D8 level" release. I'm OK with that, but if so we should try to communicate that as early as possible. One of the challenges early in D8's cycle is that there was zero clarity about how long a dev cycle it would be, how big it would be, if it would be a "stabilization" release as some were proposing, etc. That lack of clarity really messes with your head and made planning nigh on impossible.

We'd probably have to decide whether parallel implementations for a major would be worth the effort case-by-case. I'm not sure what guidelines we could or should set down there, but it would be helpful to have at least some vague idea of at what point it becomes more trouble than it's worth and it's better for all to just have a clean break at some point. (I don't have a good suggestion on that yet; I'd have to think it through further.)

timmillwood’s picture

This will in effect deprecate the Migrate module.

I think this is a good approach for Drupal 9, which I think really needs to a a clean up release. However I'm not sure if it'll be viable forever. For example imagine if we wanted to rewrite the Entity API and all entity storage. This would be a monumental upgrade path, especially if you factor in all the things contrib can do with entities. However I'm not sure I can ever argue the case of not providing some sort of upgrade path.

Maybe we should caveat this policy with "Adopt a best attempt continuous upgrade path for major version changes".

catch’s picture

For example imagine if we wanted to rewrite the Entity API and all entity storage. This would be a monumental upgrade path, especially if you factor in all the things contrib can do with entities.

Theoretically at least you could:
1. Start a new entity API in a different namespace.
2. Allow contrib to start using it
3. Provide a migration path from the old config + storage to the new one
4. Add new core implementations for the core entities that are opt-in for existing sites/opt-out for new ones
5. Deprecate the old API
6. Drop the old API

The difference if we adopt a strict-ish continuous upgrade policy is that either 1-4 would need to happen in the 8.x branch, or 5-6 would need to happen in the 10.x branch.

The hardest bit there is #4. A lot of the rest is largely what we did in the 8.x dev cycle with core entities, except many of these things happened in parallel.

larowlan’s picture

It is proposals and foresight like this that make me glad that @catch has keys to the kingdom. Keep it up mate.

Mixologic’s picture

+∞

Being able to commit to Drupal and know that its not just a matter of time before I have to incur significant cost to redo everything is awesome, and the upgrade pain of the past has been a serious hurdle for many, even in the adoption phase.

As far as techtonic shifts go, the only thing preventing us from supporting parallel systems is the parallel maintenance burden. But the cost is definitely worth the tradeoff.

vaplas’s picture

Delicious! This is a real chance to revive Drupal!

xjm’s picture

This will in effect deprecate the Migrate module.

I think this is a misunderstanding. This issue is about the code upgrade path, not the data upgrade path. However, this issue's best practice with D8 sources will also ensure a more continuous migration path for data. So it does not deprecate Migrate; they complement each other.

xjm’s picture

Title: [policy, no patch] Adopt a continuous upgrade path for major version changes » [policy, no patch] Adopt a continuous API upgrade path for major version changes
catch’s picture

Yes it doesn't preclude a hard break for data if we had to. You'd have an 8.x install, you install 9.x + modules, migrate 8.x to 9.x. This is part of the idea of having the 8-8 sources ready before 9.x is open in #2608062: [policy, no patch] Pre-requisites for opening Drupal 9.

For example say we added a completely new and optional version of forum module using a custom entity for both OP and replies instead of nodes and comments (not suggesting we do, pure example) to 8.x - we'd probably hide the old forum module from the UI in 8.x but it can't be removed, just deprecated.

We then offer an 8-8 migration path from old-forum to new-forum - but that's optional for existing 8.x sites. When 9.x is opened, old-forum module is git rm -rf but the sources and destination are in place. Therefore you have to migrate from 8.x-9.x if you haven't already switched during the 8.x cycle.

The difference is twofold though:

1. Any contrib modules/themes extending/working with forum module have had the time to update to work with the new module before 9.0.0 is released.
2. Sites built on 8.x using the new forum module don't have to run that migration to 9.x (or it'll be a no-op at least), only ones using old-forum from before it was deprecated.

If you have custom code working with old-forum, you might end up porting that as part of the move 9.x, which might be some work - but we've given the option for you to do that during the 8.x cycle, and it's your custom code that you didn't keep up-to-date that's blocking you from moving, not every single Drupal contrib module that exists definitely being broken until at least 9.0.0-betasomething.

effulgentsia’s picture

We discussed this on a call with @catch, @xjm, @Dries, @alexpott, @webchick, and @cilefen, and we're +1 to this proposal, except in terms of adopting and communicating this as an official decision, we want to make sure to leave room for the possible exception.

From the issue summary:

Some things will be hard or impossible to provide bc layers for - however we've not found those yet. Even in those cases, it should usually be possible to build a completely parallel system and leave the old one in place.

From #3:

I really want to see cases where [building a completely parallel system and leaving the old one in place] fails though, which means trying it first.

+1 to attempting this, but the issue summary's proposed resolution, as currently worded, doesn't leave room for discovering a case where building a completely parallel system and leaving the old one in place fails. So, I think the next step here is to update it to leave room for that possibility.

effulgentsia’s picture

Status: Active » Needs work
Issue tags: +Needs issue summary update

Tagging and setting status per #13.

catch’s picture

Issue summary: View changes

Tried an issue-summary update, does that help?

effulgentsia’s picture

Issue summary: View changes
Status: Needs work » Needs review
Issue tags: -Needs issue summary update

Thanks. I merged your new #5 into #2.

effulgentsia’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: +Needs product manager review, +Needs framework manager review, +Needs release manager review

Since my only change to the IS was in #16, and that was just moving catch's text around and minor grammar fixes, and I believe all the comments on the issue have been addressed and there've been no new comments since #16, RTBC!

Let's leave it in the RTBC queue for at least a couple weeks before marking it Fixed to make sure it has sufficient visibility for people to raise concerns if they have any.

Meanwhile, although per #13 this has already been +1'd by the various core committers, I'm adding each of the 3 tags, so that each one can be removed with a corresponding comment from a person of each role, either explaining why they like it, or what remaining concerns they have with it.

catch’s picture

One thing with the 'possible exception' is we also have #2608496: [policy, no patch] Drupal 7 (and 8) EOL timing open.

We should consider the following:

- once 9.0.0 is released, do we expect people to update from 7.x to 8.x or 7.x to 9.x straight away (bearing in mind some upgrade projects will be in progress when it lands)?

- what will the impact of the continuous upgrade policy (and the reality of a continuous upgrade) be on the rate of 7.x migrations to either 8.x or 9.x? We know that people intentionally skip major versions now, being able to communicate that they don't need to, and being able to say that honestly might make a difference.

Assuming we adopt and continue this policy for the next several major releases, once we only have versions > 8.x to worry about these questions start getting a lot less important hopefully. While there are still ~100k 6.x sites and ~1m 7.x sites around it really needs to be taken into account.

xjm’s picture

For #18, I assume that the guarantee of continuous upgrade path is 8.x+ only. People still are encouraged to go directly from 7.x to 9.x using Migrate for data and upgrading their code the same way they have to from 7.x to 8.x. So I don't know that there's anything we need to change? Or if there is I am not clear on what it is.

I agree with and sign off on this policy; assuming @catch does too we can remove the release manager review tag.

catch’s picture

People still are encouraged to go directly from 7.x to 9.x using Migrate for data and upgrading their code the same way they have to from 8.x. So I don't know that there's anything we need to change? Or if there is I am not clear on what it is.

I'm talking about the practicalities of upgrading real sites from 7.x to 8.x/9.x. i.e. by adopting this policy, if we're very successful it would be quite feasible to update from 7.x to the penultimate 8.x release, then the 8.x LTS, then from there to 9.0.x or 9.1.x within a total of 9-12 months or similar.

Almost no-one would do that with 6.x/7.x/8.x due to the amount of work involved. The more we diverge from a continuous upgrade path, the less feasible this becomes and the more likely people delay 7.x to 8.x migrations to wait for 9.x, or delay 8.x to 9.x migrations to wait for 10.x. Need to remember there are 1.1 million sites without any continuous upgrade path that need to get to 8.x before the project as a whole can say it has one, this makes the context of 8.x to 9.x quite different from what 10.x to 11.x might be.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

xjm’s picture

Aki Tendo’s picture

I was thinking on some of the current problems with runtime assertions (I'm obsessed, sorry) and was thinking - josh this would be so easy if 5 just went away. And I had a thought,

Could a PHP 7.0 floor be 9's marquee change, and that's it? jQuery did something similar when the decision was made to withdraw support for old IE's in 2.0 so they could optimize the code in ways old IE support prevented.

Similarly we can get a lot of performance upticks just by dropping 5 support in 9. Also, jQuery 1 & 2 where developed in parallel for awhile - this is something to consider. And perhaps put the deprecated functionality into a backwards compatibility module? I don't know if that's practical, but it would be nice to do. In principle having each major able to at least use the previous major's modules would be nice.

catch’s picture

@Aki Tendo PHP 5 will likely be completely unsupported by the PHP team by the time 9.x comes out, see #2842431: [policy] Decide and document how we deal with PHP version EOL for some discussion.

Dropping bc layers in 9.0.0 means that a fully-up-to-date 8.x module should run on 9.x . The idea is that we don't introduce new backwards compatibility breaks in 9.0.0 beyond possibly some small @internal changes that have been pushed off from minors.

This policy as written includes a clause to allow exceptions to this in case something comes up, but the goal will be that the 9.x upgrade is about the same work as say upgrading two minor releases of 8.x (or less than that).

Aki Tendo’s picture

@catch Ok. Thanks. Being self trained I don't always get the intricacies in some of these plans but I try to. The above was a late night musing before going to bed - not fully thought out but worth musing.

Thought though - why does 9 have to be pushed out that far? Why not just relabel everything targeted at 9 now to 10, and put 9 out as the PHP 7 floor version. Reason - I don't think it's appropriate to change minimum PHP major in a minor release. We have an infinite number of version numbers to use - why save them? If PHP 7 is the floor was the *only* difference between 9 and 8 that would be worth the major version number. Also, extended a deprecation's lifespan from 9.x to 10.x isn't going to break anything or upset anyone like the reverse would.

catch’s picture

Thought though - why does 9 have to be pushed out that far? Why not just relabel everything targeted at 9 now to 10, and put 9 out as the PHP 7 floor version.

So when 8.3.0 comes out, we immediately drop support for the 8.2.x branch, and this continues for every minor release.

When 9.x support comes out, we've committed to supporting an LTS release of 8.x for some period of time, and have to decide about 7.x LTS support at that point as well.

On top of that, .info.yml only support a single version for core compatibility, so even if 8.3.0 was renamed to 9.0.0, no contrib/custom modules or themes would be able to run on it without a new 9.x branch + that change.

This makes a new major release a very non-trivial thing to do, even if the only thing it did was increase PHP requirements. I personally don't consider a PHP version requirement change as requiring a new major release though, we'd just need to announce and plan for it carefully (and a requirement change doesn't mean we start using 7.x-only features necessarily, mostly means we stop testing etc.).

alexpott’s picture

I've added the core deprecation policy worked on by @catch, @dawehner, @gaborhojtsy, @xjm to our handbook https://www.drupal.org/node/2856615.

xjm’s picture

Added a stub section to the allowed changes policy:
https://www.drupal.org/node/2581501/revisions/view/10364213/10366102

catch’s picture

xjm’s picture

I added a couple more followups we need for #2575081-44: [policy, no patch] Use E_USER_DEPRECATED in Drupal 8 minor releases as well. Then I think the last things we need here are product and framework signoff.

catch’s picture

Status: Reviewed & tested by the community » Needs review

I've read this clause several times from #2822727-13: [policy, no patch] Adopt a continuous API upgrade path for major version changes and the more I read it, the more it makes the overall proposal look fragile, so moving back to CNR for a bit more discussion.

There may be cases where there's a necessary API change which absolutely cannot be made without a bc break between majors (i.e. via creating a parallel API and deprecating the old one). Should such a case come up, this would need to be discussed case by case as to whether it could be included.

This mentions discussing things case by case, but as stated this could cover wildly different things, between doing something like changing a database API interface that might affect 50 modules vs. rewriting the form and render systems without bc for the old one, which could affect hundreds of thousands of modules once you include custom ones.

The transition between 8.x to 9.x in practice has some very specific challenges outside this policy in the abstract, which I think we should take into consideration:

Contrib modules are also still being ported to 8.x, if say 98% of contrib modules are able to make zero changes from 8.x-9.x (beyond what they might need to for a minor, or to catch up with previous minor deprecations), then we've not created a large backlog of work for contrib maintainers by moving to 9.x. This in turn presents zero barriers to site builders since they can theoretically upgrade from 8.x.last to 9.x.first, without every contrib module they use also needing to be individually ready.

More concretely, the 100k+ sites still on Drupal 6 and the 1m+ sites still on Drupal 7. At time of writing, we do not support a stable upgrade path from either version to 8.x, and won't for at least another six months. This means that whenever 9.x might be released, there is likely to be a significant number of sites either fairly recently migrated to Drupal 8, in the process of migrating, or still on Drupal 6/7. Any perceived difficulty getting from 8.x to 9.x discourages people from moving from 7.x to 8.x, because they'll have to factor in another non-trivial migration again in a shorter timespan.

On top of this we have no decision (or even much discussion) yet about when 7.x and 8.x EOL are going to be - partly again because discussing EOL of 7.x when 1m sites are on it and 100k sites are on 8.x is :/

So for changes which we're unable to do in 9.x without a bc break, at least if we think there's any chance of broad disruption, I think we should seriously consider postponing those issues to 10.x. There are several reasons this isn't just 'postponing the pain':

  1. By the time 9.1.0+ is out, we'll know to what extent people are successfully (or not) migrating off 6.x and 7.x. This is like getting animals onto the ark of 8.x..
  2. By the time 9.1.0+ is out, we can expect most contrib modules to have been ported from 8.0.0 APIs to 9.0.0 APIs (i.e. no 8.x deprecations) and depending from #1, from 7.x if they weren't already or officially abandoned. This is a much shorter jumping off point than 7/8-9. If that's not the case by then, then things aren't working out as planned and we can look at that, but it limits things to one set of changes (deprecation removals) vs. two (deprecation removals + new public API breaks).
  3. It gives another release cycle to conclusively demonstrate that the changes proposed really can't be made with a bc layer or by developing a parallel API, and on a code base that's had a lot of accumulated cruft removed.
xjm’s picture

Thanks @catch. I think deciding to defer anything we really can't solve with BC to D10 is a good idea. That gives us room to really do our best to put the continuous upgrade path into practice and test our hypotheses about it. It also gives us more room to straighten out the things that are keeping people on D7 now, before we consider disrupting them again.

I know Dries had concerns about making a commitment to something that didn't allow for adapting if things don't work or that favor procedure over discretion, but maybe the compromise of saying "Yes, but D10" instead of "No, never" to anything that we can't figure out how to do with reasonable BC will address his concerns.