[policy, no patch] Allow contrib projects to specify multiple major core branches

How the module version string will look like in this schenario. Will be the core compatibility part dropped?
I mean, 8.x-3.12 becomes 3.12? I guess in composer it is already as 3.12.

Is Update module compatible with this change? Can we prepare the list of modules/systems which needs to be changed?

Closed #2856657: Add capability for modules to specify compatibility with two+ major versions as a duplicate. @Mixologic posted this comment there:

It would be nice if the core key could begin to be interpreted, internally, just like any other dependency, and begin to allow the same kinds of dependency constraints as are allowed in other dependency declarations.

I would strongly suggest that fixing the dependency constraint parsing code to support semantic versioning would be a blocker to this: #2641658: Module version dependency in .info.yml is ineffective for patch releases

Marking postponed as per Mixologic.

In preparation for d9 support on drupal.org, I did a crude audit of the locations in drupal 8 that we're using CORE_COMPATIBILITY.

They fall into a few categories of usage:

URLS to external services (updates/translations)

Its used in URL's for a couple of API's with drupal.org, specifically the urls for translations, and the urls for updates:
Translations:

/files/translations/8.x/drupal/drupal-8.x.en.po

Updates release history:
/release-history/drupal/8.x

Database update compatibility

We prevent database updates if they are incompatible with the current core version, and our update_7xxx/update_8xxx hooks are tied to the major version number.

Warn/Prevent/Disable incompatible extensions from being installed.

We Provide warnings if a theme or module is not compatible in the list of modules and themes
We disable any extensions that are not compatible with the current core version (in the case of an update from maj->maj)

Version string handling

When we have version strings, we strip out the core_compatibility string from exsiting version strings so we can do comparisons.

#2677532: Move drupal_check_incompatibility() functionality to a new Dependency class and Version component is now in

We should define how is this exactly related to #3009338: [META] Support semantic versioning for extensions (modules, themes, etc) in Drupal core, and allow modules to be compatible with Drupal 8 and 9 at the same time which was recently opened as an implementation issue :)

Yeah my main question was rather how does the Drupal 9 plan look like. Module admin_menu-8.x-2.3 will be both compatible with 8.x and 9.x in its info file and done. Or we'd also change contrib versioning so it will now be admin_menu-2.3 that is both compatible with 8.x and 9.x? (Or something else entirely).

This is perhaps a contentious suggestion, but what if we were to invert this:

1) Adjust the logic in system.module (or wherever it is) to instead of checking for DRUPAL_CORE_VERSION, check for 8 or 9.
2) Add a deprecation notice to Modules Page | Status Page | wherever that says we're gonna remove that in 9.2. (Or whenever.)
3) This gives contrib authors > 1 year to properly branch their module to 9.x. In the meantime, all D8 modules work day one, and we also seem to avoid any infra changes on the DA's side.

@webchick re: #19: There are no "infra issues on the DA side" for this change. This is only about making the core version dependency logic more flexible. It doesn't depend on anything infra-related at all. The nice list in #12 really belongs as sub-issues in the (much more complicated and thorny) issue at #3009338: [META] Support semantic versioning for extensions (modules, themes, etc) in Drupal core, and allow modules to be compatible with Drupal 8 and 9 at the same time. This issue here is but one small part of that plan. If the DA and its representatives think the big meta at #3009338 is too much trouble to implement or support, so be it. But that doesn't have much bearing on if core's compatibility checking logic only knows this:

core = 8.x

or can support something like this:

core: ~8.7 || ~9.0

Once this change is "done", there *are* some infra-related issues (like not having the packaging script add a core = 8.x line to .info.yml files automatically). But that's not a blocker, or even a dependency, on making this change here.

Cheers,
-Derek

p.s. It seems more appropriate to call this issue a child of #3009338 (which is itself a child of #2608062: [META] Requirements for tagging Drupal 9.0.0-alpha1 instead of calling this a direct child of #2608062. Feel free to move it back if y'all disagree.

It seemd like a good idea, but now I am not convinced that it really is a good idea.

It can make things complicated. Having the core version number in the module version string makes everything nice and clear. Currently there is absolutely no confusion about which module package is compatible with which core version. And this is really really useful feature of the current system.

I started to think about this when angular 6 came out, and the devs aligned the version numbers of all base systems. Now angular CLI 6 is compatible with angular 6. No confusion, no more stackoverflow questions about which version to install.

So what would be the benefit of the proposed change?
- No need to create a new package when D9 comes out.
- No need to manage 2 versions, until the codebase is still compatible with both core versions, max until D8 is supported.

I am not convinced these benefits outweight the confusion it might create.
Managing 2 packages in many cases could be easier than writing 1 codebase which stays compatible with 2 core versions.
And when I make a breaking change in the module how would I explain to users that they have used module release 4.3 with D9, but they need to switch to 5.0 because 4.4 is now only compatible with D8. Or maybe I should not even release 4.4, but 5.0 for D8 and 6.0 for D9. Choosing the right version number is confusing.

I think I would keep the current system that a module package can be compatible only with 1 core version.

@Pasqualle: You may be right, but you're in the wrong issue. You need to (try to) make those points at #3009338: [META] Support semantic versioning for extensions (modules, themes, etc) in Drupal core, and allow modules to be compatible with Drupal 8 and 9 at the same time, not here.

That said, you're probably going to be crushed by the momentum of #ProudlyFoundElsewhere, #SemverIsTheOnlyWay and #ComposerSolvesEverything attitudes among a very wide swath of the community. We had that debate for about 6 years and 400+ comments over at #1612910: [policy, no patch] Switch to Semantic Versioning for Drupal contrib extensions (modules, themes, etc). In the end, the possibility that any given extension might work with both 8.last.* and 9.0.* versions of core (I'm going to be very curious to see, in practice, how often that actually happens), the idea that "we do things differently than other software projects and that's inherently bad", and the assertion that "encoding dependencies in your version strings is Just Wrong(tm)" carried the day, and we decided to move forward with contrib extensions using pure semver.

As the person responsible for Drupal's bespoke contrib version numbers in the first place, I'm not attached to them, and I think enough things have changed in the Drupal ecosystem that using pure semver for contrib extensions actually makes more sense at this point (although not necessarily for all the same reasons everyone else does). That's why I opened #3009338 and am trying to help it succeed. I agree that the current system served us well for close to a decade, especially from D5 through D7 when each major version of core was a single branch, totally incompatible with everything else, etc. But in the era of D8, with minor branches, "8.x-2.0" doesn't really capture reality enough anymore to be as useful as it was in the past. There are all sorts of breaking changes in those minor versions (so much for "semantic" versioning, but whatever), so knowing a contrib works with "8.x" isn't specific enough. Extensions might only work with 8.5.* and higher (depending on new features/APIs only added in 8.5.*, data destroying bugs that were only fixed in later releases, an extension trying to restore the functionality to a module that was moved into core and stripped of some essential features that were deemed inappropriate for core, etc). And (in theory, at least) there will be extensions that could work with 8.last.* and 9.0.*, so having "8.x-" in the version is no longer helpful.

For a very long time, the Drupal ecosystem for extensions was a different kind of reality from most other software projects, which is why I introduced the version strings we had. Our version strings were different because the reality of our core system (what those extensions have been extending) were different. I wanted our versioning system to match our reality. Now that our reality has changed, we need to change our version strings accordingly.

Anyway, all this is off-topic from this issue. ;)

Everyone needs to think of this issue as "a given extension needs more flexibility to accurately describe what version(s) of core it can work with". That is all this issue is about. Instead of just "I work with all versions of 8.* core" (which is what we support now), core needs to be able to let extensions express that dependency more accurately. That's all this issue is trying to fix. Everything else belongs elsewhere.

Cheers,
-Derek

p.s. Given that contrib can already define more specific and accurate dependencies by depending on drupal:system or whatever, why don't we completely remove the "core = *" stuff entirely? Why have 2 similar but different systems to accomplish the same thing? Can we rename/re-scope this issue to be "Remove 'core' compatibility support from Drupal core and force extensions to use module dependencies instead"? Does the current state of the module dependency code handle multiple versions? I confess to not having studied that code in a few years and I have no idea what state it's in and what it now supports. But that seems like the right solution here. Let's completely kill the core: keyword from .info.yml files and put all our energy into making dependencies: handle everything we need. Thoughts?

I am not sure that I am on the wrong issue. What I wanted to point out, that official angular packages from angular 6 does not support two core versions. Technically they can, but they have chosen not to. Even "contrib" packages for angular started to follow this idea.
Angular packages "sacrifice" the first number (MAJOR) from semver to inform which core version they are compatible with.

This issue is about the opposite direction what angular team did. We want to support multiple core versions with one package. I think this might be a bad idea in the long run, with semver or without it, does not matter..

a) Yes, you really are in the wrong issue. :) If you're advocating that we stick with our current contrib version strings and keep it so each release is "tied" to a single version of core (more on this in point b), you need to speak up at #3009338: [META] Support semantic versioning for extensions (modules, themes, etc) in Drupal core, and allow modules to be compatible with Drupal 8 and 9 at the same time or "no one" will hear you. ;) But again, before you do that, you should probably (re)read the debate at #1612910: [policy, no patch] Switch to Semantic Versioning for Drupal contrib extensions (modules, themes, etc) since we've gone over all this already.

b) If you're advocating we dedicate the MAJOR from SemVer to binding a given contrib release to a specific version of core, we actually would need to use 2 digits for that, since as I explained above "I'm compatible with Drupal 8" isn't specific enough anymore. We're now basically back to the D4 and earlier days of hell where core minor version numbers change but introduce major API breaking problems for some contrib extensions. We try to avoid that, but we don't always succeed. So, you sometimes need to target both a MAJOR and MINOR of core. It's not *always* broken across MINOR, so sometimes (usually, in fact) a given release of a contrib can work with multiple MAJOR.MINOR combos of core.

Bottom line: core dependency is just way more complicated now than it was from D5 through D7, and trying to encode those complex dependencies in our contrib version strings is no longer sufficient or all that helpful. And again, that's not even what this issue is about. This is about letting a contrib extension define which version(s) of core it works with in a more flexible way. Whether that's to accurately capture "I work with 8.5.* and 8.6.* but not 8.4 and earlier" or "I work with 8.7.*, 8.8.* and 9.0.*" is sort of irrelevant. In both cases, a single contrib release has to say it can work with multiple branches of core. That's already the reality we all face. But we lack the tools to make that as explicit and obvious as it needs to be to help everyone (humans and robots alike) find the right versions of things to build a working site.

Side side note: 9.0.0 being a "major API breaking release" that "only" removes "deprecated APIs" and "should be compatible with 8.last.* (if everyone is keeping up with all the API changes in the minor releases)" is sort of a weird interpretation of SemVer and rev'ing the MAJOR (and MINOR!) digits. We've got a rather funny interpretation of the semantics of "semantic versioning" for core, which is not "#ProudlyFoundElsewhere". That's part of our problem, and what makes things so complicated and confusing for everyone, contrib maintainers and site builders alike.

You can be upset with the status quo. You wouldn't be alone. ;) But this truly isn't the best place to have this discussion.

All that said, thanks for your vote of confidence in the old system. :) I'm not attached to keeping it, but I'm proud that it served us so well for so long, to the point that all sorts of people would rather keep it than go to the "standard" way of versioning releases of things.

Best,
-Derek

Ok, I am here to continue, because now I am sure I am in the right issue.

I am ok with contrib semver, even if I will most probably renumber my modules after #3009338 is implemented.
I agree that core dependency should be set on drupal:system.

I disagree with "Allow contrib projects to specify multiple MAJOR core branches". And this is the issue which wants to allow that.

That means I would keep the core parameter in info file or add a dependency as drupal:system (<9.0) at packaging.

And the reason is that we are not prepared.

How contrib maintainers will define core compatibility, when core key is removed from info file? like this:

dependencies:
  drupal:system (>=8.5)

What is the problem here? This means the module is compatible with Drupal 13, like 20+ years from now. We can be sure that will not happen, the module will not be compatible. I would not like to see users trying to install a module which was never meant to be compatible with their Drupal 13, or even Drupal 9.

If we would use the composer format, then contrib maintainers would write

dependencies:
  drupal:system ~8.5

and I would not complain, as in this case it means the module is not compatible with Drupal 9 or Drupal 13.

I was curious how other systems manage this issue, when they are using similar (non composer) way of dependency settings for core.
I tried to find out how Firefox does it. It might not be 100% right, but this is what I have read about it. A Firefox addon does not need to specify any core compatibility. Firefox has a new major version like every 2 months. So addons must be incompatible with the latest major release on regular bases.
The Firefox addon website has a compatibility scanner which check a list of breaking changes on every uploaded addon. They can put a maxVersion restriction on existing packages when they know a new Firefox will not be compatible with existing addons. They are sending out emails to maintainers when they see an addon will not be compatible with the next major version.

Drupal org does not have anything like that, we are relying on contrib maintainers that they set up core compatibility correctly. Dependency like drupal:system (>=8.5) could be good today, but will be making headaches for future users and d.o infrastructure maintainers..

This issue is currently postponed on #2641658: Module version dependency in .info.yml is ineffective for patch releases, but as I see that is not the only issue which should be solved before allowing contrib projects to specify multiple MAJOR core branches.

Thanks for your persistence and desire to not see this issue end up in disaster! Much appreciated.

  drupal:system (>=8.5)

No one is proposing this. I think we can all plainly see how horrifically wrong and dangerous this would be. You're totally right about that.

The parent issue has some discussion about this. The current proposal is to use composer-esque syntax. Something like:

core: ~8.6 || ~9.0

Therefore, this issue needs a summary update to more clearly spell out what is (and isn't) being proposed.

I think the current approaches are:

A) Enhance the core keyword to use the "composer semver library" (see #3009338-16: [META] Support semantic versioning for extensions (modules, themes, etc) in Drupal core, and allow modules to be compatible with Drupal 8 and 9 at the same time comments #16 through #19 for more) to support core: ~8.6 || ~9.0 or equivalent.

B) Remove the core keyword entirely. Force all extensions to use module dependencies. Make sure module dependencies also support the "composer-esque semver syntax" so that you can do something like:

dependencies:
  drupal:system (~8.6 || ~9.0)

(or something).

Anyone else have alternate proposals? Any thoughts on B that would cause that to be a disaster in its own right? I don't want to write it up in detail if it's doomed to failure from the start.

Thanks,
-Derek

Thanks Derek, I have no more complaints. I do not have anything against making the core dependency as flexible as you have described in your comment.

I think there should be a separate issue about making info file dependencies adopt the composer format. I did not see that issue yet. It could be the next step before fully switching to composer.json for dependencies.#1398772: Replace .info.yml with composer.json for extensions

I would still recommend to contrib authors to create separate package when targetting multiple core majors, even if the package contains the same code. And my second recommendation is to use the same major number as core. But no need to enforce it, that is just my recommendation..

I would go with option B, that seems to be the more common way of defining dependency on a core system.

We're already running into this for minor branches in Drupal 8, see #3034682-8: Fatal error: trait not found, due to core not yet having been updated to the latest security release.

drupal:system (>=8.5.11, >=8.6.10)

As I see your case was not mentioned here yet, as the ticket is about Drupal 8 and Drupal 9 (multiple major branches)

Correct!

Re: #28-#30: Not to split hairs, but I did already mention this case (or something like it) in #24:

Bottom line: core dependency is just way more complicated now than it was from D5 through D7, and trying to encode those complex dependencies in our contrib version strings is no longer sufficient or all that helpful. And again, that's not even what this issue is about. This is about letting a contrib extension define which version(s) of core it works with in a more flexible way. Whether that's to accurately capture "I work with 8.5.* and 8.6.* but not 8.4 and earlier" or "I work with 8.7.*, 8.8.* and 9.0.*" is sort of irrelevant. In both cases, a single contrib release has to say it can work with multiple branches of core. That's already the reality we all face. But we lack the tools to make that as explicit and obvious as it needs to be to help everyone (humans and robots alike) find the right versions of things to build a working site.

(emphasis added).

I still think removing core: entirely and making dependencies: more flexible (#26.B) is the way to go. Thoughts?

This issue still desperately needs a summary update.

yes, 26B

I'm leaning towards supporting B over A, but mostly (I think) because I'd be keen to see #1398772: Replace .info.yml with composer.json for extensions, since it annoys me that as a module (and theme) developer I have to maintain two separate files listing all sorts of meta data, including dependencies.

Having said that though, having a separate core key does provide more emphasis on developers to make sure they specify the correct core version, lumping it in with dependencies might decrease that emphasis and result in lazy or novice devs skipping specifying the core dependency completely. Probably the most logical solution would be simply just prevent the module/theme from being installed if it doesn't specify a valid core dependency. I suppose that's kind of pointlessly stating the obvious on my part, but I figured it's an important consideration none the less. We could even go as far as providing a helpful message under the module/theme description.

Opened #3054433: Support multiple core branch functionality for contributed projects on drupal.org for the drupal.org side of this issue.

The current patch in
#2313917: Core version key in module's .info.yml doesn't respect core semantic versioning
would solve this problem by allowing semantic version constraints to used in the core: key and passed to \Composer\Semver\Semver::satisfies(). So you could just use ^8 || ^9

#2313917-138: Core version key in module's .info.yml doesn't respect core semantic versioning we figured out that using anything but '8.x' in the core key with cause major problems as the same root problem is detailed in #2917600: update_fix_compatibility() puts sites into unrecoverable state

The solution we are currently working on is using a new key core_dependency(name still up for debate) that would use semantic versioning constraints. The current core: key would only be required if needed your module to be compatible with versions before Drupal 8.8.0(assuming #2313917 gets committed to 8.8.0)

See the current #2313917 issue summary for details.

There has been some discussion on #2313917 as to whether it is good idea whether at all to have 1 branch being compatible with drupal 8 & 9, whether modules should have create a new branch, and whether Drupal 9. I don't think that the the issue is the correct place to have that discussion because it has been discussed In Detail here. So you keep the discussion one place. But also I think this has been decided. #2608062: [META] Requirements for tagging Drupal 9.0.0-alpha1, #3069795: [meta] Improve dependency management for extensions

this current issue is really not postponed on #2313917: Core version key in module's .info.yml doesn't respect core semantic versioning this is more really a policy discussion(to close?) and the other is the implementation(though not started that way)

DrupalWTF

how core compatibility is handled in Drupal (explanation for non Drupal developers)

Drupal 7
core = 7.x needed in the .info file

not possible to specify a dependency on minor core release

Drupal 8
core: 8.x needed in the .info.yml file

if specific minor core version is needed then "dependencies" can be used in the .info.yml file as:
dependencies:
- drupal:system (>=8.7)

Proposed solutions for Drupal 9 (Drupal 8.8)
core_dependency: ^8.8.4 || ^9.3
fully semver compatible, and supports multiple core versions (from 8.8)

Theoretically it is possible to use with releases older tham D8.8 with core:8x, but as D8 minor can not be specified, it is mostly meaningless in that case.

This multi-core version can be meaningful for 1 year from jun2020 until jun2021 (from D9 release until D8 end of life).

Drupal 10 (or even D9!)
The proposed solution will be deprecated and dependencies will be defined in a composer.json file. So there will be another hack to make the 3 solutions compatible..

---
I still think it is too easy to create a separate release for D9, and the feature implemented like this is a waste of money..

@Pasqualle:

First, end of Drupal 8 support is not June 2021 but November 2021. So a year and a half after Drupal 9's release, not one year. That is not a short time.

Second, while it may be easy to create a separate release for Drupal 9, it is not trivial to maintain two branches, merge fixes, etc. especially if you don't remove deprecated API use from the Drupal 8 branch but (of course) remove from the Drupal 9 branch. Most contribs don't have a set of developers and release managers standing by to do that. By using the same codebase on Drupal 8 and 9, Drupal 8 users should equally get the bugfixes, security releases will be easier to make, etc. We are talking about long term money savings here for a year and a half across the whole userbase. Who's wasted money you are worried about and how did you arrive at that conclusion?

Finally, by making the act of Drupal 9 compatibility a simple info file change by default, we discourage the all too common historic practice of "oh, new major branch, now I can refactor all the bad decisions I did before". Historically the new major branching of contrib came with the maintainers using that time to refactor everything as well and thus delaying the practical usefulness of the new major version. We need contrib to be updated much faster than that and allow refactoring of contrib on its own schedule, *not* coupled with when core makes a new major version. The eventual right solution for this will be semantic versions of contrib where the major version prefix of the version number will not be present anymore and major version compatibility will be derived from dependency data, but we are focusing on making Drupal 9's update easy first and foremost.

1. This multi-core version can be meaningful for 1 year from jun2020 until jun2021 (from D9 release until D8 end of life).

   The actual timeline for how long this is meaningful as far as module developers are concerned starts as soon as Drupal 8.8.0 is released, December 2019. At this point a module developer will be able release a version of their module that is compatible with Drupal 8 & 9.
   If they have removed deprecations with replacements just added Drupal 8.8.x then

   core_dependency: ^8.8 || ^9
   

   If the module does not use any methods that have been deprecated (about half of contrib modules according to @Mixologic) then

   core: 8.x
   core_dependency: ^8.8 || ^9
   

   So for relatively simple modules that are working but not undergoing active development this allows a module maintainer to make a release(maybe 1-line change) in December 2019 and be good until June 2021

   Even though Drupal 9 won't be available at this point there will be no deprecations after 8.8.0. So this allows a module to have a release ready as soons as 8.8.0 and they will not have do anything after June2020 to make their module compatible with Drupal 9.

   So this 1.5 years versus just 1 year.

2. The proposed solution will be deprecated and dependencies will be defined in a composer.json file. So there will be another hack to make the 3 solutions compatible..

   Hopefully #3005229: Provide optional support for using composer.json for dependency metadata will land in 8.x maybe 8.8 or 8.9.
   This will allow optionally having dependencies including Drupal core defined in the composer.json but will not deprecate dependencies and core requirements until Drupal 10.

   Therefore this would allow a module to be compatible with Drupal 8 & 9 & 10, though the number of modules that won't use anything deprecated in 8 & 9 would probably be very low. But since there will be no deprecations after Drupal 8.8.0 they would really only have worry about deprecations in Drupal 9 to be compatible with ^8.8 || ^9 || 10.0.0

   But problem with practically having a module be compatible with 3 versions of Drupal will not be that it won't be easy to specify core compatibility(if #3005229 lands in 8) but rather actually having a module that doesn't use deprecated code.

I don't want to sound arrogant, but my contribution to Drupal is limited to things I need or I am interested in. From that limit comes that all these years I've never supported 2 core versions (with D8 not even 2 minor versions).

When I will write D9 code I don't want to think about if the code is still compatible with D8 (same as currently I don't care if the code is still compatible with php5). Therefore to be a reasonable maintainer and do not break D8 websites using my modules, it is better if I create a separate branch for D9.

Devs who are supporting 2 core versions are currently doing it in 2 branches. Even Gabor described all the benefits of 2 cores in 1 branch, I am not convinced that it will be less painful. Mostly because of writing backward compatible code and the possible switch where the code must be separated when it stops being backward compatible.

As I see I am starting to repeat my comments. Last note about the money wasted; making the feature and working with it does not waste my money, so I am not worried. If you think this is what Drupal needs then go for it.
The only thing how this feature can complicate my life I have described in comment #21.

@Pasqualle: As per the issue title and the proposed implementation, the goal is to *allow* multi-version compatibility *not to require it*. You should still be able to create a new branch and set different dependency requirements. So I don't think this limits your ability to choose what you prefer to do as a project maintainer. It does reduce the startup and ongoing cost for project maintainers who do want to support multiple major core versions though. Which should be a feature available given how we do the deprecations process (also known as continuous upgrade path). We should not require maintainers to open a new branch given the same codebase could work on both Drupal 8 and 9 (which according to our tools is currently true for over half of the Drupal 8 contributed projects).

#2313917: Core version key in module's .info.yml doesn't respect core semantic versioning has committed that allows this. Closing this since I don't see any other remaining taskes here.

🥳

I verified that the implementation issue in #2313917: Core version key in module's .info.yml doesn't respect core semantic versioning is already tagged for both the release notes and the highlights, so this one doesn't need to be. :)