[meta] Improve dependency management for extensions

So I think option #3 is a good plan. Ideally we'd land #2313917: Core version key in module's .info.yml doesn't respect core semantic versioning asap so that as soon as the 9.x branch opens, contrib modules can start testing against it. We can then continue to work on the other issues but it might just mean deprecation for Drupal 10 instead of for Drupal 9.

+1 for option 3.
re

And take most of the work from the following issue, and adapt it to work in parallel with the existing system.

I think #3005229: Provide optional support for using composer.json for dependency metadata actually does allow it to work in parallel already it would just be matter of deprecating in 9 instead of 8.

It seems like we would have to introduce the support for using composer.json to replace the core and dependencies keys in info.yml in the 8.x branch though even if we weren't going to stop support those until 10.x.

Otherwise it really takes away the benefit of #2807145: [policy, no patch] Allow contrib projects to specify multiple major core branches(now covered by #3005229) because you would still need different versions of your module for 8 & 9 if you wanted to use only composer.json for dependencies in 9(unless you just duplicated in both info.yml and composer.json but then what is the point?)

It would probably be simpler though if we could wait till 9.x to do #3005229(also because there so much else to do) but then we would probably have to advice people to keep using core and dependencies in .info.yml if they want there module to compatible with both 8 & 9.

Additionally if we don't do #3005229 in 8.x then modules won't be able to set module dependency constraints once #3054391: [META] Support semantic versioning on drupal.org for contributed projects once is implemented unless we just did #2641658: Module version dependency in .info.yml is ineffective for patch releases(the last patch there simplifies this)

+1 for option 3 too

I think one thing we should consider doing before 9.0 is deprecating own Drupal specific constraint string format.

So not remove version constraints in .info.yml dependencies but rather just deprecate support for handling of formats that won't to valid composer formats. and keep our current
- 'drupal:[module] ([constraint])'
pattern.

So
- "drupal:weform (>=2.0)
or
- "drupal:weform (2.x)
Would still be valid.

What would not be valid is
- "drupal:weform (>=2.x)
Composer semver doesn't allow and operator with a range.
For composer semver 2.x gets parsed as [>= 2.0.0.0-dev < 3.0.0.0-dev]
which is the same thing it means to our constraints.

Of course this would also mean anything that started with 8.x- would also not be valid unless we put logic in just to strip that as there is now.

The latest patch in #2641658-50: Module version dependency in .info.yml is ineffective for patch releases solves that problem by using \Composer\Semver\Semver::satisfies() directly when possible(no exception) and only falling back to our current logic when it can't. So basically in Drupal 9 we would just use \Composer\Semver\Semver::satisfies() directly always.

I think this would 3 benefits

1. #3054391: [META] Support semantic versioning on drupal.org for contributed projects would not have to wait till #3005229: Provide optional support for using composer.json for dependency metadata so it would make the timing not as critical as far as timing with drupal.org changes
2. If we did #3005229: Provide optional support for using composer.json for dependency metadata after 9.0 then the update script in that is currently in that patch that moves dependencies from the info.yml file to the composer.json could be much simpler because we would know that if it is a valid Drupal 9 module the constraints would be valid for \Composer\Semver\VersionParser which \Composer\Semver\Semver::satisfies() uses. And we won't have edge case in our current format when converting to composer.json.
3. So far everyone is in favor of option 3 which depends on #2313917: Core version key in module's .info.yml doesn't respect core semantic versioning which allow any valid composer constraints in the core key for info.yml files. If we also use composer constraints in dependencies in would be much simpler to document(which really don't describe in detail now.). Otherwise we have to explain use composer constraint format for core and this drupal specific one which overlaps almost completely but not quite for dependencies.

@Mixologic thanks for the extra info.

I added a comment to #2641658-51: Module version dependency in .info.yml is ineffective for patch releases and mentioning the options now that we know should be super careful not to introduce any new capabilities such as ~/^ support accidentally.

Also related to here

Related, I wanted to see how many contrib dependencies constraints actually would throw an error if used with `\Composer\Semver\VersionParser::parseConstraints()`. So I made https://github.com/tedbow/drupal_contrib_dependencies. tldr, 99% don't throw an error.

First: thanks to everyone involved with creating this issue and discussing it. This issue brings clarity in a cluster of seemingly overlapping issues that were hard to make sense of (at least for me). The options listed in the "proposed resolution" section in the issue summary made it crystal clear. 👏👍

@tedbow was kind enough to walk me through the discussion and thought process. He pointed out one very interesting aspect of the first and second option that I had not considered yet (it may be obvious to some of you, it was not to me) that I don't see listed in the issue summary yet: even though it seems unlikely that we'd break something along the way, if we get something wrong, it may break the Drupal 8 to 9 upgrade process in a subtle way that we will only discover when it is too late. This is what convinced me beyond doubt that the risk in options one and two is unacceptable.

@Mixologic in #13: wow, impressive foresight! 😲

The plan that came out of a meeting with key people that is documented in #14 is crystal clear. 👍The only clarification that I think would be helpful there is

we can hold off on the contrib semver part until #3005229: Provide optional support for using composer.json for dependency metadata

Does this mean we want to do #3005229 only in Drupal 9, not in Drupal 8? (#3005229 is currently marked for 8.8.x-dev.)

That's crystal clear. Thanks, @Mixologic!

Relating #3074998: Add explicit information about core compatibility to update data because currently information in the update XML doesn't explicitly specify which core version and update is compatible with.

Updated the issue summary to make it clear we chose option #3.

So, how are we executing our detailed strategy from #14?

1. 🏃‍♀️#2313917: Core version key in module's .info.yml doesn't respect core semantic versioning saw many more iterations, many refinements, many edge cases discovered and addressed. It's currently RTBC to get release manager and framework manager feedback wrt the name of the new key and to get release manager feedback wrt a potential 8.7 backport.
2. 🏃‍♀️#3005229: Provide optional support for using composer.json for dependency metadata is in progress and the patch @tedbow is currently working on and I am reviewing.
3. ✅ #3004459: [PP-1] Fold in dependency parsing wisdom from project_composer was closed.
4. ✅ #2641658: Module version dependency in .info.yml is ineffective for patch releases was closed.
5. ✅ #3066448: Harden test coverage for dependency Constraint class was committed.

🤦‍♂️

I only updated the grammar of the issue description.

Not sure if this is the right place to mention this, or if I should spin off a child issue, or what, but...

#3116581: Use core_version_requirement, not a version dependency on drupal:views

VBO made a 8.x-3.5 release with bold text on the release notes saying "this version of VBO will require Drupal core >= 8.8. If for some reason 8.7 or lower is used on your project - don't update."

However, it defines this requirement via:

core: 8.x
dependencies:
  - drupal:views (>=8.8)

Not:

core_version_requirement: ^8.8

So none of our tooling knows about it. :(

Do we care? Should we do anything to try to handle this case, or do we punt and say: "that way of defining core requirements is deprecated, go fish."?

Oh, except I searched and couldn't find any issue where we actually "deprecate" using dependencies: to define a specific version of a core module you need. So what VBO did is actually totally fine, and I don't believe that's been deprecated at all. :( Seems like a big gap in this whole plan, and I don't see it mentioned in this issue. In #10 @tedbow wants to deprecate the Drupal-specific range operators in dependencies:, but there's no talk of using dependencies to refer to core. I don't think we can deprecate this behavior, since not all core module are required, and contrib needs to be able to have dependencies on optional core modules. So I think we've got to keep supporting that. But maybe we can deprecate defining a core version that way?

Can the project_dependencies world notice this kind of thing and make sure that the <core_compatibility> in the release history XML feeds takes these sorts of core compatibility constraints into account, too? Basically, define <core_compatibility> as the most restrictive requirements from all of core, core_version_requirement and dependencies ?

Also, does anything stop contrib authors from getting this totally wrong and doing things like:

core_version_requirement: ^8.7 || ^9
dependencies:
  drupal:system (>=8.8)

? Not sure where we can/should try to notice that and prevent them from doing it. It'd sorta be nice to notice before they try to make a broken release, but I don't think that's possible. I know we've got code that makes sure you don't do core_version_requirement: ^8.5 or something, since 8.5.x core didn't support that key. But I don't think we've got anything that tries to notice conflicts / incompatibilities between core_version_requirement and dependencies.

Thoughts?

Thanks/sorry,
-Derek

Clarifying that this is about downstream modules and themes, not upstream dependencies.