Problem/Motivation

As part of #3262573: Update our yarn dev dependencies to the extent allowed by current constraints we noticed that some of the production dependencies allow minor updates within the version constraints. This prevents installing patch updates without also installing minor updates.

Proposed resolution

Review version constraints for production yarn dependencies to ensure that within version constrains, only patch level updates can be installed.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

Forward branches:

Drupal core's yarn dependency constraints for production dependencies. The latest minor versions of all development dependencies are now required by the constraints. Additionally, the constraints have been changed to only allow patch-level updates. This allows yarn upgrades can be done easily and safely when there are security issues with the dependencies, without accidentally making disruptive updates to production dependencies. The constraints will be deliberately increased as necessary for future updates and future Drupal minor versions.

9.3.x:

Drupal core's yarn dependency constraints for production dependencies have been changed to only allow patch-level updates. This allows yarn upgrades can be done easily and safely when there are security issues with the dependencies, without accidentally making disruptive updates to production dependencies. The constraints will be deliberately increased as necessary for future updates and future Drupal minor versions.

Issue fork drupal-3266912

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

lauriii created an issue. See original summary.

Wim Leers’s picture

Assigned: Unassigned » Wim Leers
Issue tags: +JavaScript

AFAICT the scope of this issue is to change all entries in core/package.json to only allow patch-level changes.

We have two choices to do that:

  1. ^x.y.z to ~x.y.z
  2. ^x.y.z to ^x.y

Since core/composer.json does the latter, I think we want the same here.

Wim Leers’s picture

Assigned: Wim Leers » Unassigned
Status: Active » Needs review
nod_’s picture

Status: Needs review » Needs work

So the caret thing doesn't do what the IS is after.

"@babel/core": "^7.0" => this would make it possible to update to 7.17 without problem. I think what the IS is after is that we can only update to 7.0.1 with the version constraint.

What would work with regard to the IS is either: 7.0.x or ~7.0. It will introduce quite a bit more changes to packages.json since JS are pretty fast with releasing minor releases but should be safer.

xjm’s picture

Priority: Normal » Major

This is at least major; borderline critical because we need to update 9.3.x and 9.2.x for security updates again, and yarn upgrade on 9.3.x or 9.2.x introduces a number of minor version updates that we should avoid if possible.

mmjvb’s picture

Indeed, option 2 is irrelevant. Only option 1 allows for patch level updates (stay within major.minor (x.y.*)).

Same thing for ~x.y, which is equivalent to ^x.y allowing you to stay within major (x.*.*).

Suggest to only use ~ for patch level updates (~x.y.z), use ^ for minor updates as well as patch level (^x.y.z)

nod_’s picture

Status: Needs work » Needs review

went for the 1.2.x format, pretty self explanatory compared to ~ and at least we don't "lie" on the patch version number with what's in the package.json and what is actually installed.

Was on the fence about updating all of the things, i think i'll put them back to previous minors, they probably deserve their own issues.

xjm’s picture

One thing we should make sure of is that we increase the minor constraint to at least whatever is in the lock file for each branch. 10.0, 9.5, and 9.4 should get the latest minor. 9.3 and 9.2 should get the current minor in the package .json. This might result in minor downgrades in the lockfile for some dependencies, but probably that is less disruptive than minor updates being a hard requirement in the production branches.

It looks like the latest MR is already addressing that at least for Babel but we should make sure on a case by case basis that we do so everywhere.

Don't we need to rebuild all of the assets also?

nod_’s picture

nod_’s picture

updated the MR and patch to only update production dependencies

nod_’s picture

didn't touch at "@popperjs/core": "^2.9.2 <2.11.0", don't remember what that was about.

lauriii’s picture

I thought that was added to prevent updating to the next minor. I was able to track where that was added: https://www.drupal.org/project/drupal/issues/3262573#comment-14426410. Based on that, we could update the constraint to 2.10.x but I think what we have there does the same so not a big deal.

Status: Needs review » Needs work

The last submitted patch, 12: core-3266912-13-9.3.x.patch, failed testing. View results

nod_’s picture

Status: Needs work » Needs review
FileSize
244.22 KB

update tests for 9.3.x

Wim Leers’s picture

I actually looked the syntax + meaning up before writing #2, and apparently still got it wrong! 😅 (It doesn't help that npm and composer handle ~ constraints differently 💩)

I do VERY strongly agree with:
went for the <code>1.2.x format, pretty self explanatory compared to ~

→ especially given the fact that npm and composer interpret these differently.

One thing I'm surprised to not see asked yet: why do we use a different kind of semver constraint for PHP dependencies than for JS dependencies? 🤔

lauriii’s picture

#16: I'm not a composer package expert but I assume that we have different requirements for our composer package vs npm package given that for us npm is supposedly an internal dev tool for core development, whereas we actually ship Drupal via composer. Drupal core itself is not shipped as a package on npm so the version constraints can be optimized for our internal workflows. In this case we are optimizing for being able to run yarn upgrade for patch releases without having to worry about accidentally updating to a new minor in a patch release.

Wim Leers’s picture

That seems worth documenting 🤓

(But not sure where, since JSON cannot contain comments.)

lauriii’s picture

Wim Leers’s picture

The changes in https://www.drupal.org/node/2996785/revisions/view/12615848/12634473 are fine, but they do not yet document the rationale behind the different semver constraint concerns.

lauriii’s picture

I'm happy to extend the documentation. I'm wondering what is the use case you have in mind where someone would need that information? I believe I had a different use case in mind which was updating dependencies. Just trying to figure out what is the best way to document that.

Wim Leers’s picture

I'm thinking about the next core contributor who passes by and starts wondering just like me why we handle upstream dependencies differently depending on whether they're PHP or JS ones.

I'm not certain https://www.drupal.org/about/core/policies/core-change-policies/frontend... is the best place for that. I don't know if there is a good place for that.

lauriii’s picture

I'm not sure it's 100% required for us to document that. There are many things that are different in JavaScript vs PHP and we are not documenting why they are different. 🤔 Either way I hope that won't block this issue since this is borderline critical and we are urgently trying to land this ASAP.

Wim Leers’s picture

Status: Needs review » Reviewed & tested by the community

Oh, definitely not saying that should be blocking! I just think that it's important for Drupal core security's sake that we clearly document the rationale for differing strategies in dependency management, to reduce the bus factor 😅

Speaking of which: I feel comfortable RTBC'ing this one, after having seen so many smart eyes on it. Let's land it, one less worry for shipping 9.4 then 😊

nod_’s picture

reroll for 10.x

Wim Leers’s picture

#25 is double the size of #15. Why? 🤔

lauriii’s picture

That patch is for 10.0.x and it updates Popper.js which the 9.3.x patch doesn't do since it's on the previous minor. The patch is similar in size to #11 which is the most recent 10.0.x patch.

  • lauriii committed a97433e on 10.0.x
    Issue #3266912 by nod_, Wim Leers, lauriii, xjm, mmjvb: Review version...

  • lauriii committed 5b047c3 on 9.4.x
    Issue #3266912 by nod_, Wim Leers, lauriii, xjm, mmjvb: Review version...

  • lauriii committed ad87092 on 9.5.x
    Issue #3266912 by nod_, Wim Leers, lauriii, xjm, mmjvb: Review version...

  • lauriii committed 9d0d480 on 9.3.x
    Issue #3266912 by nod_, Wim Leers, lauriii, xjm, mmjvb: Review version...

lauriii’s picture

Version: 9.4.x-dev » 9.3.x-dev
Status: Reviewed & tested by the community » Fixed

Committed a97433e and pushed to 10.0.x. Committed the MR to 9.5.x and 9.4.x and committed #15 to 9.3.x. Thanks!

xjm’s picture

Status: Fixed » Needs review
Issue tags: +10.0.0 release notes, +9.4.0 release notes, +9.3.13 release notes
+++ b/core/core.libraries.yml
@@ -7,10 +7,10 @@ internal.backbone:
   remote: https://github.com/jashkenas/backbone
-  version: "1.4.0"
+  version: "1.4.1"

@@ -1027,10 +1027,10 @@ picturefill:
 popperjs:
-  version: "2.11.2"
+  version: "2.11.5"

@@ -1078,10 +1078,10 @@ internal.underscore:
   remote: https://github.com/jashkenas/underscore
-  version: "1.13.2"
+  version: "1.13.3"

Just making note of the prod dep updates (10.0.x).

Meanwhile, I realized this did not get backported to 9.2.x. Hopefully the other issue didn't include accidental minor updates for that branch. (I did review for that, but the issue took all day so I might've missed somethig.) We should still consider backporting it.

xjm’s picture

9.5 and 9.4 have the same updates as #35. For 9.3.x it is is only Backbone and Underscore. (GitLab does not cp well.)

+  version: "1.4.1"
  license:
    name: MIT
    url: https://raw.githubusercontent.com/jashkenas/backbone/1.4.0/LICENSE
 
underscore:
  remote: https://github.com/jashkenas/underscore
-  version: "1.13.2"
+  version: "1.13.3"
lauriii’s picture

This issue is not backportable to 9.2.x because #3219088: Use package.json to manage third party JS libraries was not backported to 9.2.x. For the same reason there isn't a risk of doing accidental minor updates there.

Good catch on the release notes!

xjm’s picture

Status: Needs review » Fixed

Thanks @lauriii.

I diffed core/assets/vendor, core/package.json, and core/core.libraries.yml against 9.2.0 just to make sure there were no out-of-sync updates to the prod dependencies for 9.2.x in this or other commits.

I think we are safe not trying to backport anything further since 9.2.x only has 6 more weeks of support. We just need to be careful if there are any yarn dependency updates between now and then. Easy to forget @nod_'s cool update thing is 9.3.x+ only. :)

xjm’s picture

Release note worthy; belatedly tagging.

xjm’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.