Update from Yarn 1 to Yarn 4

Is it a must have to update to Yarn 2?

Given some community backlash and the fact that Yarn 1 isn't going to be abandoned anytime soon, I'd encourage to stick to Yarn 1 for now.

As a build tool it should be relatively easy to update at a later stage : )

(If really needed yarn set version can be used to specify a Yarn version on project level)

BTW reason I am asking is we have plenty to do for Drupal 9 (beta), and a dependency coming out with a new version does not and should not automatically mean we need to update. For example, Symfony 5 is very well out and we are not going to update to it due to support timelines. CKEditor 5 is out but we cannot update to it due to incompatibilities and ecosystem concerns, etc. So we need to look at a must have/need to scale as well as consider this in context of the timeline we are aiming for with Drupal 9 (both how soon we want the beta and how long we want to support Drupal 9).

Tagging as a requirement for Drupal 9.0.0-beta1.

I don't think we need to worry about this right now, the discussion in the yarn community is still very much ongoing on how this would be released (maybe even as a different binary). yarn 1 isn't going to disappear and will still receive security updates.

Should we consider splitting this into two: a) make a decision on what happens for Drupal 9, just so there can be a decision on what happens with Yarn for D9 and be able to close that issue, and b) make a plan for Yarn 2? We could rename this one to "Determine Yarn plan for Drupal 9" and create a separate one for Yarn 2?

#9:

I do still think that at minimum we have to ensure that Drupal either works with Yarn 2 when it's released, or that we have a proper warning in place for users who try to use Yarn 2 with Drupal.

Another option comes to mind, which is enforcing the use of Yarn 1 and providing that version via Core. I believe this should be possible with something like yarn policies set-version '^1', but haven't yet validated this as I'm having trouble getting Yarn 2 working in my dev environment. I'm going to try this out on a VM and report back, provided someone doesn't comment here explaining why enforcing Yarn 1 is not a good option.

gonna review it.

1) Added per project installed yarn 2. Almost all works with some build changes.
It is possible to use both Yarn 1 and 2 in same time. In this patch Yarn2 force enabled. To use Yarn1 rename .yarnrc.yml into anything
Details: https://yarnpkg.com/getting-started/install#per-project-install

2) I grouped build\watch\log script files into `/core/scripts/common/`
Because they are equal for both postcss/js and needed for same filesystem operations.
and `yarn build:css & yarn build:js & yarn build:jqueryui` didn't worked after update to yarn 2.
compile css and compile js tasks are still individual and extendable.

3) Only Nightwatch task still broken, not sure how to check binary file yet.
Because in yarn cache it looks like `/core/.yarn/$$virtual/nightwatch-virtual-5ae4fbd7b4/0/cache/nightwatch-npm-1.3.4-277dc0a684-2.zip/node_modules/nightwatch/bin/nightwatch`
Maybe we need to use https://nightwatchjs.org/guide/running-tests/#programmatic-api which was added `Since v1.0, Nightwatch`
Current nightwatch script was added before 1.0 was released

4) Keeped jquery UI build still same because there is no need to group it. I think it will be removed one day.

5) Added `eslint-import-resolver-node` because of fail in eslint tasks

6) removed "stylelint-checkstyle-formatter" because of fail in stylelint.
Changed this task output to `verbose` own stylelint.
This plugin seems very outdated.
https://plugins.jenkins.io/checkstyle/ is deprecated anyway.

7) Removed "stylelint-no-browser-hacks" for same reasons.

Today during the committer meeting we discussed this and agreed that, given that yarn 1 is no longer being marked legacy, we will stick with yarn 1 (rather than yarn 2 or npm) in 8.9/9.0.

But we can still take some part of this patch related to optimisation of build scripts.

Tagging for D10 consideration: https://yarnpkg.com/getting-started/migration#why-should-you-migrate

This wasn't in the correct issue tree so it got overlooked. Do we need to start looking into this?

Yarn 3.2 is out and in that post they're saying that they're now working towards Yarn 4.

farbtastic and joyride prevent us from using a newer version as far as i can see.

We can remove farbtastic if we do either #1651344: Use color input type in the color.module or #2808151: [policy] Move the Color module to a contributed project when Bartik is deprecated.

For joyride, #3238507: Drupal 10 JavaScript dependency plan says "Remove from Drupal 10... Needs issue". Is that true? Does Tour module not use joyride anymore? If so, let's get that removal issue opened and started.

Moving to 10.0.x on the assumption that #24 prevent us from doing it in 9.4.

Also, retitling for what I think is more accurate. The "and" in the issue title might be a signal that we should split this into two issues :)

followed the steps in https://yarnpkg.com/getting-started/migration#step-by-step for now. This is still using the node_modules folder. So it's not using the Zero-Install feature.

Yarn 1 is in maintenance mode only since 2020, and while there is no date for the end of support, the next step for yarn 1 is end of support so let's get ready before that.

Note that the test failure was a known random, but I can't requeue tests because there's a merge conflict.

using the new yarn version means we're committing the yarn cli tool (in core/.yarn/releases/yarn-3.2.0.cjs) which is about 2.5MB of JS. might need to exclude that from the release tarball or something maybe?

drupal ci uses yarn 1.x in the setup steps before the commit check, will need to be updated at some point.

Tried to exclude the binary from the repo, but that makes yarn crash. We do have to keep that in the source.

The title talks about Yarn 3 but the IS only mentions upgrading to Yarn 2.

Adding our starting point (Yarn 1) to the issue title for clarity. Also, removing the "prepare for Yarn 4" part, since that seems like it should be a separate issue if/when we want to do that, and Yarn 4 doesn't look particularly close (https://github.com/yarnpkg/berry/issues/3591 still has a bunch of empty checkboxes).

Am I reading the MR correctly that it removes the Farbtastic library (per #24), but not the Color module, and yet Color module tests still pass even with the Farbtastic library removed?

If so, is that an indication that we're missing some important test coverage in Color module?

Updated the issue summary. Setting status to "Needs work" because the merge request needs a rebase.

Also raising the priority to Major and tagging as a should-have for beta. In #3118149-74: [meta] Requirements for tagging Drupal 10.0.0-beta1, I added it to the must-have list, but I don't know if that's accurate, so tagging here as a should-have to surface the uncertainty, and then we can make the two match once we decide which it is.

Discussed in #d10readiness meeting the result being this is moved to D11, #3118149-89: [meta] Requirements for tagging Drupal 10.0.0-beta1

Reparenting to the D11 roadmap,and updating tags.

we don't need to checkin the binary anymore thanks to corepack

The preferred way to manage Yarn is through Corepack, a new binary shipped with all Node.js releases starting from 16.10. It acts as an intermediary between you and Yarn, and lets you use different package manager versions across multiple projects without having to check-in the Yarn binary anymore.

bit too optimistic there, binary still around :/

Ok, we need to wait for yarn 4 to not have the binary checked in, so postponing on that.

https://github.com/yarnpkg/berry/issues/3591

Yarn 4 in rc. So we may test it in 11.x

Seems updating caused some test errors.

Looks like yarn fixed in Drupal ci

The IS needs a (hopefully minor) update to reflect the current status of affairs. Yarn 4 is now fully released.

It can be reviewed and tested manually because pipeline failure related to old version of yarn used in gitlab CI
#49

Do we really have to ship the code for yarn inside the core directory? Can we instead rely on the user installing yarn themselves? Many sites won't need to run it at all.

We also need to make the GitLab CI scripts (which also live in the repo) use the new version.

Scripts are in repo but Docker templates outside.
https://git.drupalcode.org/project/drupalci_environments/-/blob/dev/dock...

I don't think we need to run extra scripts in CI container in this ticket.

Regarding the pipeline failure: I ran into this same issue in trying to use Yarn 3.x and 4.x for the current 2.x revamp of the RefreshLess module (see #3393105: Hotwire Turbo minimum viable implementation if you're interested by the way); here's our current .gitlab-ci.yml that took me a lot of swearing and late hours last night to finally get working; it's well documented so should explain everything.

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Removed mentions of color module from IS since it was removed in 10.0.x. Also removed comment about waiting for Yarn 4 to be released since that is also old news.

RE #52

About yarn files which are stored in repo.

1. It allows us not to define which version of yarn should be installed. Should be only node image. Even with pre installed different version of yarn.

In current MR:

docker run -it --rm --name yarn -v "$PWD":/usr/src/app -w /usr/src/app node:20-alpine yarn && yarn build

Will work fine.

And we will don't care about this https://github.com/nodejs/docker-node/blob/62c2e3cfb17ba8d9167b0daebbff9...

2. About `Many sites won't need to run it at all.`
We can say same about https://git.drupalcode.org/project/drupal/-/tree/11.x/core/scripts/css?r...
or a lot of directories that are in core.

yarn.cjs is 2.6Mb. Those other scripts that you have spotted are about 1kb each.

#54 looks like it might have a solution where we can install Yarn purely inside GitLab CI?

I guess it will help ;)

CI mostly fine now

we need to remove --cwd from yarn commands and call them only form core dir
https://github.com/yarnpkg/berry/issues/4908

1. Spell-checking command should be configured.
2. .gitlab-ci/pipeline.yml Nightwatch command needs some love.

I really don't think that 2.6MB is critical.

Especially if we compare this with the opportunities it gives us

Regarding committing the yarn-x.x.x.cjs: It's what modern Yarn recommends to do so that you always have a working copy of Yarn. Yes, it's a couple of megabytes, but if it's only in the Git repository to be used for compiling assets and other dev tasks, it can be left out of releases.

The problem is that the release process will have to be modified to take that into account, at present this would be shipped with all future releases.

Asking a bigger question: what does yarn do for us that npm (or perhaps pnpm) does not? They do not depend on living in the repository, as far as I know.

I think this question has own discussion.

Finally brute forced :)

Adding this to review not in terms of code.
(Some DevOps gurus can make it better I'm sure)

But in terms of approach, seems we are ready to go.

Some js files changes included to MR because yarn linters not silent now ;)

Please review.

Why have the JS files been reformatted? IIRC we use prettier to format, so it should be the same if the locked version of prettier is unchanged? You say "yarn linters not silent now", was there a bug somewhere before?

@longwave
Yes for some reason linter failed.
https://git.drupalcode.org/issue/drupal-3109556/-/jobs/334117

I only wanted to pass all tests here. Js linting can be moved to own ticket.

all right, wanted to make sure we could use yarn with corepack only, it's simpler than #54 because we have a package.json with a packageManager entry.

Using yarn like this present some issues:

• Seems like what gets installed by corepack is not validated (or maybe setting the yarn version makes it ok, haven't look deep enough yet)

  Allow checking hashes for exact versions, and check them for default versions #37 and feat: download the latest version instead of a pinned one #134

• When using recommanded yarn init commands, yarnPath is still set to make sure things don't break for people. Meaning that for our users running corepack enable is necessary for the whole thing to work (as shown by the CI scripts needing it)

Not sure if we should go with this or with the executable committed to the repo, in any case it works as expected.

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Is it feasible to do corepack enable in the initial Yarn step and then pass whatever it does as an artifact to the child jobs?

Or we could just move it to a global before_script so it runs automatically on all jobs?

Or even, do it in the base images so it's always available?

I don't think we can pass whatever corepack enable does as an artifact. Probably a before script makes more sense, I don't have strong preference, whatever works is good with me. I'm more concern of existing devs not having this working. I think we need the yarnPath if an existing yarn exec already exists. Can someone test it?

@longwave:

Asking a bigger question: what does yarn do for us that npm (or perhaps pnpm) does not? They do not depend on living in the repository, as far as I know.

You're right that npm and pnpm don't rely on placing themselves in the repository like Yarn does. It took me a bit of time to adjust when moving from npm. The Yarn documentation does a pretty good job of detailing what it offers. One big difference is that, by default, it doesn't create a node_modules directory at all, but abstracts calls to packages, which are actually stored in archive files; again, the Yarn Plug'n'Play documentation has a lot of info about their architecture and the benefits of it, and of particular interest are performance and predictability, and of particular interest to CI:

Shared installs across disks

Related to the previous point, Yarn PnP allows to reuse the same package artifacts across all projects on the disk. Unlike pnpm, which uses a content-addressable store where each file from each package needs to be hardlinked into its final destination, the PnP loader directly references packages via their cache path, removing a lot of complexity.

what's a recommended way for testing this one?

Believe issue summary has been updated so removing that tag.

@smustgrave

Just need to be runned /core yarn scripts in "node": ">= 18.12" for first MR.

For second MR also should be https://nodejs.org/api/corepack.html enabled

So I applied MR 4698 and just clicked around. Performed some basic tasks like create a content type, view block, etc. Didn't notice anything that broke. Think this may have missed the 10.2 window but maybe get in early for 10.3?

I have only skimmed the Issue Summary. I see that there are two MR on this issue and the remaining tasks states that there is a failing test. However, the tests for the two MR are passing. The issue summary is out of date.

I am tagging this for an issue summary update. In that, include which MR is considered RTBC and then rebase it so it is up to date.

Thanks!

Rebased both MRs

Seems were some random pipeline failures. Now `both branches ready for review

Yet another approach
https://www.drupal.org/project/drupal/issues/3420041

Seems to have a merge block.

But with 11.x now open wonder if it would be good to try and get in.

It was easier to create new MR instead of rebase corepack branch.
I think we ready to go with this approach even experimental.

Keeping executable branch for example of approach for review.

I've added yet another MR for tests that js/css/yaml linters will triggered.
Seems all tasks triggered and passed well

Mind updating the issue summary with the option you’re going with. And maybe quick sentence for which MR should be reviewed. Or if any can be hidden.

Thanks! Really do think it would be good to get in to 11.x

Some scaffold error here. Not sure how to fix

Seems it is not mine
https://www.drupal.org/project/drupal/issues/3424644#comment-15490289

I think we need a change record that tells core developers what to do. I also wonder about the impact of working on 11.x and 10.x - is this going to make it harder?

@alexpott

git co 3109556-4-1-1-corepack && yarn -v
Already on '3109556-4-1-1-corepack'
Your branch is up to date with 'drupal-3109556/3109556-4-1-1-corepack'.
4.1.1

then

git co 11.x && yarn -v                  
Switched to branch '11.x'
Your branch is up to date with 'origin/11.x'.
1.22.21

Switch happends easy.

Was expecting to tag this for docs updates but we actually already tell developers to run corepack enable: https://www.drupal.org/about/core/policies/core-change-policies/frontend...

A change record is still worthwhile for those that already had Yarn 1 and haven't used corepack yet.

I've just swapped over to using corepack locally and now my core/package.json file changes whenever I run a command. That's really annoying. If we're telling people to use corepack great but we need to update package.json in 10.3 and 10.2 to have the "packageManager": "yarn@1.22" key.

Opened #3428122: Add packageManager key to core/package.json now we recommend corepack enable

can't commit this one but RTBC +1

Change record for review: https://www.drupal.org/node/3428571

CR reads well, tested on my macbook that it worked

Alright, time to land this. Tested locally back and forth between Yarn 1 and 4 and haven't run into any issues, and we backported the packageManager setting already to previous branches. The changes to .gitlab-ci.yml might make future CI changes trickier to backport, I think we should consider copying those to 10.3.x as well so we have corepack on both sides, but that can be done in a followup.

Committed ad10de3 and pushed to 11.x. Thanks!

Also published the change record.

Adding a release note snippet since the CR has important info for core contributors.