Problem/Motivation

Today we have once, autocomplete that are either released or going to be as npm packages.

We have 2 general projects to track bugs host code, etc. the DA needs to do some custom configuration as far as gitlab is concerned to support our testing/npm publishing.

Proposed resolution

use a single general project making use of workspaces (either with yarn workspaces or lerna) like wordpress and many other do.

This would simplify d.o integration: only one project to configure for credit weight for the ranking, only one gitlab project to configure.

This would simplify greatly the maintenance of JS packages, only one queue to follow, code is in one place, same gitlab config for publishing all @drupal packages, etc.

Remaining tasks

Agree on:

  1. using a monorepo (with this issue, I'm assuming yes)
  2. What the name of the project (core_js, something else?)
  3. Yarn itself or lerna?

Comments

nod_ created an issue. See original summary.

nod_’s picture

nod_
Primary language French
Location Lille
commented
Related issues: +#3201195: Migrate from npm to yarn
justafish’s picture

justafish
she/her
Primary language English
Location London, UK
commented

1. +1
2. core-ui-js ? (to avoid confusion with https://www.npmjs.com/package/core-js)
3. yarn + lerna

nod_’s picture

nod_
Primary language French
Location Lille
commented

Since "core-core" is drupal, maybe "core-js" can be drupal-js(drupal_js is taken and only underscore allowed) drupal_javascript?

anything you can point to for the benefits of lerna over "vanilla" yarn workspaces? been doing some reading and still not clear what are the responsibilities of the different tools.

nod_’s picture

nod_
Primary language French
Location Lille
commented

From what I read yarn workspaces = dependencies deduplication, lerna = tools for facilitating packaging/publishing of actual packages.

https://doppelmutzi.github.io/monorepo-lerna-yarn-workspaces/ is pretty good for an overview.

droplet’s picture

droplet commented

Why not just put it back to Drupal-CORE? You can publish packages from CORE also

- It's already no github.com like a portal for marketing, easy contribution
- The repo almost controlled by the same DrupalCore maintainers (100% now)
- most things still on d.org. Gitlab is almost useless for normal contributors

The only Pros is diff Testing system?

Just make a new rule for CORE commit. Package maintainers can self-review their own stuff. That's all done.

nod_’s picture

nod_
Primary language French
Location Lille
commented

I think there are also security concerns with putting that in the existing core repo. If it's a different repo it's easier to use more exotic packages than if we were in core, and potentially be flexible with access (to add more people, not saying it's planned but it would be easier if we wanted to).

Making it a separate repository means we'll be able to adopt gitlab features faster than current core can. Like if gitlab issues are turned on, we wouldn't have 10+ years of history to handle and we can adopt that much faster, same goes for static pages for some of the documentation/examples, etc.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

nod_’s picture

nod_
Primary language French
Location Lille
commented

so far we're looking into getting the drupal_js namespace on d.o to start the monorepo with learna+yarn workspaces.

@droplet, it's the testing system, it's also avoiding a lot of drupal core "history", relaxing commit rules and so on is just not possible right now in drupal core. It's at least an option in a repo where the code is less "mission critical" and releases are automated.

justafish’s picture

justafish
she/her
Primary language English
Location London, UK
commented

I've opened a request in the site moderators issue queue https://www.drupal.org/project/site_moderators/issues/3217884

nod_’s picture

nod_
Primary language French
Location Lille
commented

Test over at https://github.com/theodoreb/drupal_js_monorepo_test

@droplet: getting a separate repo means we'll be able to try using https://www.npmjs.com/package/commitizen and conventional commits for js packages to automate a few things :D

nod_’s picture

nod_
Primary language French
Location Lille
commented

We have the drupal_js namespace on d.o need to transform it into a general project.

nod_’s picture

nod_
Primary language French
Location Lille
commented
Related issues: +#3221263: [policy] Monorepo packages release management, +#3221261: [policy] Decide what goes in the monorepo, +#3220688: [policy] Use conventional commit format
drumm’s picture

drumm
he/him
Location NY, US
commented
Related issues: +#3220446: Change project 'drupal_js' into a general project

We have the drupal_js namespace on d.o need to transform it into a general project.

This was done #3220446: Change project 'drupal_js' into a general project

xjm’s picture

xjm
she/her
Primary language English
commented

Would the monorepo be used for the upstream-of-core projects only (once, a11y_autocomplete) or also the downstream ones (theoretically like decoupled_menus_parser)?

Edited for incorrect project name.

nod_’s picture

nod_
Primary language French
Location Lille
commented

I would think everything that is maintained by core drupal either up or downstream.

lauriii’s picture

lauriii
he/him
Primary language Finnish
Location Finland
commented

I personally like using monorepos but I think there are some downsides in them. I'm also wondering how much of a burden the current situation is where code is split into multiple repositories. It's possible to view the issue queues in a single search https://www.drupal.org/project/issues/search?projects=once.js%2C+Accessi.... Small repositories make it easier for people to contribute since they don't have to worry about other packages. As an infrequent contributor to some projects, I've personally found it much harder to contribute to projects that are in a monorepo due to the cognitive overhead.

Since we don't have many dependencies between our packages, some of the overhead on package management and build tooling in monorepo might be unwarranted. Also there are some limitations to versioning because using branches become more complex. It seems like that is being discussed in #3221263: [policy] Monorepo packages release management.

The fact that all packages in a monorepo are required to move at the same speed can cause some issues when the monorepo grows. For example, if you want to update the node version, it needs to happen at once in all of the packages. It’s not a big problem now when we don’t have that many packages. However, when the number of the packages increase, and there’s more maintainers, it might cause some issues. For example, if there’s one person mainly maintaining one of the packages, but they are not available to help, it might prevent us from updating rest of the packages while we are waiting for their help.

We need to also consider the impact on security releases. One downside is that when there's a security release, we'd need to freeze commits to all packages in the monorepo.

Have we considered adding committers to individual packages? If we move packages to a monorepo, it would be something we couldn't do anymore.

justafish’s picture

justafish
she/her
Primary language English
Location London, UK
commented

we'd need to freeze commits to all packages in the monorepo

I don't believe that would be the case but maybe I'm misunderstanding - could you elaborate?

Have we considered adding committers to individual packages? If we move packages to a monorepo, it would be something we couldn't do anymore.

We could use codeowners for this in GitLab

nod_’s picture

nod_
Primary language French
Location Lille
commented

In my mind I already replied, didn't make it to the internet though.

Small repositories make it easier for people to contribute since they don't have to worry about other packages. As an infrequent contributor to some projects, I've personally found it much harder to contribute to projects that are in a monorepo due to the cognitive overhead.

The maintenance overhead is pretty significant though. We only have once and a11y_autocomplete and I'm already over having to swich all the time between the two codebase/queue. it's only going to get worst if we implement everything in #3221261: [policy] Decide what goes in the monorepo.

What was the cause of the cognitive overhead specifically? I'm pretty sure we can address some of the issues.

Since we don't have many dependencies between our packages

That is true for now but once we publish eslint, prettier, stylelint, browserlist, etc. config we're going to have a few deps between our pakcages.

The fact that all packages in a monorepo are required to move at the same speed can cause some issues when the monorepo grows

Not sure that is the case, we don't plan on using the same version for all packages (like babel is doing), each package has it's own version and we can update it when we want.

We need to also consider the impact on security releases. One downside is that when there's a security release, we'd need to freeze commits to all packages in the monorepo.

That is the current core policy but in our case we'd need to freeze commits that touch the package in question, not everything. We'll need to update some core guidelines but don't think it's a showstopper.

Like Sally said, we can leverage the codeowner feature to give more access to subsections of the repo.

lauriii’s picture

lauriii
he/him
Primary language Finnish
Location Finland
commented

What was the cause of the cognitive overhead specifically?

I think the cause for the cognitive overhead is the additional tooling required for managing monorepos, and understanding how changes are managed in a monorepo. I don't think I have any very specific examples but I feel like overall I've had hard time contributing to projects using monorepo due to its complexity. I think this is something that could be tackled by providing documentation, but it will certainly take conscious effort to do that.

That is true for now but once we publish eslint, prettier, stylelint, browserlist, etc. config we're going to have a few deps between our pakcages.

Makes sense! 👍

Not sure that is the case, we don't plan on using the same version for all packages (like babel is doing), each package has it's own version and we can update it when we want.

But for example if we are updating to Webpack 4 and we have multiple packages utilizing it. Wouldn't we have one issue that will upgrade all of the packages using Webpack at one issue so that we don't need to depend on multiple versions of Webpack?

That is the current core policy but in our case we'd need to freeze commits that touch the package in question, not everything. We'll need to update some core guidelines but don't think it's a showstopper.

How is that different from core? If there's a security release to one of the core modules, all of core is frozen at the time of a security release. Maybe that's something that could be changed, but it would be useful to have someone from the security team provide input on that.

justafish’s picture

justafish
she/her
Primary language English
Location London, UK
commented

all of core is frozen at the time of a security release

That's a consequence of core's branching policies rather than a technical limitation, which we can approach differently in a new repo

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.