Problem/Motivation

There is no way to easily manage frontend libraries that are related to Drupal but not dependent on Drupal, such as the new vanilajs once, the jQuery UI fork, or all the work that is coming from the js-menu-component initiative.

The best we have today is the jquery-once library maintained by RobLoach on Github:

  • It's made for Drupal by Drupal folks, but doesn't need Drupal to be useful
  • It is well maintained, and kept up to date when necessary
  • The npm package only ship necessary files (no test files, github-specific files, etc.)
  • There is exhaustive test coverage
  • There is documentation shipped with the code
  • It is published to NPM, and various cdns
  • There is a typescript type declaration (nice for IDE integration)
  • There are commands to build the code, docs, test
  • Tests are run with Github actions so testsuite can be launched before merging a PR
  • The resulting file is actually minified (the generated core JS is not minified today)
  • The release cycle is not tied to Drupal's

And while it's pretty extensive and RobLoach has been in the community for a long time, because of the way we integrate with the jquery-once lib, Third party library risks applies: maintainer availability, security fixes, release schedule, etc.

Proposed resolution

Create an official space in github to "host" JS libraries produced by Drupal that does not have a strong dependency on Drupal. Essentially creating a space for "second party" code that is loosely coupled to the Drupal community:

  • Project/code overseen by Drupal maintainers
  • people contributing have to follow the Drupal community guidelines
  • Follows a set of requirements (essentially the JS gates). I'd start with:
    • GPL and/or MIT code (to be defined)
    • There must be a comprehensive test suite
    • Code in all Drupal supported browsers (can rely on polyfill but don't ship with them)
    • If there is a UI, it must be accessible (for an agreed upon definition of accessible)
    • Documentation must be exhaustive and up to date (in code, probably generated from jsdoc, not on a wiki page in github)
    • Require a few commands to be defined and working: build, docs, test regardless of what is used to perform those actions
    • Drupal maintainers/committers are the only ones that can push new releases to npm
    • (a bonus: some metadata that makes generating the Drupal libraries files easier)

The expectation is that a project manages it's own lifecycle and tooling, the "contract" is to ship quality code ready to use. For example a project can choose the testing framework that fits with the library, issue queue, the labels, tags, and everything else that is relevant. Drupal then uses the result as an external library within the core codebase.

Benefits

  • Much easier to contribute to libraries. No "drupal" overhead, just the standard github experience
  • Almost all JS tooling assumes code is hosted on github, easier to use existing tooling without drupal specific tweaks
  • This would solve several problems related to JS library testing that we have currently. Nightwatch or the JS test methods we have today are not well suited for standalone libraries.
  • Github is where most JS developers already are, more people to find and fix JS bugs
  • Release cycle of libraries are not tied to Drupal release cycle, JS in general moves faster than core (for better or worst), making those packages (possibly) more useful to client projects outside of core
  • There is less of a need to audit development dependencies (within reason of course), core will only make use of the generated result so security issues in the testing framework will not impact core for example.
  • Almost a detail: we would ship with more minified code

Questions/risks

  • Create a "parallel" community and fragment maintenance: True in part, today we don't have a strong JS community in core, we tried for years to get more interest but it's just not happening with the current situation. JS folks will just not comme to drupal.org create an account and read how to make a patch, merge request, find things in our issue queue to fix a bug.
  • We will have many different tools to do one thing (different test tools, build pacages): will be necessary for some libs, not enforcing a tool doesn't mean we can's suggest one "by default"
  • Credit system is less good on github: while it's true, it is not worst than on other github
  • Depend on github infrastructure: we can set up a mirror from github to d.o projects and manage the npm part from drupal.org so that if github goes down, we can still push new code.
  • Security coverage: level of integration to existing security process to be defined
  • Community moderation: level of integration to the existing CWG to be discussed (how to deal with people breaking community guidelines within the github projects, etc.) might be a big one to figure out, more exposure => more people

Remaining tasks

  1. Agree this is a good idea
  2. Try it out on a library once
  3. Define a set of policy recommendations in this Google doc
  4. Update the doc with the new core leadership team names, for instance, change BDFL
  5. Clarify which guide the doc will be in in the core policies
  6. Agree upon the recommendations made above and canonicalize them here: https://www.drupal.org/about/core/policies

Changes

Ideally we'll find a way to require third party libraries with yarn/npm within core to make updating them easier, and generate the libraries definitions for second/third party libraries automatically.

Release notes snippet

Comments

nod_ created an issue. See original summary.

nod_’s picture

Issue summary: View changes
gabesullice’s picture

Title: Publishing / Maintaining JS libraries produced by Drupal that do not have a dependency on Drupal » [policy, no patch] Publishing / Maintaining JS libraries produced by Drupal that do not have a dependency on Drupal
Issue tags: +policy
zrpnr’s picture

+1 to this idea! Having an "official" Drupal Github space would be an excellent step for hosting source code that shouldn't need to be in core.
Not only would it allow us to standardize on the expectations for a Drupal flavored javascript project, but it could open up future integrations with d.o such as links to project pages, maybe even some form of issue credits, and allow for the possibility of a core "ecosystem" - projects that don't live directly in core but extend it.

Suggestions for existing libraries to try this with:

There are future / in progress projects which would benefit from this organization as well:

  • CKEditor5 - there's discussion and a prototype in #2966864: Add optional support for CKEditor 5 in D9 so we can remove CKE 4 from Drupal 10 but because it requires a build step we would need to consider having the source files for just the "build" in core. Putting that build on github would be cleaner and allow for faster iterations.
  • JS Menu component - there will likely be several smaller js projects which won't even necessarily be used directly in core, having an "official" home would be a big benefit to give these projects legitimacy and have a single place for developers to find them.
nod_’s picture

The once patch is hold up on library tests, so I think it'll be a requirement for the code and tests to be put on github and have at least as much as jquery-once provides before we can get it in core in a "third party" kinda way.

Very good point about CKEditor, might make is easier for people to customize the build while also keeping the build step out of core. We'll have to find a way to allow this type of override somehow.

And I think that will be a way for contib to provide 'prebuilt' version of library dependencies, I'm thinking of the workbox lib for my PWA module for example. It's not written anywhere than it's only for core that we can use this, but it's a different discussion, next step :)

gabesullice’s picture

Create an official space in github to "host" JS libraries produced by Drupal

I believe the DA has ownership of the drupal GitHub organization.

people contributing have to follow the Drupal community guidelines

GitHub has a norm of putting this sort of thing in a file named CODE_OF_CONDUCT.md. We could link directly to the Drupal Code of Conduct from that file as I've done here.

Documentation must be exhaustive and up to date (in code, probably generated from jsdoc, not on a wiki page in github)

💯

Require a few commands to be defined and working: build, docs, test regardless of what is used to perform those actions

❤️ this.

Drupal maintainers/committers are the only ones that can push new releases to npm

+1 to the restriction on who can commit. I think this is vital to the stability and "official-ness" of these projects.

Do you mean Drupal maintainers AND committers or do you mean maintainers OR comitters?

IMO, I think only committers should have push access, but I think we should consider a small expansion the number of Drupal committers.

The expectation is that a project manages it's own lifecycle and tooling, the "contract" is to ship quality code ready to use.

I think I agree with this, but "manages its own lifecycle" is a bit concerning. I don't think we want these libraries to increment 3 major versions within a single major version of Drupal—unless we clearly document what constitutes the public JS APIs (i.e. there aren't any) such that Drupal can upgrade to newer major versions of these libraries in minor versions of Drupal.

I see that you later you say:

Release cycle of libraries are not tied to Drupal release cycle, JS in general moves faster than core (for better or worst), making those packages (possibly) more useful to client projects outside of core

and it does make sense that we should be able to release more quickly than say, once a month, but that should only be applicable to minor and patch releases.

Perhaps there's a middle ground whereby these projects promise to maintain a particular major version as an LTS branch? It's really only the major versions that concern me.

There is less of a need to audit development dependencies (within reason of course), core will only make use of the generated result so security issues in the testing framework will not impact core for example.

Frankly, I think we should maintain a loose no-3rd-party dependency policy (excluding dev deps). I.e. no dependencies outside the Drupal org except in well-justified circumstances. We could start with this policy and relax it if it turns out to be unsustainable.

Almost a detail: we would ship with more minified code

I don't understand this. Can you explain why this is a benefit?

We will have many different tools to do one thing (different test tools, build pacages): will be necessary for some libs, not enforcing a tool doesn't mean we can's suggest one "by default"

Did you consider a JS mono-repo? If you did, what made you decide against it? I say that with a lot of trepidation because I don't want to derail this conversation with a bikeshed.

nod_’s picture

Do you mean Drupal maintainers AND committers or do you mean maintainers OR comitters?
IMO, I think only committers should have push access, but I think we should consider a small expansion the number of Drupal committers.

Sort of vague on purpose :p Like you said, committers only for NPM release + increase number of committers to take up the additional workload.

I think I agree with this, but "manages its own lifecycle" is a bit concerning. I don't think we want these libraries to increment 3 major versions within a single major version of Drupal—unless we clearly document what constitutes the public JS APIs (i.e. there aren't any) such that Drupal can upgrade to newer major versions of these libraries in minor versions of Drupal.

Since those lib are standalone the "exhaustive documentation" would give a clear definition of the public API of the library, whatever you document for other people to use is the public API (unless you tag something as @private and document just for reference).

As you said the issue is with the support of the version used by core. Since core committers will be involved in publishing new releases (major or minor) it'll be their job to figure this out. I'd go for a LTS version for the one that is used by core as well.

Frankly, I think we should maintain a loose no-3rd-party dependency policy (excluding dev deps). I.e. no dependencies outside the Drupal org except in well-justified circumstances. We could start with this policy and relax it if it turns out to be unsustainable.

I agree with that: standalone vanilla-js libs only to start with and see how it goes.

> Almost a detail: we would ship with more minified code
I don't understand this. Can you explain why this is a benefit?

today all the *.js files are not minified, if we move some code outside, we can provide *.min.js files for core to use. Like the once patch in the other issue, as it is today everyone would download the "verbose" version of the script while it'd be better for everyone to use a minified version.


Did you consider a JS mono-repo? If you did, what made you decide against it? I say that with a lot of trepidation because I don't want to derail this conversation with a bikeshed.

I did for 5 seconds :p for one thing it's not that widespread, it adds another layer of tooling just to manage unrelated vanilla-js libs. I like to have as little tooling layers as possible to get the job done. With 1 lib = 1 repo it's boring and reliable, most people know what to expect: fork, npm install, patch, npm test, send a PR. I don't see how it would make things better for us overall, it might help lib maintainers a little (and even that is debatable) at a cost to everyone else. Also I don't think we're good enough at JS stuff to manage a monorepo properly right away.

Might revisit later when we have just one test framework for most libs, more than 4/5 libs to manage, and too much friction with the 1 lib = 1 repo setup.

bnjmnm’s picture

I discussed this with @nod_ shortly before this issue was posted. @zrpnr in #4 already brought the main points I was planning to make in support of this proposal. But a few additional benefits come to mind that I think are worth mentioning.

  • Without this sort of policy, adding sufficient test coverage would likely require the addition of many new JS dependencies. An example of how unwieldy this can get can be seen in this agnonizing dependency evaluation of over 60 items for the jQueryUI fork tests. Drupal doesn't need the burden of additional test-only dependencies for a single library, and no contributor should have to create a 60 item table listing them.
  • Earlier it was mentioned that this could "make those packages (possibly) more useful to client projects outside of core". There are specific issues that could potentially benefit from this a great deal. There are several issues, such as #2346973: Improve usability, accessibility, and scalability of long select lists where we are searching for UI libraries and are having difficulty finding ones that meet Drupal's accessibility standards. The option of creating a new library that meets our requirements is more feasible with the policy suggested here. Drupal is far from the only project looking for multiselects/datepickers/etc, there is legitimate demand for this. If the UI libraries developed for Drupal's needs are not inherently tied to Drupal (and D.O.), these libraries can benefit from a much wider range of contributors, and boost overall awareness of the project.

Two small things regarding the policy come to mind:

  • Would "second party" libraries benefit from a change to the directory they live in, to distinguish them from third party ones? something like core/assets/vendor/drupal/<library dir>? A specific place this could help is to provide an additional cue that jquery.ui is a custom fork and not the main libray
  • Similarly, would this benefit from introducing a documentation standard, perhaps something in the opening @fileblock that makes it evident that this project is under the Drupal umbrella, despite not being something that should be altered within core?
gabesullice’s picture

Sort of vague on purpose :p Like you said, committers only for NPM release + increase number of committers to take up the additional workload.

👍. I'm curious what @lauriii thinks would be a good policy suggestion to make regarding this.

whatever you document for other people to use is the public API (unless you tag something as @private and document just for reference).

I assume this is reference to https://jsdoc.app/ and #2182153: [Meta] Document Drupal JavaScript using JSDoc. Let's be sure to include this in each project's standard CONTRIBUTING.md file. Idea: once we have a few examples, perhaps we should make a template for new libs.

I agree with that: standalone vanilla-js libs only to start with and see how it goes.

👍

today all the *.js files are not minified, if we move some code outside, we can provide *.min.js files for core to use. …

Thanks for the explanation.

… monorepo …

Thanks for the reasoning. This satisfies my curiosity. I like the idea of reconsidering this after we have a few libs to manage.

For context, my thoughts were not very well collected when I brought this up. I was thinking that it would be nice to share a single .prettierrc file between all the projects, for example. It might also make it easier to manage evolution of the libs when they have interdependencies. But, like I said, I like the idea of reconsidering this later after we have some experience with these thing. A monorepo feels like and early optimization for problems we may not actually have.

Drupal doesn't need the burden of additional test-only dependencies for a single library, and no contributor should have to create a 60 item table listing them.

😱

Drupal is far from the only project looking for multiselects/datepickers/etc, there is legitimate demand for this.

This is a wonderful example. Providing high-quality, open source libs for these things will probably do more for Drupal's penetration into the JS developer community than any other thing we've tried!

Similarly, would this benefit from introducing a documentation standard, perhaps something in the opening @fileblock that makes it evident that this project is under the Drupal umbrella, despite not being something that should be altered within core?

I must be lacking some context because I'm not exactly sure what this is suggesting. @bnjmnm, can you expand on this a little bit? I like the idea of a documentation standard, do we already have a similar standard somewhere for PHP (or even JS) that we could draw from?


The remaining tasks at this point are:

Remaining tasks

  1. Agree this is a good idea
  2. Try it out on a library
  3. Define the policy in details, find answers to the questions above and
    • which licence are allowed? GPLv2, MIT only?
    • Mirror code from github to drupal.org
    • Are we building the package published to npm from the code in gitlab or the one in drupal.org, etc.
  1. Who do we need to get agreement from? I think everyone who has responded thinks this is a good idea. Are we done with this step?
  2. I think the consensus is that the "once" library is the obvious first candidate for such a library. Agreed? If so, what is a concrete step we can take to "try it out"? Do we want to do this using the drupal GH org? If so, we need someone from the DA to help us get a project created. If not, maybe we should create an x-drupal github org (or reuse the jsdrupal org) and pretend we're using the drupal or for all intents and purposes. WDYT?
  3. It seems like this is blocked on #1, above.
nod_’s picture

For 1. we need core committer (especially release managers) to have a look and see if they agree with the proposal. I'd like to have the thoughts of the people involved with the modernize initiative too because if I'm not mistaken, that's pretty much what they've been asking for this whole time.

2. We can put it wherever doesn't really matter for now, i'm going to put some things in here: https://github.com/theodoreb/once and it can be moved in the proper namespace later

On the configuration sharing I need to update eslint-config-drupal with the latest stuff, that'd be useful for contrib as well. updated

nod_’s picture

That would also allow us to clean up the scripts/js folder in a way that makes what we have in core useful for contrib modules. Starting with the eslint config.

droplet’s picture

Doesn't drupal.org move to gitlab (self-hosted) soon? Rejected?

I'm leaving Drupal contribution because the slow commit rate and unfair treatment. (In another word: waste my time.)

Drupal maintainers/committers are the only ones that can push new releases to npm

If I don't own my code, I can't see how can I keep my productivity at contribution. Most great JS libraries are opinionated code. It's developed and maintained by a single person usually.

It's only benefit for current passion contributors + sub-maintainers. They can skip the commiter's approval now. Less reroll :s

nod_’s picture

If I don't own my code, I can't see how can I keep my productivity at contribution. Most great JS libraries are opinionated code. It's developed and maintained by a single person usually.

Agreed, the jquery-once lib is a good example. My assumption was to relax some rules around commit access on those libraries, essentially having the "maintainer" status for the libraries mean something more than adding access to a couple of fields in the issue queue.

gabesullice’s picture

we need core committer (especially release managers) to have a look and see if they agree with the proposal.

I will ping @catch and @xjm in their capacity as release managers. Also @lauriii and @larowlan since they most often interact with JS issues.

I'd like to have the thoughts of the people involved with the modernize initiative too because if I'm not mistaken, that's pretty much what they've been asking for this whole time.

I will also ping @webchick as a committer and as a PM for the JS modernization initiative. Finally, @justafish as a coordinator of that initiative and as a JS maintainer.

drumm’s picture

Doesn't drupal.org move to gitlab (self-hosted) soon? Rejected?

Drupal.org has been using self-hosted GitLab for over a year now. Per-issue forks for workspaces, #2488266: [META] Improve Git workflow on Drupal.org by implementing issue workspaces and merge requests, #2205815: Merge requests in Drupal.org issue queues?, are in opt-in beta testing right now, and we plan to enable them for all issues on all projects on November 10.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

e0ipso’s picture

Cross posting from Slack

Other stacks and frameworks rely on independently developed components. Many projects even produce them so they are used cross-frameworks. They are typically hosted in whatever code hosting platform the people working on them feel more comfortable with. Then the project using them grabs those from a registry (npm, packagist, ...). This has the benefit to:

  • Allow independent code hosting, ticketing, workflow, CI, etc.
  • Provide independent governance.
  • Allow different progress speeds.
  • Attract devs from outside the parent project (realistically this is not ofthen the case).

Is the main concern the impact on governance?

hestenet’s picture

Version: 9.2.x-dev » 9.1.x-dev

Not speaking on behalf of the core maintainers, but knowing some of their concerns from prior discussions about Drupal.org tooling - I would say that the concerns are, yes, about governance, but also about fragmentation of effort/contribution.

Here's what I mean - for a project hosted on Drupal.org:

  • The maintainer has registered an account on d.o (we have contact info)
  • The maintainer has agreed to the Code of Conduct, Terms of Service, and Git Access Agreement
  • Drupal.org maintainers can handle a mainatainership migration should the code be abandoned
  • Issues are part of the same queue (credit is part of the same queue)
  • The project is searchable on Drupal.org
  • The project page + documentation + issue relationship is the same as any other Drupal project.
  • All projects have the standardized GitLab repositories and collaboration tools (and as of Nov 10th, all projects have merge requests)
  • All projects have access to DrupalCI and are included in our Composer endpoint.
  • There is a standard process for security coverage for D.O hosted projects, and some integrated tooling (s.d.o)

That said, several of these points are much less relevant to JavaScript projects:

  • We don't really have robust javascript testing infrastructure
  • We don't need packagist endpoints for javascript components, we need npm
  • And, as was said above, we already have other components of core that are not D.O hosted (Symfony, etc)

The Drupal Association does already control the Drupal namespace on GitHub, which gives us some, but not all, of the governance tools:

  • We can control the maintainers of the Drupal namespace, and the repositories underneath it
  • We could set up any necessary mirroring
  • We can control CRUD operations for repositories and forks
  • We could integrate into GitHub's existing tooling for javascript testing
  • We could integrate with GitHub's existing npm listing capability

We would lose a lot of the workflow and change management processes if this were all hosted on GitHub, but we would be able to enforce the most fundamental governance that we might need.

I suppose the question is - do the benefits outlined in the IS outweight the decentralization of Drupal contribution in this area - and could many of those benefits be replicated in our now much more modern GitLab stack?

For myself, I find the proposal in the IS compelling, particularly since we're talking *specifically* about components that aren't dependent on Drupal and might be used outside Drupal. But I can certainly see both sides.

Not proposing an answer yet, just more food for thought.

hestenet’s picture

Version: 9.1.x-dev » 9.2.x-dev
wim leers’s picture

Just wanted to say that this is a superb discussion. I can't think of any additional points to raise 😊I think this is the right time to reassess how we handle this.

catch’s picture

#3176918-18: [policy, no patch] Publishing / Maintaining JS libraries produced by Drupal that do not have a dependency on Drupal is a good summary of what I'd have written. We've been very concerned about fragmentation of PHP code (and while we have non-Drupal PHP dependencies, we have a lot of trouble managing their varying release/support cycles these days), but this isn't really an issue with JavaScript - our problem is really the opposite that it gets stagnant in core.

Also when we added jQuery, it was a nice centalized solution along with jQuery UI, but whatever we do now to get off it is going to rely on multiple different libraries, since there's no modern equivalent that handles everything - so even a project maintained by two Drupal developers on github may quickly end up more active than some of the third party dependencies we've been looking at.

So broadly in favour of them being hosted and developed as separate projects - also most of our targets for this, at least initially, are going to be relatively small and self-contained.

I don't think the projects should rely on the existing set of core committers for commits - since there are only about three of us who can usefully review JavaScript as it is, so we'd either need to add additional core maintainers with a specific focus on js, or come up with some kind of new system where the github maintainers have an official role (like subsystem maintainers in core) that gives them commit access on github. Initial commits to core and updates would still end up going through the core issue queue.

gabesullice’s picture

we'd either need to add additional core maintainers with a specific focus on js, or come up with some kind of new system where the github maintainers have an official role (like subsystem maintainers in core) that gives them commit access on github. Initial commits to core and updates would still end up going through the core issue queue.

The second part of your remark makes me worry that we would still be treating core as the One True Project while holding these JavaScript libraries at arm's length. There's value in the perception of security and stability that comes with being part of Drupal core and I would like these JS libs to earn that same reputation. IOW, saying "that guarantee comes from it being committed and brought into core," it's slightly different than saying "that guarantee comes with this code because it _is_ core."

I think we can foster that perception by having some ritual for packaging releases of these libraries and pushing them to NPM so that they get the imprimatur of being an "official Drupal project".

e0ipso’s picture

It is probably too early to start solutioning but I don't imagine Drupal will bring in the dependencies with loose dependency contraints. In other words, to have Drupal core rely on @drupal/super-library-they-write from v1.2.3 to v1.3.4 there will be a core committer in the Drupal project pushing that button. So the "sign-off" per se will likely still exist.

catch’s picture

Yeah we have issues like #2874198: Create and run dependency regression tests for core but it's not up and running yet, so at least until then we'd be having a Drupal.org issue every time we want to update to a specific version.

The second part of your remark makes me worry that we would still be treating core as the One True Project while holding these JavaScript libraries at arm's length. There's value in the perception of security and stability that comes with being part of Drupal core and I would like these JS libs to earn that same reputation.

So one possibility would be that people maintaining the js libraries on github are listed as core committers, but the minimum expectation would be that they review PRs on github (i.e. they wouldn't also be expected to review PHP or even front end patches on d.o, but someone could do both if they wanted to). Like a 'JavaScript library committer' role.

larowlan’s picture

For what it's worth, I agree with #18, #21 and #24

And I think #24 is one of the most salient points here.

nod_’s picture

this isn't really an issue with JavaScript - our problem is really the opposite that it gets stagnant in core.

I don't want to derail the issue so I opened for further discussion of this point #3178115: [policy, no patch] Maintenance of JS in core

About what should be out or in core as of today what I have in mind:

  • once: github
  • libs to create for ajax/events/autocomplete replacement: github
  • config eslint/prettier/stylelint: keep on drupal.org (put the source back in coder?) and publish estlint-config-drupal, prettier-config-drupal, etc. on npm
gabesullice’s picture

Sorry to be exhaustive about this, but I think it's important to be very explicit in order to avoid issues like these from getting caught in limbo states.

#24 and #25 indicate to me that, in principle, you both (@catch and @larowlan) think this is a good idea. Is that true? "This" being the proposed resolution minus the exact policy details we have yet to flush out.

If so, we'd probably like to to hear from @webchick and @justafish before we strike through the first step of the remaining tasks and move to try this out on a library, per #9, #10, and #14.

justafish’s picture

I'd like to have the thoughts of the people involved with the modernize initiative too because if I'm not mistaken, that's pretty much what they've been asking for this whole time.

Yes, I would love to do this! As well as hestenet's list in #18, in my mind there's been two major sticking points for this in the past -

  • We'll need to hook into the script which packages Drupal as a tarball, as well as requiring node/npm|yarn for installation for people checking out Drupal from source
  • If we can't do the above, core committers need to run the JavaScript build step when we change something, which they have not been comfortable with doing (which is totally fair - they already have a lot to do as it is!)
webchick’s picture

I keep getting pinged about this... I'm right in the throes of being absolutely slammed at work. so not sure when I can reasonably get around to responding (going to try hard to do so by the November 6 meeting)... no need to hold things up on me.

I will say though that I've read the IS and the comments so far and really appreciate the nuances that have gone into the brainstorming/discussion. In general, it sounds like folks are trying really hard to strike a good "middle ground" between quality control, governance, and developer empowerment.

catch’s picture

@justafish #28

If we can't do the above, core committers need to run the JavaScript build step when we change something, which they have not been comfortable with doing (which is totally fair - they already have a lot to do as it is!)

There's a couple of (positive) caveats to this.

At the moment we only package specific versions of js dependencies in core, and each update requires a core patch. So development can happen on git***, and when there's a new tag to integrate into core, the core patch might have to build the js still. But this is one awkward patch each time core gets updated, not each commit to the library itself. A bit less work without actually changing anything (except the changes in this proposal...)

We also have an open issue for a core packaging build pipeline:

#3099100: Convert the release script to a Drupal.org build pipeline

gabesullice’s picture

Issue summary: View changes

WOOT! I think that's consensus on point 1!

Let's try this out on a library! I think we also have consensus that the jQuery once library is a good first candidate. @nod_, do you want to keep it under your namespace for the moment or should we move it to github.com/drupal? I'm indifferent.

As of this comment, we're here:

Remaining tasks

  1. Agree this is a good idea
  2. Try it out on a library
  3. Define the policy in details, find answers to the questions above and
    • which licence are allowed? GPLv2, MIT only?
    • Mirror code from github to drupal.org
    • Are we building the package published to npm from the code in gitlab or the one in drupal.org, etc.
  1. Done!
  2. We're going to try this out w/ a jQuery.once() replacement.
  3. Let's begin researching and drafting these policies.

Let's begin drafting these policies.

Where? Google docs? Drupal.org? Here?

hestenet’s picture

Let's begin drafting these policies.

Where? Google docs? Drupal.org? Here?

My vote would be a Google Doc listed in the issue summary here for the initial policy draft - and then the canonical place should be part of the Core documentation on Drupal.org: https://www.drupal.org/about/core/policies

gabesullice’s picture

Issue summary: View changes

Your wish is my command @hestenet!

I started a doc here (I added it to the issue summary too). I took the liberty of ripping off the style and format of an IETF document (we'll call it "emulating" 🤣). I think this format makes it easier to communicate technical requirements and to discuss them.

droplet’s picture

StatusFileSize
new1.22 MB

Stopped discussions already? The current draft policy still very Drupal-specified. (I hope I'm wrong.) It's a dead circuit.

We have to make libraries reusable everywhere and able to acquire NON-drupal developers.

@see Chart below (click the link above for large version)

  • To maintain MAIN repo in github.org, not shallow of d.org.
  • Lean Code. Clean, Small, Fast!
  • No same code requirements as Drupal is required. No Drupal Core maintainers only repo. (which can be 100% detached from Drupal I meant. You need not study a story book from d.org/doc to get started.) But yeah, reach a consensus on a basic standard is good for long-term development.
  • Forked main repo as repo.drupal to add Drupal-specified code. Drupal Core Maintainers make their magic here to do slow releases to fulfill Drupal's requirements. Or the same workflow as now in d.org. Above main repo picked up 90~95% code development, only 5 ~ 10% left for Drupal-only. Drupal should always remove this 5~10 % to show it's flexible.) Sometimes, we should fork other libs before they join the Drupal foundation or upstream updates. For example, the VSCODE team forked and maintained their https://github.com/microsoft/chokidar version years before upstream pick up their requests.

Make Drupal Theme the same way, then we have a good Bootstrap-like framework for everyone. It's good for Drupal.

** chart source: https://excalidraw.com/#json=5726368989970432,GpM7O2zIggaa43ZUOBUdGg

nod_’s picture

I think that's what we have in mind (that's what I have in mind at least), now it's mainly a matter of formalizing the proposal (in the google doc) and figuring out the details. The aim is not to create new Drupal things. I see this as moving existing Drupal things that are better, in a different community (code of conduct, core gates around accessibility/documentation/code quality, handling of security issues, etc.)

To me this is not really about code, today we can still go the way RobLoach did for jquery-once and have a faster releases, there is nothing stopping us to create the once lib and submit it as a third party lib, same for autocomplete, events, etc. What's missing is an official way of managing "Drupal-supported" code outside drupal.org, for smaller libs it's not critical, but it's critical for the js menu component initiative because the third party approach is not what is wanted.

Fork to github to add Drupal-specific things to libraries is already the policy on the PHP side #2157189-26: [Policy, no patch] How to handle upstream fixes that don't get committed fast enough, and I think we didn't go this way for jQuery UI because those forks are supposed to be temporary. Having an official policy would solve that.

nod_’s picture

Had a chat with gabesullice and lauriii about this and we discussed the different assumptions/expectations we had since a few things didn't match with what I tried to address in the summary. To me what's in the doc linked in #33 goes a little too far for a first step and try to address some issues that are out of scope for what I want to solve.

I spelled out what I had in mind for the policies in this a separate doc the difference is that it's a smaller scope. Essentially it is to give the use of the "drupal" space in github/npm to trusted maintainers that will manage libraries used by Drupal core and make sure they work well and keep a good quality. The document is not editable it is only there for discussion, main work should happen in the existing google doc.

hestenet’s picture

The Drupal Association had our regular monthly check-in call with core maintainers today, and naturally this issue was a topic of some discussion.

@Lauriii raised some really thoughtful points, in particular, about some of the intangible impacts of our decision making.

Intangible Impacts

To paraphrase quickly:

  • All of the considerations of tooling and testing are obviously hard requirements
  • But we also need to improve that sense that javascript components and the initiatives to make them are first class citizens of the project
  • And one of the let-downs of the last admin-ui javascript initiative is that the presence on github left them siloed from the Drupal project in terms of governance, visibility, brand, etc.

This is raising some new questions for me, and makes me want to hear from the current initiative participants, and javascript devs in general about what's important to them.

  1. Setting aside the tooling - what are the intangible considerations that are important to you?
  2. Is it important that these projects have a project page on Drupal.org (regardless of what toolset it links out to?)
  3. Is it important that the work is recognized with Drupal.org contribution credits?
  4. Is it important that the work is simply *visible* to the rest of the Drupal community?
  5. Is it important that the work is happening in a 'familiar way' to other javascript projects?
  6. Is it important that the work can receive easy participation from Drupal outsiders?

I know these intangible questions are very squishy and touchy feely - but understanding the answers would help bring us a lot closer to understanding the best way to solve the tooling and governance issues, I think.

For example, if the most important intangible element is 'being able to receive contribution from Drupal outsiders' that would suggest a different route than if the most important thing is 'being present within the Drupal community'.

Middle ground solution: Drupal.org + GitLab

When I think about all the ways that we've recently worked on making different kinds of contribution recognizable within Drupal, it at least makes me want to *offer* the best possible foot forward that Drupal.org could provide.

Remember, Drupal.org's git backbone is now GitLab and that does give us a lot of power.
We could:

  • Make a new project content type for 'components' or 'libraries' with a drop-down for 'type' (e.g: javascript)
  • Enable additional GitLab features for that project, such as GitLabCI and GitLab Pipelines, which can be integrated with many of the same javascript dev tools as GitHub(https://docs.gitlab.com/ee/ci/examples/), and could be configured by the maintainers, rather than being bottlenecked on the central DrupalCI config.
  • Enable either the built-in GitLab NPM endpoint (https://docs.gitlab.com/12.10/ee/user/packages/npm_registry/), or automatically mirror to GitHub's /drupal namespace and publish to NPM that way (like we do for Drupal core components and packagist)
  • Build new tools into api.drupal.org to handle javascript documentation parsing
  • Suggest still using d.o issues - but we could even turn on GitLab's more simple issue queue so it could be used for kanban boards

These are all things we could do so that these Javascript efforts (and any other non-php Drupal efforts) can be first-class citizens of the project - with credit, d.o visibility, d.o documentation support, governance, etc - and the DA is absolutely willing to invest in that.

In the meantime, pre-release development of the new component could continue on GitHub, and we can compare feature by feature as the DA adds new capabilities.

How does that strike you all? Very curious for all of your thoughts.

nod_’s picture

Very interesting questions, I was feeling like I didn't get my point across, with these questions I'm starting to see from a different point of view and better understand what other concerns there are around the topic.

First, let's acknowledge that a Drupal JS community already exists on github, folks making projects with React/Vue use github, not drupal.org. I want to support that and not try to force them back on drupal.org. We can help by bringing some pieces of the Drupal community on github and put some official framing around all that (the policy document). This issue is not going to create something that is not already there, what I'm proposing is aimed at improving the situation both for core and folks using github.

Ideally, I want people without a drupal.org account to be Drupal contributors, I want them to be able to contribute to some parts of Drupal JS code, documentation, architecture, etc. from where they are already comfortable.. This is pretty central to what I have in mind, and the policy and different suggestions are a pretty much a way to define what a Drupal contributor is outside Drupal.org, and explain to other developers what to expect when working with and working on Drupal JS code. You're contributing to github.com/drupal/once => boum you're a Drupal contributor. You can expect X/Y from us, and we need Z from you.

With my JS maintainer hat (not talking on behalf of any of the initiatives), to me there are 2 different situation here:

  1. "Second-party" libraries that are standalone and mainly used by drupal (jquery-once, event/dom helpers we come up with)
  2. SDK type libraries/components (the new initiative) that have a dependency on Drupal (either the API or architecture)

Back to your questions:

  1. Setting aside the tooling - what are the intangible considerations that are important to you?

    Have a healthy Drupal JS community that is in touch with the rest of the Drupal community.

  2. Is it important that these projects have a project page on Drupal.org (regardless of what toolset it links out to?)

    For A: no, it doesn't matter. these are libraries that have Drupal as the main user so they need to cater to Drupal needs but not only.
    For B: no, the main people supposed to use it are not on drupal.org so there is little benefit, also everything is already on github today.

  3. Is it important that the work is recognized with Drupal.org contribution credits?

    No, people contributing might not even have a drupal.org account. It's better but if it create any friction to contribution it's better to do without.

  4. Is it important that the work is simply *visible* to the rest of the Drupal community?

    Work should be visible to the target users. So I would say yes to some extend, but not at the cost of the people working on the project (fed up with lack of progress, reactivity, etc.). And for A, I'd want the work to be visible to the whole JS community, not just Drupal.

  5. Is it important that the work is happening in a 'familiar way' to other javascript projects?

    It's the most important point. and probably why everything is already on github.

  6. Is it important that the work can receive easy participation from Drupal outsiders?

    Yes, second most important point. Although like I proposed above, they wouldn't be outsiders anymore.

I would like to point out the waterwheel effort from Acquia which was similar to what is underway for the new JS initiative, there was a drupal.org module then replaced with a github repo. People involved there would have some ideas about why a move was necessary.

Quick note on the middle ground solution: Drupal.org + GitLab: I appreciate the willingness to be open about alternatives. As a maintainer I would welcome the possibility to run common JS tools with gitlab CI and have the JSDoc api published officially. This wouldn't solve the "take people where they already are" point, but it'd be much better than what we do now for folks still on drupal.org.

To me we're well past the point of trying to stop people from going outside of drupal.org. I'm for supporting them because it'll take way less effort than trying to make a home on drupal.org with a patchwork of tools. Why not try it this way? The most visible thing we have to show for our past efforts is a trail of burnouts, pretty sure we can do better this time.

nod_’s picture

Quick note on github, by now there are many IDE integrations which means PR can be created, reviewed and merged within PhpStorm, or VSCode. Gitlab plugins exists but they have much less features. If a middle ground is found with d.o + Gitlab, the developer experience will not be close to what can be achieved on github.

mirom’s picture

I added some licensing recommendations based on the JS modernization iniatiative to the document - https://docs.google.com/document/d/1LkTeRac_UmEsXNx5du3IfXDwHEeR2nvw9zei...

droplet’s picture

Something happened today:

1.
I got the Drupal's Gitlab email today. At first sight, this is adding another extra layer to block your contribution more than help. I can't 100% tell what that Drupal Gitlab is and how it works by the way.

Simple tooling has never been a problem for developers, at least not for senior developers(, real contributors). Now, I can still fork it to use github/gitlab.com and that's FREE also.

2.
https://www.drupal.org/project/ideas/issues/2913628#comment-13686894
This is my first time knowing that an issue about Menu thing. That's 2 months late.

I want people without a drupal.org account to be Drupal contributors

Who are they? Drupal site builder? or anyone?

nod_’s picture

About 2. I think in general there is a lot going on in chat (IRC and now Slack) that you end up missing out on. There are some discussions that are not well suited for the d.o issue queue. Getting an initiative organized is one of them, the process to propose a new initiative is documented and has to go through this different issue queue on purpose to not get in the way of technical core work. That is a discovery problem you're correct, the way people hear about the initiatives somethings like :
People hear about the topic in Dries keynote, or read about it in Dries blog post (which is cross posted on the drupal blog), then they get on Slack and start discussing in #contribute channel, or they see a tweet about it, and start discussing in the dedicated slack channel to organize. If you're not (or can't ) be on twitter or slack it'll be hard to be informed. Where would you see it?

On the Drupal contributor thing it's my personal opinion but I would like anyone to be a drupal contributor. A rando on github fixing a typo in the documentation of the once lib => they are a Drupal contributor now. I don't know how to associate contribution credits for that but we can find something later, right now I'm interested to see if we can agree on the principle.

hestenet’s picture

I've made a quick and dirty demo of how the D.O issue queue integration with merge requests presently works, just to provide something more concrete around that option: https://youtu.be/NIWCXE-aM6Y

gabesullice’s picture

If you have not been paying careful attention to the google doc, now is the time to sit down with it.

Please be sure to read the section on page 1 titled How to read and contribute to this document. Following those suggestions will ensure that no discussion gets forgotten/overlooked.

Our goal should be to alter the document until it represents something that we all agree on, not to decide up or down on exactly what it looks like today.

The first thing on your mind will probably be: Why does the doc look the way it does?

Since we tried to be extremely thorough and to consider every implication of our recommendations, from licensing to security coverage to governance (however small), it would be very difficult to discuss all the details via comments in this issue.

By using the RFC style (borrowed from the IETF and many, other, OSS projects) we can have discussions about specific details and collaboratively work toward a consensus on them without anything getting lost in the noise.

justafish’s picture

And one of the let-downs of the last admin-ui javascript initiative is that the presence on github left them siloed from the Drupal project in terms of governance, visibility, brand, etc.

I strongly disagree with this!

I'm +1 to @nod_'s sentiments

fgm’s picture

Just went through the whole issue, and now I wonder what in this is really JS-specific. Couldn't the consideration be extended to PHP components too, for the same reasons ?

Consider a very popular one like commerceguys/intl, for example. Used by lots of projects beyond Drupal, but still very much tied to Drupal, although for a contrib, not for core. Or the scaffold projects, currently on github.

Wouldn't these benefit from the same approach ? This would allow us to export some code (thinking of core/lib/Drupal/Component, specifically), to allow them independent maintenance instead of being just splits from core.

nod_’s picture

I think that could be the case, it's just that starting with JS where there is no history or "official" integrations with existing package systems is easier in that we don't have to deal with existing things. I don't have a good answer for php, might need to ping catch or alexpott about that :)

gabesullice’s picture

Issue summary: View changes
Status: Active » Needs review
StatusFileSize
new124.95 KB

I think all the stakeholders have had time to weigh in on this. Uploading the Google Doc as a PDF so we never lose access to it (hopefully). There are still a few remaining TODOs and QUESTIONs, but I think we need to hash those out in practice within the library itself and the DA.

The DA has begun implementing many of the recommendations that it makes, @justafish and @nod_ have been appointed as new core committers to govern JS.

https://www.drupal.org/project/once has been published.

The next major milestone will be publishing that library to NPM.

Hopefully we can refer back to the doc to resolve some questions in the future :)

Our remaining tasks are as follows:

Remaining tasks

  1. Agree this is a good idea
  2. Try it out on a library
  3. Define a set of policy recommendations in this Google doc
  4. Agree upon the recommendations made above and canonicalize them here: https://www.drupal.org/about/core/policies

IOW, I think it's a little too soon to say we've have completed Try it out on a library since the project linked above doesn't have any releases on NPM, but we're close!

Once that happens, I think we can move to canonicalization/adoption of the RFC/policies and fill in any gaps and remove the TODOs/QUESTIONs in it.

gabesullice’s picture

Status: Needs review » Active

Actually, Active is still more accurate, I think.

droplet’s picture

I scanned 20 pages. My summary is: "To share Drupal code to non-Drupal only"

It's very different from my expectations: To encourage and acquire more contributions [easy access & management + less Drupalism + better marketing portal].

I suspect this is what the original poster wanted also. Whatever, if this issue ends up in the current direction, I have another (better) suggestion.

First, isn't it more strict rules than Core development? (only 3.2.1 is more than Core.)

If so, why not change a bit of test standard in the current CORE development (, the only blocker) and build an automatic way to push part of code (components) to NPM. (It's very common in open sources) (Even as @fgm in #46 said: Make it to every components)

In Drupal (JS), the problem isn't a lack of committer to trigger Git. We have 100% RTBC patches commit rate. That's not enough Reviewer to set RTBC. Old contributors have no motivation and left the community or seek their own way to maintain patches privately
https://www.drupal.org/project/drupal/issues/2706483

Above 20 pages creating new (less power) maintainers. (and I can predict that's a very small group and may be the same current Core maintainers, no new players)

Now you can create a new role "SUPER REVIEWER" (as a virtual maintainer of this issue's suggestion.) and commit their work first. It's less work & confuses than the current new SPEC way.

nod_’s picture

I must say it ended up different from my early expectations too (to be clear, I'm ok with how it ended up), I made the proposal to create a new "position" with the idea that it was hard to add people to the core committer team, thanks to this document, it was not hard.

The main benefit of this document is that it explained to core committers what it means to maintain a JS library. You know what it means, justafish know, I know, many other know in the JS issue queue, but it was not clear for all core committers. It made it less "scary" or "worrying" and made it easy to add people to the core committer team, because it's clear what they will need to do.

We ended up with justafish and I added to the core committer team as (provisional) core committer with an area of focus on JavaScript, we're not product/release/framework manager, "just" core committers (kind of like the "super reviewer" you talked about) and we're going to put what this document talks about in practice.

A few things I learned/realized during this whole thing that can help explain the situation:

  1. The main thing that was made clear by this issue: keeping Drupal Core development on Drupal.org is a non-negotiable policy, it is enforced by Dries and core committers. There are many reasons for this, justafish and I talked about this with Dries directly before we were onboarded, and that is the reality of today. I think it was never made as clear before, which caused confusion, difficulties, crushed hopes, and frustration. Knowing there is no hope of this changing any time soon allowed us to move past this and find a more suitable compromise
  2. While keeping things on Drupal.org is required, user/developer experience is still very important, this is why the DA is spending a significant amount of time improving tools for us, it's actually their top priority these days:
    • There is a new "general purpose" project type for standalone JS/PHP libs (down the line, non-core lib could be hosted if people want to have some of the other nice things that go with being on d.o)
    • We just had CI/CD enabled on a few gitlab projects to try out making test/npm publishing workflows
    • We're starting discussions about gitlab pages to publish documentation and things under a drupal-controlled domain name (hopefully a path to have JSDocs published?)
    • I heard there is work underway to allow people to register/login to drupal.org with a github account
    • Discussions are open with the DA on how we can use more of gitlab features in our workflows and reduce the "Drupal specific" things we do around managing code
  3. Then there is balance between code integrity (stability) and velocity. as a contributor I want velocity, as a maintainer I want stability. To sum up:
    • Before: 90% code integrity, 10% velocity (we don't get much code in but it doesn't brake often anymore)
    • In the initial proposal I asked for: 20% code integrity, 80% velocity (the "super reviewer")
    • Right now: 60% code integrity, 40% velocity (added more people to the committer team, but with stricter rules than current core)

    It's only a first step, we will keep improving things

Now I know that things are still slow and I have made a few things to start addressing the review problem, because I agree with you that "adding more people" will not solve the issue (but it will help). My personal worry about JS patches is that a review and RTBC doesn't guarantee a "quick" commit. You are correct that we have a 100% commit rate for RTBC patches, the problem is velocity. I personally need to know that when I invest time in a patch, it'll be addressed quickly and the path to RTBC/Commit is clear and "fast". Having the ability to commit patches means that when justafish or I make a review, once the feedback is addressed we can commit and move on to other patches (if we didn't forget something during review, always possible :)

So to address the velocity:

  • To make the issue queue situation clear, I have a single detailed JS issue queue it lists everything that is opened and it also picks up issues that have JS patches/merge requests but not the js tag/componenent. We won't miss a JS patch thanks to this.
  • The fact that we are 2 people added to the core commit team for JS topics means that we have everything needed to get a "hard" patch committed (like the ajax command execution order, and a few others), with some alignement we can't get blocked on waiting for someone else to review/RTBC
  • In the scope of the decoupled menu initiative, we're talking about how to get people outside drupal community to use/contribute to drupal in different ways (we have a survey for the initiative to help us figure out what people want).

There is progress, not as radical as we thought/hoped but we got some pretty important policies cleared up and it's still huge progress compared to just a few months ago.

nod_’s picture

Issue summary: View changes
Status: Active » Needs review
Issue tags: -JavaScript menu component initiative +Needs issue summary update, +Decoupled menus initiative

Update

At the moment, and thanks to justafish, we have a workflow that can publish to @drupal/once from drupal.org gitlabci instance, and security scanner configured to warn us when there are security releases of package dependencies.

TODO

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

borisson_’s picture

Issue tags: -JavaScript +JavaScript

Do the updates in #51 and #52 mean that this policy issue can be closed?

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

Status: Needs review » Postponed (maintainer needs more info)

According to #52 there are documentation updates to do. What needs to be documented? Since this is tagged for an issue summary updated, it isn't clear if the information is the proposed resolution has been agreed to or not. At least, not to me.

What are the decisions here that need to be documented?

smustgrave’s picture

Wanted to bump this 1 more time or if it can be closed?

quietone’s picture

Assigned: Unassigned » quietone
Status: Postponed (maintainer needs more info) » Needs review

Working on reviewing this in detail with support from @nod_.

quietone’s picture

Issue summary: View changes
Status: Needs review » Needs work
Related issues: +#3559157: Add 'Core JavaScript packages' core committers

Created #3559157: Add 'Core JavaScript packages' core committers

The document needs to be reviewed to make sure it is using the new names for the core leadership team. For example, Drupal now longer has a BDFL.

And which guide will this be put into?

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.

quietone’s picture

Assigned: quietone » Unassigned