Problem/Motivation

Currently, we are using yarn package manager to manage node dependencies.

This works fine but the problem is that if you have multiple projects each project and your custom theme will have separate node_modules. This is a big waste of disk space because each node_module can be around 500MB of the size!

Additionally, for each new custom theme, you need to download again all these dependencies - not a big problem if you are using yarn which is cashing libraries locally.

There is relatively new package manager pnpm which claims to fix this and also it should be drop in replacement for mpn or yarn.

Proposed resolution

Evaluate pnpm and switch to it if is better.

Comments

pivica created an issue. See original summary.

ytsurk’s picture

Would be great to save disk space, vote for this - yes !

pivica’s picture

Just tested this and it's quite trivial to implement. First, you install pnpn

npm add -g pnpm

And then install all node modules with

pnpm install

Instead of `yarn install` or `npm install`.

To execute a gulp task you then do

pnpx gulp sass.

The only thing we should add is a new lock file pnpm-lock.yaml and update the documentation so people can use any method they prefer, pnpn, yarn or npm.

pivica’s picture

Issue tags: +next-release
primsi’s picture

Status: Active » Needs review
StatusFileSize
new178.29 KB

Initial patch. As per chat we would need to update https://www.drupal.org/docs/8/themes/bs-base/quick-start#s-usualtasks too

pivica’s picture

Status: Needs review » Needs work

Took some time to think about this and how can we solve this in a better way so we allow custom projects to pick their own package system and also we allow easier changes when next better package management system is developed and we want to implement it.

We could do two things.

1. Instead of hardcoding package install and build commands in code we could move this to package.json scripts section. For example, we could add to our package.json next section

"scripts": {
    "build-css": "yarn install && yarn gulp clean:css && yarn gulp sass"
}

This is for old yarn and for a new pnpm it could be (note using of npx which is bundled in node from 5.2 version):

"scripts": {
    "build-css": "pnpm install && npx gulp clean:css && npx gulp sass"
}

Not sure should we have all in one command or maybe in two (one for install and another for build). I guess having just one is enough?

Then we can change drush to:

function _bs_base_drush_build_css($path) {
  // Install npm packages and execute gulp sass compilation.
  drush_log("Installing any missing package and rebuilding sass files", LogLevel::OK);
  exec('cd ' . $path . ' && npm run build-css');
}

We should also remove yarn.lock if we are switching by default to pnpm lock.

primsi’s picture

Status: Needs work » Needs review
StatusFileSize
new617.61 KB
new794.13 KB

Implemented the above + what discussed.

pivica’s picture

Issue tags: +ContributionWeekend2020, +ContributionWeekendCH
StatusFileSize
new1.83 KB
new794.24 KB

Some minor text fixes, let's test this now a bit more.

pivica’s picture

StatusFileSize
new799.83 KB

Here is a patch updated against patches in other issues, suitable for composer.

pivica’s picture

Status: Needs review » Needs work

It would be nice that we fallback to npm in build-css script so we don't fail hard if pnpm is not installed on the system.

sasanikolic’s picture

Agree, this is currently failing when trying to execute the theme update. What would be also helpful to have is a better error message and aliases for compiling sass, instead of pnpx gulp sass, yarn gulp sass etc., we could just write compile-css or something similar.

pivica’s picture

> Agree, this is currently failing when trying to execute the theme update.

We could try something like this maybe:

"scripts": {
    "build-css": "((type pnpm && pnpm install) || npm install) && npx gulp clean:css && npx gulp sass"
}

> we could just write compile-css or something similar.

Yes that is the whole idea of having package.json script for this so you can do

npx run build-css

Instead of build-css we can do compile-css but it's pretty much the same.

pivica’s picture

StatusFileSize
new1.59 KB
new794.33 KB
new799.92 KB

Here is a new patch, added fallback from pnpm to npm.

pivica’s picture

We need to update documentation before closing this :)

pivica’s picture

Status: Needs work » Needs review

Documentation is updated.

sasanikolic’s picture

StatusFileSize
new37.86 KB

With the latest patch applied, it tries to run npm install (I think) on every theme compilation. In the case of multiple subthemes, this slows the compilation time significantly. Could we improve this somehow? Maybe run this only once at the start - save it in a local variable or so?

pivica’s picture

> With the latest patch applied, it tries to run npm install (I think) on every theme compilation.

I don't think that this is the case. The compilation is slower because of internal color contrast algorithm we are using now - it requires more time. If that script is triggering npm instead of pnpm then you would see package-lock.json in theme folders and you don't see them - just tried that.

To additionally verify that is going on you can go to the theme folder and execute `npm run build-css`:

➜  primer_demo_theme git:(master) ✗ npm run build-css

> primer_demo_theme@ build-css ../primer/web/profiles/primer/themes/primer_demo_theme
> ((type pnpm && pnpm install) || npm install) && npx gulp clean:css && npx gulp sass

pnpm is /usr/local/bin/pnpm
Lockfile is up-to-date, resolution step is skipped
Already up-to-date
node_modules                             |  WARN  Cannot link bin "gonzales" of "gonzales-pe-sl" to "/home/pivica/moje/www/md-systems/primer/web/profiles/primer/themes/primer_demo_theme/node_modules/.pnpm/node_modules/.bin". A package called "gonzales-pe" already has its bin linked.
[14:36:06] Using gulpfile ~/moje/www/md-systems/primer/web/profiles/primer/themes/primer_demo_theme/gulpfile.js
[14:36:06] Starting 'clean:css'...
[14:36:06] Finished 'clean:css' after 16 ms
[14:36:07] Using gulpfile ~/moje/www/md-systems/primer/web/profiles/primer/themes/primer_demo_theme/gulpfile.js
[14:36:07] Starting 'sass'...
[14:36:08] Finished 'sass' after 1.1 s
[14:36:11] gulp-postcss: components/node.css

Drush is using the same script run. Note that 1.1s for sass is not correct value, try to execute `time npx gulp sass` and you will see what i mean:

➜  primer_demo_theme git:(master) ✗ time npx gulp sass
[14:43:11] Using gulpfile ../primer/web/profiles/primer/themes/primer_demo_theme/gulpfile.js
[14:43:11] Starting 'sass'...
[14:43:12] Finished 'sass' after 1.17 s
npx gulp sass  11.29s user 0.21s system 119% cpu 9.589 total

So it says 1.17s but in reality it needs 11.29s.

  • pivica committed c5b11fa on 8.x-1.x
    Issue #3045552 by pivica, Primsi, sasanikolic: Evaluate pnpm package...
pivica’s picture

Status: Needs review » Fixed
Issue tags: -next-release

Committed.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.