Problem/Motivation

Since #3312089: Run phpcs in parallel in commit-code-check.sh, the commit check script throws this error on Macs:

/Users/xjm/git/maintainer/core/scripts/dev/commit-code-check.sh: line 270: nproc: command not found

Proposed resolution

Use a portable equivalent. According to a GitHub issue for another project, the Darwin-friendly replacement is:
sysctl -n hw.logicalcpu

Remaining tasks

Test to make sure the above also works on Linux; otherwise, make the command conditional.

Issue fork drupal-3407360

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

xjm created an issue. See original summary.

xjm’s picture

Issue summary: View changes
xjm’s picture

Status: Active » Needs review

alexpott’s picture

I have it installed as part of homebrew - it's part of coreutils - so brew install coreutils would fix this.

OTOH if we are going to replace nproc here we should also replace it in the root composer.json

quietone’s picture

The suggested change does not work with ddev so will need some logic.

$ ddev ssh
$ vendor/bin/phpcs -ps --parallel=$(sysctl -n hw.logicalcpu) --standard=core/phpcs.xml.dist
sysctl: cannot stat /proc/sys/hw/logicalcpu: No such file or directory
............................................................    60 / 12877 (0%)
smustgrave’s picture

Status: Needs review » Needs work

For both #5 and #6. Though not sure if it's fixed in #5 how that impacts #6

alexpott’s picture

I think given #6 we're going to need tell mac users to do install something that makes nproc available. It looks like this could be an issue sometimes on linux too - https://www.howtodojo.com/nproc-command-not-found/

quietone’s picture

Then, lets add, at least, a comment to the script that nproc is needed and how to install it. Or, do an early check for it an output an nice message for the user.

longwave’s picture

Does this command print the expected output on machines without nproc?

$ nproc 2>/dev/null || sysctl -n hw.logicalcpu
8

On Linux with nproc this works as expected, if this also works on Mac then we can perhaps make it a bit more portable this way.

(This says: run nproc, but do not display errors, and if it fails, run sysctl instead)

alexpott’s picture

I moved my nproc...

nproc 2>/dev/null || sysctl -n hw.logicalcpu
fish: Unknown command: nproc

So that doesn't look like that will work - it'll need to be something longer hand that checks for a commands existence first.

longwave’s picture

How about:

$ which nproc >/dev/null && nproc || sysctl -n hw.logicalcpu
8

although https://unix.stackexchange.com/questions/85249/why-not-use-which-what-to... implies this is not really solvable across all shells and operating systems.

xjm’s picture

I think it would be good to make this as portable as possible even if it's not everywhere. #12 is also suggested elsewhere online. Does that work for you @quietone/@alexpott?

Edit: Forgot to say that #12 does work on OOTB Darwin.

alexpott’s picture

I think we should consider doing command -v nproc && nproc || sysctl -n hw.logicalcpu - there no piping any error stuff. Works on Darwin and *nix. And we already use it via Symfony in our tests and other places - see usages of \Symfony\Component\Process\ExecutableFinder.

alexpott’s picture

Ah we'll have to do command -q nproc && nproc || sysctl -n hw.logicalcpu ... -v will output stuff we don't want.

alexpott’s picture

Testing...

$ command -q  nproc   && nproc || sysctl -n hw.logicalcpu
10
$ mv /opt/homebrew/bin/nproc /opt/homebrew/bin/nproc2
$ command -q  nproc   && nproc || sysctl -n hw.logicalcpu
10
alexpott’s picture

Ho hum... command -q is a nicety from using fish shell...

So command -v nproc >/dev/null && nproc || sysctl -n hw.logicalcpu looks the best option. We're using a command we're already using and it'll work on Darwin and *nix and doesn't using niceties from fish shell...

quietone’s picture

#12 and #16 work in ddev.

Is there a reason not to 'hash' like tag.sh does?

if hash pbcopy 2>/dev/null; then
    echo "$notes" | pbcopy
    echo -e "\n** Your releases notes have been copied to the clipboard. **\n"
else
    echo "$notes"
fi
alexpott’s picture

@quietone hash is definitely an option. Better than which... and about equal to command -v imo... prior art for both of them. hash seems to be a little quieter and concise. Tested
hash nproc 2>/dev/null && nproc || sysctl -n hw.logicalcpu
on fish and bash on Darwin without issue.

quietone’s picture

Yes, definitely quieter which I prefer.

mstrelan’s picture

We could always add a final fallback value of 1 right? So if both checks fail just use 1 thread.

alexpott’s picture

#21++ that's a nice idea

xjm’s picture

Yes #22 is the "oh duh why didn't I think of that" robustness that we should have.

longwave’s picture

So the final version should be something like

hash nproc 2>/dev/null && nproc || sysctl -n hw.logicalcpu 2>/dev/null || echo 1
longwave’s picture

Or maybe, a simpler idea is to compromise: assume everyone has either 4 or 8 threads/CPUs available, and just set a fixed value?

longwave’s picture

Or given the renewed interest in maintaining PHPCS, wait for them to automate it like PHPStan already has: https://github.com/PHPCSStandards/PHP_CodeSniffer/issues/10#issuecomment...

quietone’s picture

I prefer to get a change it quickly here so everyone has working script without having to install extra packages. I would go with '1' for now and if that turns out to be a problem it can be changed.

alexpott’s picture

Status: Needs work » Needs review

Let's not change it to only use 1 process - that'll slow builds down on gitlab

I'd do

hash nproc 2>/dev/null && nproc || echo 1

It's more concise and if people want quicker runs on darwin they can install coreutils. I've pushed to the MR to fix it this way.

xjm’s picture

Status: Needs review » Needs work

Why would we not include as an "or" the command that allows Macs to run 8 processes in parallel? I don't see what the downside is of including that command in the condition chain.

xjm’s picture

NW to go back to #24. "Concise" is not preferable over "still fairly concise and allowing improvements to all platforms without forcing them to install new packages".

alexpott’s picture

#30 - my reasoning was because it is something else to maintain that is never run on any of our CIs so will be broken when something changes as inevitably it will.

And FWIW the command has to be hash nproc 2>/dev/null && nproc || hash sysctl 2>/dev/null && sysctl -n hw.logicalcpu 2>/dev/null || echo 1 because if we don't check for sysctl's existence then if a machine does not have nproc or systl it wouldn't echo the 1. You'd get an error about sysctl not existing.

alexpott’s picture

Maybe we should echo something other than 1. There probably is now harm in going for 4 here as most PCs have at least these days. See https://store.steampowered.com/hwsurvey/cpus/

longwave’s picture

For me 2>/dev/null hides the "command not found" error if I deliberately change the command names, and falls back correctly because "command not found" has a non zero exit code:

hash nprocx 2>/dev/null && nproc || sysctlx -n hw.logicalcpu 2>/dev/null || echo fallback
fallback

Wonder if we can even just shorten it:

$ nprocx 2>/dev/null || sysctlx -n hw.logicalcpu 2>/dev/null || echo fallback
fallback

e.g.

nproc 2>/dev/null || sysctl -n hw.logicalcpu 2>/dev/null || echo 4

I know this doesn't work on the CLI in fish (via #11), but is fish actually used for the shell when run from the composer or commit-code-check script? (which specifies bash in the first line)

Or we could try:

bash -c "nproc 2>/dev/null || sysctl -n hw.logicalcpu 2>/dev/null || echo 4"

Or even:

bash -c "nproc || sysctl -n hw.logicalcpu || echo 4" 2>/dev/null 
alexpott’s picture

Status: Needs work » Needs review

There's a space between $( ( due to https://github.com/koalaman/shellcheck/wiki/SC1102

smustgrave’s picture

Tested this with ddev and still seems to work there just FYI.

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Used this for about a day on a few tickets and at least for ddev I didn't hit any issues.

  • longwave committed 65d3ab96 on 10.2.x
    Issue #3407360 by alexpott, xjm, longwave, quietone, smustgrave,...

  • longwave committed a5c4ebf2 on 11.x
    Issue #3407360 by alexpott, xjm, longwave, quietone, smustgrave,...
longwave’s picture

Status: Reviewed & tested by the community » Fixed

Committed and pushed to 11.x and 10.2.x, thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.