Problem/Motivation
Since #3312089: Run phpcs in parallel in commit-code-check.sh, the commit check script throws this error on Macs:
/Users/xjm/git/maintainer/core/scripts/dev/commit-code-check.sh: line 270: nproc: command not found
Proposed resolution
Use a portable equivalent. According to a GitHub issue for another project, the Darwin-friendly replacement is:
sysctl -n hw.logicalcpu
Remaining tasks
Test to make sure the above also works on Linux; otherwise, make the command conditional.
Issue fork drupal-3407360
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3407360-commit-check-script
changes, plain diff MR !5750
Comments
Comment #2
xjmComment #3
xjmComment #5
alexpottI have it installed as part of homebrew - it's part of coreutils - so
brew install coreutilswould fix this.OTOH if we are going to replace nproc here we should also replace it in the root composer.json
Comment #6
quietone commentedThe suggested change does not work with ddev so will need some logic.
Comment #7
smustgrave commentedFor both #5 and #6. Though not sure if it's fixed in #5 how that impacts #6
Comment #8
alexpottI think given #6 we're going to need tell mac users to do install something that makes nproc available. It looks like this could be an issue sometimes on linux too - https://www.howtodojo.com/nproc-command-not-found/
Comment #9
quietone commentedThen, lets add, at least, a comment to the script that nproc is needed and how to install it. Or, do an early check for it an output an nice message for the user.
Comment #10
longwaveDoes this command print the expected output on machines without nproc?
On Linux with nproc this works as expected, if this also works on Mac then we can perhaps make it a bit more portable this way.
(This says: run nproc, but do not display errors, and if it fails, run sysctl instead)
Comment #11
alexpottI moved my nproc...
So that doesn't look like that will work - it'll need to be something longer hand that checks for a commands existence first.
Comment #12
longwaveHow about:
although https://unix.stackexchange.com/questions/85249/why-not-use-which-what-to... implies this is not really solvable across all shells and operating systems.
Comment #13
xjmI think it would be good to make this as portable as possible even if it's not everywhere. #12 is also suggested elsewhere online. Does that work for you @quietone/@alexpott?
Edit: Forgot to say that #12 does work on OOTB Darwin.
Comment #14
alexpottI think we should consider doing
command -v nproc && nproc || sysctl -n hw.logicalcpu- there no piping any error stuff. Works on Darwin and *nix. And we already use it via Symfony in our tests and other places - see usages of \Symfony\Component\Process\ExecutableFinder.Comment #15
alexpottAh we'll have to do
command -q nproc && nproc || sysctl -n hw.logicalcpu... -v will output stuff we don't want.Comment #16
alexpottTesting...
Comment #17
alexpottHo hum... command -q is a nicety from using fish shell...
So
command -v nproc >/dev/null && nproc || sysctl -n hw.logicalcpulooks the best option. We're using a command we're already using and it'll work on Darwin and *nix and doesn't using niceties from fish shell...Comment #18
quietone commented#12 and #16 work in ddev.
Is there a reason not to 'hash' like tag.sh does?
Comment #19
alexpott@quietone hash is definitely an option. Better than which... and about equal to command -v imo... prior art for both of them. hash seems to be a little quieter and concise. Tested
hash nproc 2>/dev/null && nproc || sysctl -n hw.logicalcpuon fish and bash on Darwin without issue.
Comment #20
quietone commentedYes, definitely quieter which I prefer.
Comment #21
mstrelan commentedWe could always add a final fallback value of 1 right? So if both checks fail just use 1 thread.
Comment #22
alexpott#21++ that's a nice idea
Comment #23
xjmYes #22 is the "oh duh why didn't I think of that" robustness that we should have.
Comment #24
longwaveSo the final version should be something like
Comment #25
longwaveOr maybe, a simpler idea is to compromise: assume everyone has either 4 or 8 threads/CPUs available, and just set a fixed value?
Comment #26
longwaveOr given the renewed interest in maintaining PHPCS, wait for them to automate it like PHPStan already has: https://github.com/PHPCSStandards/PHP_CodeSniffer/issues/10#issuecomment...
Comment #27
quietone commentedI prefer to get a change it quickly here so everyone has working script without having to install extra packages. I would go with '1' for now and if that turns out to be a problem it can be changed.
Comment #28
alexpottLet's not change it to only use 1 process - that'll slow builds down on gitlab
I'd do
It's more concise and if people want quicker runs on darwin they can install coreutils. I've pushed to the MR to fix it this way.
Comment #29
xjmWhy would we not include as an "or" the command that allows Macs to run 8 processes in parallel? I don't see what the downside is of including that command in the condition chain.
Comment #30
xjmNW to go back to #24. "Concise" is not preferable over "still fairly concise and allowing improvements to all platforms without forcing them to install new packages".
Comment #31
alexpott#30 - my reasoning was because it is something else to maintain that is never run on any of our CIs so will be broken when something changes as inevitably it will.
And FWIW the command has to be
hash nproc 2>/dev/null && nproc || hash sysctl 2>/dev/null && sysctl -n hw.logicalcpu 2>/dev/null || echo 1because if we don't check for sysctl's existence then if a machine does not have nproc or systl it wouldn't echo the 1. You'd get an error about sysctl not existing.Comment #32
alexpottMaybe we should echo something other than 1. There probably is now harm in going for 4 here as most PCs have at least these days. See https://store.steampowered.com/hwsurvey/cpus/
Comment #33
longwaveFor me
2>/dev/nullhides the "command not found" error if I deliberately change the command names, and falls back correctly because "command not found" has a non zero exit code:Wonder if we can even just shorten it:
e.g.
I know this doesn't work on the CLI in fish (via #11), but is fish actually used for the shell when run from the composer or commit-code-check script? (which specifies bash in the first line)
Or we could try:
Or even:
Comment #34
alexpottThere's a space between
$( (due to https://github.com/koalaman/shellcheck/wiki/SC1102Comment #35
smustgrave commentedTested this with ddev and still seems to work there just FYI.
Comment #36
smustgrave commentedUsed this for about a day on a few tickets and at least for ddev I didn't hit any issues.
Comment #39
longwaveCommitted and pushed to 11.x and 10.2.x, thanks!