Investigate realistic resource requests for core jobs

Container memory usage could be found with: cat /sys/fs/cgroup/memory.current

This is easyer, just add time -v [command], also used this for performance analysis of project analysis. This should tell us things like; cpu usage, and memory usage/peak etc.

Baseline:

CSS linting

Step runtime: 13 seconds

Command being timed: "yarn run --cwd=./core lint:css --color --custom-formatter node_modules/stylelint-junit-formatter"
	User time (seconds): 12.31
	System time (seconds): 1.27
	Percent of CPU this job got: 146%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.25
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 225340
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 109161
	Voluntary context switches: 158353
	Involuntary context switches: 168
	Swaps: 0
	File system inputs: 0
	File system outputs: 4336
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Compilation check

Step runtime: 12 seconds

Command being timed: "yarn run --cwd=./core build:css --check"
	User time (seconds): 8.06
	System time (seconds): 0.39
	Percent of CPU this job got: 159%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.28
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 217608
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 95107
	Voluntary context switches: 36188
	Involuntary context switches: 30
	Swaps: 0
	File system inputs: 0
	File system outputs: 4336
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Javascript linting (eslint)

Step: 32s

	Command being timed: "yarn --cwd=./core run -s lint:core-js-passing --format junit"
	User time (seconds): 40.18
	System time (seconds): 0.95
	Percent of CPU this job got: 140%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:29.24
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 344756
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 190039
	Voluntary context switches: 14282
	Involuntary context switches: 668
	Swaps: 0
	File system inputs: 0
	File system outputs: 4336
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

PHP Coding standards (phpcs)

step: 14s

	Command being timed: "composer phpcs -- --report-junit=junit.xml --report-full --report-summary"
	User time (seconds): 210.99
	System time (seconds): 2.22
	Percent of CPU this job got: 1986%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:10.73
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 64260
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 6
	Minor (reclaiming a frame) page faults: 269674
	Voluntary context switches: 552
	Involuntary context switches: 2236
	Swaps: 0
	File system inputs: 0
	File system outputs: 20824
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

PHPStan

Step: 1:14 (74 seconds)

	Command being timed: "php -d apc.enabled=0 -d apc.enable_cli=0 vendor/bin/phpstan analyze --configuration=./core/phpstan.neon.dist --error-format=junit"
	User time (seconds): 1881.21
	System time (seconds): 31.87
	Percent of CPU this job got: 2728%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:10.11
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 647436
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 96
	Minor (reclaiming a frame) page faults: 3409610
	Voluntary context switches: 5980
	Involuntary context switches: 6229
	Swaps: 0
	File system inputs: 0
	File system outputs: 751472
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

⚙️️ PHPUnit Kernel

Step 405s

	Command being timed: "sudo MINK_DRIVER_ARGS_WEBDRIVER=["chrome", {"browserName":"chrome","chromeOptions":{"args":["--disable-dev-shm-usage","--disable-gpu","--headless"]}}, "http://localhost:9515"] -u www-data php ./core/scripts/run-tests.sh --color --keep-results --types PHPUnit-Kernel --concurrency 32 --repeat 1 --sqlite ./sites/default/files/.sqlite --dburl mysql://drupaltestbot:drupaltestbotpw@database/mysql?module=mysql --url http://localhost/subdirectory --verbose --non-html --all --ci-parallel-node-index --ci-parallel-node-total 1"
	User time (seconds): 6766.96
	System time (seconds): 2219.35
	Percent of CPU this job got: 2241%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 6:40.89
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 193020
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 24912
	Minor (reclaiming a frame) page faults: 112557265
	Voluntary context switches: 2544460
	Involuntary context switches: 218525
	Swaps: 0
	File system inputs: 0
	File system outputs: 92046288
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

⚡️ PHPUnit Unit

Step: 69s

Command being timed: "sudo MINK_DRIVER_ARGS_WEBDRIVER=["chrome", {"browserName":"chrome","chromeOptions":{"args":["--disable-dev-shm-usage","--disable-gpu","--headless"]}}, "http://localhost:9515"] -u www-data php ./core/scripts/run-tests.sh --color --keep-results --types PHPUnit-Unit --concurrency 32 --repeat 1 --sqlite ./sites/default/files/.sqlite --dburl mysql://drupaltestbot:drupaltestbotpw@database/mysql?module=mysql --url http://localhost/subdirectory --verbose --non-html --all --ci-parallel-node-index --ci-parallel-node-total 1"
	User time (seconds): 650.91
	System time (seconds): 419.07
	Percent of CPU this job got: 1658%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:04.49
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 374800
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 5880
	Minor (reclaiming a frame) page faults: 18246808
	Voluntary context switches: 65672
	Involuntary context switches: 30446
	Swaps: 0
	File system inputs: 0
	File system outputs: 24108320
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

🌐️️ PHPUnit Functional 1/6

Step: 11:43 (703s)

	Command being timed: "sudo MINK_DRIVER_ARGS_WEBDRIVER=["chrome", {"browserName":"chrome","chromeOptions":{"args":["--disable-dev-shm-usage","--disable-gpu","--headless"]}}, "http://localhost:9515"] -u www-data php ./core/scripts/run-tests.sh --color --keep-results --types PHPUnit-Functional --concurrency 32 --repeat 1 --sqlite ./sites/default/files/.sqlite --dburl mysql://drupaltestbot:drupaltestbotpw@database/mysql?module=mysql --url http://localhost/subdirectory --verbose --non-html --all --ci-parallel-node-index 1 --ci-parallel-node-total 6"
	User time (seconds): 3789.59
	System time (seconds): 445.97
	Percent of CPU this job got: 607%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 11:37.64
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 252148
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 3633
	Minor (reclaiming a frame) page faults: 22989432
	Voluntary context switches: 3515180
	Involuntary context switches: 74737
	Swaps: 0
	File system inputs: 0
	File system outputs: 13988032
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

🌐️️ PHPUnit Functional 2/6

Step: 312s

	Command being timed: "sudo MINK_DRIVER_ARGS_WEBDRIVER=["chrome", {"browserName":"chrome","chromeOptions":{"args":["--disable-dev-shm-usage","--disable-gpu","--headless"]}}, "http://localhost:9515"] -u www-data php ./core/scripts/run-tests.sh --color --keep-results --types PHPUnit-Functional --concurrency 32 --repeat 1 --sqlite ./sites/default/files/.sqlite --dburl mysql://drupaltestbot:drupaltestbotpw@database/mysql?module=mysql --url http://localhost/subdirectory --verbose --non-html --all --ci-parallel-node-index 2 --ci-parallel-node-total 6"
	User time (seconds): 3049.68
	System time (seconds): 444.16
	Percent of CPU this job got: 1138%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 5:06.99
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 267616
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 3804
	Minor (reclaiming a frame) page faults: 22597116
	Voluntary context switches: 3268739
	Involuntary context switches: 68417
	Swaps: 0
	File system inputs: 0
	File system outputs: 14436616
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

🌐️️ PHPUnit Functional 3/6

Step: 392s

	Command being timed: "sudo MINK_DRIVER_ARGS_WEBDRIVER=["chrome", {"browserName":"chrome","chromeOptions":{"args":["--disable-dev-shm-usage","--disable-gpu","--headless"]}}, "http://localhost:9515"] -u www-data php ./core/scripts/run-tests.sh --color --keep-results --types PHPUnit-Functional --concurrency 32 --repeat 1 --sqlite ./sites/default/files/.sqlite --dburl mysql://drupaltestbot:drupaltestbotpw@database/mysql?module=mysql --url http://localhost/subdirectory --verbose --non-html --all --ci-parallel-node-index 3 --ci-parallel-node-total 6"
	User time (seconds): 4364.99
	System time (seconds): 557.75
	Percent of CPU this job got: 1268%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 6:27.93
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 200120
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 4286
	Minor (reclaiming a frame) page faults: 28466563
	Voluntary context switches: 3885051
	Involuntary context switches: 95557
	Swaps: 0
	File system inputs: 0
	File system outputs: 17522640
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

🌐️️ PHPUnit Functional 4/6

Step: 195s

	Command being timed: "sudo MINK_DRIVER_ARGS_WEBDRIVER=["chrome", {"browserName":"chrome","chromeOptions":{"args":["--disable-dev-shm-usage","--disable-gpu","--headless"]}}, "http://localhost:9515"] -u www-data php ./core/scripts/run-tests.sh --color --keep-results --types PHPUnit-Functional --concurrency 32 --repeat 1 --sqlite ./sites/default/files/.sqlite --dburl mysql://drupaltestbot:drupaltestbotpw@database/mysql?module=mysql --url http://localhost/subdirectory --verbose --non-html --all --ci-parallel-node-index 4 --ci-parallel-node-total 6"
	User time (seconds): 2324.35
	System time (seconds): 368.11
	Percent of CPU this job got: 1413%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 3:10.46
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 169996
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 4190
	Minor (reclaiming a frame) page faults: 21647653
	Voluntary context switches: 2601738
	Involuntary context switches: 53703
	Swaps: 0
	File system inputs: 0
	File system outputs: 16145864
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

🌐️️ PHPUnit Functional 5/6

Step: 253s

	Command being timed: "sudo MINK_DRIVER_ARGS_WEBDRIVER=["chrome", {"browserName":"chrome","chromeOptions":{"args":["--disable-dev-shm-usage","--disable-gpu","--headless"]}}, "http://localhost:9515"] -u www-data php ./core/scripts/run-tests.sh --color --keep-results --types PHPUnit-Functional --concurrency 32 --repeat 1 --sqlite ./sites/default/files/.sqlite --dburl mysql://drupaltestbot:drupaltestbotpw@database/mysql?module=mysql --url http://localhost/subdirectory --verbose --non-html --all --ci-parallel-node-index 5 --ci-parallel-node-total 6"
	User time (seconds): 2715.96
	System time (seconds): 393.95
	Percent of CPU this job got: 1255%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 4:07.63
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 188332
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 3784
	Minor (reclaiming a frame) page faults: 22998089
	Voluntary context switches: 2289010
	Involuntary context switches: 80428
	Swaps: 0
	File system inputs: 0
	File system outputs: 15514136
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

🌐️️ PHPUnit Functional 6/6

Step: 308s

	Command being timed: "sudo MINK_DRIVER_ARGS_WEBDRIVER=["chrome", {"browserName":"chrome","chromeOptions":{"args":["--disable-dev-shm-usage","--disable-gpu","--headless"]}}, "http://localhost:9515"] -u www-data php ./core/scripts/run-tests.sh --color --keep-results --types PHPUnit-Functional --concurrency 32 --repeat 1 --sqlite ./sites/default/files/.sqlite --dburl mysql://drupaltestbot:drupaltestbotpw@database/mysql?module=mysql --url http://localhost/subdirectory --verbose --non-html --all --ci-parallel-node-index 6 --ci-parallel-node-total 6"
	User time (seconds): 2774.54
	System time (seconds): 353.06
	Percent of CPU this job got: 1033%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 5:02.51
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 163196
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 3483
	Minor (reclaiming a frame) page faults: 23714515
	Voluntary context switches: 2350699
	Involuntary context switches: 63635
	Swaps: 0
	File system inputs: 0
	File system outputs: 14582136
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

👷️️️ PHPUnit Build

Step: 154s

	Command being timed: "sudo MINK_DRIVER_ARGS_WEBDRIVER=["chrome", {"browserName":"chrome","chromeOptions":{"args":["--disable-dev-shm-usage","--disable-gpu","--headless"]}}, "http://localhost:9515"] -u www-data php ./core/scripts/run-tests.sh --color --keep-results --types PHPUnit-Build --concurrency 32 --repeat 1 --sqlite ./sites/default/files/.sqlite --dburl mysql://drupaltestbot:drupaltestbotpw@database/mysql?module=mysql --url http://localhost/subdirectory --verbose --non-html --all --ci-parallel-node-index --ci-parallel-node-total 1"
	User time (seconds): 162.16
	System time (seconds): 52.87
	Percent of CPU this job got: 144%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 2:28.30
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 110320
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 2217
	Minor (reclaiming a frame) page faults: 9591258
	Voluntary context switches: 54927
	Involuntary context switches: 8045
	Swaps: 0
	File system inputs: 0
	File system outputs: 11583064
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

🖱️️️ PHPUnit Functional Javascript

Step: 443s


	Command being timed: "sudo MINK_DRIVER_ARGS_WEBDRIVER=["chrome", {"browserName":"chrome","chromeOptions":{"args":["--disable-dev-shm-usage","--disable-gpu","--headless"]}}, "http://localhost:9515"] -u www-data php ./core/scripts/run-tests.sh --color --keep-results --types PHPUnit-FunctionalJavascript --concurrency 32 --repeat 1 --sqlite ./sites/default/files/.sqlite --dburl mysql://drupaltestbot:drupaltestbotpw@database/mysql?module=mysql --url http://localhost/subdirectory --verbose --non-html --all --ci-parallel-node-index --ci-parallel-node-total 1"
	User time (seconds): 2902.74
	System time (seconds): 390.86
	Percent of CPU this job got: 753%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 7:17.33
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 234076
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 2911
	Minor (reclaiming a frame) page faults: 20716435
	Voluntary context switches: 2794470
	Involuntary context switches: 200874
	Swaps: 0
	File system inputs: 0
	File system outputs: 12042096
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

🦉️️️ Nightwatch

Step: 373s

	Command being timed: "sudo BABEL_DISABLE_CACHE=1 -u www-data yarn --cwd ./core test:nightwatch"
	User time (seconds): 133.97
	System time (seconds): 22.29
	Percent of CPU this job got: 42%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 6:08.26
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 232500
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 366
	Minor (reclaiming a frame) page faults: 2372862
	Voluntary context switches: 226248
	Involuntary context switches: 3598
	Swaps: 0
	File system inputs: 0
	File system outputs: 1416728
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Postponed on #3388392: Allow setting resource limits

Ok, i can automate this a little better and add the reports to artifacts.

Usage: /usr/bin/time [OPTIONS] COMMAND [ARG]...
Run COMMAND, then print system resource usage.

  -a, --append              with -o FILE, append instead of overwriting
  -f, --format=FORMAT       use the specified FORMAT instead of the default
  -o, --output=FILE         write to FILE instead of STDERR
  -p, --portability         print POSIX standard 1003.2 conformant string:
                              real %%e
                              user %%U
                              sys %%S
  -q, --quiet               do not print information about abnormal program
                            termination (non-zero exit codes or signals)
  -v, --verbose             print all resource usage information instead of
                            the default format
  -h,  --help               display this help and exit
  -V,  --version            output version information and exit

Commonly usaged format sequences for -f/--format:
(see documentation for full list)
  %%   a literal '%'
  %C   command line and arguments
  %c   involuntary context switches
  %E   elapsed real time (wall clock) in [hour:]min:sec
  %e   elapsed real time (wall clock) in seconds
  %F   major page faults
  %M   maximum resident set size in KB
  %P   percent of CPU this job got
  %R   minor page faults
  %S   system (kernel) time in seconds
  %U   user time in seconds
  %w   voluntary context switches
  %x   exit status of command

Default output format:
%Uuser %Ssystem %Eelapsed %PCPU (%Xavgtext+%Davgdata %Mmaxresident)k
%Iinputs+%Ooutputs (%Fmajor+%Rminor)pagefaults %Wswaps

NOTE: your shell may have its own version of time, which usually supersedes
the version described here.  Please refer to your shell's documentation
for details about the options it supports.

GNU Time website: <https://www.gnu.org/software/time/>
Full documentation at: <https://www.gnu.org/software/time/manual>
E-mail bug reports to: bug-time@gnu.org

Using --output=./time-report.txt and adding that to the artifacts means we'd have statistics on this and we don't break output from commands. This might even be usefull to add in the general runs until we find a good balance.

I rebased (which was no fun). So it will run again.

Although perhaps just applying the "time" commands on a fresh branch is better to get a new baseline based on the current state.

Added a clean MR with only the yaml changes, since rebase was hard and failed i think. It's running, and there should be artifacts for each output of the time commands (although also available in the pipeline). I wont have time to administer this much more in the next week.

I've added a package artifacts step like this:
https://git.drupalcode.org/issue/drupal-3388162/-/jobs/259844

Also one in the child pipeline, but that is still running. Then you can just download those artifacts and set resource usage based on that.

Ok, full baseline with artifacts for tests and lint jobs with statistics:

https://git.drupalcode.org/issue/drupal-3388162/-/pipelines/42687

Also added the artifacts as files.