Test-only job sometimes report as successfull even if there are failures in tests, see:
https://git.drupalcode.org/project/drupal/-/jobs/506479
There were 4 errors:
1) Drupal\Tests\system\Functional\Pager\PagerTest::testActiveClass
Error: Class "Drupal\Core\Cache\MemoryCache\MemoryCacheFactory" not found
...
There was 1 failure:
1) Drupal\Tests\user\Kernel\UserPassRehashTest::testUniqueHashNoPasswordValue
...
And still:
Job succeeded
From the Slack discussion with @fjgarlin:
test-only tests are run using
phpunit, whereas the whole suite usesrun-tests.sh, so that might explain whyphpunitis not catching the exception.we have sometry/catchblocks onrun-tests.shand on some test base classes that might capture that and not reported
See: https://git.drupalcode.org/project/drupal/-/jobs/506479
Fix the error checking.
Show commandsStart within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
changes, plain diff MR !8466
Comments
Comment #2
fjgarlin commentedNote that the report shows them as errors: https://git.drupalcode.org/project/drupal/-/pipelines/65788/test_report
I think the reason for this is that the job is taking the exit code of the last command being run (https://stackoverflow.com/questions/70820135/gitlab-make-the-ci-cd-faile...). If we see the reported job, the last tests being run are actually successful, so it's taking the status from that last command.
So we need to capture if there was any error during the script and then add an
exit 1in that case.Comment #5
james.williamsThanks for your guidance @fjgarlin - hopefully !8466 is along the right lines?
Comment #6
fjgarlin commentedThe code changes look correct to me and hopefully that's all that's needed. Now we need to verify it. Two options:
- Bring the test files from that other job to this MR, run the test only, then remove those changes.
- Bring the changes from this MR to that other issue, run the test only, then remove the changes.
With that, I'd be happy to RTBC.
Comment #7
james.williamsThanks - I've done the second option; so let's see how https://git.drupalcode.org/issue/drupal-3454196/-/pipelines/203616 comes out for #3454196: Filter placeholders without arguments are not replaced when HTML corrector filter applied afterwards...
Comment #8
fjgarlin commentedBased on the code review I did before and this run https://git.drupalcode.org/issue/drupal-3454196/-/pipelines/203616 which verifies the fix in both the top level Test-only job and the per-version test-only job, I am marking this RTBC.
Comment #13
longwaveCommitted and pushed 0ac3d2a036 to 11.x and 1c3d9629f4 to 11.0.x and 2d0757fde1 to 10.4.x and cdd8d3cc14 to 10.3.x. Thanks!