Early Bird Registration for DrupalCon Portland 2024 is open! Register by 23:59 PST on 31 March 2024, to get $100 off your ticket.
Problem/Motivation
Currently we get a DrupalCI Error if a test runs but there are no tests to run. This is different than old pifr which would run tests and return 0 results as a pass.
Since we rely on the behavior of run-tests.sh for our exit code, we have to either bracket run-test.sh with our own logic, or change the behavior of run-tests.sh.
This behavior of run-tests.sh was introduced in #2189345: run-tests.sh should exit with a failure code if any tests failed
Proposed resolution
Some options:
- Count the tests available and if it's 0 then don't run the simpletest job. This would be in-scope for drupalci_testbot.
- Modify run-tests.sh to allow either behavior, perhaps by using a flag. This would be in scope for drupal core.
- Have drupal.org determine pass/fail status based on a full-fledged Job Result object passed back by the test runner, instead of run-test.sh exit code and xml output.
Comments
Comment #2
MixologicI'll add this to my todo list
Comment #3
Mile23See also: #2645590: Ensure that simpletest job doesn't "fail" testing if no tests are present
Comment #4
Mile23Comment #5
jthorson CreditAttribution: jthorson commentedComment #6
jthorson CreditAttribution: jthorson commentedTest pass/fail status should not be tied directly to the exit code or output generated by any particular script executed within the job ... a given job can run many individual scripts, and a job might be considered to have passed even if some of those scripts do not return cleanly (for example, advisory code review jobs).
Thus the proper solution to this issue is to decouple run-tests.sh from the final job status, and only relate the two through job-specific logic in the SimpletestJob class. To support extensibility of the DrupalCI architecture and other job types, the current drupal.org integration logic needs to be refactored to operate off of a generic job result object passed back by the test runner.
Comment #7
jthorson CreditAttribution: jthorson commentedProposed approach for the decoupled Job Results object was posted at #2515078: Job Types need to be able to specify job-specific build artifacts to collect/publish and #2489530: Add getArtifacts to all BuildTask plugins, add test.
Comment #8
drummComment #9
Mile23#2683013: Better functional test framework added an (admittedly imperfect) test for this: http://cgit.drupalcode.org/drupalci_testbot/tree/tests/DrupalCI/Tests/Ap...
Comment #10
MixologicComment #11
MixologicThis duplicates this : #2645590: Ensure that simpletest job doesn't "fail" testing if no tests are present