Problem/Motivation

Currently we get a DrupalCI Error if a test runs but there are no tests to run. This is different than old pifr which would run tests and return 0 results as a pass.

Since we rely on the behavior of run-tests.sh for our exit code, we have to either bracket run-test.sh with our own logic, or change the behavior of run-tests.sh.

This behavior of run-tests.sh was introduced in #2189345: run-tests.sh should exit with a failure code if any tests failed

Proposed resolution

Some options:

  • Count the tests available and if it's 0 then don't run the simpletest job. This would be in-scope for drupalci_testbot.
  • Modify run-tests.sh to allow either behavior, perhaps by using a flag. This would be in scope for drupal core.
  • Have drupal.org determine pass/fail status based on a full-fledged Job Result object passed back by the test runner, instead of run-test.sh exit code and xml output.

Remaining tasks

User interface changes

API changes

Data model changes

Comments

Mixologic created an issue. See original summary.

Mixologic’s picture

Assigned: Unassigned » Mixologic

I'll add this to my todo list

Mile23’s picture

Mile23’s picture

Issue summary: View changes
jthorson’s picture

Issue summary: View changes
jthorson’s picture

Test pass/fail status should not be tied directly to the exit code or output generated by any particular script executed within the job ... a given job can run many individual scripts, and a job might be considered to have passed even if some of those scripts do not return cleanly (for example, advisory code review jobs).

Thus the proper solution to this issue is to decouple run-tests.sh from the final job status, and only relate the two through job-specific logic in the SimpletestJob class. To support extensibility of the DrupalCI architecture and other job types, the current drupal.org integration logic needs to be refactored to operate off of a generic job result object passed back by the test runner.

jthorson’s picture

Mile23’s picture

Mixologic’s picture

Component: Code » Jobs and Job Handling
Mixologic’s picture

Status: Active » Closed (duplicate)