Problem/Motivation

If a pipeline fails, you have to click through to the failed job and ctrl-f to find the test failures.

DrupalCI's Drupal.org integration would (usually) show you which class failed and assertion errors or error messages at quite a high level.

For example:

https://git.drupalcode.org/project/drupal/-/pipelines/25459 shows a functional javascript failure.

If I click on 'failed jobs', I get:

      ./sites/simpletest/browser_output: found 2989 matching artifact files and directories 
Uploading artifacts as "archive" to coordinator... 201 Created  id=132615 responseStatus=201 Created token=64_dfbux
Uploading artifacts...
./sites/default/files/simpletest/phpunit-*.xml: found 204 matching artifact files and directories 
Uploading artifacts as "junit" to coordinator... 201 Created  id=132615 responseStatus=201 Created token=64_dfbux

Cleaning up project directory and file based variables

ERROR: Job failed: command terminated with exit code 1

If I click on Functional Javascript tests, I get the last lines of the console log:

https://git.drupalcode.org/project/drupal/-/jobs/132615

I ctrl-f for 'fail', I get to:

Drupal\Tests\media_library\FunctionalJavascript\FieldUiInteg   0 passes             1 exceptions             
FATAL Drupal\Tests\media_library\FunctionalJavascript\FieldUiIntegrationTest: test runner returned a non-zero error code (2).
Drupal\Tests\media_library\FunctionalJavascript\FieldUiInteg   0 passes   1 fails

If ctrl-f again, I get the actual error:

 PHPUnit Test failed to complete; Error: PHPUnit 9.6.8 by Sebastian Bergmann
    and contributors.
    
    Testing
    Drupal\Tests\media_library\FunctionalJavascript\FieldUiIntegrationTest
    E                                                                   1 / 1
    (100%)
    
    Time: 00:31.405, Memory: 4.00 MB
    
    There was 1 error:
    
    1)
    Drupal\Tests\media_library\FunctionalJavascript\FieldUiIntegrationTest::testFieldUiIntegration
    Behat\Mink\Exception\ExpectationException: Checkbox "set_default_value" is
    not checked, but it should be.
    
    /builds/project/drupal/vendor/behat/mink/src/WebAssert.php:794
    /builds/project/drupal/vendor/behat/mink/src/WebAssert.php:736
    /builds/project/drupal/core/modules/media_library/tests/src/FunctionalJavascript/FieldUiIntegrationTest.php:100
    /builds/project/drupal/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
    
    ERRORS!
    Tests: 1, Assertions: 23, Errors: 1.
    
    HTML output was generated

Where you actually need to go to is the 'tests' tab, then click into the type you want to check, like:

https://git.drupalcode.org/project/drupal/-/pipelines/25513/test_report

However, I wonder if we can make this easier.

Steps to reproduce

Proposed resolution

Get the class, assertion failure and backtrace into the 'failed jobs' tab.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

CommentFileSizeAuthor
Screenshot from 2023-10-01 08-55-11.png185.53 KBcatch

Comments

catch created an issue. See original summary.

catch’s picture

catch
he/him
Primary language English
commented

Looks like https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html

catch’s picture

catch
he/him
Primary language English
commented

OK so there is a summary here, but it is a couple of clicks to find, and the failure details are in a modal:

https://git.drupalcode.org/project/drupal/-/pipelines/25513/test_report

catch’s picture

catch
he/him
Primary language English
commented
Issue summary: View changes
Parent issue: » #3387107: [meta] GitLab CI feature parity with DrupalCI
jonathan1055’s picture

jonathan1055 commented

This is also tedious on contrib testing too. Sometime even the 'tests' tab does not have the error. For example, this 'tests' tab /scheduler_content_moderation_integration/-/pipelines/25812/test_report - there is no phpunit job row in the table. But from the pipeline overview you can see that the phpunit job ran and failed.

[edit] The above might be due to gitlab config on that project, because even when the phpunit passes there is no row in the 'tests' tab. So it might not be an error with all contrib project results.
catch’s picture

catch
he/him
Primary language English
commented

Reading https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html a bit more properly it's what we're already doing (for core at least) so is not helpful particularly for what I'm thinking of here.

ressa’s picture

ressa
he/him
commented
Related issues: +#3396710: Bypass migration warning 'There is translated content of these types' if Language module is not installed

Thanks for creating this issue @catch, and the tip about Ctrl+F'ing until the actual error pops up, it was very helpful.

joachim’s picture

joachim commented

> the tip about Ctrl+F'ing until the actual error pops up, it was very helpful.

Sort of -- if you know what a failure looks like in the output of the particular test runner. I was recently working on a MR which added JS and I had no idea what string to search for in the test output.

ressa’s picture

ressa
he/him
commented

It would still be great, if the reason why a test is failing, could be made clearer for the user :)

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.