"Build Successful" is treated as a test failure

testbot says this core patch failed, but there are no failures.

and the message is BUild successful (with no results), but have a look at the console output.
https://www.drupal.org/project/drupal/issues/2878046#comment-12785748

I'm hitting this problem with Backup Migrate too, most of its branch tests are indicating that they fail because of this, even though the actual code tests pass.

This needs attention.

Setting it to "major" as there's no dataloss involved.

This just hit Metatag's branch tests too. Dagnammit.

ya it's everywhere, someone ping Dries.

Tests are failing with "Build Successful" even if they have tests. #3001981: Tests fail with "Build Successful" message

Hello,

I experienced the same issue.
Can someone confirm that is a CI related issue?

I notice that my colleague that pushed previously a patch on the same issue, did receive a full green when he tested against PHP 5.3 & MySQL 5.5 while I tested against PHP 7 & MySQL 5.5 and PHP 5.6 & MySQL 5.5.

This was in the wrong queue, that is why we get no response.

The person responsible for the Drupal.org testing infrastructure is Mixologic , I recommend someone here please contact him

FYI: I sent a message to Mixologic
I'm not sure who else works with him on this, I imagine he's with Aquia?

this is an important issue because like me, a lot of maintainers rely on the automated testing system, it automatically filters out broken patches in issue queues and provides valuable debugging information for the community .

Basically my maintainer work is pretty much halted until this gets fixed.

this might have been fixed, can anyone else please confirm?

Webform is still failing with "Build Successful".

Hello,

I triggered a retest on the last patch of the #1657886: Filter "Convert URLs into links" doesn't support multilingual web addresses issue, using PHP 5.6 & MySQL 5.5 and PHP 7 & MySQL 5.5.

Both are green: PHP 5.6 & MySQL 5.5 2,042 pass PHP 7 & MySQL 5.5 2,042 pass.

This still affects some contrib as mentioned by Liam (webforms) , I just checked Panels.

Can pinpoint when this issue started, September 19th 2018 to be exact. Last known working state, September 18th 2018.

see panels automated tests:
https://www.drupal.org/pift-ci-job/1082840

I think I may have fixed this this morning.

The issue was that coder created a new, backwards incompatible release, which everything started using (8.3.1). I've reverted drupalci to use 8.2.12 and it has fixed some of the problems people were seeing with phpcs failing to run.

However, that is only half of the issue, and separate from the original one that TR is reporting.

When a build has no tests it used to work, but we had to upgrade jenkins, and the plugin we were relying on to allow a 'no tests build' to pass was broken in the upgrade (rather it insisted on absolutely valid XML for a non-existant xml spec), so we reverted to the other junit processing plugin.

However, I had forgotten that the other junit plugin fails when there are no tests, so, Im currently working on a fix for how to make a 'no tests' build still pass.

Sorry for the delays.

Testing is now working for Webform and my other modules that I have tried.

appears to be working again, can others please confirm?

cool, ya the problems I noticed started approximately September 19th,
it appears to be fixed now, I think we can mark this as fixed. I haven't looked into the other issue, doesn't seem to affect my stuff.

I can confirm that the branch tests for at least some of the modules I comaintain are working correctly now, e.g. Date's tests ran fine this evening: https://www.drupal.org/pift-ci-job/1088139

Thanks Mixologic!

As I mentioned before, we cannot mark this as fixed. This issue was opened back in August, and is unrelated to the problem that started on september 19th, which is now fixed.

Another failure https://www.drupal.org/pift-ci-job/1093942

Another failures
- https://www.drupal.org/pift-ci-job/1097062
- https://www.drupal.org/pift-ci-job/1097079
- https://www.drupal.org/pift-ci-job/1097118

I also just hit this same issue here:

https://www.drupal.org/project/readmore_formatter/issues/3013948#comment... - https://www.drupal.org/pift-ci-job/1122032

Specifics of PHPCS failing to install is being worked on here: #3013813: Coder failing for all PHP 5 tests

I was going to be super-smug and tell people to add drupalci.yml files to their project, but you can see the PHAIL-WHALE results of that here: #3014492-4: Adopt Gitlab CI and fix CS violations detected

The tail of the build log looks like this:

18:55:14 ---------------- Finished assessment in 2.043 seconds ---------------- 
18:55:14 chown -R 1001 /var/www/html
18:55:14 chown -R 1001 /var/www/.composer/cache
18:55:15 chmod -R 777 /var/www/html
18:55:15 sudo chown -R 1001 /var/lib/mysql
18:55:16 chmod -R 777 /var/lib/mysql
18:55:23 Archiving artifacts
18:55:23 Checking console output
18:55:23 Recording test results
18:55:23 ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
18:55:23 Finished: FAILURE

This tells me that the cure should be in the JUnit publish stage, which is not a plugin but probably lives in the realm of Jenkins. So it expects JUnit but doesn't find any, which is enough to cause the build to fail. This confirms #23.

So the workaround is: Add a passing test.

I just noticed another one here: https://www.drupal.org/pift-ci-job/1131242

They often occur in the core after RTBC. I saw a few.

Example of Build Successful:
#3001997: Transliteration a string containing an unknown character (e.g. 0x80) is not valid

#18
#22

Both after RTBC.

https://www.drupal.org/pift-ci-job/1145127
https://www.drupal.org/pift-ci-job/1138297

Re #35 - most likely because all RTBCs are tested periodically whilst Needs Review are tested only once per status change.

The remaining cases of "Build Successful" might have something to do with a failing Nightwatch test being not correctly reported as test failure, see #1149078-104: States API doesn't work with multiple select fields.

@TR: Sure. Just saying that while introducing Nightwatch, the buggy behaviour might have been introduced as a stop gap to Nightwatch not correctly reporting its results. However I can't find any commit on:

• Project
• Project issue tracking
• PIFT
• DrupalCI TestRunner

that corresponds to the test results you gave on your initial report.
Then again, between July 18, 2018 02:16 and July 19, 2018 03:34, there's been 11 commits on DrupalCI: Environments that possibly might have caused this. I'm not sufficiently into the PIFT infrastructure to be of any help here.