Inconsistent patch test results for old and new testbot

I can confirm things are inconsistent, when using DCI.
I just ran the test multiple times (10 times some with different concurrency) and it had failures once.

I've seen unreproducible failures [which I was eventually able to figure out] in one of my contrib modules and could have saved time if I had installed DrupalCI locally - maybe next time :)

Google Analytics with SimpleTest has ~386 Tests FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 381 pass(es), 5 fail(s), and 0 exception(s). CI tells me I have 15 pass, 2 fail.

CI is wrong and does not properly run tests for unknown reasons. reCaptcha looks like the same.

Drupal core has ~113.000 tests, but CI reports just ~14.000 as passing.

Drupal core has ~113000 assertions, not tests. CI is now properly reporting the ~14000 test methods that are being executed. Since the assertions always included every assertion from every parent class (like all of the setup assertions), this "number of tests" was always traditionally miscounted.

Every assertion tests one feature or at least verify a functionality.

That means it is one test.

Please count and display assertions, too. The current behavior is very confusing and do not motivate me to write any more test.

Here's another case https://www.drupal.org/node/2423341#comment-10401233

@andypost: that particular case is a known issue with the pift/pifr bots that we're not going to spend any time solving. Migrate likes to segfault, and we don't have an explanation for it.

The other consistently failing problem with the old bots all testbots is sometimes they will do an http request and get back a 0 instead of any real status code.

I just noticed, for the first time, that this happens on drupalci: https://www.drupal.org/pift-ci-job/38105

So, I will investigate that issue (which is a problem with testing in general, not with old vs new bots, as such is not a blocker for disabling pift/pifr)

Follow along here: #2579183: GET requests on testbots sometimes return 0

Also, Hass, there is a plan to expose those assertion counts as well: #2568137: Expose Assertion Count to Jenkins

seeing a bunch of not matching results with the old testbot and the new one. usually, fails on the old one, and passes on the new one.

examples:
#2513534-52: Remove the 'disabled' region from Block UI
"FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 113,855 pass(es), 4 fail(s), and 0 exception(s).
PHP 5.5 & MySQL 5.5 14,137 pass, 1 fail"
[edit: this might actually be consistant, both fail, just probably count differently]

#1810394-180: Site configuration with domain based language negotiation results in redirecting authenticated users to a different domain when accessing a content entity route for translation language different from the interface language
"Previous result: FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 116,797 pass(es), 496 fail(s), and 23 exception(s).
PHP 5.5 & MySQL 5.5 14,218 pass"

#2581817-2: Follow-up for #2505263: clean up comments
"FAILED: [[SimpleTest]]: [PHP 5.5 MySQL] 116,797 pass(es), 428 fail(s), and 22 exception(s).
PHP 5.5 & MySQL 5.5 14,212 pass"

All three of the test results in #13 are fine. The first has the same failures - remember the old counts were *assertion* counts, the new counts are *test* counts. The numbers will not, and should not match. If you click through to the results it shows that both have the same errors.

The latter two are examples of pift/pifr that have been biting us lately where pift/pifr is randomly failing. I.e we are *only* concerned if drupalCI has failures that it shouldn't, or is passing when it shouldn't.

Are we still concerned with PIFR inconsistencies?

Please re-open if we are.

Give people a chance to see.

The only relevant thing left is #2568137: Expose Assertion Count to Jenkins