Disable Legacy Testbots and use drupalCI as our testing infrastructure

@tim.plunkett suggested I comment here with things that I think need fixing for DrupalCI before deployment.

I've been trying to follow the emailed results, but I find them entirely ungrokkable. I never really understand if HEAD is passing on DrupalCI or not, nor why it seems so arbitrary whether the builds are successful or not. Not sure if there is an issue for improving the reporting.

Also it's difficult to drill down on what specifically failed on a test run. You get one useless page and then a second page that may crash because so much data is on it.

On the first point, fully agreed that I can't really parse those emails either. They make it sound like Jenkins is broken, which I can do nothing about, and then require reading var_dump()ed arrays to get any meaning.

On the second point, there is a FANTASTIC interface that shows real-time results, but it's buried under "Console output" in the sidebar which no one would think to click on. :( Can we make that the default place the link goes? (I think I made an issue for this at one point.)

Another thing that came up today, in trying to re-test #2498599-7: Remove sdboyer/gliph since it is unused by core PIFT tested it fine, but DrupalCI (?) returned an error saying the patch not "eligible" for re-testing (sorry, didn't catch the exact message). I attempted to re-test it after and got no such error, so this appears to be intermittent.

Here's an example of PIFT passing and DrupalCI giving an error: #2498599-18: Remove sdboyer/gliph since it is unused by core.

Quick link: https://dispatcher.drupalci.org/job/default/1479/console

The problem in #8 is covered in #2534152: Drupalci testbot sometimes fails with container X is not running.

Ok great, sorry ... missed that. Also found my issue from above.

I noticed that the old testbots run on PHP 5.3 (for Drupal 7), but PHP 5.3 is not an option in the dropdown for the new testbots. Does that mean there's currently no way to test with PHP 5.3 in the new system?

If so, that seems like a blocker for disabling the legacy testbots (or at least for doing it anytime soon). PHP 5.3 is much more heavily used on production sites than it is on patch authors' machines, so people kind of rely on the testbot to help catch any 5.3-only regressions.

I and more than a couple other core devs had psuedo-admin rights on qa.d.o, and were able to cancel and requeue tests, and force-queue tests that had failing branches.
Is that sort of functionality going to be present for Drupal CI?

Errors on D7 testing of Dynamic Entity Reference https://www.drupal.org/pift-ci-job/5898 and https://www.drupal.org/pift-ci-job/5900 also unable to test D8 patches of Dynamic Entity Reference https://www.drupal.org/node/2469609#comment-10143582

@tim.plunkett - we have an issue open for canceling tests: #2518056: Need the ability to cancel a test job - there's a short term option of granting some elevated rights to core devs in jenkins and a longer term one for creating a ui. Wouldn't hurt to articulate some more explicit use cases that canceling is most needed for - it's slightly de-prioritized for us now that the bots can auto-scale and a bad test shouldn't cause a big blockage.

Does the already available re-test/add-test ui cover the re-queuing needs?

@tim.plunkett re: #12, we decided previously to limit the list to committers for the initial launch. We might expand it later.

Edit: @hesenet, @tim.plunkett doesn't have access to that UI I think; I haven't had opportunity yet to test it myself.

Adding to the summary for #11 and #5.

Ubercart's 8.x-4.x branch tests failed to run: https://www.drupal.org/pift-ci-job/5910

19:46:33 cd /var/www/html && sudo -u www-data php /var/www/html/core/scripts/run-tests.sh  --color --concurrency 27 --sqlite /var/www/html/results/simpletest.sqlite --php /opt/phpenv/shims/php --keep-results --dburl mysql://drupaltestbot:drupaltestbotpw@drupaltestbot-db-mysql-5-5/jenkins_default_1562 --module ubercart
19:46:33 Command created as exec id e4085507
19:46:34   ERROR: No valid tests were specified.

There is no "ubercart" module, instead our tests are spread among the many submodules in the project.

Ubercart 7.x-3.x failed with a different error: https://www.drupal.org/pift-ci-job/5964

21:13:55 cd /var/www/html && sudo -u www-data php /var/www/html/scripts/run-tests.sh --color --concurrency 27 --sqlite /var/www/html/results/simpletest.sqlite --php /opt/phpenv/shims/php --keep-results --dburl mysql://drupaltestbot:drupaltestbotpw@drupaltestbot-db-mysql-5-5/jenkins_default_1586 --module ubercart
21:13:55 Command created as exec id c87659a7
21:13:55   ERROR: Unknown argument '--sqlite'.

Unit tests passing even though they failed.

Due to the recent Guzzle upgrade, Feeds tests failed.

The results here, https://dispatcher.drupalci.org/job/default/1635/console don't show a failure in the unit tests, just a PHP fatal. I can't tell exactly, but it seems like I wouldn't have been able to tell without the integration tests which fail later on.

Updating the issue summary with links to issues about better e-mail format + better results links.
I do not know what "Not obvious where the exceptions are displayed" means.

As soon as we reach a consensus on this issue and all child issues are closed or postponed, we will close this issue and disable the qa.drupal.org testbots.

Maybe consider a trial period where both the old and the new run at the same time, so that contrib developers aren't completely blindsided and disrupted by the switchover? An announcement and documentation published *well before* this switchover happens would also be greatly appreciated.

This has the potential to be highly disruptive to those of us who have large numbers of tests we rely on.

As of today, both are running, for core and contrib. check out the bottom of your project's "automated testing" tab.

Yes, I noticed - that's why I posted here ... and it's not working at all for Ubercart, see #18 and #19, and there's no documentation at all, and no announcement of this feature. The "Automated testing" tab on my projects has shown "Continuous integration tests" for several weeks, but they haven't worked at all until today; the tests always showed up as queued but never ran. And if you search for "Continuous integration tests" on drupal.org, you find nothing to explain this feature or how to use it - indeed there is no mention at all that it just turned on, and no explanation of what it means. My guess is that most contrib maintainers don't know about it yet.

So as I said, this has the potential to be highly disruptive. I welcome the improvement, but please manage the rollout responsibly so you don't screw over contrib developers like me.

Also note, I posted #2536786: Continuous integration tests just two days ago, after two weeks of finding no information about this feature.

Re: #23 @TR - Some draft documentation of configuring DrupalCI as a project maintainer has been added here: https://www.drupal.org/node/2538838

Feedback welcome, of course.

#2495411-115: Make simpletest fail a test when it detects pages that need more than 64MB seems to be showing worrisomely different results on DrupalCI versus PIFR.

I still find the DrupalCI test results very difficult to understand.

Take this example in: https://dispatcher.drupalci.org/job/default/6071/testReport/junit/Module...

Regression

Module.Drupal\system\Tests\Module\DependencyTest.testEnableRequirementsFailureDependency (from TODO_ Set Package.Module)

Failing for the past 1 build (Since Unstable#6071 )
Took 0 ms.
Error Message

Modules status has been updated.
Standard Output

pass: [Other] Line 104 of core/modules/system/src/Tests/Module/DependencyTest.php:
Module "requirements1_test" is not enabled.

pass: [Other] Line 105 of core/modules/system/src/Tests/Module/DependencyTest.php:
Module "requirements2_test" is not enabled.

pass: [Browser] Line 115 of core/modules/system/src/Tests/Module/DependencyTest.php:
GET http://localhost/admin/modules returned 200 (405.67 KB).

pass: [Browser] Line 115 of core/modules/system/src/Tests/Module/DependencyTest.php:
Valid HTML found on "http://localhost/admin/modules"

pass: [Browser] Line 115 of core/modules/system/src/Tests/Module/DependencyTest.php:
GET http://localhost/admin/modules returned 200 (405.93 KB).

pass: [Browser] Line 115 of core/modules/system/src/Tests/Module/DependencyTest.php:
Valid HTML found on "http://localhost/admin/modules"

pass: [Other] Line 119 of core/modules/system/src/Tests/Module/DependencyTest.php:
Module "requirements1_test" is not enabled.

pass: [Other] Line 120 of core/modules/system/src/Tests/Module/DependencyTest.php:
Module "requirements2_test" is not enabled.

pass: [Other] Line 124 of core/modules/system/src/Tests/Module/DependencyTest.php:
Module "comment" is enabled.

It took running the test locally to figure out what the fail was, because it says the test fails, and then lists a bunch of passes. Really confusing. Furthermore, the passes all have the full file and line number with the assertion, but the fail does not.

#2337191-15: Split up EntityManager into many services has some disconcerting results.
The legacy bot failed (legitimately), but DrupalCI didn't catch it:
https://qa.drupal.org/pifr/test/1119278
https://www.drupal.org/pift-ci-job/16043

Oh, actually it had the error, it just didn't flag it. See https://dispatcher.drupalci.org/job/default/6144/consoleFull, search for "PHP Fatal error".

We had a pow-wow today with most of the core committers (@xjm, @alexbronstein, @alexpott, @catch) as well as most of the DA tech team (@drumm, @basic, @isntall, @joshuami, @Mixologic, @hestenet, @japerry... hope I didn't forget anyone!) to go through the list at #2267715: [meta] Drupal.org (websites/infra) blockers to a Drupal 8 release.

We didn't get a chance to delve into this topic too much; planning to do so on another call in a couple of weeks. But here were a few things that came up:

1. Stability issues were raised; test results currently seem very inconsistent. From the PostgreSQL log:

August 7, 2015 - 11:37	

    PHP 5.5 & PostgreSQL 9.1 13,054 pass, 111 fail

August 6, 2015 - 11:34	

    PHP 5.5 & PostgreSQL 9.1 13,061 pass, 67 fail

August 5, 2015 - 11:36	

    PHP 5.5 & PostgreSQL 9.1 13,003 pass, 98 fail

August 4, 2015 - 11:34	

    PHP 5.5 & PostgreSQL 9.1 13,005 pass, 78 fail

August 3, 2015 - 11:33	

    PHP 5.5 & PostgreSQL 9.1 12,980 pass, 97 fail

While you might expect some sort of minor variation if we accidentally commit a regression one day, or fix some test failures another, it's extremely odd that 5 subsequent days would result in 5 completely different pass/fail numbers. It's basically very hard to ever find more than two days in a row in the log that match. :\

2. Another place this shows up is in discrepancies between DrupalCI / PIFT, which currently rely on a manual reporting process by a tiny, tiny subset of people who a) are developing against a project with DrupalCI turned on, b) are very observant, c) know this issue exists (all 17 of them :P), and d) have the time to write down what they are observing. Suggestion was for someone with access to write a script to automate these comparisons (e.g. compare a dump from the previous day from both PIFT and Jenkins) to more quickly draw out the areas that mismatch so that analysis can be performed on the diff to figure out if we have any other nasties like #2538478: PHP Fatal Errors during unit tests should show as a failure lurking about.

3. We also noted that PHP 5.3 (and 5.2?) support is required for D7 prior to shutting off PIFT, so opened #2551233: Ensure that DrupalCI supports D6 testing on the 5.3 environment for parity with PIFT/PIFR for that.

DA staff have created a plan to phase out the PIFT/PIFR testbots in two stages. First disabling the old bots for D8 testing - then disabling the D7 and D6 bots.

This will allow us to disable the bulk of testing on the old bots sooner - without compromising functionality for d6/d7 testing while we're still bringing DCI testing up to parity for those environments.

The goal is to have D8 testing disabled on old testbots around September 9th, so there is enough time to find any unexpected regressions before DrupalCon Barcelona sprints. We want to do this as soon as possible to avoid tripping up any release candidate sprinting that is likely to be occurring at that time.

I've updated the summary above to reflect the two phases.

_________________________

As a side note - there are a few issues that would ideally need to be fixed in run_tests.sh in core. (Not only in D8, but in D7 and D6 as well). Existing PIFT/PIFR testbots work around these issues in code - but those are not ideal long term solutions.

We want to fix these issues the 'right way' but will likely have to re-implement similar work arounds in DrupalCI for the short term.

The affected issues are:
#2538478: PHP Fatal Errors during unit tests should show as a failure
#2538462: Contrib Testing Test discovery not working for d6/d7

Look for follow up issues after this meta has been closed for core fixes in these areas.

#2538462: Contrib Testing Test discovery not working for d6/d7 should be considered a blocker for this. None of the Ubercart tests will run in DrupalCI because of that issue. And if that issue doesn't get fixed in the next few days, September 9th will be too soon - you need to allow a reasonable amount of time AFTER #2538462: Contrib Testing Test discovery not working for d6/d7 is fixed before turning off the old testbot for D8 so that we can work out any bugs in the new bot. Right now I have no idea what the new bot does with my tests because they won't run at all.

@TR - That is absolutely the goal. I'm sure you're already following #2538462: Contrib Testing Test discovery not working for d6/d7 - but as soon as we have something there it would be great to have your feedback as to how that looks. Any representatives from contrib-land that can help us validate these things are appreciated.

I think #2484119: Create a 'syntax linting' build step plugin probably belongs in the D8 blocker list. There's no reason we should ever be committing PHP syntax errors to D8.

Also, are you sure that a little over a week is enough time to code, review, test, deploy, and actually test all the things in that blocker list? I'm a bit skeptical, myself, given only one of the things in that list has even been started yet. :\

What's the fallback strategy if that date is missed? Post-Barcelona?

@webchick: It's ambitious. We think this is the best-scoped path we have to being able to get some bots shut off prior to Barcelona. If we can't clear these by that time frame than we'll probably have to leave the full suite of old bots running in parallel and take the 2x cost hit during BCN sprints.

Being non-disruptive is the first priority- so if it comes to that I think we'll have to see if we can find some cost savings elsewhere.

I had a look just now for somewhere in the jenkins output that shows what the fatal errors are when a test doesn't complete due to a fatal error - PIFR showed these at the bottom of the console log. Not sure if I'm just not finding it or it actually isn't there though.

Is this issue (or something similar) already known: #2565567: Inconsistent patch test results for old and new testbot?
In my eyes, that would be a definitive blocker for disabling the old testbot, but it would seem this doesn't happen for most other projects?

Tagging, so this shows up on the Composer board

Runtime assertions must be enabled. They are currently left off. In my research (and drupal's documentation as a result) I stated that assert.active on is the default. Turns out that's true only insofar as PHP's default ini file has them turned on. If a value is never passed for the setting it defaults to off. The test bot's have the assert.active line in their config, but it's commented out turning the setting off.

No test directly testing runtime assertions will pass with this setting as is. I'm surprised the SimpleTestTest hasn't been uncovered as a problem - does this new system test simpletest?

@Aki Tendo which bots have the assert.active disabled?

According to the PHP documentation this value defaults to true
http://php.net/manual/en/info.configuration.php
assert.active "1" PHP_INI_ALL

Looking at the DCI 5.3, 5.4, 5.5, 5.6, and 7
assert.active => 1 => 1

All D8 projects which had legacy testing enabled have now been set up with DrupalCI testing.

Disabling legacy testing for D8 might be as simple as removing it from http://cgit.drupalcode.org/project_issue_file_test/tree/pift.cron.inc#n443.

Investigation of my issue is continuing - it isn't the assert.active setting as first thought but a failure of the program code flow to reliably register the PHP 5 assert.callback used to cause assertion failures to throw \AssertionError (and set assert.exception = 1 on PHP 7) I'll be able to look at this closer this weekend.

I filed https://github.com/unicorn-fail/dreditor/pull/265 to reset some of Dreditor's integration to not emphasize the legacy tests.

Disabling legacy testing for D8 might be as simple as removing it from http://cgit.drupalcode.org/project_issue_file_test/tree/pift.cron.inc#n443.

The actual way to do that will be updating the pift_core variable.

I've heard there is some concern over losing #2484119: Create a 'syntax linting' build step plugin from when we disable legacy D8 testing to when DrupalCI has it deployed. Should #2484119: Create a 'syntax linting' build step plugin be a blocker to disabling legacy D8 testing?

#2579417: Only check pift_core_api_versions() for legacy tests is the blocker for disabling these on Drupal.org

I don't think we need syntax listing for shutoff. Tests should fail anyway, that just makes them fail really quickly. Which in the case of head being broken would mean an immediate warning instead of 20+ minutes. So useful but we are all ignoring pifr entirely now for core so not missing this much.

Added another 2 critical blockers.

#2579417: Only check pift_core_api_versions() for legacy tests is now deployed, so unchecking the 8.x PIFT configuration on Drupal.org will be okay.

Unless there are objections, I plan to deploy #2579977: Remove legacy automatic retesting later today or tomorrow. Then the automatic RTBC retests would be DrupalCI-only.

Had a conversation in #drupal-infrastructure about turning legacy off for 8.x core.

catch, alexpott, xjm, and I were all in agreement: let's do it. Of the two issues added in #69, #2579985: [Policy] Should tests fail based on strict standards warnings is apparently not something PIFT does today, so would be a new feature (?) (and even if not, feature parity here is not a deal-breaker; we have workarounds like PHPCS, etc. to make corrections en masse), and #2580293: Patch having test with "PHP Fatal error" is marked as PASSED sounds a bit worse than it actually is. There is handling for PHP fatal errors (see http://cgit.drupalcode.org/drupal/tree/core/modules/simpletest/src/TestB...) but this edge case got missed. @Mixologic has a plan for adding generic "stuff we didn't think of" checking to Jenkins.

So barring any last-minute objections by anyone, legacy testbots will aim to be disabled on 8.x core Thursday morning Eastern US time. That way it's as soon as possible (we've been effectively ignoring PIFT since during DrupalCon, and lost hours of time in the past couple of days in the pre-RC sprint because PIFT comes up 2 hours after the fact and marks an issue "needs work" that was already committed :P), but doesn't present any risk to the RC1 window tomorrow.

\m/ YEAH! \m/

So barring any last-minute objections by anyone, legacy testbots will aim to be disabled on 8.x core Thursday morning Eastern US time.

It is now Thursday morning.

8.x legacy testing, for both core and contrib is now disabled.

\m/ Awesome, thank you :D :D

super awesome.

@drumm
thanks for the tweet too!
https://twitter.com/drupal_infra/status/652141162375938049

we should also update the irc factoids...

YesCT
qa?
Druplicon
qa is Drupal Quality Assurance, newly https://www.drupal.org/node/3060/qa and used to be PIFR testing patches http://qa.drupal.org
YesCT
testbot?
Druplicon
Drupal.org's patch QA (a.k.a "testbot") is composed of several projects. For details, or for help troubleshooting testbot issues, see: http://drupal.org/project/testbot | See testbot health at http://qa.drupal.org/ - See: testbot broken? | Get test result notifications. See: notifier?
YesCT
testbot broken?
Druplicon
testbot broken is https://www.drupal.org/node/2395675

New IRC factoids:

qa?

qa is Drupal Quality Assurance, now DrupalCI - Drupal Continuous Integration. How to use DrupalCI: https://www.drupal.org/node/2538838. Understanding results on Drupal.org: https://www.drupal.org/node/2532396. Core’s automated testing page: https://www.drupal.org/node/3060/qa.  The old testbots used PIFR:  http://qa.drupal.org. 


DrupalCI?
DrupalCI is Drupal Continuous Integration. How to use DrupalCI: https://www.drupal.org/node/2538838. Understanding results on Drupal.org: https://www.drupal.org/node/2532396. Core’s automated testing page: https://www.drupal.org/node/3060/qa.  The old testbots used PIFR:  http://qa.drupal.org. 


testbot?

Testbot is shorthand for Drupal.org's testing infrastructure. The testbots are part of DrupalCI. For details, or for help troubleshooting testbot issues, see: http://drupal.org/project/drupalci | See testing details on the automated testing tab of Projects. See: testbot broken? | Get test result notifications: https://www.drupal.org/node/2583775

testbot broken?

testbot broken is https://www.drupal.org/node/2395675

notifier?

notifier is Email notifications for DrupalCI tests: https://www.drupal.org/node/2583775. | qa.drupal.org tests notifier is a plugin https://chrome.google.com/webstore/detail/drupal-qa-notifier/fcgjigcnkbhpjdhimnhlieoplnpoojhi to subscribe to testbot results, by justafish |  For firefox https://github.com/larowlan/drupal_qa_notifier by larowlan

Infrastructure currently blocks the Drupal 8 run-tests.sh script exiting with the correct status code, opened this: #2378713: Results broken when run-tests exits with appropriate error code

#2618238: Enable DrupalCI for Drupal 7 just made for d7

Taking this for the next phase of work. I've disabled legacy testing for all Drupal core versions with David_Rothstein's approval. And I've motivated moving all projects maintained by Association staff members.

That leaves automated migration of the remaining 1,499 D7 and 316 D6 releases to be done next week.

Once that's done, there's code cleanup and static archiving QA.Drupal.org to do.

I opened #2635846: Remove legacy test UI (except for existing results) for the cleanup work in project_issue_file_test module. The initial deployment, disallowing new tests from being configured with DrupalCI, is being deployed.

Legacy D6 testing is now shut off on Drupal.org and the replacement DrupalCI tests are queueing.

Legacy D7 testing is migrating right now. The DrupalCI tests have been queued up.

QA.Drupal.org is now inactive.

Huzzah!!