Add email notifications

Marking as critical since I believe it's a blocker to #2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure.

Here's an old PIFT email:

Subject: [failed] Drupal core - 8.0.x on Wed, 04/15/2015 - 17:58:19

Body:

Drupal core - 8.0.x fail: https://qa.drupal.org/8.0.x-status

Overall Summary
==============================================

FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 90,487 pass(es), 522 fail(s), and 294 exception(s).

Individual Environment Summaries
==============================================
-- [[SimpleTest]]: [PHP 5.4 MySQL] --

* Drupal\comment\Tests\CommentFieldsTest (122 pass(es), 4 fail(s), and 2 exception(s))
   - [exception] [Warning] "simplexml_import_dom(): Invalid Nodetype to importsimplexml_import_dom(Object)
Drupal\simpletest\WebTestBase->parse()
Drupal\simpletest\WebTestBase->drupalPostForm('admin/config/people/accounts/fields/user.user.field_user_comment/storage', Array, 'Save field settings')
Drupal\comment\Tests\CommentFieldsTest->testCommentFieldCreate()
Drupal\simpletest\TestBase->run(Array)
simpletest_script_run_one_test('103', 'Drupal\comment\Tests\CommentFieldsTest')
" in AssertContentTrait.php on line 133 of Drupal\simpletest\WebTestBase->parse().
   - [fail] [Other] ""An illegal choice has been detected. Please contact the site administrator." found" in CommentFieldsTest.php on line 119 of Drupal\comment\Tests\CommentFieldsTest->testCommentFieldCreate().
   - [exception] [Warning] "simplexml_import_dom(): Invalid Nodetype to importsimplexml_import_dom(Object)
Drupal\simpletest\WebTestBase->parse()
Drupal\simpletest\WebTestBase->drupalPostForm('admin/config/people/accounts/fields/user.user.field_user_comment/storage', Array, 'Save field settings')
Drupal\comment\Tests\CommentFieldsTest->testCommentFieldCreate()
Drupal\simpletest\TestBase->run(Array)
simpletest_script_run_one_test('103', 'Drupal\comment\Tests\CommentFieldsTest')
" in AssertContentTrait.php on line 133 of Drupal\simpletest\WebTestBase->parse().

[snip]

For more information please visit the test detail page [https://qa.drupal.org/8.0.x-status].

-- 
Your friendly automated testing system 

What I like about this:

1) The subject line tells you everything you need to know at a glance: what happened ("[failed]"), which project, which branch, what time.
2) The link to get more detailed info is right front-and-center at the beginning of the mail.
3) It then goes into a very useful summary so I can tell in one line what happened.
4) Then a barf of details if I care to dig in further.
5) The barf of details is fairly human-readable.

Contrast this with DrupalCI's notifications:

Subject: Build failed in Jenkins: php5.5 mysql5.5 Simpletest Runner #129

See <http://dispatcher-origin.drupalci.aws:8080/job/php5.5_mysql5.5/129/>

------------------------------------------
Started by remote host 140.211.10.34
Building remotely on 455a815d-5fa4-4462-b9fb-11adea904fae (testrunner) in workspace <http://dispatcher-origin.drupalci.aws:8080/job/php5.5_mysql5.5/ws/>
[web] $ /bin/bash /tmp/hudson271845043092153349.sh
+ cd /opt/drupalci_testbot
+ sudo git pull --rebase
Current branch production is up to date.
+ env
+ grep DCI
DCI_CheckoutDir=<http://dispatcher-origin.drupalci.aws:8080/job/php5.5_mysql5.5/ws/jenkins-php5.5_mysql5.5-129/checkout>
DCI_RunScript=/var/www/html/scripts/run-tests.sh
DCI_PHPVersion=5.5
DCI_JobType=simpletest
DCI_CoreBranch=7.x
DCI_Color=True
DCI_JunitXml=xml
DCI_Concurrency=27
DCI_CoreRepository=git://git.drupal.org/project/drupal.git
DCI_DBVersion=mysql-5.5
+ env
+ grep -v DCI
BUILD_URL=http://dispatcher-origin.drupalci.aws:8080/job/php5.5_mysql5.5/129/
HUDSON_SERVER_COOKIE=f9f94f9baaa33b04
TERM=linux
BUILD_TAG=jenkins-php5.5_mysql5.5-129
WORKSPACE=<http://dispatcher-origin.drupalci.aws:8080/job/php5.5_mysql5.5/ws/>
JOB_URL=http://dispatcher-origin.drupalci.aws:8080/job/php5.5_mysql5.5/
JENKINS_HOME=/var/lib/jenkins
PATH=/sbin:/usr/sbin:/bin:/usr/bin
runlevel=2
PWD=/opt/drupalci_testbot
HUDSON_URL=http://dispatcher-origin.drupalci.aws:8080/
JOB_NAME=php5.5_mysql5.5
PREVLEVEL=N
BUILD_DISPLAY_NAME=#129
previous=N
JENKINS_URL=http://dispatcher-origin.drupalci.aws:8080/
BUILD_ID=129
Drupal_JobID=https://www.drupal.org:5456
SHLVL=1
JENKINS_SERVER_COOKIE=f9f94f9baaa33b04
EXECUTOR_NUMBER=0
NODE_LABELS=455a815d-5fa4-4462-b9fb-11adea904fae testrunner
UPSTART_EVENTS=runlevel
HUDSON_HOME=/var/lib/jenkins
NODE_NAME=455a815d-5fa4-4462-b9fb-11adea904fae
BUILD_NUMBER=129
UPSTART_JOB=rc
HUDSON_COOKIE=9773ef03-eb5d-4b8c-aabf-57b1d5bb3c45
OLDPWD=<http://dispatcher-origin.drupalci.aws:8080/job/php5.5_mysql5.5/ws/>
_=/usr/bin/env
+ ./drupalci run simpletest
Calculating job definition
Loading DrupalCI platform default arguments:
DCI_CodeBase
Loading job type default arguments:
DCI_DBVersion,DCI_PHPVersion,DCI_CoreRepository,DCI_CoreBranch,DCI_GitCheckoutDepth,DCI_RunScript,DCI_DBUser,DCI_DBPassword,DCI_DBUrl,DCI_TestGroups,DCI_SQLite,DCI_Concurrency,DCI_PHPInterpreter,DCI_RunOptions
Loading local namespaced environment variable override arguments.
DCI_CheckoutDir,DCI_RunScript,DCI_PHPVersion,DCI_JobType,DCI_CoreBranch,DCI_Color,DCI_JunitXml,DCI_Concurrency,DCI_CoreRepository,DCI_DBVersion
Creating codebase data volume directory
Checkout Directory created at <http://dispatcher-origin.drupalci.aws:8080/job/php5.5_mysql5.5/ws/jenkins-php5.5_mysql5.5-129/checkout>
Parsing required database container image names ...
Adding image: drupalci/mysql-5.5
Validating container images exist
Found image drupalci/mysql-5.5 with ID 8a68e487
Found existing drupalci/mysql-5.5 service container instance.
Parsing required Web container image names ...
Adding image: drupalci/5.5
Validating container images exist
Found image drupalci/web-5.5 with ID 51766a25
Populating container codebase data volume.
Entering setup_checkout_git().
Performing git checkout of git://git.drupal.org/project/drupal.git 7.x branch to <http://dispatcher-origin.drupalci.aws:8080/job/php5.5_mysql5.5/ws/jenkins-php5.5_mysql5.5-129/checkout.>
Git Command: git clone -b 7.x --depth 1 git://git.drupal.org/project/drupal.git '<http://dispatcher-origin.drupalci.aws:8080/job/php5.5_mysql5.5/ws/jenkins-php5.5_mysql5.5-129/checkout'>
Cloning into '<http://dispatcher-origin.drupalci.aws:8080/job/php5.5_mysql5.5/ws/jenkins-php5.5_mysql5.5-129/checkout'...>
Checkout complete.
Container /fervent_cray created from image drupalci/web-5.5 with ID 9e2a34a1
Executing on container instance 9e2a34a1:
mkdir -p /var/www/html/results
Command created as exec id a324257e
HTTP/1.1 200 OK
Content-Type: application/vnd.docker.raw-stream


Executing on container instance 9e2a34a1:
mkdir -p /var/www/html/artifacts
Command created as exec id a96d51df
HTTP/1.1 200 OK
Content-Type: application/vnd.docker.raw-stream


Executing on container instance 9e2a34a1:
mkdir -p /var/www/html/sites/simpletest/xml
Command created as exec id 28dbf8d4
HTTP/1.1 200 OK
Content-Type: application/vnd.docker.raw-stream


Executing on container instance 9e2a34a1:
chown -fR www-data:www-data /var/www/html/sites /var/www/html/results
Command created as exec id a8f43dbf
HTTP/1.1 200 OK
Content-Type: application/vnd.docker.raw-stream


chmod 0777 /var/www/html/artifacts
Command created as exec id fdc69be7
HTTP/1.1 200 OK
Content-Type: application/vnd.docker.raw-stream


Executing on container instance 9e2a34a1:
mysql -h drupaltestbot-db-mysql-5-5 -u drupaltestbot -pdrupaltestbotpw -e 'CREATE DATABASE jenkins_php55_mysql55_129'
Command created as exec id 1b794342
HTTP/1.1 200 OK
Content-Type: application/vnd.docker.raw-stream


Executing on container instance 9e2a34a1:
php /var/www/html/core/scripts/run-tests.sh --list --php /opt/phpenv/shims/php > /var/www/html/artifacts/testgroups.txt
Command created as exec id 5743a60b
HTTP/1.1 200 OK
Content-Type: application/vnd.docker.raw-stream


cd /var/www/html && sudo -u www-data php /var/www/html/scripts/run-tests.sh --color --concurrency 27 --sqlite /var/www/html/results/simpletest.sqlite --php /opt/phpenv/shims/php --keep-results --dburl mysql://drupaltestbot:drupaltestbotpw@drupaltestbot-db-mysql-5-5/jenkins_php55_mysql55_129 --all
Command created as exec id 29ec72c5
  ERROR: Unknown argument '--sqlite'.
HTTP/1.1 200 OK
Content-Type: application/vnd.docker.raw-stream


Gathering job build artifacts in a common directory ...
Executing on container instance 9e2a34a1:
mkdir -p /var/www/html/artifacts
Command created as exec id 10b923b2
HTTP/1.1 200 OK
Content-Type: application/vnd.docker.raw-stream


Executing on container instance 9e2a34a1:
cat >/var/www/html/artifacts/jobDefinition.txt <<EOL 
Array
(
    [environment] => Array
        (
            [db] => Array
                (
                    [0] => mysql-5.5
                )

            [web] => Array
                (
                    [0] => 5.5
                )

        )

    [setup] => Array
        (
            [checkout] => Array
                (
                    [0] => Array
                        (
                            [protocol] => git
                            [repo] => git://git.drupal.org/project/drupal.git
                            [branch] => 7.x
                            [depth] => 1
                            [checkout_dir] => .
                        )

                )

            [mkdir] => Array
                (
                    [0] => /var/www/html/results
                    [1] => /var/www/html/artifacts
                    [2] => /var/www/html/sites/simpletest/xml
                )

            [command] => Array
                (
                    [0] => chown -fR www-data:www-data /var/www/html/sites /var/www/html/results
                    [1] => chmod 0777 /var/www/html/artifacts
                )

        )

    [dbcreate] => Array
        (
            [mysql] => 
        )

    [install] => 
    [execute] => Array
        (
            [command] => Array
                (
                    [0] => php /var/www/html/core/scripts/run-tests.sh --list --php /opt/phpenv/shims/php > /var/www/html/artifacts/testgroups.txt
                    [1] => cd /var/www/html && sudo -u www-data php /var/www/html/scripts/run-tests.sh --color --concurrency 27 --sqlite /var/www/html/results/simpletest.sqlite --php /opt/phpenv/shims/php --keep-results --dburl mysql://drupaltestbot:drupaltestbotpw@drupaltestbot-db-mysql-5-5/jenkins_php55_mysql55_129 --all
                )

        )

    [publish] => Array
        (
            [gather_artifacts] => /var/www/html/artifacts
            [junit_xmlformat] => xml
        )

)

EOL
Command created as exec id a44949ff
HTTP/1.1 200 OK
Content-Type: application/vnd.docker.raw-stream


Skipping /var/www/html/artifacts/testgroups.txt, as it already exists in the build artifact directory.
Executing on container instance 9e2a34a1:
cp -r /var/www/html/results/simpletest.sqlite /var/www/html/artifacts/simpletest.sqlite
Command created as exec id 7e9cdfc1
cp: cannot stat '/var/www/html/results/simpletest.sqlite'HTTP/1.1 200 OK
Content-Type: application/vnd.docker.raw-stream


PHP Warning:  SQLite3::prepare(): Unable to prepare statement: 1, no such table: simpletest in /opt/drupalci_testbot/src/DrupalCI/Plugin/BuildSteps/publish/JunitXMLFormat.php on line 82
PHP Fatal error:  Call to a member function execute() on a non-object in /opt/drupalci_testbot/src/DrupalCI/Plugin/BuildSteps/publish/JunitXMLFormat.php on line 83
Build step 'Execute shell' marked build as failure
Recording test results
ERROR: Publisher 'Publish JUnit test result report' failed: No test report files were found. Configuration error?

What I like about this

1) The subject line has the environment details in it (php5.5 mysql5.5)
2) Super detailed information is available, including what commands were passed to run-tests.sh so we can check for any weirdness.
3) I get a follow-up email when things are working properly so I know I don't need to take any action if I get this e-mail while on a walk with my family.

What I don't like about this:

1) The rest of the subject line is extremely obtuse.
2) More importantly, the project name / branch isn't mentioned, which is the most critical info.
3) The link at the top for more info goes to some http://dispatcher-origin.drupalci.aws:8080 site that I do not have access to see.
4) There's no summary of what happened. I'm left instead to parse through a barf of details to figure it out.
5) The barf of details is not human-readable. Or I guess it is, but not until the very end:

PHP Warning:  SQLite3::prepare(): Unable to prepare statement: 1, no such table: simpletest in /opt/drupalci_testbot/src/DrupalCI/Plugin/BuildSteps/publish/JunitXMLFormat.php on line 82
PHP Fatal error:  Call to a member function execute() on a non-object in /opt/drupalci_testbot/src/DrupalCI/Plugin/BuildSteps/publish/JunitXMLFormat.php on line 83
Build step 'Execute shell' marked build as failure
Recording test results
ERROR: Publisher 'Publish JUnit test result report' failed: No test report files were found. Configuration error?

Can we invert the details there at all?

6) Those subsequent emails can sometimes be confusing... this evening I got several of them in rapid succession:

Build failed in Jenkins: php5.5 mysql5.5 Simpletest Runner #125
Build failed in Jenkins: php5.5 mysql5.5 Simpletest Runner #129
Jenkins build is unstable: php5.5 mysql5.5 Simpletest Runner #126 (this one had no details, just the link to something I can't see.)
Jenkins build is back to stable : php5.5 mysql5.5 Simpletest Runner #127 (likewise.)
Jenkins build is back to normal : php5.5 mysql5.5 Simpletest Runner #130 (likewise.)

This leaves with a bunch of questions:

a) What do those numbers mean? Are they in reference to specific testbots? Do they just count up? (can we place a label there?)
b) What's the difference between unstable and failed?
c) What's the difference between stable and normal?
d) Can I do anything about unstable/stable conditions? I'm assuming not, so why am I notified about them?

---

Sorry for all the barf, but I wanted to try and add some details around (for me at least) what "we need better emails" means.