Random testbot aborts [#2607494]

We have added a new ami, and DrupalCI is not happy with it. The tests begin running, then timeout and abort after 50 minutes:

This appears to be the first occurrence of this phenomenon: (which was ~ Oct 29th at 5pm pdt)
https://dispatcher.drupalci.org/job/default/29615/console

We are seeing timeout aborts on the following jobs:
29636
29656
29681
29682
29705
29713
29731
29753
29792
29810
29873
29882
29909
29910
29919
29921
29929
30035
30039
30046
30083
30155
30164
30173
30184
30238
30318
30350
30375
30378
30399
30402
30419
30485
30602
30607
30619
30636
30658
30661
30689
30692
30717
30766
30785
30786
30824
30835
30838
30854
30859
30860
30938
30949
31034
31055
31069
31088
31102
31106
31169
31172
31179
31180
31190
31213
31216
31221
31242
31281
31295
31311
31337

Comments

Comment #1

3 November 2015 at 16:20

Mixologic created an issue. See original summary.

Comment #2

isntall commented 3 November 2015 at 18:41

Issue summary:

View changes

Looking at the timestamps this happened on Oct 29, around 5pm pdt, pretty much as soon as we switched to the new AMI.

Comment #3

isntall commented 3 November 2015 at 23:58

We think we've isolated the problem to a new version of mysql (5.5.46) and a fix will be applied (revert to the older container with mysql 5.5.44).
The fix will be done in stages, to confirm that it helps, we'll use docker pulls to update the images and if that fixes things we'll rebuild the AMI with the mysql 5.5.44 container.

A little more info, on what seems to be random occasions mysql seems to hang and testing stops. This does not seem to happen with enough regularity that we can pinpoint it at this time.