Both core 8.0.x and 8.1.x branch tests have random fails, all with the same issue:

https://www.drupal.org/pift-ci-job/143946

https://www.drupal.org/pift-ci-job/145094

Example failed job:
https://www.drupal.org/pift-ci-job/139574

Comments

Mixologic created an issue. See original summary.

xjm’s picture

Issue tags: +Random test failure
isntall’s picture

Assigned: Unassigned » isntall

As an update to this issue.

The Apache2 children are seg faulting

[Wed Jan 20 21:56:09.484749 2016] [core:notice] [pid 15] AH00051: child pid 3268 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Jan 20 21:56:25.502063 2016] [core:notice] [pid 15] AH00051: child pid 3282 exit signal Segmentation fault (11), possible coredump in /etc/apache2
......
[Wed Jan 20 22:02:44.905457 2016] [core:notice] [pid 15] AH00051: child pid 3251 exit signal Segmentation fault (11), possible coredump in /etc/apache2
isntall’s picture

The failing test class that I've been focusing on is
Update.Drupal\system\Tests\Update\UpdatePathTestBaseFilledTest

This issue seems to mainly affect AWS. I've tried re-creating it on a virtualbox and actual hardware.

I think I've narrowed the issue down to APCu, though upgrading to the latest stable version that is compatible, 4.0.10, yielded the same intermittent issue.

I haven't been able to get a core dump file; I'm not sure if that is my inexperience with core dump files or the docker container.

[Sun Jan 24 05:59:10.173506 2016] [core:notice] [pid 6] AH00051: child pid 535 exit signal Segmentation fault (11), possible coredump in /var/apache2-gdb-dump
[Sun Jan 24 05:59:15.180937 2016] [core:notice] [pid 6] AH00051: child pid 649 exit signal Segmentation fault (11), possible coredump in /var/apache2-gdb-dump
[Sun Jan 24 05:59:25.192451 2016] [core:notice] [pid 6] AH00051: child pid 534 exit signal Segmentation fault (11), possible coredump in /var/apache2-gdb-dump
[Sun Jan 24 06:00:09.236549 2016] [core:notice] [pid 6] AH00051: child pid 652 exit signal Segmentation fault (11), possible coredump in /var/apache2-gdb-dump
zend_mm_heap corrupted

With APCu disabled the test runs fine. The tests seem to run just as fast with APCu turned off.

catch’s picture

I haven't seen this segfault, but nearly every issue I've run into with APCu has usually been because it's got full/segmented and can't handle evictions - like the "can't allocate memory for pool' bug. Seems very likely that could happen with a full test run.

isntall’s picture

@catch I've been able to reproduce the error with running just one test group.

dave reid’s picture

Seeing a lot of failures with D7 core update tests as well: https://www.drupal.org/pift-ci-job/189110

Mixologic’s picture

The php 5.6 errors are, so far as we can deduce, all bugs in php5.6 related to garbage collection. We've attempted upgrades from 5.6.7 all the way to 5.6.18, and get different segfaults but always with the same tests (even in non-concurrency environments).

If there are issues on d7 with 5.3, I'd be hesitant to consider them related. Also, it looks like it extends beyond just the update tests in d7 - intermittently failing on Translation issues as well as Taxonomy issues.

Oh, garbage.

alexpott’s picture

So does this always occur in Drupal\system\Tests\Update\UpdatePathTestBaseFilledTest or it is always happening in Drupal\system\Tests\Update\UpdatePathTestBaseTest which quite a few tests extend? Also it looks like the fail is happening here core/update.php/start?id=4&op=do_nojs - so this is where we build a graph of update dependencies - maybe this might give us a way to repeat this.

alexpott’s picture

What's weird is that some similar random fails seem to go away if ConfigDependencyManager.php is patched in anyway... see #2762549: Drupal\field\Tests\Update\FieldUpdateTest, Drupal\views\Tests\Update\EntityViewsDataUpdateTest and Drupal\comment\Tests\CommentFieldsTest fail on 8.1.x

borisson_’s picture

Facets has similar random test fails for php5.6 as well, there are no failures on 5.5 or 7.0. https://www.drupal.org/pift-ci-job/369229

Drupal\Tests\facets\FunctionalJavascript\WidgetJSTest::testAddFacet
PHPUnit_Framework_Exception: Segmentation fault (core dumped)

I haven't looked into this yet, but I figured an extra datapoint might be useful.

Mixologic’s picture

Component: Code » Unexplained Failures/Support
xjm’s picture

Mixologic’s picture

Status: Active » Closed (cannot reproduce)

Okay, *this* particular php 5.6 + mysql 5.5. seems to not have returned since feb 29th of 2016. Not sure what changed, but lets check this particular one off the list as 'cannot reproduce'

xjm’s picture