Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
So @xjm might be on to something with #8.
As it looks like the 'tested on commit' isn't really testing each of the triggering commits, but whatever given branch is at when it starts executing.
PHP Warning: file_put_contents(): Only 0 of 575 bytes written, possibly out of free disk space in /opt/drupalci/testrunner/src/DrupalCI/Build/Build.php on line 481
Re: #17 There is definitely a bug in drupalci where its not cleaning up after its own coredumps, so the same bot will re-report the existence of core dump files. I'll look into that today. (#2899031: Core dumps need to be cleaned up properly.
One way to tell, currently, if the core dump is related to the current run of tests is if there is a line in the console output that says: Cannot access memory at address 0x7ef95e818708
(with varying addresses, of course) -> that indicates that the core dump is from a different docker container, and not related to this test run.
Is there any reason not to make PHP 7 the default 'on commit' and 'issue testing' branch, the advantages are obvious and it represents the majority of the market.
Otherwise to support some (imaginary?) minority, we'll be commiting to a daunting task of trying to audit the whole codebase just not to step on some obsolete version's toes.
It is one thing to limit yourself to a given version's feature level but whole another to be limited by its bugs, if there's a distribution that is unwilling to update it should be its job to patch things up.
Comments
Comment #2
timmillwoodAdding https://www.drupal.org/pift-ci-job/728179 which looks to be the same issue.
Comment #3
tedbowComment #4
tacituseu CreditAttribution: tacituseu commentedComment #5
tacituseu CreditAttribution: tacituseu commentedComment #6
xjmIt's a segfault. Goody!
Comment #7
xjmThis is happening in a high rate in HEAD on 5.6 environments:
8.4.x: https://www.drupal.org/pift-ci-job/728200
8.5.x: https://www.drupal.org/pift-ci-job/728316
Comment #8
xjmOkay looking at the history of fails on both branches, this was most likely introduced by #2849674: Complex job in ViewExecutable::unserialize() causes data corruption. Uploading both a revert of this and a baseline patch, and queuing a bunch of runs to see if reverting will resolve the problem.
Comment #9
tacituseu CreditAttribution: tacituseu commentedhttps://www.drupal.org/pift-ci-job/728146 from #2891911: Random fail in Drupal\Tests\locale\Functional\LocaleTranslationUiTest::testStringTranslation failed even before that.
Comment #10
xjm@tacituseu, good catch. Hmm.
Comment #11
xjmHere's a revert of that one just to test. This is weird.
Comment #12
mpdonadioHave we gotten a core dump yet? Wonder if #2828559: UpdatePathTestBase tests randomly failing / #2842393: Discover why gc_disable() improves update.php stability are related? That fix only went into update.php. We may want to try a patch where we disable GC for all test runs...
Comment #13
xjmSo far I have only seen the segfault on:
#2891911: Random fail in Drupal\Tests\locale\Functional\LocaleTranslationUiTest::testStringTranslation was backported to 8.3.x though.
Comment #14
mpdonadioComment #15
tacituseu CreditAttribution: tacituseu commented@mpdonadio: almost all of them are in
gc_collect_cycles()
.Comment #16
tacituseu CreditAttribution: tacituseu commentedhttps://www.drupal.org/pift-ci-job/728169 was the first failing (in issue testing), did so 2 hours before the one from HEAD (https://www.drupal.org/pift-ci-job/728146).
The thing is, console logs say they both checked out the same commit:
So @xjm might be on to something with #8.
As it looks like the 'tested on commit' isn't really testing each of the triggering commits, but whatever given branch is at when it starts executing.
Comment #17
tacituseu CreditAttribution: tacituseu commentedAnother weird thing, even the passing runs from #8 (both baseline and revert) contain coredumps (stale data ??):
24725/artifacts/simpletest.standard/
24727/artifacts/simpletest.standard/
24732/artifacts/simpletest.standard/
Comment #20
catchGiven the revert of 43c568a71 is all green, I'm going to go ahead and do that to see if it improves the situation.
If people notice any more segfaults from now on, please post here since that'll mean it's not the culprit.
Comment #21
tacituseu CreditAttribution: tacituseu commentedPlenty of
Requeued after CI error
,CI aborted
, looks like testbots ran out of disk space:https://dispatcher.drupalci.org/job/php-7.1.x-apache_mysql5.5/397/consoleText
https://dispatcher.drupalci.org/job/php-5.6-apache_postgres9.1/449/consoleText
Comment #22
xjmAgreed on the revert for #2849674: Complex job in ViewExecutable::unserialize() causes data corruption to stop the damage (which catch already did). However, the segfault still did happen on
14c2920589cc9
which is the commit immediately before it happened. Also see #16.https://www.drupal.org/node/3060/qa is not pretty right now.
Rescoping. I think we should keep this open to continue to disucss the garbage collection angle (we can use the patch from #2849674: Complex job in ViewExecutable::unserialize() causes data corruption to stress-test) but I'll ping Mixologic again about the ongoing issues. https://www.drupal.org/node/3060/qa is not pretty right now.
Comment #23
xjmStill critical really.
Comment #24
tacituseu CreditAttribution: tacituseu commented@xjm: I don't think they happened on
14c2920589cc9
, see #16 for details, it wasn't actually testing14c2920589cc9
.Comment #25
MixologicIm pretty sure these segfaults are related to a bug that upstream does not plan on fixing: https://bugs.php.net/bug.php?id=72286
Re: #17 There is definitely a bug in drupalci where its not cleaning up after its own coredumps, so the same bot will re-report the existence of core dump files. I'll look into that today. (#2899031: Core dumps need to be cleaned up properly.
Comment #26
MixologicOne way to tell, currently, if the core dump is related to the current run of tests is if there is a line in the console output that says:
Cannot access memory at address 0x7ef95e818708
(with varying addresses, of course) -> that indicates that the core dump is from a different docker container, and not related to this test run.
Comment #28
xjmI dug into https://bugs.php.net/bug.php?id=72286 and eventually managed to find its scenario:
https://github.com/lightster/php-circular-reference-segfault/commit/ce85...
The author of the above says they're not sure if it's the same bug as https://bugs.php.net/bug.php?id=71958 or not.
The test result of #14 in https://www.drupal.org/pift-ci-job/728450 is an OOM and shows that we do need and benefit from garbage collection. ;)
Would #2899031: Core dumps need to be cleaned up properly. affect the segfaults at all, or only the subsequent (possibly unrelated) swath of "CI error" test results?
Comment #29
mpdonadioFor giggles, let's see if it is really this test or just the alignment of the way concurrent tests are going.
Comment #30
tacituseu CreditAttribution: tacituseu commentedIs there any reason not to make PHP 7 the default 'on commit' and 'issue testing' branch, the advantages are obvious and it represents the majority of the market.
Otherwise to support some (imaginary?) minority, we'll be commiting to a daunting task of trying to audit the whole codebase just not to step on some obsolete version's toes.
It is one thing to limit yourself to a given version's feature level but whole another to be limited by its bugs, if there's a distribution that is unwilling to update it should be its job to patch things up.
Comment #31
Anonymous (not verified) CreditAttribution: Anonymous commentedI sent the #2849674-49: Complex job in ViewExecutable::unserialize() causes data corruption patch with checking only FileFieldWidgetTest (200 times):
Comment #32
xjmSince #2849674: Complex job in ViewExecutable::unserialize() causes data corruption was reverted, this segfault is no longer happening in HEAD and so we have to work around it in that other issue. So, we can move this issue to fixed and focus discussion over there.
I've also changed the patch testing default environment to PHP 7 (as per #2607222: [policy, no patch] Default to PHP 7 for Drupal core patch testing). So this fail won't explode the entire patch queue if reintroduced.
Comment #33
LendudeWe've been running some tests in this in #2879048: Ignore: patch testing issue for #2919863 and not sure if I would call this 'fixed'. FileFieldWidgetTest is highly unstable, since the mere act of adding an empty test module to the code base can break it, see https://www.drupal.org/node/2879048#comment-12202738
Yeah the random fails are gone, but any patch that adds a test module can easily bring these back.
Comment #34
Anonymous (not verified) CreditAttribution: Anonymous commentedToday I ran the test on Windows without any patches and got a failure:
488 passes, 1 fail, 1 exception, 138 debug messages
Exception:
Fail:
Confirmed that access is denied for the file without the needed permission.
After re-run all good:
492 passes, 0 fails, 0 exceptions, 139 debug messages
Comment #35
Anonymous (not verified) CreditAttribution: Anonymous commentedThe last 8.5.x-dev, php 7.1.7, mysql 5.7.19. And maybe i'm not clear cache after patch with removed themes (#2879048-139: Ignore: patch testing issue for #2919863). But it's still suspicious.
Comment #37
Gábor HojtsyThe right status would be closed duplicate given another issue supposedly fixed what this issue said was a problem.
Comment #38
xjm