Problem/Motivation

Follow-up to #2828559: UpdatePathTestBase tests randomly failing. We added gc_disable() to update.php because it improved stability of running update tests on PHP5.5.

We should determine:

  • if we should do gc_disable() for all update.php and not just test runs.
  • why gc_disable() is effective

After #2749955: Random fails in UpdatePathTestBase tests, we are continuing to see random fails in UpdatePathTestBase tests. For example, from #2627512-139: Datetime Views plugins don't support timezones:

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Comments

alexpott created an issue. See original summary.

catch’s picture

@alexpott mentioned that the test failures appear to be in the dfac() after updates have run (and before post_updates have run), not necessarily when running the updates themselves. Our update tests don't do any large-scale content updates, which would be the obvious culprit if they did.

tacituseu’s picture

http://git.php.net/?p=php-src.git;a=commitdiff;h=f29c98c1289b00e0dbb58631df6a5e006f5311d1 was fixed in PHP 5.5.24/5.6.9, testbot uses 5.6.7 for testing and yet it wasn't failing, so unless there is some other factor it would exclude it as culprit.

mpdonadio’s picture

Did we explore using USE_ZEND_ALLOC=0 to see if that helped matters? Needs to be in the environment itself, so I don't think we can make a patch to test this.

alexpott’s picture

Well USE_ZEND_ALLOC=0 is not going to help matters... it might elucidate where the problem is coming from - or might not.

berdir’s picture

I think as a starting point, some numbers on what the impact is on memory_get_peak_usage() for having gc enabled or not would be good. going to be different again when you actually have a serious amount of data to process but it might be a useful starting point

tacituseu’s picture

StatusFileSize
new2.03 KB

Not sure if there is less invasive way to do it.

tacituseu’s picture

Status: Active » Needs review
StatusFileSize
new2.49 KB
tacituseu’s picture

StatusFileSize
new2.03 KB

Once more.

tacituseu’s picture

StatusFileSize
new38.16 KB

Top offenders are: system_update_8004, system_update_8013, node_update_8003.

mem-per-update.png

Legend:
Before - peak before given update
After - peak after given update
Used - memory used by the update
Request - peak of the whole request (with couple of other updates)
all in MiB

So worst-case is 4 times as much memory used.
In the meantime testbot has been updated and is now using PHP 5.5.38, so it might no longer be an issue.
Notice: system_update_8004 is the one found in backtraces of segfaults from #2828559-115: UpdatePathTestBase tests randomly failing.

wim leers’s picture

Observations:

  1. system_update_8004() and node_update_8003() are both updating entity definitions.
  2. system_update_8013() is installing a module

Perhaps we can disable GC specifically for updating entity definitions and module (un)installation?

alexpott’s picture

Thing is according to the test results all the hook_update_N ran - otherwise the test fails would have been different. The tests showed that the hook_update_N had finished but the post updates had not run. So either it crashed in drupal_flush_all_caches() or the first post update. The first post update is quite innocuous BUT rebuilding all the caches that it might require is not.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

xjm’s picture

Priority: Critical » Major
Issue tags: -Triaged D8 critical +Triaged core major

This was accidentally opened as a triaged critical including with the tag because of an issue clone. We should be really careful not to do that as that means the criticals don't actually get discussed in triage.

Since this is a followup to resolve future test/stability issues without a failure in HEAD currently, it should be major.

tacituseu’s picture

StatusFileSize
new1.85 KB

Checking environments for garbage collector issues by forcing gc_collect_cycles() after each test.

Status: Needs review » Needs work

The last submitted patch, 15: 2842393-gc-collect-check.patch, failed testing.

Pavan B S’s picture

StatusFileSize
new1.85 KB
new759 bytes
+++ b/core/tests/Drupal/Tests/Listeners/DrupalStandardsListener.php
@@ -141,6 +141,8 @@ public function checkValidCoversForTest(\PHPUnit_Framework_TestCase $test) {
+    gc_collect_cycles();
...
     // subclasses.

index 59e808e..3ffd367 100644
--- a/update.php

+++ b/update.php
@@ -13,15 +13,6 @@
 

Line exceeding 80 characters
There is one line which exceeds more than 80 characters.Applying the patch, please review

Pavan B S’s picture

Status: Needs work » Needs review

The last submitted patch, 7: 2842393-7-gc-disabled.patch, failed testing.

tacituseu’s picture

@Pavan B S: these are just testing/debugging patches, not meant to be commited.

xjm’s picture

Version: 8.3.x-dev » 8.4.x-dev
StatusFileSize
new578 bytes

Thanks @Pavan B S and @tacituseu. Unchecking credit for @Pavan B S as per #20.

In #2828559: UpdatePathTestBase tests randomly failing we never conclusively determined if PHP 7 was affected by the same failure or if it was a different random failure at the time, but the fix does not do a PHP version check. (This is unlike the fix in #2859704: Intermittent segfaults on DrupalCI (some "did not complete due to a fatal error" with no additional info) where we only disabled it for PHP versions less than PHP 7.) Here's a patch that reverts #2828559: UpdatePathTestBase tests randomly failing to check the baseline of fails on both PHP 5.6 and PHP 7.

tacituseu’s picture

PHP versions on testbot changed since the time #2828559: UpdatePathTestBase tests randomly failing was committed, at the time they were PHP 5.5.23, PHP 5.6.7 and PHP 7.0.16-dev, one of the updates might've very well fixed that, or growth of the codebase just moved the segfault to FileFieldWidgetTest.
I tried testing for previous FrontPageTest failure with reverted DrupalKernel patch and it didn't trigger (see: #2879048-97: Ignore: patch testing issue for #2919863).

tacituseu’s picture

I think we got it, keeping references to byproducts of
core/lib/Drupal/Core/Extension/ModuleHandler.php::buildModuleDependencies()
stabilized it ;)
Attached with control patch from @Lendude and another not limited to FileFieldWidgetTest to check if it just moved to other place.
Results in other issue: test and control.

mpdonadio’s picture

Based off bugs references in https://www.drupal.org/node/2898721#comment-12198555, I think this indefinitely plausible since any graph can result in circular references.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

alexpott’s picture

Status: Needs review » Closed (outdated)

This no longer exists in 8.8.x as we've dropped support for PHP5