Updated: Comment #0

Problem/Motivation

Since November 18 or 19, numerous unrelated tests have been failing randomly on the testbots with the following error:

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 268435456 bytes) in /var/lib/drupaltestbot/sites/default/files/checkout/core/lib/Drupal/Core/Database/Connection.php on line 336

It is not may always be in Connection.php.

Examples:
https://qa.drupal.org/pifr/test/674688
https://qa.drupal.org/pifr/test/674683

The fatal occurs in completely random tests, sometimes just one fail, sometimes hundreds.

Proposed resolution

Track down and revert the cause of this regression, and then profile to determine how to fix that patch.

Remaining tasks

TBD

API changes

TBD

CommentFileSizeAuthor
#9 revert_only_of_fc04601.patch86.8 KBxjm
#6 [failed] Drupal core - 8.x on Thu, 11:21:2013 - 14-19-02 .txt701 bytesxjm
#6 [failed] Drupal core - 8.x on Thu, 11:21:2013 - 12-44-40 .txt259.89 KBxjm
#6 [failed] Drupal core - 8.x on Thu, 11:21:2013 - 11-15-02 .txt701 bytesxjm
#6 [failed] Drupal core - 8.x on Thu, 11:21:2013 - 01-28-38 .txt701 bytesxjm
#6 [failed] Drupal core - 8.x on Wed, 11:20:2013 - 23-36-02 .txt114.34 KBxjm
#6 [failed] Drupal core - 8.x on Wed, 11:20:2013 - 15-03-11.txt273.5 KBxjm
#6 [failed] Drupal core - 8.x on Wed, 11:20:2013 - 13-03-41 .txt107.33 KBxjm
#6 [failed] Drupal core - 8.x on Wed, 11:20:2013 - 10-05-11 .txt734 bytesxjm
#6 [failed] Drupal core - 8.x on Tue, 11:19:2013 - 17-36-48 .txt273.39 KBxjm
#6 [failed] Drupal core - 8.x on Mon, 11:18:2013 - 19-08-01.txt10.89 KBxjm
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

xjm’s picture

xjm
she/her
Primary language English
CreditAttribution: xjm commented
Issue summary: View changes
rfay’s picture

rfay
Primary language English
Location Palisade, CO, USA
CreditAttribution: rfay commented

We can easily increase the testbot memory_limit, which hopefully could give some breathing space to chase this. Unless it has to do with recursion or something that has nothing to do with actual memory space.

We can make available a dedicated testbot for doing some git bisect action to try to sort this out.

xjm’s picture

xjm
she/her
Primary language English
CreditAttribution: xjm commented
Issue summary: View changes
xjm’s picture

xjm
she/her
Primary language English
CreditAttribution: xjm commented

@amateescu reverting a suspect commit:
https://drupal.org/comment/8200987#comment-8200987

rfay’s picture

rfay
Primary language English
Location Palisade, CO, USA
CreditAttribution: rfay commented

@amateescu pointed out https://qa.drupal.org/pifr/test/675143 so I grepped on that server and found no sign of the error message (grep -r in /var/log after failing in /var/log/apache2). Then I realized this message must come from run_tests.sh instead of Apache/php

xjm’s picture

xjm
she/her
Primary language English
CreditAttribution: xjm commented
FileSize
[failed] Drupal core - 8.x on Mon, 11:18:2013 - 19-08-01.txt10.89 KB
[failed] Drupal core - 8.x on Tue, 11:19:2013 - 17-36-48 .txt273.39 KB
[failed] Drupal core - 8.x on Wed, 11:20:2013 - 10-05-11 .txt734 bytes
[failed] Drupal core - 8.x on Wed, 11:20:2013 - 13-03-41 .txt107.33 KB
[failed] Drupal core - 8.x on Wed, 11:20:2013 - 15-03-11.txt273.5 KB
[failed] Drupal core - 8.x on Wed, 11:20:2013 - 23-36-02 .txt114.34 KB
[failed] Drupal core - 8.x on Thu, 11:21:2013 - 01-28-38 .txt701 bytes
[failed] Drupal core - 8.x on Thu, 11:21:2013 - 11-15-02 .txt701 bytes
[failed] Drupal core - 8.x on Thu, 11:21:2013 - 12-44-40 .txt259.89 KB
[failed] Drupal core - 8.x on Thu, 11:21:2013 - 14-19-02 .txt701 bytes

Here's the HEAD failures for the past several days. (Unfortunately these do not include the full log so we can't check the actual PHP error in each one.)

xjm’s picture

xjm
she/her
Primary language English
CreditAttribution: xjm commented

Bisecting on the bot:
https://drupal.org/node/2126447#comment-8201283

amateescu’s picture

amateescu CreditAttribution: amateescu commented

Continued: https://drupal.org/node/2126447#comment-8201631

xjm’s picture

xjm
she/her
Primary language English
CreditAttribution: xjm commented
Status: Active » Needs review
Related issues: +#1954892: Replace 'theme callback' and hook_custom_theme() with a clean theme negotiation system
FileSize
revert_only_of_fc04601.patch86.8 KB

This is probably the one (from @amateescu). You can see the results of testing it in https://drupal.org/comment/8202607#comment-8202607. The one fail appears to be legitimately a bot issue and not related to the regression, see: https://qa.drupal.org/pifr/test/675568

catch’s picture

catch
he/him
Primary language English
CreditAttribution: catch commented
Status: Needs review » Fixed

Rolled that back. Thanks for the bisecting!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.