Here is the full output from the most recent occurance:

<h1>Uncaught exception thrown in shutdown function.</h1><p>PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away: DELETE FROM {semaphore} 
WHERE  (value = :db_condition_placeholder_0) ; Array
(
    [:db_condition_placeholder_0] =&amp;gt; 188110706353494fe7577377.98358558
)
 in dblog_watchdog() (line 160 of /home/travis/build/panopoly/drupal/modules/dblog/dblog.module).</p><hr />PHP Fatal error:  Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 2006 MySQL server has gone away' in /home/travis/build/panopoly/drupal/includes/database/database.inc:2168
Stack trace:
#0 /home/travis/build/panopoly/drupal/includes/database/database.inc(2168): PDOStatement->execute(Array)
#1 /home/travis/build/panopoly/drupal/includes/database/database.inc(680): DatabaseStatementBase->execute(Array, Array)
#2 /home/travis/build/panopoly/drupal/includes/database/database.inc(2347): DatabaseConnection->query('SELECT expire, ...', Array, Array)
#3 /home/travis/build/panopoly/drupal/includes/lock.inc(167): db_query('SELECT expire, ...', Array)
#4 /home/travis/build/panopoly/drupal/includes/lock.inc(146): lock_may_be_available('schema:runtime:...')
#5 /home/travis/build/panopoly/drupal/includes/bootstrap.inc(424): lock_acquire('schema:runtime:...')
#6 /home/travis/build/panopoly/drupal/includes/bootstrap.inc(446): DrupalCacheArray->set(Array)
#7 [internal function]: DrupalCacheArray->__destruct()
#8 in /home/travis/build/panopoly/drupal/includes/database/database.inc on line 2168
PHP Stack trace:
PHP   1. {main}() /home/travis/build/panopoly/drupal/profiles/panopoly/tests/behat/vendor/behat/behat/bin/behat:0
PHP   2. Symfony\Component\Console\Application->run() /home/travis/build/panopoly/drupal/profiles/panopoly/tests/behat/vendor/behat/behat/bin/behat:32
Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 2006 MySQL server has gone away' in /home/travis/build/panopoly/drupal/includes/database/database.inc on line 2168
Call Stack:
    0.0003     241984   1. {main}() /home/travis/build/panopoly/drupal/profiles/panopoly/tests/behat/vendor/behat/behat/bin/behat:0
    0.0154    1438072   2. Symfony\Component\Console\Application->run() /home/travis/build/panopoly/drupal/profiles/panopoly/tests/behat/vendor/behat/behat/bin/behat:32
PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away in /home/travis/build/panopoly/drupal/includes/database/database.inc on line 2168
Call Stack:
    0.0003     241984   1. {main}() /home/travis/build/panopoly/drupal/profiles/panopoly/tests/behat/vendor/behat/behat/bin/behat:0
    0.0154    1438072   2. Symfony\Component\Console\Application->run() /home/travis/build/panopoly/drupal/profiles/panopoly/tests/behat/vendor/behat/behat/bin/behat:32
  931.8270   82666328   3. DrupalCacheArray->__destruct() /home/travis/build/panopoly/drupal/includes/bootstrap.inc:0
  931.8271   82667856   4. DrupalCacheArray->set() /home/travis/build/panopoly/drupal/includes/bootstrap.inc:446
  931.8272   82668160   5. lock_acquire() /home/travis/build/panopoly/drupal/includes/bootstrap.inc:424
  931.8276   82689560   6. lock_may_be_available() /home/travis/build/panopoly/drupal/includes/lock.inc:146
  931.8276   82690120   7. db_query() /home/travis/build/panopoly/drupal/includes/lock.inc:167
  931.8276   82690512   8. DatabaseConnection->query() /home/travis/build/panopoly/drupal/includes/database/database.inc:2347

Take one: I think the solution is just explicitly shutting down the webserver (ie. drush runserver) after finishing the Behat tests. That way it's shutdown before the MySQL server.

Take two: It's got to be comming from some cleanup function which uses the Drupal API driver.

CommentFileSizeAuthor
#19 panopoly-mysql-server-gone-away-2231631-19.patch1.14 KBdsnopek
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

lsolesen’s picture

lsolesen CreditAttribution: lsolesen commented

Can we tweak mysql? https://drupal.org/node/259580.

Asked via an issue in Behat on Github: https://github.com/Behat/Behat/issues/491

  • Commit 0d84b34 on 7.x-1.x by dsnopek: 
    Issue #2231631: Tests randomly fail with "MySQL server has gone away".
dsnopek’s picture

dsnopek
he/him
Primary language English
Location USA
CreditAttribution: dsnopek commented
Status: Active » Fixed

I've just made a commit that I hope will fix this problem!

http://drupalcode.org/project/panopoly.git/commit/0d84b3451f8cf3cb45ba41...

However, only time will tell. :-)

dsnopek’s picture

dsnopek
he/him
Primary language English
Location USA
CreditAttribution: dsnopek commented
Title: Tests randomly fail with "MySQL server has gone away" » Tests randomly fail with "MySQL server has gone away" (take 2)
Issue summary: View changes
Status: Fixed » Active

Gah! Unfortunately, this didn't fix the issue. :-/

I was assuming that this error was coming from the webserver which was still running when MySQL was already shutdown. However, on a closer look, it appears to be from 'behat' during it's shutdown process. It's got to be some cleanup function which uses the Drupal API driver.

I've updated the error in the issue description to be more detailed and show all the back traces.

Ari Gold’s picture

Ari Gold CreditAttribution: Ari Gold commented

I took a look at https://travis-ci.org/openscholar/openscholar to see if they have the same problem and it didn't look like it. I wonder if adding apache2 and/or cat after_script will prevent the mysql server from going away issue?

From https://github.com/openscholar/openscholar/blob/SCHOLAR-3.x/.travis.yml

before_script:
....
- sudo apt-get install apache2
....

script:
- ./bin/behat --tags=$TEST_SUITE

after_script:
- sudo cat /var/log/apache2/error.log

Although now I looked at https://travis-ci.org/acquia/commons3 which seems to have taken the approach that's already committed as part of Panopoly's .travis.yml and they don't seem to have this issue either?

From: https://github.com/acquia/commons3/blob/7.x-3.x/.travis.yml

after_script:

# Stop the webserver so that it's not still running when MySQL is stopped.
- kill $SERVER_PID

dsnopek’s picture

dsnopek
he/him
Primary language English
Location USA
CreditAttribution: dsnopek commented

It'd be difficult to tell if other distributions are affected because it's normal to simply re-run tests that fail with this error. We always do! And after it passes the old history about it failing is removed from Travis-CI. I know that Commerce Kickstart's tests also suffer from this issue (because @lsolesen is the one working on them ;-)).

I think if anything is to blame, it's probably the Drupal Behat Extension.

Ari Gold’s picture

Ari Gold CreditAttribution: Ari Gold commented

Ah, that makes sense! I was about to ask in which builds this issue is happening because I didn't see where it was still failing.

jhedstrom’s picture

jhedstrom
Primary language English
Location Portland, OR
CreditAttribution: jhedstrom commented

I posted a possible solution here #2247461: Mysql server gone randomly after all Behat tests have run. Shutdown functions tend not to behave properly on long-running Behat runs since Drupal may be bootstrapped hundreds of times in a single run.

lsolesen’s picture

lsolesen CreditAttribution: lsolesen commented

I am trying the suggestion for #8 out on Commerce Kickstart at the moment. https://travis-ci.org/lsolesen/commerce_kickstart/builds/24240108. There is a long history o at least one of the jobs in the build history to fail from the Mysql Server Gone Away error.

https://travis-ci.org/commerceguys/commerce_kickstart/builds

UPDATE: It did not seem to do any difference https://travis-ci.org/lsolesen/commerce_kickstart/jobs/24240109#L811.

Trying another approach by disabling dblog, but that did not work either: https://travis-ci.org/lsolesen/commerce_kickstart/jobs/24241724#L815 and it does consistent in all jobs for a build it seems: https://travis-ci.org/lsolesen/commerce_kickstart/jobs/24241728#L937. However, the build actually exists with a success status: https://travis-ci.org/lsolesen/commerce_kickstart/jobs/24241724. This is a lot better than what we are experiencing now, as we can actually get a green light - and dblog isn't used at all for the tests.

Let me know if I should make a patch for this :)

UPDATE 2: Still a problem so it is not fixed: https://travis-ci.org/lsolesen/commerce_kickstart/jobs/24250542#L1010

lsolesen’s picture

lsolesen CreditAttribution: lsolesen commented

The real problem might be, though, that MySQL's max_allowed_packet is too low. I do not know whether that can be changed anywhere in Travis? https://github.com/travis-ci/travis-ci/issues/2250

lsolesen’s picture

lsolesen CreditAttribution: lsolesen commented

Just an update: I had fairly good results for Commerce Kickstart just disabling dblog. See here https://travis-ci.org/commerceguys/commerce_kickstart/builds.

Might be worth doing before getting a lot of commits to be tested from the sprint today?

dsnopek’s picture

dsnopek
he/him
Primary language English
Location USA
CreditAttribution: dsnopek commented

Thanks, Lars! I'll try and get this in at the beginning of the sprint: https://github.com/commerceguys/commerce_kickstart/compare/4032313c0ef1....

dsnopek’s picture

dsnopek
he/him
Primary language English
Location USA
CreditAttribution: dsnopek commented
Status: Active » Fixed

I committed a change to disable dblog! Closing until we see this problem again.

dsnopek’s picture

dsnopek
he/him
Primary language English
Location USA
CreditAttribution: dsnopek commented
Title: Tests randomly fail with "MySQL server has gone away" (take 2) » Tests randomly fail with "MySQL server has gone away" (take 3)
Status: Fixed » Active

Blergh! This error happened on the first test. :-/

lsolesen’s picture

lsolesen CreditAttribution: lsolesen commented

Doh. However, I think the prevalence will go down. That it certainly did on CK.

  • Commit de753e7 on 7.x-1.x authored by lsolesen, committed by dsnopek: 
    Issue #2231631: Tests randomly fail with "MySQL server has gone away" (...
dsnopek’s picture

dsnopek
he/him
Primary language English
Location USA
CreditAttribution: dsnopek commented
Status: Active » Fixed

I haven't seen this one for a while. I'm going to close it for now - of course, I'll re-open if it recurs.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

dsnopek’s picture

dsnopek
he/him
Primary language English
Location USA
CreditAttribution: dsnopek commented
Title: Tests randomly fail with "MySQL server has gone away" (take 3) » Tests randomly fail with "MySQL server has gone away" (take 4)
Status: Closed (fixed) » Needs review
FileSize
panopoly-mysql-server-gone-away-2231631-19.patch1.14 KB

Alright! I'm re-opening this unholy issue. The last fix attempted was disabling dblog, and actually, since then, I really haven't seen this error practically at all.

However, we have a test for #2423841: "Top search phrases" report not showing any data that needs dblog in order to work!

So, I'm going try an alternate fix for this. A new patch is attached!

I'm trying it on this Travis-CI build now:

https://travis-ci.org/dsnopek/panopoly/builds/52315872

dsnopek’s picture

dsnopek
he/him
Primary language English
Location USA
CreditAttribution: dsnopek commented
Status: Needs review » Reviewed & tested by the community

I also tried it with #2398347: Panopoly Magic abuses FPP revisions which was also experiencing this issue during one of its tests:

https://travis-ci.org/dsnopek/panopoly/builds/52323403

So far, it has solved the problem on each of the tests I've tried it with!

dsnopek’s picture

dsnopek
he/him
Primary language English
Location USA
CreditAttribution: dsnopek commented
Status: Reviewed & tested by the community » Fixed

Committed!

  • dsnopek committed 638a8c3 on 7.x-1.x 
    Issue #2231631 by dsnopek: Tests randomly fail with "MySQL server has...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.