The testbot used to give reliable pass/fail results for Drupal 7 core patches, but lately there are lots of intermittent failures. Anecdotally, I think this only occurs when the testbot is retesting an already-RTBC patch, although I could be completely wrong about that.

Here are a couple examples of random failures:

  1. #2186281-4: Adding a library without a version number causes 'Undefined index: version in drupal_get_library()'
    https://qa.drupal.org/pifr/test/717678
  2. #952394-126: "No available releases found" - fetching update information exceeds timeout
    https://qa.drupal.org/pifr/test/702793

In both cases the failures are clearly unrelated to the patch (and will go away if the test is rerun).

If you look at the test logs, you can see it's a different failure with a different underlying error for each (DatabaseSchemaObjectExistsException for the first, and PDOException for the second). However, what they have in common is that in both cases there seems to be a severe mismatch between the state of the test database and what the code thinks should be there (missing database tables, or trying to create a database table during install that is already there).

Comments

sun’s picture

sun
Primary language German
Location Karlsruhe
CreditAttribution: sun commented
Priority: Major » Critical
Issue tags: +Random test failure
jthorson’s picture

jthorson CreditAttribution: jthorson commented
Priority: Critical » Major

Once the patch is re-tested, I have no visibility into the actual failure ... now that I'm back from vacation, there's really no way for me to look into this.

That said, tonight I did find an issue on testbot 664, which would be one of three testbots dedicated to D7 tests ... there was a frozen long-running (i.e. weeks) git fetch process chugging away on the server; so that bot wasn't 100% sane.

That's not to say that this was definitely the cause or even potentially related ... but after cleaning that up, hopefully things will be a bit more stable.

In any case, next time you see this, ping me in IRC so that I can flag the test run and look into the actual errors.

David_Rothstein’s picture

David_Rothstein CreditAttribution: David_Rothstein commented

Thanks. Yeah, I didn't realize that when a patch is retested, the results take over the same URL as the old one.

In the meantime, here's one that failed only a couple days ago that might be worth looking at, and the patch needs work for other reasons so it's not too likely to be retested:

https://drupal.org/node/965078#comment-7500792
https://qa.drupal.org/pifr/test/537388

jthorson’s picture

jthorson CreditAttribution: jthorson commented

From the quoted test:

exception 'PDOException' with message 'SQLSTATE[42S02]: Base table or view not found: 1146 Table 'drupaltestbotmysql.simpletest861933field_config' doesn't exist' in /var/lib/drupaltestbot/sites/default/files/checkout/includes/database/database.inc:2171
Stack trace:
#0 /var/lib/drupaltestbot/sites/default/files/checkout/includes/database/database.inc(2171): PDOStatement->execute(Array)
#1 /var/lib/drupaltestbot/sites/default/files/checkout/includes/database/database.inc(683): DatabaseStatementBase->execute(Array, Array)
#2 /var/lib/drupaltestbot/sites/default/files/checkout/includes/database/select.inc(1264): DatabaseConnection->query('SELECT fc.*?FRO...', Array, Array)
#3 /var/lib/drupaltestbot/sites/default/files/checkout/modules/field/field.crud.inc(372): SelectQuery->execute()
#4 /var/lib/drupaltestbot/sites/default/files/checkout/modules/field/field.module(364): field_read_fields(Array, Array)
#5 /var/lib/drupaltestbot/sites/default/files/checkout/includes/module.inc(1101): field_system_info_alter(Array, Object(stdClass), 'module', NULL)
#6 /var/lib/drupaltestbot/sites/default/files/checkout/modules/system/system.module(2422): drupal_alter('system_info', Array, Object(stdClass), 'module')
#7 /var/lib/drupaltestbot/sites/default/files/checkout/modules/system/system.module(2450): _system_rebuild_module_data()
#8 /var/lib/drupaltestbot/sites/default/files/checkout/includes/bootstrap.inc(3176): system_rebuild_module_data()
#9 /var/lib/drupaltestbot/sites/default/files/checkout/includes/common.inc(7456): registry_rebuild()
#10 /var/lib/drupaltestbot/sites/default/files/checkout/modules/simpletest/drupal_web_test_case.php(1586): drupal_flush_all_caches()
#11 /var/lib/drupaltestbot/sites/default/files/checkout/modules/simpletest/drupal_web_test_case.php(1502): DrupalWebTestCase->resetAll()
#12 /var/lib/drupaltestbot/sites/default/files/checkout/modules/simpletest/tests/error.test(16): DrupalWebTestCase->setUp('error_test')
#13 /var/lib/drupaltestbot/sites/default/files/checkout/modules/simpletest/drupal_web_test_case.php(501): DrupalErrorHandlerTestCase->setUp()
#14 /var/lib/drupaltestbot/sites/default/files/checkout/scripts/run-tests.sh(368): DrupalTestCase->run()
#15 /var/lib/drupaltestbot/sites/default/files/checkout/scripts/run-tests.sh(22): simpletest_script_run_one_test('1', 'DrupalErrorHand...')
#16 {main}

Uncaught exception thrown in shutdown function.

PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'drupaltestbotmysql.simpletest861933cache_block' doesn't exist: DELETE FROM {cache_block}
WHERE (expire <> :db_condition_placeholder_0) AND (expire < :db_condition_placeholder_1) ; Array
(
[:db_condition_placeholder_0] => 0
[:db_condition_placeholder_1] => 1399163924
)
in cache_clear_all() (line 165 of /var/lib/drupaltestbot/sites/default/files/checkout/includes/cache.inc).

Uncaught exception thrown in shutdown function.

PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'drupaltestbotmysql.simpletest861933cache_bootstrap' doesn't exist: DELETE FROM {cache_bootstrap}
WHERE (cid = :db_condition_placeholder_0) ; Array
(
[:db_condition_placeholder_0] => variables
)
in cache_clear_all() (line 165 of /var/lib/drupaltestbot/sites/default/files/checkout/includes/cache.inc).

FATAL FeedIconTest: test runner returned a non-zero error code (1).

The only real changes lately has been the apc_cli addition, and I do know that the D7 bots are throwing 'already loaded' APC errors, so there could be something there ... but I can't think of anything else at the moment.

David_Rothstein’s picture

David_Rothstein CreditAttribution: David_Rothstein commented

Since it looks like this will be hard to solve, I created #2319997: Don't automatically retest Drupal 7 core RTBC patches as a proposal for how to work around the problem.

dcam’s picture

dcam CreditAttribution: dcam commented

I can confirm that this can happen any time a patch is tested. I've seen patches fail the first time they're tested and their status is Needs review.

isntall’s picture

isntall CreditAttribution: isntall at Drupal Association commented
Project: Drupal.org Testbots » DrupalCI: Dispatcher (Modernizing Testbot Initiative)
Component: unexplained test failure » Code
jthorson’s picture

jthorson CreditAttribution: jthorson commented
Project: DrupalCI: Dispatcher (Modernizing Testbot Initiative) » Project Issue File Review
Version: » 6.x-3.x-dev
Status: Active » Closed (works as designed)

This retesting is a PIFR thing. Neil disabled retesting of D7 patches a couple weeks ago.

David_Rothstein’s picture

David_Rothstein CreditAttribution: David_Rothstein commented
Project: Project Issue File Review » Drupal.org Testbots
Version: 6.x-3.x-dev »
Component: Code » unexplained test failure
Priority: Major » Normal
Status: Closed (works as designed) » Active

You're probably thinking of #2319997: Don't automatically retest Drupal 7 core RTBC patches? It was just deployed today though, not a couple weeks ago.

That issue definitely helps make this one less of a problem, but it's a separate issue.

David_Rothstein’s picture

David_Rothstein CreditAttribution: David_Rothstein commented
Title: Intermittent "test did not complete due to a fatal error" failures when retesting Drupal 7 RTBC patches » Intermittent "test did not complete due to a fatal error" failures when testing Drupal 7 patches
Project: Drupal.org Testbots » Drupal core
Version: » 7.x-dev
Component: unexplained test failure » simpletest.module

Let's move this to the Drupal core issue queue for now, though. We don't know for sure it's an issue with the testbot (it could be, but it could also be caused by a bug in Simpletest that only triggers very rarely).

Also retitling this based on @dcam's comment in #6.