Fix tests on 7.x-3.x branch

How were the tests broken by that issue?

What are the symptoms?

Can I replicate this on a local machine?

I have to admit, after looking into this for about an hour, I still understand very little of this. It seems to me however, that if we revert (just to see what happens) the changes from #2415421, these specific failures should stop?

So let us try that first.

I am still getting the same (type of) fails as before, i.e.

FAIL query

exception: [PDOException] Line 524 of sites/all/modules/backup_migrate/includes/sources.db.mysql.inc:
SQLSTATE[42S02]: Base table or view not found: 1146 Table 'jenkins_drupal_d7_33498.simpletest116906watchdog' doesn't exist: SHOW CREATE TABLE `simpletest116906watchdog`; Array
(
)

FAIL testFirstBackup

exception: [Notice] Line 102 of sites/all/modules/backup_migrate/tests/BmTestBasics.test:
Undefined offset: 0

I guess it could be argued that line 102 of tests/BmTestBasics.test should not fail the way it does, because it is causing a PHP error (maybe use !empty() instead of isset()?), but the result is the same: the test script appears to be unable find the value of an href attribute for a link called 'download' - which is one of the things I do not understand, because an earlier assertion that a link called 'download' exists appears to succeed.

One more note, ever since I started contributing tests to this module, I have had the occasional "Base table or view not found" error. Since these errors did not show up on local tests (i.e. on my own PC) and restarting the test led to a successful test, I assumed the problem was with the drupal.org test runner - maybe it could not load the database on time? The table that failed to load also tended to be 'actions', which reinforced my feeling that the entire database was not online yet.

Recently (the past month), failures have been much more common and often it is not 'actions', but a later table that is unavailable.

I am not sure if this is related to the current issue, but I thought it noteworthy.

I cannot say I have ever seen that. I run XAMPP on Windows 10 Home.

I have installed Ubuntu 18.04 on a VM and manually installed a LAMPP.

Then I git cloned Drupal, B&M (7.x-3.x) and Ctools to an Apache site and ran the tests.

All tests passed with "330 passes, 0 fails, 0 exceptions, and 78 debug messages".

I would love to get these failing tests fixed because they are currently causing havoc with the issue queue, but I cannot reproduce. Does anybody have any suggestions what else I could try?

As suggested in #10, I installed DrupalCI locally. The results are erratic.

The tests that fail:

I ran a test against #5 ten times in a row, with a reset of Vagrant half-way to help flush out any settings that might be influencing the tests.

The following 4 tests were run:

• Ctools tests.
• Email handling.
• Basic tests.
• Profiles test.

Of these, Ctools and Email always passed.

1 run, all tests passed.
4 runs, only Basic tests failed.
1 run, only Profiles test failed.
4 runs, both Basic tests and Profiles test failed.

It is not clear to me in which order these tests are run, nor if that matters.

The failures that occur:

I also looked at the issue queue for the past four months, at the open issues of which the most recent regular patch failed, to see where the tests started failing.

Again I have no reason to believe that the order in which failures are reported are the same order in which tests were executed.

However, the first failure is important, because it can have knock-on effects for all sub-sequent tests.

https://www.drupal.org/project/backup_migrate/issues/3027709#comment-133...

exception: [PDOException] Line 524 of sites/all/modules/backup_migrate/includes/sources.db.mysql.inc:
SQLSTATE[42S02]: Base table or view not found: 1146 Table 'jenkins_drupal_d7_33080.simpletest981770field_revision_body' doesn't exist: SHOW CREATE TABLE `simpletest981770field_revision_body`; Array
(
)

https://www.drupal.org/project/backup_migrate/issues/3102218#comment-133...

exception: [PDOException] Line 524 of sites/all/modules/backup_migrate/includes/sources.db.mysql.inc:
SQLSTATE[42S02]: Base table or view not found: 1146 Table 'jenkins_drupal_d7_32667.simpletest155240users_roles' doesn't exist: SHOW CREATE TABLE `simpletest155240users_roles`; Array
(
)

https://www.drupal.org/project/backup_migrate/issues/2885094#comment-134...

exception: [PDOException] Line 526 of sites/all/modules/backup_migrate/includes/sources.db.mysql.inc:
SQLSTATE[42S02]: Base table or view not found: 1146 Table 'jenkins_drupal_d7_37183.simpletest95446image_styles' doesn't exist: SHOW CREATE TABLE `simpletest95446image_styles`; Array
(
)

https://www.drupal.org/project/backup_migrate/issues/3102506#comment-134...

exception: [PDOException] Line 524 of sites/all/modules/backup_migrate/includes/sources.db.mysql.inc:
SQLSTATE[42S02]: Base table or view not found: 1146 Table 'jenkins_drupal_d7_33498.simpletest116906watchdog' doesn't exist: SHOW CREATE TABLE `simpletest116906watchdog`; Array
(
)

One patch, not shown here, started with a different type of error.

Also, these failures are important because in cases, the patch doesn't appear to do anything with the database, so these are the failures that suggest to us that it is not the patch, but something else.

I repeated my tests from #11, except this time against a patch that pauses the runBackup() method from BmTestBase.text for 10 seconds just before it runs a backup, using PHP's sleep() function (patch not attached).

The idea was to see if I could find further clues that enforce the idea from #11, part 2, that the tests fail because the test database is not ready yet.

The results were:

4 runs, all tests passed.
6 runs, only Basic tests failed.
0 runs, only Profiles test failed.
0 runs, both Basic tests and Profiles test failed.

The results are different from last time in that they contain:

• More instances of all tests passing.
• Zero instances of both tests failing.
• Zero instances of Profiles tests failing.

The results are similar in that:

• The results are erratic.

There may be other reasons than the patch that cause these differences, for example the host system may have had more resources available at the right time this time around.

With regards to my question last time about the order in which tests are run, Neil Drumm replied in the #drupal Slack channel Testing: "Tests are run concurrently, so order is not guaranteed at all. Each test should be able to run independently without relying on other tests, and shouldn’t conflict with other tests being run simultaneously."

I checked and noticed that each .test file is ran against its own copy of the database.

Nice work!