findTables returns an empty array when attempting to list tables for an external DB. findTables works as expected for both PostgreSQL and MySQL external databases. The bug doesn't impact core usability.

The following code works for MySQL and PostgreSQL, but fails for SQLite:

// fetch all databases in settings.php
$databases = Database::getAllConnectionInfo();
foreach ($databases as $database => $config) {
  // connect to the database.
  $connection = Database::getConnection('default', $database);
  // fetch all tables in the database
  $tables = $connection->schema()->findTables('%');
}

For now I'm using the following work around:

// fetch all databases in settings.php
$databases = Database::getAllConnectionInfo();
foreach ($databases as $database => $config) {
  // connect to the database.
  $connection = Database::getConnection('default', $database);

  if ($config['default']['driver'] == 'sqlite') {
    // Create a table query because schema()->findTables('%') returns
    // an empty array for SQLite. The problem is inherent to SQLite the driver.
    $connection->query("ATTACH DATABASE '" . $config['default']['database'] . "' as " . $database . ";");
    $q = "SELECT name FROM " . $database . ".sqlite_master ";
    $q.= "WHERE type = 'table' AND name LIKE '%' AND name NOT LIKE 'sqlite_%'";
    $tables = $connection->query($q)
    ->fetchAllKeyed(0, 0);
  }
  else {
    $tables = $connection->schema()->findTables('%');
  }
}

I simply copied the SQL query from the SQLite driver. Essentially, the SQLite code works, the problem is in how it is attaching the database.

I believe bug is inherent in the way in which the driver is constructed. Unfortunately, I'm not familiar enough with Drupal internals to track it further.

Drupal 7 issue is #1713332: The SQLite database driver fails to drop simpletest tables

Comments

Mike A. created an issue. See original summary.

andypost’s picture

Version: 8.4.x-dev » 8.8.x-dev

Confirm this

$ drush ev "var_dump(db_find_tables('batch%'));"
array(0) {
}
$ drush ev "var_dump(db_find_tables('%'));"
array(0) {
}
andypost’s picture

Priority: Minor » Major
Status: Active » Needs review
Issue tags: -sqlite +Needs tests
StatusFileSize
new1004 bytes

Here's a quick fix

andypost’s picture

I'm getting empty list of tables on default local setup with sqlite

amateescu’s picture

Status: Needs review » Needs work

How about adding a $include_main = FALSE parameter to \Drupal\Core\Database\Driver\sqlite\Connection::getAttachedDatabases(), and calling it with TRUE in findTables()? This way anyone who calls getAttachedDatabases() should be aware that it doesn't return the main schema by default :)

Also, it's really strange that \Drupal\KernelTests\Core\Database\SchemaTest::testFindTables() doesn't fail in HEAD..

andypost’s picture

It does not fail because tests always have $attached_dbs non-empty, running this test I always get kind of that in $attached_dbs:

$attached_dbs array (
  'test93593384' => 'test93593384',
  'test93593384_shared_' => 'test93593384_shared_',
)
$attached_dbs array (
  'test93593384' => 'test93593384',
  'test93593384_shared_' => 'test93593384_shared_',
)
$attached_dbs array (
  'test93593384' => 'test93593384',
)
amateescu’s picture

Issue tags: -Needs tests

Right, we can't test this in a kernel or functional test because of how we instantiate the database in the a environment by using prefixes. I guess manual testing will have to do for this issue :)

What do you think about the fist part of #5?

andypost’s picture

Assigned: Unassigned » andypost

I find in nice workaround for specific driver to manage schemas/attached databases for sqlite (recall few issues with it in pgsql meantime)
Will work on it tonight

andypost’s picture

Assigned: andypost » Unassigned
Status: Needs work » Needs review
Issue tags: +Needs manual testing
StatusFileSize
new2.61 KB

Used to add tests but that really not possible, probably it need CR

To test it manually - use quickstart install (sqlite) and

drush ev 'var_dump(\Drupal::database()->schema()->findTables("%"));'

which returns nothing without patch and all tables on patched core

amateescu’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: -Needs manual testing

I also tested it manually and it works fine!

alexpott’s picture

+++ b/core/tests/Drupal/KernelTests/Core/Database/SchemaTest.php
@@ -1246,6 +1246,13 @@ public function testFindTables() {
+    // Make sure prefixed table is not visible for default connection.
+    Database::removeConnection('test');
+    $schema = Database::getConnection()->schema();
+    $tables = $schema->findTables('test_%');
+    sort($tables);
+    $this->assertEquals(['test_1_table'], $tables);

So HEAD will pass this test on SQLite?

andypost’s picture

@alexpott yes, that's just extra case for existing coverage
Basically in tests we always have kinda test93593384 database attached ( as #6 trying to explain

alexpott’s picture

Status: Reviewed & tested by the community » Needs review

So I've read

    // The SQLite implementation doesn't need to use the same filtering strategy
    // as the parent one because individually prefixed tables live in their own
    // schema (database), which means that neither the main database nor any
    // attached one will contain a prefixed table name, so we just need to loop
    // over all known schemas and filter by the user-supplied table expression.

carefully many times. And I'm not sure the solution here is correct. I think we should do something like

$attached_dbs = $this->connection->getAttachedDatabases();
// If there are no attached databases use the main database.
if (empty($attached_dbs)) {
  $attached_dbs['main'] = 'main';
}

The reason I don't think the solution is correct is because if you did - var_dump(\Drupal\Core\Database\Database::getConnection('my_prefix')->schema()->findTables("%")); this would include main and that's not right either.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

daffie’s picture

StatusFileSize
new743 bytes

To test the patch from comment #9 with only the added test.

andypost’s picture

Test patch does not catch the issue, any idea how to cover it?

daffie’s picture

Status: Needs review » Postponed

I would like to postpone this issue on #3106531: Notify in Status Report that per-table database prefixes are no longer supported, and will throw errors in Drupal 10.0. The two issue might seem not to be related, only they both use the class variable $this->prefixes. And the other issue needs to be fixed before the release of 9.0.

andypost’s picture

StatusFileSize
new749 bytes

test-only reroll for 9.0.x

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

andypost’s picture

Status: Needs work » Needs review
StatusFileSize
new638 bytes
new732 bytes
new1.7 KB

Fix for fail-test (sqlite only) and patch with early fix

The last submitted patch, 21: 2949229-21-fail.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 21: 2949229-21.patch, failed testing. View results

andypost’s picture

D7 issue was commited with extra tests that could be forward-ported from #1713332: The SQLite database driver fails to drop simpletest tables

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.