Problem/Motivation

We have a method to determine the db driver namespace if it is not provided - \Drupal\Core\Database\Database::getDatabaseDriverNamespace() - which supports supports fallling back to the core db driver namespace. This results in the connection info sometimes having the driver namespace and sometimes not.

Proposed resolution

Always set $info['namespace'] instead of using a method.

Remaining tasks

User interface changes

None

API changes

\Drupal\Core\Database\Database::getDatabaseDriverNamespace() is deprecated in Drupal 9.1.x

Data model changes

None

Release notes snippet

N/a

Comments

alexpott created an issue. See original summary.

alexpott’s picture

Status: Active » Needs review
StatusFileSize
new3.33 KB
mondrake’s picture

StatusFileSize
new492 bytes
new3.78 KB
daffie’s picture

If I understand this patch correctly, the namespace part of the driver "settings" will become required. Is this what we want?

alexpott’s picture

@daffie well the only driver's that don't need it are core drivers. So it's one more special case for core drivers. I think we should try to minimise them as much as possible.

But you're right in that code like

 * @code
 * $databases['default']['default'] = [
 *   'database' => 'databasename',
 *   'username' => 'sqlusername',
 *   'password' => 'sqlpassword',
 *   'host' => 'localhost',
 *   'port' => '3306',
 *   'driver' => 'mysql',
 *   'prefix' => '',
 *   'collation' => 'utf8mb4_general_ci',
 * ];
 * @endcode

would have to change .... hmmm...

mondrake’s picture

Actually, I think that as things go atm, the 'namespace' key would be the important one, and maybe the 'driver' one is not that necessary anymore...

See Database::openConnection, the $driver variable is unused.

daffie’s picture

Sorry, to you @mondrake. I was not talking about the "driver" setting, but I was talking database driver "settings". Just like @alexpott is talking about it in comment #5.

mondrake’s picture

#7 @daffie exactly, I am pushing further and IMHO the 'driver' key in the database settings is redundant and this would suffice:

 * @code
 * $databases['default']['default'] = [
 *   'database' => 'databasename',
 *   'username' => 'sqlusername',
 *   'password' => 'sqlpassword',
 *   'host' => 'localhost',
 *   'port' => '3306',
 *   'namespace' => 'Drupal\Core\Database\Driver\mysql',
 *   'prefix' => '',
 *   'collation' => 'utf8mb4_general_ci',
 * ];
 * @endcode

:)

Then if you need a label for the driver, you can get it from Connection::driver()...

mondrake’s picture

StatusFileSize
new693 bytes
new4.45 KB
daffie’s picture

For me is making the namespace setting required very bad for the Drupal end user/site owner. The driver setting is simple for them to understand and changing that will require a lot of communication and also the namespace setting is more difficult. Therefore am I against this change.

For me this issue is a small part of a bigger problem. Drupal is divided in different parts. Everything in the directory /core/lib is/should be general code for that is not part of a specific module or theme. The specific code for a module or theme belongs in that module or theme. For database drivers this is different. All code is in the directory /core/lib, including all database driver specific code. For me this is wrong. Just like with modules and themes, the database driver specific stuff needs to be moved to core/databases or for contrib to /databases.
There are in Drupal core a lot places where there is code that starts with something likeif (Database::getConnection()->databaseType() == 'mysql/pgsql/sqlite') {, not only in code but also in tests. Those code parts should be moved to their specific database driver. It should not be anywhere else. For that database drivers need module like capabilities. Just like themes are a template directory with module capabilities, database drivers should be a driver directory with module like capabilities. Just like the contrib database drivers are that already. The three by code supported should be changed in the same way.
Problems like contrib database drivers that need some test to do something different now cannot do that. They cannot make changes to test like the by core supported drivers can (see: #3110546: Allow contributed modules (mostly database drivers) to override tests in core).
The problem we have in this issue should then be solved very easily. Lets start by assuming that we will require that the driver name and the module name must be the same. The namespace for any database driver will then be Drupal\$module_name\Driver.

daffie’s picture

alexpott’s picture

StatusFileSize
new2.25 KB

I think much of what #10 is saying makes sense. Special casing core makes the contrib space very painful.

I also think we can take another approach here - we can provide the default for namespace in the connection info adder - that way we know it is always set.

daffie’s picture

Status: Needs review » Needs work
+++ b/core/lib/Drupal/Core/Database/Database.php
@@ -224,6 +224,10 @@ abstract class Database {
+    // Fallback for Drupal 7 settings.php if namespace is not provided.
+    if (empty($info['namespace'])) {
+      $info['namespace'] = 'Drupal\\Core\\Database\\Driver\\' . $info['driver'];
+    }

If I understand this change correctly, if the (contrib) database driver uses another namespace then the default we are adding here, you are required to set the namespace part in the driver "settings". This is to me not the right solution.

alexpott’s picture

Status: Needs work » Needs review

@daffie this is already the case. This change is just a baby step towards all drivers declaring a namespace by ensuring that all drivers always have the namespace key set - thereby increasing consistency in code that interacts with connection settings and make core just a tiny bit less special.

daffie’s picture

Status: Needs review » Reviewed & tested by the community

@alexpott: You are right. I agree with this baby step. Therefor for me it is RTBC.

This change is just a baby step towards all drivers declaring a namespace by ensuring that all drivers always have the namespace key set

I am not sure what you mean:
1. Do want to have all drivers add a method that will return its namespace. Something like Connection::getNamespace(). If this is want you mean then lets do this in 9.0 as is a BC break.
Also add a hook_requirements that tests if the namespace key in the "driver settings" is set and if that is the case then return a REQUIREMENT_WARNING with the message that the setting of the namespace will be ignored.
2. If we want for all drivers that users must set the namespace key in the "driver settings" then that would be for me bad for the Drupal end user/site owner experience. For me the Drupal end user/site owner should have nothing to do with the namespace of the database driver. It will be confusing to a lot of them and this will not be good for the usability of Drupal. Please lets not do this.

mondrake’s picture

Version: 9.1.x-dev » 9.0.x-dev
Status: Reviewed & tested by the community » Needs work

IMHO this can be done now, without the deprecation, possibly also backported to D8. Then, deprecate it later in D9.1, with a test that is missing here.

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new2.26 KB

Doing #16.

daffie’s picture

Status: Needs review » Needs work

@mondrake: Could you create the followup for deprecating the method getDatabaseDriverNamespace() and updating the patch with the link to the followup. For me it would be RTBC again.

alexpott’s picture

Title: Deprecate \Drupal\Core\Database\Database::getDatabaseDriverNamespace as it is no longer needed » Always set $info['namespace'] on database connection info
Issue summary: View changes

Updated the issue summary. This patch can remove \Drupal\Core\Database\Log::getDriverNamespace() and \Drupal\Core\Database\Log::$driverNamespace too.

alexpott’s picture

Status: Needs work » Needs review
StatusFileSize
new10.57 KB
new12.75 KB

Addressing #19 and also making the test added by #2867788: Log::findCaller fails to report the correct caller function with non-core drivers. way more realistic - real backtraces and a real db connection info - plus not mocking the thing under test. And making it a unit test.

alexpott’s picture

+++ b/core/tests/Drupal/KernelTests/Core/Database/LoggingTest.php
@@ -138,193 +137,4 @@ public function testGetLoggingWrongKey() {
-      // Extreme case, should not happen at normal runtime - if the driver
-      // namespace is not in the stack trace, the first entry to a method
-      // in core database namespace is returned.
-      'missing driver namespace' => [
-        'Drupal\\Driver\\Database\\fake',
-        $stack,
-        [
-          'class' => 'Drupal\\Driver\\Database\\dbal\\Statement',
-          'function' => 'execute',
-          'file' => '/var/www/libraries/drudbal/lib/Statement.php',
-          'line' => 264,
-          'type' => '->',
-          'args' => [
-            0 => 'test',
-          ],
-        ],
-      ],

This is not extreme. It's impossible. There's no way a Drupal\\Driver\\Database\\dbal\\Statement could be in a stack trace when using a driver namepsace like Drupal\\Driver\\Database\\fake. It looks like a construction of the over-complex mocking in the existing test. As far as I can see this is not really testing anything. If a site uses a driver namespace that doesn't exist it is going to crash way before we can to log a query and if the database query logger is used outside of the database driver code (either core or contrib) then the stack entry returned is going to be one that called the logger. If we want to test that we can - but we don't need a fake driver namespace to do that.

alexpott’s picture

StatusFileSize
new13.06 KB
new1.35 KB

Here's a test for calling Log::findCaller() outside DB code.

The last submitted patch, 20: 3112476-2-20.patch, failed testing. View results

alexpott’s picture

@daffie @mondrake - a review of this would be great. This will be easier to get done prior to the beta.

andypost’s picture

Related issues: +#3118679: Allow driver name to be different then the PDO driver name during site install proces
mondrake’s picture

I find the changes in the Log tests OOS here, and a bit difficult to follow. I understand that current state can be improved, but I was finding having an sample backtrace easier to read and follow. My suggestion would be to defer the Log test changes to #2999962: Unify Database/Log::findCaller and Utility/Error::getLastCaller and just get rid of the usage of reflection in Log::getDriverNamespace here.

alexpott’s picture

@mondrake I regret committing that patch with the test as it was. The changes are in scope because we no longer have to mock or even maintain the protected methods. This is our last chance to get rid of the this unnecessary API because those methods are only in 9.0.x so the class has not being released. Mocking the actual class under test should really be an option of last resort because it always leads to fragile tests.

daffie’s picture

Status: Needs review » Needs work

Reviewed as requested on Slack.

  1. +++ b/core/lib/Drupal/Core/Database/Database.php
    @@ -502,6 +508,8 @@ public static function getConnectionInfoAsUrl($key = 'default') {
    +   * @todo https://www.drupal.org/node/3115388 Deprecate this in drupal:9.1.0.
    

    Should we do deprecate this in 9.0 and remove in 10.0?

  2. +++ b/core/lib/Drupal/Core/Database/Log.php
    @@ -181,31 +176,4 @@ public function findCaller() {
    -  /**
    -   * Gets the debug backtrace.
    -   *
    -   * Wraps the debug_backtrace function to allow mocking results in PHPUnit
    -   * tests.
    -   *
    -   * @return array[]
    -   *   The debug backtrace.
    -   */
    -  protected function getDebugBacktrace() {
    -    return debug_backtrace();
    -  }
    

    I do not think that this should be removed in this issue.

  3. +++ b/core/tests/Drupal/KernelTests/Core/Database/LoggingTest.php
    --- /dev/null
    +++ b/core/tests/Drupal/Tests/Core/Database/LogTest.php
    

    I think we should test with "full" stack. This test does not give me the confidence that the Log class will do what it should do.

I think that the Log testing can be improved. But for me it does not belong in this issue. I agree with @mondrake. Lets do that in #2999962: Unify Database/Log::findCaller and Utility/Error::getLastCaller.

alexpott’s picture

Status: Needs work » Needs review

Thanks for the reviews

  1. New deprecations are happening in 9.1.x and not 9.0.0 where possible - that is policy.
  2. Yes it should. It's resulting in a fake test that gives confidence where none is deserved.
  3. I disagree fundamentally about the log testing. The test in HEAD is completely faked. The test this patch adds is real and does much more of a full stack test than currently.
alexpott’s picture

+++ b/core/tests/Drupal/Tests/Core/Database/LogTest.php
@@ -0,0 +1,55 @@
+    $pdo = $this->prophesize(StubPDO::class)->reveal();
+    $result = (new StubConnection($pdo, []))->testLogCaller();
+    $this->assertSame([
+      'file' => __FILE__,
+      'line' => 33,
+      'function' => 'testContribDriverLog',
+      'class' => 'Drupal\Tests\Core\Database\LogTest',
+      'type' => '->',
+      'args' => [],
+    ], $result);
+
+    // Test calling the database log from outside of database code.
+    $result = (new Log())->findCaller();
+    $this->assertSame([
+      'file' => __FILE__,
+      'line' => 44,
+      'function' => 'testContribDriverLog',
+      'class' => 'Drupal\Tests\Core\Database\LogTest',
+      'type' => '->',
+      'args' => [],
+    ], $result);

Here we have *proof* that if the logger is called inside or outside a db function the caller is determined to be outside the database layer.

Currently the test doesn't prove that because we faking the entire backtrace and database driver namespace.

mondrake’s picture

I tested #22 with a contrib driver, and --group Database,views tests pass. OK for RTBC for me, but I cannot. Would be good to have input from guys developing MSSQL or Oracle contrib drivers.

alexpott’s picture

StatusFileSize
new7.07 KB
new7.74 KB

Even though I think it is not correct to maintain \Drupal\Core\Database\Log::getDebugBacktrace() that it never should have been added - rather than argue about that here here's a patch that maintains it and the false sense of test coverage added by \Drupal\KernelTests\Core\Database\LoggingTest::testContribDriverLog(). But I'm also leaving in core/tests/Drupal/Tests/Core/Database/LogTest.php as that provides demonstrably better and non mocked cover of the Log class. Hopefully this is enough to get this to rtbc because funnily enough this makes core drivers more like contrib and in fact it's the core drivers whose info arrays have been changed here and not contribs.

Status: Needs review » Needs work

The last submitted patch, 32: 3112476-2-32.patch, failed testing. View results

alexpott’s picture

Status: Needs work » Needs review

Unrelated random fail. Typical.

daffie’s picture

Status: Needs review » Reviewed & tested by the community

Apparently it is policy not to deprecate in 9.0. The added comment is therefor correct.
The removal of the backtrace method has been removed from the patch.
We are going to improve the testing of Database/Log in #2999962: Unify Database/Log::findCaller and Utility/Error::getLastCaller. Which is great.
The changing of the namespace stuff is not a problem for a contrib driver. See the coment of @mondrake in comment #31.
The testbot failure is not related to this patch.
All code changes look good to me.
For me it is RTBC.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

Realised whilst starting at \Drupal\Core\Database\Connection::getDriverClass for other reasons we can do some simplification here.

alexpott’s picture

Status: Needs work » Reviewed & tested by the community

Actually let's handle that in a follow-up. Since we're now always setting the namespace we can deprecate now passing it into \Drupal\Core\Database\Connection::__construct() in the $connection_options array. Which will allow us to simplify \Drupal\Core\Database\Connection::getDriverClass and remove the override in \Drupal\database_statement_monitoring_test\LoggedStatementsTrait

alexpott’s picture

I've added the follow tasks to the follow-up #3115388: PP-1: Deprecate \Drupal\Core\Database\Database::getDatabaseDriverNamespace as it is no longer needed as doing it properly requires deprecations.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 32: 3112476-2-32.patch, failed testing. View results

daffie’s picture

Status: Needs work » Reviewed & tested by the community

Back to RTBC.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 32: 3112476-2-32.patch, failed testing. View results

neslee canil pinto’s picture

Status: Needs work » Reviewed & tested by the community
mondrake’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs reroll
neslee canil pinto’s picture

Status: Needs work » Needs review
StatusFileSize
new7.6 KB
new1.58 KB
daffie’s picture

Status: Needs review » Needs work

Hi @Neslee: The interdiff file you have added does not correspond with the patch file changes. Your interdiff file let me to believe that you where making fundamental changes to the patch, also because you did not remove the reroll tag.

The patch from comment #32 needs a reroll, because there is one hunk that is failing. You have updated that hunk, only not in right way. The original hunk was:

+++ b/core/lib/Drupal/Core/Database/Database.php
@@ -488,7 +494,7 @@ public static function getConnectionInfoAsUrl($key = 'default') {
     if (empty($db_info) || empty($db_info['default'])) {
       throw new \RuntimeException("Database connection $key not defined or missing the 'default' settings");
     }
-    $connection_class = static::getDatabaseDriverNamespace($db_info['default']) . '\\Connection';
+    $connection_class = $db_info['default']['namespace'] . '\\Connection';
     return $connection_class::createUrlFromConnectionOptions($db_info['default']);
   }
 

You have changed it into:

+++ b/core/lib/Drupal/Core/Database/Database.php
@@ -614,7 +620,7 @@ public static function getConnectionInfoAsUrl($key = 'default') {
       $db_info['default']['module'] = explode('\\', $namespace)[1];
     }
 
-    $connection_class = $namespace . '\\Connection';
+    $connection_class = $db_info['default']['namespace'] . '\\Connection';
     return $connection_class::createUrlFromConnectionOptions($db_info['default']);
   }
 

The essence of the change was to remove the calling of static::getDatabaseDriverNamespace() and replace it by $db_info['default']['namespace'].

neslee canil pinto’s picture

Status: Needs work » Needs review

@daffie, ya i have rerolled the #32.

daffie’s picture

Status: Needs review » Needs work

The essence of the change was to remove the calling of static::getDatabaseDriverNamespace() and replace it by $db_info['default']['namespace'].

@Neslee: The change you made in your reroll was wrong, because your change does not do that and it should do that.

neslee canil pinto’s picture

Status: Needs work » Needs review
StatusFileSize
new7.22 KB
new515 bytes
daffie’s picture

Status: Needs review » Needs work

In the method Drupal\Core\Database\Database::getDatabaseDriverNamespace() there is a call to static::getDatabaseDriverNamespace. That call needs to be replaced by $db_info['default']['namespace']. Just like in the patch from @alexpott in comment #32.

neslee canil pinto’s picture

Status: Needs work » Needs review
StatusFileSize
new7.75 KB
new676 bytes

@daffie, got your point, updated the patch. Thanks for mentoring👍🏻

daffie’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: -Needs reroll

Reroll looks good.
Back to RTBC.

catch’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/core/lib/Drupal/Core/Database/Database.php
@@ -628,6 +634,8 @@ public static function getConnectionInfoAsUrl($key = 'default') {
    *
    * @return string
    *   The PHP namespace of the driver's database.
+   *
+   * @todo https://www.drupal.org/node/3115388 Deprecate this in drupal:9.1.0.
    */

It's now possible to do the 9.1.x deprecation if we have a separate patch.

andypost’s picture

Status: Needs work » Reviewed & tested by the community

@catch the deprecation is the follow-up to this patch

andypost’s picture

Or do you mean not split it?

catch’s picture

Status: Reviewed & tested by the community » Needs work

@andypost I mean not splitting it, the follow-up was created when the 9.1.x branch wasn't open yet (but after we stopped committing new deprecations to 9.0.x).

alexpott’s picture

Status: Needs work » Needs review
StatusFileSize
new4 KB
new10.77 KB
new4.34 KB
new11.41 KB

Here's patches for 9.0.x and 9.1.x with \Drupal\Core\Database\Database::getDatabaseDriverNamespace() deprecated in the 9.1.x. I don;t think we can reasonable test the deprecation because the method is protected.

I don't think we should deprecate calling Connection::__construct() without a namespace because this only occurs in tests - in regular runtime this is always created by a factory which gets its db settings via \Drupal\Core\Database\Database::parseConnectionInfo(). And providing a fallback is simple and helpful for the unit tests.

alexpott credited salah1.

alexpott’s picture

Crediting @salah1 and @neelam_wadhwani for their work on #3115388: PP-1: Deprecate \Drupal\Core\Database\Database::getDatabaseDriverNamespace as it is no longer needed which I've closed.

alexpott’s picture

Issue summary: View changes

Updated the issue summary with the deprecation information. I've not added a release note as I really don't think anyone if going to have to change anything because of this change.

daffie’s picture

Status: Needs review » Needs work

The patch for 9.0 is for me RTBC. The one for 9.0 is missing one small thing:

+++ b/core/lib/Drupal/Core/Database/Database.php
@@ -628,8 +634,14 @@ public static function getConnectionInfoAsUrl($key = 'default') {
+    @trigger_error(__METHOD__ . " is deprecated in drupal:9.1.0 and is removed from drupal:10.0.0. There is no replacement as \$connection_info['namespace'] is always set. See https://www.drupal.org/node/3127769.", E_USER_DEPRECATED);

I am missing a test for this deprecation message.

alexpott’s picture

@daffie as per #56

I don't think we can reasonable test the deprecation because the method is protected.

For me testing protected stuff should only be done when it involves logic. Triggering a deprecation message doesn't involve logic - all calls to the method will fire the deprecation - what are we testing?

alexpott’s picture

Status: Needs work » Needs review
StatusFileSize
new1.74 KB
new12.51 KB
new958 bytes
new12.35 KB

Fixing the tests. The SQLite driver in 9.1.x already doesn't unnecessarily set $this->connectionOptions

daffie’s picture

Status: Needs review » Reviewed & tested by the community

I thought that we should have testing for all deprecation message's, only @alexpott is saying that in this case it is not necessary.
No problem with the removal of the unnecessary setting of $this->connectionOptions in SQLite driver.
All code changes look good to me.
The testbot is happy.
For me it is RTBC.

  • catch committed 2989dd0 on 9.1.x
    Issue #3112476 by alexpott, Neslee Canil Pinto, mondrake, daffie, salah1...

  • catch committed aebcf1d on 9.0.x
    Issue #3112476 by alexpott, Neslee Canil Pinto, mondrake, daffie, salah1...
catch’s picture

Status: Reviewed & tested by the community » Fixed

We never add tests for constructor deprecations, agreed in this case too where there's no actual code path to test.

Committed 2989dd0 and pushed to 9.1.x. Thanks!

Committed aebcf1d and pushed to 9.0.x. Thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

quietone’s picture

Publish the change record