Convert database drivers into a regular extension type

+++ b/core/includes/install.inc
@@ -136,9 +136,9 @@ function drupal_get_database_types() {
   $files = file_scan_directory(DRUPAL_ROOT . '/core/lib/Drupal/Core/Database/Driver', $mask, array('recurse' => FALSE));
...
+  $listing = new ExtensionDiscovery();
+  $files += $listing->scan('driver');

Shouldn't the core drivers be moved to /core/drivers then?

This falls under the must-for-MongoDB umbrella so I will take it on.

How is this a must for MongoDB? DBTNG is specifically for SQL databases. Using it for a document DB has *never ever* been a supported feature, nor is it now, nor will it be. (That chx was able to do horrible unholy things to it in the past to make it work doesn't change that fact.)

Honestly given how much work we've done to reduce the importance of SQL databases in general (we're close to being able to run SQL-free, I think) I don't see how this is even necessary. There's really only 2 viable contrib drivers that could exist (Oracle and MS SQL).

With the recent changes to the extension system this is no longer a major rewrite of the entire extension system, as demonstrated by the patch above.

There's really no reason to do this now. Even if there are not as many as there are modules it is so much nicer to drop them into /modules instead of /core/includes/database or wherever. Most sites don't have an awful lot of themes either, but we still treat them equally to modules.

Field storage engines are gone because storage is all handled at the entity level now. They can't be DB drivers anyway because that's a horrid mixing of layers. DB divers are *just* for abstracting differences in SQL servers. That's *it*. Everything else MUST be done at a higher level. (Document DBs, Field storage engines, etc.)

We can't even keep up with supporting SQLite and Postgres in core now; moving MSSQL and Oracle into core would in no way help them get support, and would probably just slow the release of D8 because we'd feel compelled to actually test them, something extremely few people even have the tools to do. (Plus, I am firmly against us including drivers for non-free databases in core.)

Do 2) and additionally move the built-in core drivers into /core/drivers.

We should absolutely do this. Not sure if it's easier or harder to do this directly in this issue, but we should definitely do it.

> How is this a must for MongoDB?

I will be shocked if every SQL query will become, indeed pluggable by the time of the release. One can hope of course, with the recent forumrepository and path patches... we will see. It's Plan B, essentially: revive mongodb_dbtng for the very few remaining queries (and map them by query string to mongodb helper).

Oh just realized now that this is still assigned to you @chx, hope this didn't duplicate any work. Also, I just now realized there's a patch already here, so sorry that there's no interdiff.

This installs on my system. It goes a step further as the original patch in that it also converts the existing driver types to the new system.

I wasn't sure how to do the namespacing of the respective classes, would love some thoughts on that.

We also probably want to move label and description of the database drivers into the YAML files, that is not done so far.

Let's see.

Edit: Also the core: 8.x validation that is mentioned in the IS is missing so far.

Ahh so I forgot to register the driver namespaces during regular runtime. Not sure where to do that. Also, should we just register all available namespaces (which is what we do now, implicitly) or should there be some sort of installation status for drivers?

This just hacks the namespace-registering into DrupalKernel::boot(). Let's see if it's green. Not sure if there's a better place.

Awesome, thanks a lot! Makes perfect sense to register the namespace in addConnectionInfo().

+++ b/core/includes/install.inc
@@ -135,9 +135,10 @@ function drupal_detect_database_types() {
+  $discovery->setProfileDirectories(array());

So if I understand this correctly, this will allow selecting database drivers from profiles other than that selected in a previous step during the installation. Is there a specific reason for this?

Reviving this one.

Miraculously, Git was able to merge this without any problems.

This also fixes the MigrateTestCase issues by registering database drivers in unit tests.

Ahh, this is one of those patches where forgetting -C -M is not a good idea. This is the same patch except a teeny-tiny bit more readable.

Interestingly I get completely different fails than the testbot but this let's me install both through the UI and Drush. Let's see.

I don't know how I managed to do it, but somehow I switched up the Truncate classes between MySQL and PostreSQL.

While I was at it I changed all the moved @file declarations from "Definition of" to "Contains".

Thus, 2 interdiffs.

And this time with -C -M again. Although beware that for the Merge classes Git will tell you that I moved the MySQL one to SQLite and vice versa. This is because those files are identical but for the namespace, which changes anyway, so that they have the same similarity index.

Meh, I have no idea yet how to resolve these hacks properly, but let's see if we can get this green first.

This should fix the remaining test failures except the one that caused the fatal, which I didn't figure out which one it was.

I've added some debug code (not in the interdiff, to avoid confusion) to help sort that out. Let's see.

Meh, so much for that debug code. Next try.

Let's see.

Yeah...

Updated patch re-rolled for comment #34

wrong merge

I think this issue needs new patch, not a re-roll. I checked the patch code and there is no core/lib/Drupal/Core/Database/Driver/fake folder.

Closed it as outdated but feel free to reopen it.

I still sure that db drivers should be extensions, there's more then "fake" in last patch

@tstoeckler: I have tried to reroll your patch and I am doing something wrong, because I am unable to do a complete run of the install process. Can you reroll your own patch? I think this is a important issue and I will do reviews to get the issue fixed.

I think this is an important issue, because I want to make Drupal better. Lets look at Drupal with a outside-in approach from the viewpoint of contrib database driver creator and its user.

Creating a contrib database driver is not an easy task. Finding what to do and how to do it is not easy. An extra problem is that it is not only the driver any more. There are also services that need to be overridden. Which services that need to be overridden and which ones aren’t is also not easy to determine.
As contrib database driver developer you want your driver to pass all the core tests. Only there is no way to make minor changes to those test because they are in core. The by core supported database drivers do make minor changes to core tests because they can and they are needed. Contrib database drivers should be able to do the same.

The next problem is that you should not hack Drupal core in production site. A contrib database driver needs to place their contrib database driver files in DRUPAL_ROOT/core/lib/Drupal/Core/Database/Driver. For me this is hacking core and we should never have to do that.

If I where to create a contrib database driver I would use the by Drupal core supported drivers as my main example. And those drivers are completely integrated in core. The different parts are hidden/scattered in the Drupal core. The driver is easily found, but the overridden services are not and the same for the driver changed tests. A contrib database driver developer should not be searching the whole of Drupal core to find out how to make his/her contrib database driver.

How to make it better.

Move the Drupal core database drivers to DRUPAL_ROOT/core/databases directory and the contrib ones should live in DRUPAL_ROOT/databases. First: the directory name “databases” is more easily understood then “drivers”. Second: There is more in the directory then the driver alone.
The location for the for the driver should be DRUPAL_ROOT/core/databases/mysql/src/Driver for the MySQL driver and DRUPAL_ROOT/databases/oracle/src/Driver for the Oracle driver. The specific overridden services go to DRUPAL_ROOT/core/databases/mysql/src and DRUPAL_ROOT/databases/oracle/src. The overridden tests should go to DRUPAL_ROOT/core/databases/mysql/tests and DRUPAL_ROOT/databases/oracle/tests.
Now the database drivers for MySQL, PostgreSQL and SQLite have become the examples for the contrib ones. Those are not only Oracle and MS-SQL but also other RDBD’s as DB2, Teradata, SAP-server etc. There are also cloud based databases as Amazon Redshift, AzureSQL etc. In the future we should start supporting NoSQL databases as MongoDB, Cassandra and others.

It is not that I personally would like to add as many databases as possible to Drupal for the fun of it, but I think that the usage of Drupal will grow much by supporting many more databases. One of the great strengths of Linux is that it will run almost anywhere. Lets do the same with Drupal. That does not mean that we should add more databases in Drupal core. A better option is it to make easier for others to add their own contrib database driver.

Hi daffie

how can i connect to redshift database as earlier you mention we need an connection with AWS redshift as well

@ankit: The AWS Redshift database is based on the Postgres 8 database and we have a Drupal driver for Postgres 9. So what I would do is to use our Postgres 9 driver and test how well that does. If you find any problem, then please report them by creating bug reports.

From #47:

The next problem is that you should not hack Drupal core in production site. A contrib database driver needs to place their contrib database driver files in DRUPAL_ROOT/core/lib/Drupal/Core/Database/Driver. For me this is hacking core and we should never have to do that.

Actually you do not need to "hack" core, contrib database driver are supposed to be sitting in a "\drivers\lib\Drupal\Driver\Database" directory, which is out of core.

Database access is isolated and easy to plug into (I can tell that from being the MS SQL Server driver maintainer), but it's true that to get it running the first time you need a lot of reverse engineering. But for anyone that wants to write a new database driver they could easily take a look at one of the current contrib implementations and elaborate on top of that.

It is true that some parts of core have specific database logic (such as doing one thing for SQLite and other stuff for the rest of drivers) but there is no approach I can think of to deal with that, except for overriding whatever part of core is doing that (which should by the way be easier now in Drupal 8 as overriding core code/services is much much much more easier than it was in Drupal 7).

What I really miss the ability for the driver to tell the testing framework to skip some tests (or to override them with custom implementations).

Actually you do not need to "hack" core, contrib database driver are supposed to be sitting in a "\drivers\lib\Drupal\Driver\Database" directory, which is out of core.

@david_garcia: I did not know there is a specific location for contrib database drivers. Thank you for the info!

What I really miss the ability for the driver to tell the testing framework to skip some tests (or to override them with custom implementations).

@david_garcia: If you want to work on that, create an issue for it and I will help by doing the reviewing.

This has been partially achieved in #3120096: Support contrib database driver directories in a fixed location in a module, and wkll be completed in #3129043: Move core database drivers to modules of their own.