Third-party database drivers are not tracked by update-status

As noted on the linked blog, Cassandra, MongoDB, CouchDB, and other such systems are not SQL servers. Therefore they are not relevant to this discussion, as the DB layer in Drupal 7 is SQL-specific. There's nothing wrong with Document DBs, but they are a different animal and not a drop-in replacement for MySQL or Oracle or Postgres by any means. You don't use those with Drupal's "DBTNG" database layer.

That leaves only a tiny number of SQL databases of relevance. PDO, the PHP database system on which Drupal's DB layer is built, theoretically has support for ten drivers: MS SQL, Firebird, "IBM", Informix, MySQL, Oracle, ODBC, PostgreSQL, SQLite, and 4D.

Of those Firebird and 4D are marked Experimental and "don't entirely work yet". 4D I know in particular is not a full SQL database, and so could not be used to run Drupal anyway. Firebird, as far as I'm aware, has been broken for a long time. I can't say that I've ever heard of someone using Informix or "IBM" (whatever that is; isn't it DB2?) with PHP.

So realistically, the DBs that are even relevant to support are: MySQL, PostgreSQL, SQLite, Oracle, MS SQL, and ODBC. Of those six, three already exist in core; that's a step up from the one and a half we supported in Drupal 6 (MySQL and, half-heartedly, PostgreSQL).

So that means we're talking about, really, three drivers at most: Oracle, MS SQL, and ODBC. That's it. I don't believe therefore that this is the significant issue you're making it out to be. Certainly creating a new category of project on Drupal.org to support a category that has an upper bound of three items is unnecessary overkill.

Additionally, the changes necessary to support DB drivers as components alongside modules and themes, or as modules as was previously discussed, would necessitate non-trivial work along the critical path. The "find a DB driver to use" logic is executed on every single request of any sort, and therefore we do need to be "counting cycles" for that. Disk IO is a slow operation, and we cannot cache a DB driver location in the DB the way we can for module locations, so that's a non-trivial amount of work Drupal would have to do on every single page request. Drupal 7 has enough performance issues already. :-) Such changes are also substantially out of scope for Drupal 7 anyway when we are approaching beta and could not even be considered until Drupal 8.

The current driver structure was written with contrib-extensibility in mind (Oracle in particular). Unlike in previous versions of Drupal, adding DB drivers requires modifying not one single existing line of Drupal core; all it requires is adding a directory to a specific location in a supported way. That is not hacking core; it is a new supported extension mechanism. If someone thinks that constitutes hacking core, well, they're wrong. :-) That's no more hacking core than the Libraries module in Drupal 6 was hacking core.

As for moving the non-MySQL drivers out of core, it's been noted before that when something is moved out of core it tends to die from lack of attention. The only reason that PostgreSQL and SQLite have gotten the attention they've gotten to date is because they're in core, and therefore their 100% compatibility is considered a release-blocking issue. If those move out of core, I fully expect the support for them would drop dramatically. We've gone to great lengths to get support for more DBs in core; let's not roll back that work now.

Which then leaves the question of moving the extant MS SQL or Oracle drivers into core proper. I don't believe that is necessary. For one, both MS and Oracle have volunteered and been working to set up test boxes of their own to run the core unit test suite on those databases; I am not sure of the status of those efforts but I have spoken with reps from both companies about it. That's something that SQLite or PostgreSQL cannot do as there is not a single vendor behind them. For another, and I fully understand that not all may share this viewpoint, Drupal is a Free Software project. If we put a driver in core, that means we're committing to the Drupal community supporting it. As a Free Software community we cannot reliably support a proprietary, closed source product in a community fashion. Thus we cannot properly support it in core, nor should we be expected to. Drivers for proprietary products do not belong in the core of a leading Free Software product. We should support them if at all possible, and we do, and leading members of the Drupal community have been working with both firms.

In conclusion:

- We cannot make this sort of change this late in the Drupal 7 development cycle.
- We are talking about, in practice, an extremely small number of use cases: 2, perhaps 3.
- Those use cases are already addressed.
- The question of which drivers are accepted to core it not arbitrary: Free Software DBs are supported in core, proprietary ones are not.

I believe this is a won't fix, but will leave it open briefly to allow others to respond. I don't believe there is anything to do here.

That is not an officially adopted policy; it's more an informal policy of the DB team, one that as DB maintainer I fully support. :-) I am open to documenting that somewhere if we can determine an appropriate somewhere.

I still don't consider adding DB drivers to be hacking core, any more than adding install profiles (which live in /profiles) to be hacking core. Install profiles have been available since Drupal 5.

Making DB drivers support the update module (essentially what you're asking for) would require them to have .info files. It would also require reconfiguration on Drupal.org. I cannot speak to the viability of the latter, but I highly doubt that infra team would be inclined to do so when we're on the cusp of completely changing both Drupal.org itself and our entire code management infrastructure (the Git Migration). That would be up to dww, probably, and would likely have to wait for Drupal 8 either way.

I strongly disagree here.

Because database engines are not modules, themes or installation profiles, there is no way to properly package them with Drupal. They are essentially nothing more then core patches.

Because the database layer doesn't use standard Drupal plugin mechanisms, we cannot, as noted above:

• Use drush_make to package distributions of Drupal containing database drivers
• Have update status notifications
• Have usage status for those projects

The way forward is to make the database layer use the standard mechanisms offered by Drupal core, by creating a new project type for those drivers, in addition to 'module', 'theme' and 'profile'. On the drupal.org side, we will have to create a new project type also, but this should be as easy as adding a taxonomy term to the correct vocabulary.

Promoting to major.

Here is a first shot at a patch. Honestly, the changes look very minor.

We are talking about, in practice, an extremely small number of use cases: 2, perhaps 3.

I don't believe so. There will be several proper drivers in contrib (SQL Server, Oracle, Drizzle, MongoDB), and probably several pseudo-drivers (extending existing drivers to add stuff like caching, different load-balancing strategy, health control for slaves, etc.).

MongoDB is not a DBTNG driver, nor is any other Document DB. chx has been experimenting with MongoDB as one just to prove that it can be done, not because any sane person would ever actually do so. :-) Drizzle has no PDO driver that I'm aware of and has forked from MySQL so it cannot use the MySQL PDO driver.

I don't know how many generic (ie, not site-specific) subclassed drivers we're likely to have and then you're talking about a very small percentage of sites that would be using them. So I definitely don't see this as major.

This is also technically an API change (adds required files to DB drivers), and dear god in heaven this is alpha-7 with a beta coming soon. Stop with the API changes.

Also, using the "sometimes the parameters mean X, sometimes Y, depending on if the first is NULL" pattern for db_autoload() gets an immediate won't fix from me. That's an incredibly bad pattern that serves no purpose other than to make code harder to follow.

I'm inclined at the very least to move this to Drupal 8, although I still lean toward won't fix.

Well, this is definitely major if we want to support third-party database drivers properly *and we definitely want to*. I'm not sure why justify your aggressivity against the whole idea of properly supporting this.

MongoDB is not a DBTNG driver, nor is any other Document DB. chx has been experimenting with MongoDB as one just to prove that it can be done, not because any sane person would ever actually do so. :-)

I was explicitely speaking about the experimental "mongodb caching" driver from chx. This is definitely a DBTNG driver. I never said nor implied that Document DB are DBTNG drivers.

Drizzle has no PDO driver that I'm aware of and has forked from MySQL so it cannot use the MySQL PDO driver?

Drizzle supports connecting by PDO_MySQL perfectly. It actually has two interface, the legacy mysql client protocol and the new drizzle protocol. A Drizzle driver can and should be done. See #809810: Support the Drizzle database engine for context.

I don't know how many generic (ie, not site-specific) subclassed drivers we're likely to have and then you're talking about a very small percentage of sites that would be using them. So I definitely don't see this as major.

My guess is that most big hosts (including Acquia Hosting) will provide a site-specific driver. There is currently no way to do that properly, and it's a big failure of the whole revamped database layer.

This is also technically an API change (adds required files to DB drivers), and dear god in heaven this is alpha-7 with a beta coming soon. Stop with the API changes.

This is definitely a (small) API change, but you cannot argue at the same time that "this is an API change that will affect database drivers writers" and "there will be very few database driver writers".

Also, using the "sometimes the parameters mean X, sometimes Y, depending on if the first is NULL" pattern for db_autoload() gets an immediate won't fix from me. That's an incredibly bad pattern that serves no purpose other than to make code harder to follow.

Yes, this particular pattern is not very pretty. Sure. But we have it all over the place already, and there is no clean way to do this that I can see.

Subscribing, to better track for DBTNG backport.

I'd really be in favor of some sort of standardized location for these things, as D7 is released I expect the number of folks using the DBTNG backport in increase, and one of the main reasons they do so, it to use some of the new drivers. I'd love to have a standard way to install these both for 7 and 6.

Can we please stop arguing semantics about what a driver connecting to? If I write a carrier pigeon driver, I don't care what you say abou it, but it's using the same APIs to connect to Drupal. That's all that matters.

It would be nice to get support for this as getting more database companies to adopt Drupal seems like a really smart strategy. At the same time, we're way beyond the point to figure this out, and I'm wary that there will be at least 3 follow-up issues to get this right (i.e. properly integrated with status update and whatnot). The proposed patch only implements the bare minimum. It's a bit sad but we should postpone this to Drupal 8.

+++ modules/update/update.fetch.inc
@@ -144,6 +144,7 @@ function _update_process_fetch_task($project) {
+    $h = fopen('/tmp/update', 'a'); fputs($h, print_r($xml, TRUE)); fclose($h);

Debug output?

subscribe

To clarify: http://drupal.org/project/mongodb_dbtng Drupal installs, I can log in and see tabs, node creation, user creation and filter tests pass.

Yes. It's crude. It's insane. But there is no reason to say it does not work.

So here is how we can solve this in D7 without patching core. Create a .module file, 0 byte, write a .info file, put the database driver into the aptly named directory below the module and then add a line into README.txt "copy or symlink this directory into includes/database".

So here is how we can solve this in D7 without patching core. Create a .module file, 0 byte, write a .info file, put the database driver into the aptly named directory below the module and then add a line into README.txt "copy or symlink this directory into includes/database".

That gives you nothing:

• Packaging: you have to add a symlink, which is not supported by drupal.org packaging system
• Update status: you will only get update status if the module is enabled, but you have zero reason to enable the module (because it's not required by the installation and the site will work without it)
• Usage data: same applies

Everyone should realize that not properly supporting database drivers like this means that the only way to get a third-party driver into a distribution is by patching core. Drupal.org doesn't (yet) support patches in .make files, but might one day.

Anyway, having no packaging, no update status and no usage data means that third-party drivers are not supported in Drupal 7. It's not really a big deal, but we all have to be clear about this.

@Damien Making changes to the API to make database drivers a top level entity (like a module, theme, or install profile) would be great. I previously looked into this as well. But, to do so would not be simple, would be an API change, and is past the point in the D7 release cycle to include it. I look forward to seeing patches in D8. Not sure there is a solution you'll be happy with until then.

We will definitely make API changes up until 7.0 is released, and we will probably do so in 7.1, 7.2, 7.3, 7.4 etc.

The mere argument of "API change" is not an argument at all. API changes have to be avoided, but we make them all the time, on the argument that they bring more benefit then trouble. The changes described in the patch in #5 are pretty much self-contained, and will not impact *anyone* except the writers of database engines, who are mostly all participating in this issue.

There's also performance implications to consider, and the potential for introducing new bugs. We're talking about the critical path here so have to be careful, more careful than I want to be when we're desperately trying to squeeze out a beta. :-)

Pardon me for this thought but I like this. Let Drupal 7 be a release when drivers are not yet modules. There are going to be such problems every time someone tries a Drupal on a non-MySQL database... let it mature. With plugins in D8, we will be able to do a lot more, anyways and then those drivers will need to become modules anyways so they can implement plugins.

DamZ asked me to comment here about the update status + usage statistics implications of this patch. I don't have time or head-space to do a thorough review, but off the top of my head, the following would have to be addressed:

=== update manager/status ===

- Although the changes in the patch should make it so we fetch data and process the status for any DB drivers, we'd need changes in update.report.inc to display that information properly.

- There's obviously nothing here to teach the update manager how to upgrade a DB driver via authorize.php and friends. That'd require (at least):
-- a DatabaseDriverUpdater class in modules/system/system.updater.inc:

class DatabaseDriverUpdater extends Updater implements DrupalUpdaterInterface

-- changes in modules/update/update.manager.inc
-- maybe some other stuff, I'd have to look more carefully

=== drupal.org ===

- Yeah, we'd probably just need a new term in the Project types vocab for the new project type
- It's possible that the release history XML generation script would Just Work(tm) for releases of the new project type
- Certainly serving the release history XML will Just Work(tm)
- I'm pretty sure that the usage tracking stuff isn't going to care about the project type
- It's possible that the http://drupal.org/project/usage page would also Just Work(tm)

So, pretty easy on the d.o side. Mostly 1 trivial change, and then checking that everything else behaves as expected.

I'm not in a great space right now to say much else about this issue, so instead of flu-induced rambling, I'll just leave it at that for now... ;)

As a problem that affects a very small percentage of Drupal sites, and even then only in a minor way, I don't think we can classify this as major. Per #1050616: Figure out backport workflow from Drupal 8 to Drupal 7, major and critical bugs are now preventing development in other areas of Drupal core. If we'd like this issue to prevent development of features entirely in Drupal 8, please re-categorize again, but I don't see the actual harm here.

hi,

I have done a Cassandra driver based on cassandra-pdo driver. What I understand here is that I can't share it easily to the community or things have changed on drupal8 and I can share it as a project without asking users to hardlink the project to core/lib/Drupal/Core/Database/Driver ?

Unless I'm mistaken, #2209307: Convert database drivers into a regular extension type would help to resolve ~80% of this?

#3129534: Automatically enable the module that is providing the current database driver converted contrib database drivers to extensions, so this should be resolved now. Marking outdated.