bootstrap from sqlite

UI team is looking at this process with awe. http://groups.drupal.org/node/16616

I'd appreciated some benchmarks on MySQL with this patch applied. What does this mean for things like 'static file caching'? :)

Subscribing.

Subscribing. Very interesting and cool idea! Hope that by changing to database bootstrap first that this will still allow a graceful error page if PDO is not enabled (if a users ignores the requirement for PHP 5.2).

Heh, testbot can be quite insistent! =)

"Awesome" just about sums it up. I think we'd set a record for improving the installation process between versions: 4.7 had an SQL file, 5 had a simple database installation script, 6 lets you set up your site/admin user (+ neat progress bar), and 7 may work out of the box without installation entirely?

Two things I'm sure you've thought of: DB error handling would probably need to be bootstrapped before the database, because spilling PDO errors on the users is very bad.

Secondly, will the MySQL install replace SQLite entirely, or will SQLite remain in place after installation as a fallback (for error-handling, cache-serving etc.) when the database server goes down?

subscribing

See: #335122: Test clean HEAD after every commit and http://pastebin.ca/1258476

What's the exception handling like with this patch if there's no pdo (or drivers). In HEAD the failures are pretty nasty, and #299308: Installing Drupal by visiting index.php (rather than install.php) leads to a fatal error when PDO is not enabled and #315901: Add server capabilities check to installation/upgrade tasks both attempt to deal with that in complementary ways - but if we lose install.php then we need another way to do the same thing (inform people in clear terms if their server doesn't meet requirements and direct them to help).

Unusual. I've tried out SQLite on my shared server with no fancy performance tweaking, and it was not perceptibly slower than MySQL. Of course, I didn't do rigorous benchmarking with a big database.

subscribe ... will we still prompt the admin to create uid=1?

I have nothing to add to #23: opening two database connections, one for configuration+registry, the other for virtually everything else seems like the way to go.

That way offers the most flexibility:

- in small blog-like setup, you can choose to have two SQLite databases
- in large setups, each cluster node will have a local SQLite database for configuration, registry, etc. and will hit the common MySQL/PostgreSQL/Oracle/SQL Server database server for everything else

As discussed with chx on the IRC, we need to think more about how deployment tasks (enable/disabling a module, changing database configuration, etc.) will work in that kind of setup.

I agree with #25. The whole design of "Targets" in the database layer is to enable transparent fallback. That means that we just need to flag selected queries as "system" target, and if a system target is defined it gets used. If not, default gets used. Then we just need to make sure we clone the DB information appropriately when needed.

Right now there is no way to route selected cache tables to another target or subsystem without horrid ickiness, hacks, or wasting layers of function calls with a contrib module. I don't like any of those options.

Also, killes has pointed out that the cache may not always be so read-heavy. The page cache on d.o, for instance, clears on a very regular basis so we'd actually lose performance by moving that to SQLite.

@chx - I'm not clear from your comment how this comparison would be done. Another/additional approach would be to timestamp the info in sqlite and always truncate after a certain time.

We have other caches that are more certain to be read heavy - such as cache_menu. Also what about the menu router table itself?

So first SQLite was used to speed up Hierarchical Select and make it more scalable (although indirectly and only in WebKit since no other browser supports HTML 5 client-side databases yet), and now it might be used to speed up Drupal core. SQLite ftw! :)

Very interesting work chx :) Lots of potential.

Subscribing.

chx, sounds interesting but i'm confused:
If a "relatively simple uploading bug" leading to this result and if you do md5 and store the result in the sqlite itself, that's almost the same degree of complexity to inject code (since a potential offender could also build md5 correctly). Still if you add a local secret: If you could upload a php file too, one could also extract the secret.
Right: first that code needs to be called at all (which possibly won't happen in case of mismatch...). And if we still consider the secret to be save.. A md5 of the SQLite itself is always wrong if you store the value in its own table. Well a second db for the md5 finally would lead to further complexity.
And even if you update something and if drupal refuses to include, who is allowed to rebuild and empty md5 caches? (Will drupal switch off completely? A site maintainer would always hit "rebuild" if he's getting such a message.. and still we need bootstrap for the user 1 to login) ... or do we need another rescue tool then to reset SQLite md5 protection variables?
Module development and module replacements could become very cumbersome.

Finally the most important warning: Executing md5 on every file to check before including it would lead to massive impact in speed!
To protect you'd need to do this all the time. The only pretty speedy check would rely on stat() and as such file timers. but these could be open to an offenders manipulation too (but hopefully not as long as he's only capable of SQLite overwriting)...

This has become reality with the new DB layer: http://api.drupal.org/api/drupal/INSTALL.sqlite.txt/7. :)