Warning: this patch is nowhere near ready and requires expertise and patience to test. With that said, we have DRUPAL_BOOTSTRAP_DATABASE as the first bootstrap phase.

This patch presumes you have SQLite support, #67349: Support SQLite databases and #331213: DBTNG: Make it easier to write weird database drivers patches installed. Before installing this patch, install Drupal into sites/default/files/.ht.sqlite, an expert profile is more than enough. A smaller binary DB and a generator script will be provided later. Now, destroy settings.php and install a MySQL Drupal. Apply patch. Nothing breaks. That's a start!

Files: 
CommentFileSizeAuthor
#20 bootstrap_new.txt20.03 KBchx
#20 ab.txt2.22 KBchx
#18 bootstrap_new.txt17.16 KBchx
#18 sqlite_db.txt44 KBchx
#13 bootstrap_new.patch.txt19.9 KBchx
#12 bootstrap_new.patch.txt18.81 KBchx
#10 empty.patch0 byteschx
Passed on all environments. View
#8 bootstrap_new.patch.txt8.67 KBchx
#8 dump.txt5.28 KBchx
#1 bootstrap_new.patch3.94 KBchx
Failed: Failed to install HEAD. View
bootstrap_new.patch4.37 KBchx
Failed: Failed to install HEAD. View

Comments

chx’s picture

FileSize
3.94 KB
Failed: Failed to install HEAD. View

A debug hunk was left in the original. Also, the profile that you install to MySQL and Drupal should be the same as the registry now uses SQLite.

chx’s picture

Our ultimate goal is to make Drupal run out of the box and turn the installer into a normal module. Small sites can run from SQLite , once the write concurrency outgrows that, users can run the installer and install into MySQL.

kika’s picture

UI team is looking at this process with awe. http://groups.drupal.org/node/16616

Dries’s picture

I'd appreciated some benchmarks on MySQL with this patch applied. What does this mean for things like 'static file caching'? :)

Anonymous’s picture

Status: Needs review » Needs work

The last submitted patch failed testing.

Crell’s picture

Subscribing.

Dave Reid’s picture

Subscribing. Very interesting and cool idea! Hope that by changing to database bootstrap first that this will still allow a graceful error page if PDO is not enabled (if a users ignores the requirement for PHP 5.2).

chx’s picture

Status: Needs work » Needs review
FileSize
5.28 KB
8.67 KB

I keep this at CNR to solicit code reviews. Understand please that this is not, in any sense, complete. I attach the patch as .txt to avoid testingbot. I added an SQLite dump as well.

Anonymous’s picture

Status: Needs review » Needs work

The last submitted patch failed testing.

chx’s picture

Status: Needs work » Needs review
FileSize
0 bytes
Passed on all environments. View

Keeping the issue CNR for real.

cburschka’s picture

Heh, testbot can be quite insistent! =)

"Awesome" just about sums it up. I think we'd set a record for improving the installation process between versions: 4.7 had an SQL file, 5 had a simple database installation script, 6 lets you set up your site/admin user (+ neat progress bar), and 7 may work out of the box without installation entirely?

Two things I'm sure you've thought of: DB error handling would probably need to be bootstrapped before the database, because spilling PDO errors on the users is very bad.

Secondly, will the MySQL install replace SQLite entirely, or will SQLite remain in place after installation as a fallback (for error-handling, cache-serving etc.) when the database server goes down?

chx’s picture

FileSize
18.81 KB

Some progress. I suspect I will move the form to system.install.inc later. It's coming along quite nicely.

chx’s picture

Title: Bootstrap database first » Remove install.php, bootstrap from sqlite
FileSize
19.9 KB

This begins to, you know, work. The first two steps are there, and then I will add the rest and then benchmark and stuff.

meba’s picture

subscribing

Status: Needs review » Needs work

The last submitted patch failed testing.

lilou’s picture

Status: Needs work » Needs review
catch’s picture

What's the exception handling like with this patch if there's no pdo (or drivers). In HEAD the failures are pretty nasty, and #299308: Installing Drupal by visiting index.php (rather than install.php) leads to a fatal error when PDO is not enabled and #315901: Add server capabilities check to installation/upgrade tasks both attempt to deal with that in complementary ways - but if we lose install.php then we need another way to do the same thing (inform people in clear terms if their server doesn't meet requirements and direct them to help).

chx’s picture

Title: Remove install.php, bootstrap from sqlite » bootstrap from sqlite
FileSize
44 KB
17.16 KB

I back down -- better not to do install.php overhaul now. The sqlite_db.txt should be copied to sites/default/files/.ht.boot.sqlite -- a further revision will do that automatically and complain if the file is readonly. Drupal installed for me. The results are terrible, on a fresh install, HEAD gives me 3.43 requests second the patch gives me 0.95... and I can't see why really. Is SQLite that much slower? I'd be surprised! If that's so then we can only rely on SQLite during maintanence which would be a major disappointment.

cburschka’s picture

Unusual. I've tried out SQLite on my shared server with no fancy performance tweaking, and it was not perceptibly slower than MySQL. Of course, I didn't do rigorous benchmarking with a big database.

chx’s picture

FileSize
2.22 KB
20.03 KB

Unusual, because I am moron and forgot to patch variable_del which caused the menu to rebuild every pageload. That hurts. Apache bench attached: HEAD is 122 +- 5 vs SQLite is 125+-7 here. It's the same.

beejeebus’s picture

subscribe.

moshe weitzman’s picture

subscribe ... will we still prompt the admin to create uid=1?

chx’s picture

The problem is this -- as a site grows, SQLite will be too slow -- and this mostly has to do with writes so there is no hard point. So if someone starts a Drupal site in an SQLite database at some point (s)he might be disappointed by mysterious problems much later on. While a migration is possible with a batch process (SELECT * -- drupal_write_record -- repeat, rinse) it 's a nontrivial process and might not fix the real cause for a site slowdown (crappy written queries will be slow on mysql too). So while technically we *could* ship Drupal to run from the box this dream might not be feasible. Maybe some demo profile can ship like that but I'm afraid that's it.

After the above, I think we should ship with a very small SQLite database which holds registry and registry_files and a new settings table. The settings table will hold the DB conenction data and the list of enabled modules. This SQLite DB will sit in the root of Drupal install as we ship it and when you install it will copy itself into the files dir as it can. This will require that conf_path .'/files' to exist no matter what you set files later but ah well. These tables are enough for DBTNG to work all the time -- we need the module list for schema to work (as schema needs a module_invoke_all).

I think a followup patch should also have a very small menu_router table in there with paths like 'install' and 'down' so we can get rid of the dreaded maintenance mode (and there was much much rejoicing).

Damien Tournoud’s picture

I have nothing to add to #23: opening two database connections, one for configuration+registry, the other for virtually everything else seems like the way to go.

That way offers the most flexibility:

- in small blog-like setup, you can choose to have two SQLite databases
- in large setups, each cluster node will have a local SQLite database for configuration, registry, etc. and will hit the common MySQL/PostgreSQL/Oracle/SQL Server database server for everything else

As discussed with chx on the IRC, we need to think more about how deployment tasks (enable/disabling a module, changing database configuration, etc.) will work in that kind of setup.

chx’s picture

I think that the registry and the system table both can live in the primary DB and the settings DB merely keeps a copy of it.

Crell’s picture

I agree with #25. The whole design of "Targets" in the database layer is to enable transparent fallback. That means that we just need to flag selected queries as "system" target, and if a system target is defined it gets used. If not, default gets used. Then we just need to make sure we clone the DB information appropriately when needed.

chx’s picture

Just got an idea: we keep the cache_registry in SQLite and we keep a registry_version variable which upon rebuild is increased, this has a copy in the settings table (in the SQLite DB) and if versions do not match, the cache in SQLite is truncated so this instance of the cache gets rebuilt as well.

For maintenance we shall have the registry tables and the menu_router table in the SQLite database but cache_set will check whether the 'default' DB is active and not write the cache if it is not.

Following on this versioned-cache-truncate, we could, if we want to, move more cache tables in the SQLite DB and serve cached pages out of it without bootstrapping to MySQL. For scalability this can be quite good. But, that's a followup.

Crell’s picture

Right now there is no way to route selected cache tables to another target or subsystem without horrid ickiness, hacks, or wasting layers of function calls with a contrib module. I don't like any of those options.

Also, killes has pointed out that the cache may not always be so read-heavy. The page cache on d.o, for instance, clears on a very regular basis so we'd actually lose performance by moving that to SQLite.

pwolanin’s picture

@chx - I'm not clear from your comment how this comparison would be done. Another/additional approach would be to timestamp the info in sqlite and always truncate after a certain time.

We have other caches that are more certain to be read heavy - such as cache_menu. Also what about the menu router table itself?

Wim Leers’s picture

So first SQLite was used to speed up Hierarchical Select and make it more scalable (although indirectly and only in WebKit since no other browser supports HTML 5 client-side databases yet), and now it might be used to speed up Drupal core. SQLite ftw! :)

Very interesting work chx :) Lots of potential.

Subscribing.

chx’s picture

There is a problem with this plan: if the SQLite database with the registry is inside an Apache writeable directory then we are back to the problem where a relatively simple uploading bug leads to PHP execution. Upload a modified SQLite DB and a PHP file and then bummer, you can have your PHP file included in Drupal. What I will do here, md5 everything cache_get'd from the cache_registry in SQLite DB and store the MD5 of that into variable table. This is also good to trigger the rebuilds, just delete the variable.

miro_dietiker’s picture

chx, sounds interesting but i'm confused:
If a "relatively simple uploading bug" leading to this result and if you do md5 and store the result in the sqlite itself, that's almost the same degree of complexity to inject code (since a potential offender could also build md5 correctly). Still if you add a local secret: If you could upload a php file too, one could also extract the secret.
Right: first that code needs to be called at all (which possibly won't happen in case of mismatch...). And if we still consider the secret to be save.. A md5 of the SQLite itself is always wrong if you store the value in its own table. Well a second db for the md5 finally would lead to further complexity.
And even if you update something and if drupal refuses to include, who is allowed to rebuild and empty md5 caches? (Will drupal switch off completely? A site maintainer would always hit "rebuild" if he's getting such a message.. and still we need bootstrap for the user 1 to login) ... or do we need another rescue tool then to reset SQLite md5 protection variables?
Module development and module replacements could become very cumbersome.

Finally the most important warning: Executing md5 on every file to check before including it would lead to massive impact in speed!
To protect you'd need to do this all the time. The only pretty speedy check would rely on stat() and as such file timers. but these could be open to an offenders manipulation too (but hopefully not as long as he's only capable of SQLite overwriting)...

chx’s picture

It's much simpler: the cache_registry table is in SQLite, when cache_get that then its MD5 is compared with a variable_get which is in MySQL. if the MD5 does not match then the cache_registry is rebuilt from the registry table. No idea when I will have time to code all this discussed here.

chx’s picture

catch’s picture

Status: Needs review » Needs work
Issue tags: +Needs architectural review
Wim Leers’s picture

Status: Needs work » Closed (fixed)

This has become reality with the new DB layer: http://api.drupal.org/api/drupal/INSTALL.sqlite.txt/7. :)