Implement fastpath database caching mechanism

Looking good to me. I'm really interested in this so subscribing now to see updates and post feedback.

Good to see you taking a look at this. Thanks.

I know that this is a first draft, but I think we'll want to document this a little bit better. The aggresive cache is cool but has some drawbacks as the _init and _exit hooks won't run. Depending on what modules you use, this may or may not be a problem. For example, poormanscron will no longer run, node counters will stop working, statistics logging will become inaccurate, etc.

More cleanup with documentation.

• Can't we tidy up the bootstrap process a little bit more? It feels a little bit "slapped on" right now. I wouldn't mind to massage the bootstrap process a little to make this a tad more elegant. (Not sure if it is possible at all.)
• Even with the eplanation, I don't think people will be able to predict the impact of 'aggressive mode'. For example, node counters will stop working. I bet many people won't be able to tell. Maybe we should add 'recommended, no side-effects' after 'standard', and 'expert mode, possible side-affects' after 'aggressive'? I think we should advise against using 'aggressive' caching.
• I would avoid saying 'twice as fast'. It depends on the modules you shortcut, your LAMP-configuration, etc.
• To make the form descriptions a little bit shorter, maybe we should move the 'introduction into the page cache' (currently under the 'Page cache'-setting) to the top of the page (i.e. make it a regular help text)?

My main issue is with the bootstrap code. I wish we could make it a bit more elegant and easier to follow. :-)

Two more details:

1. The title of the administration page reads 'page cache', and the title of the first configuration option is 'Page caching:'. It looks kinda dull, don't you think? It is a redundant repetition. I suggest we rename the title of the form element to 'Caching strategy:', or something a bit more descriptive.
2. The sentence "Enabling the cache will offer a sufficient performance boost for most low-traffic and medium-traffic sites." is redudant (people already figured that, hopefully) and should either be removed, or moved to the "introduction" at the top of the page (see previous comment).

Here is a useful snippet for the documentation:

print 'The aggresive caching strategy is incompatible with the following modules: '. implode(', ', module_list(TRUE, TRUE));

Needs to be Drupal-ised but it tells the site administrator exactly what modules conflict with the aggressive caching.

Does disabling these modules render the aggressive mode useless? If so, we could just as well advise people to disable these?

I am surely blind but if you call your cache before loading the session how do you know whether the user is anonymous or not?

From a usability perspective, if a user has modules installed that implement init and exit hooks, shouldn't we set the advanced setting to "disabled" so they *can't* click it?

Reason being, if they have modules enabled that are implenting these hooks, going with aggressive caching will obviously break these in some sort of form. Don't want someone accidently turning this on. Similar to how we prevent users from turning on clean urls.

subscribing

Updated patch. Fix the description on the cache settings page.

It will now show in the color green if you can safely enable expert mode. If not, it's shown in red. I cleaned up the wording quite a bit. Also, I changed the title on the page to "cache mode" instead of "cache strategy". Minor other tweaks to the interface.

Matt, you and Jeremy created the first fastpath module (fastpath_fscache) how did you solve the session problem there? I can see cache_get called from the fastpath function and global $user; in there. I think that's also broken.

When creating the EARLY_PAGE_CACHE step, I was looking at Jeremy's stuff and how he used a $_COOKIE['drupal_uid'] . I still think that's a better -- even if a bit clunky -- solution for certain environments than creating a database connection.

So -1 for removing EARLY_PAGE_CACHE.

Hm, I did not really grok you are now calling fastpath from config. That's even worse than I thought.

First of all, forget variable_get before you load the setting.php. variable_get uses the $conf in settings.php. Second, configuration must be a separation step if you want Drupal to be reusable. I am not even sure whether install does not break with this bold step.

I benchmarked this patch and it gives me a 55% performance improvement.

The latest patch got me to think: maybe we should remove the 'expert mode' radio button, and automatically switch to 'expert mode' when there are no conflicts ... That is, hide it all from the user. :)

Dries I thought about that too. However, if you do use expert mode it if works--what happens when a user then installs statistics, poormanscron, etc... and then notices that the site is behaving a bit odd.

We *shouldn't* automatically give it to them--then they blame Drupal and us :-) Rather, we give them all the warnings and the green light, but ultimately let them make the decision. If it breaks, well they shouldn't have taken the green pill ;-)

subscribe ... there are module enable/disable hooks now so it does seem possible to automatically disable/enable this mode modules chnaged. unfortunately, it seems difficult to know when a user wants to keep expert enabled even though he has devel or statistics enabled. maybhe the UI is more like a checkbox to 'force enable aggressive cache'.

i haven't closely reviewed the logic here yet.

Moshe, that is why I updated the patch-- you can still enable expert mode even if it affects some modules. However, there is a warning in nice bold red text that says this is not advised. If you are good to go, you see some bold green light text saying you have nothing to worry about.

I think that makes the most sense--users are free to do what they want, we provide the necessary information to let them know if it is a good or bad decision.

Marking back for review.

mav3rck: what I mean is this. If someone enables caching, Drupal will automatically try to use the aggressive path. If, however, the administrator enables a 'conflicting module', Drupal will automatically switch back to normal caching. In other words: an automated approach without extra user interface.

To check if there are conflicting modules, simple use:

  if (module_list(TRUE, TRUE)) {
    // conflicting modules
  }
  else {
   // no conflicting modules
  }

I think this approach is beneficial for various reasons: it will always behave correctly (administrators won't think their newly enabled modules are broken when aggressive caching is on), it automatically optimizes performance and there is less documentation to read. :)

Dries, what if enable Devel and Statistics module but I *know* how they break and are ok with the breakage, since I know what is going on. I don't Drupal to turn off aggresive caching now for me :-/

Just a note on this - as an enthusiastic drupal user but relative newbie, I'd rather have control over "aggressive" mode, with the warning, than have the decision made for me.

Two reasons:

For a start, there are things that devel does (clear cache, php info, showing you query information for each page you view (like those 600 on my default forums page with 50 subforums!), probably other stuff I've not got to yet) that wouldn't be affected by the caching to anonymous users if I've got it right. Why disable aggressive caching only because of query logging or whatever when people might be using devel for other reasons anyway?

If statistics module with aggressive caching only collected stats for logged in users, then it could also be useful for comparing page views or referrals for logged in and anonymous in users against external statistics software like google analytics/awstats etc. - google/awstats won't tell you how many users who visited your site were logged in or anonymous, this sort of would - even if it's unpredictable it's better than nothing.

Variable init simply does not belong to the database step -- what if I write code outside of Drupal and want to simply reuse the DB layer? And cache in session, what if I write a little script which serves file thus boots up until the session, what does cache do there? Add more steps if you want or simply move your cache from session to the beginning of late cache, it looks belonging there. This solves your variable init problem as well as that can move back to late cache.

Otherwise, it's a very promising patch.

variable_get to the rescue. Dries can have his automatic switching to non-aggressive -- this will result in the variable being non set to CACHE_EXPERT. m3avrck being the advanced user can force the setting to CACHE_EXPERT in his $conf in settings.php.

So, let's do the automatic stuff.

The patch is looking better and better. However, weird stuff is going on with the latest version:

- Without the patch, I can serve 101 requests/second.
- With the patch, and with the caching strategy set to 'recommended', I can serve 172 requests/second (?).
- With the patch, and with the caching strategy set to 'aggressive', I can serve 151 requests/second.

It is strange that I can serve that many pages with the strategy set to 'recommended'.

Update: turns out I get a PHP error with the strategy set to 'recommended'.

Move the cache.inc inclusion down from the database step to the late_page_cache step where it belongs.

I start to dislike the _drupal_cache_init function. I can see three totally distinct function named one for no real good reason. Down with spaghetti!

We do not need to take into consideration that a module can load code outside of function definitions. Our motto is: "we do not babysit broken code". And that kind of code is seriously fubar.

Committed a modified version of your patch to CVS HEAD.

Eh, with this patch, I can't install Drupal anymore because of this one line change in bootstrap.inc:
- foreach ($GLOBALS as $key => $value) {
+ foreach ($GLOBALS as $key) {

Committed to HEAD.