The attached patch is a very early pass at a plugin caching architecture for Views. It allows each display to have a specified caching mechanism, modeled after the access plugins. (See screenshot below.) The plugin architecture is in place and working, with a no-op "no caching" implementation, and a dirt-simple "cached for [n] minutes" plugin. The latter is unfinished, but should show the direction things are going in.

After conversations with Earl, I'm anticipating three points for caching: the build_info that stores a rendered query and its arguments, the raw results of a query/pager_query combination, and the fully rendered output of a given display. The first mechanism is in place in views.inc, but work to get the query-result and display-rendering cache hooks in place are continuing.

Members fund testing for the Drupal project. Drupal Association Learn more

Comments

eaton’s picture

FileSize
21.15 KB

I've ported this to the DRUPAL-6--3 branch as per merlinofchaos' request. There are a number of hitches, though, that make things tricky. Queries and raw results in particular are very difficult to cache using the new plugin caching mechanism. I've manged to implement rudimentary output caching, but I'm not sure that it will work with any filters or arguments that use contextual information (the currently logged in user, the current date, etc).

Further updates as events warrant.

eaton’s picture

In the light of day, there are still some thorny problems that need figuring out.

  1. Caching of results and output requires first generating the unique aspects of the particular view instance -- essentially, the query that will be run. We can then use that as a key to look up/store the rendered results. Unfortunately, extraction of context (like exposed filter data and arguments) is done deep within the display handler code, and so far I haven't found an easy way to let it get that far, then bail out and use cached output.
  2. Similarly, caching queries is promising but it appears that generating a query results in a LOT of properties on the view and display being modified. IE, a $query object is created, but lots of random flags and props get flipped in the process, and unless they're all examined and set explicitly when the cache is retrieved the view fails.

Those are the two biggies I'm looking at right now. The existing views_plugin_cache_time.inc implementation is very dumb -- it just uses the view name, the display ID, and a hash of $_GET to generate its key. That means it can generate a key very early in the views building process, but it only works for page displays and also ignores defaulted arguments and context-sensitive filters like 'author is current user'.

Work continues.

eaton’s picture

Version: 6.x-2.x-dev » 6.x-3.x-dev
FileSize
17.41 KB

Woo! After chatting with Earl for a while and poking around, I stepped back from trying to cache the queries and focused on output caching. It's definitely a bit easier to work with -- the attached version of the patch includes ONLY output caching implementation, but it reduces a page with the example 'frontpage' view from 107 queries to 34. It currently spits out a 'Cached output used' message via dsm(), and thus will choke if you don't have devel module installed. Further testing to follow....

eaton’s picture

FileSize
17.73 KB

Here's a cleaned up version of the patch that uses vpr() instead of dsm() for better compatability. I still need to address cache invalidation; right now old data is never cleared out. We should probably clear anything related to a given view or display whenever it's validated. But it's definitely ready for testing at this point.

eaton’s picture

FileSize
19.8 KB

Now both result AND rendered output caching are working, including paged views.

The *_time cache implementation allows users to set a lifespan independently for results and rendered output -- depending on the output style, there can be problems caching the output so it's nice to be able to turn that off while still getting the result caching.

Next up will be cache invalidation so the tables get cleared out automatically when saving views, etc. Right now, though, it's a perfectly usable patch and can probably be used for some benchmarking.

eaton’s picture

Status: Needs work » Needs review

...And here's a version that ignores caching when the view is being previewed, flushes the cache for a given view whenever it's saved, and properly clears out old cache data when cache plugins are changed.

I'm doing to say that it's starting to hit 'needs review' rather than 'needs work' -- it might not be perfect but it's now fully functional, and doesn't have any major areas of scary wrongness that I'm aware of.

eaton’s picture

FileSize
20.83 KB

And here's the actual patch.

eaton’s picture

FileSize
23.72 KB
eaton’s picture

Whoops. Looks like the notes got lost for that one -- the latest version of the patch properly handles clearing out the cache when the 'Clear cache' button is pressed on the Performance page, or in Devel module. In addition, it properly stores JS, CSS, and HEAD additions made by output styles that use them. Carousels and AJAX pagers now cache properly. Huzzah!

eaton’s picture

FileSize
29.54 KB

Here's a re-rolled version of the patch for Views 2.x, as well. The patches applied cleanly to that branch with on tweak in $views::execute(), where the logic had been dramatically simplified. It just required wrapping a larger chunk of code in the caching, since it hadn't been pulled into the query class yet.

Tested with exposed filters, ajax paging, and CSS/JS. w00t.

merlinofchaos’s picture

Status: Needs review » Fixed

Committed! Let's see how this place in the real world.

merlinofchaos’s picture

Status: Fixed » Needs review
FileSize
10.69 KB

Updated patch, putting here for testing before I commit:

merlinofchaos’s picture

FileSize
16.62 KB

More refactoring

ChrisBryant’s picture

WOW, nice work here Eaton & Earl! This really rocks and I'm looking forward to test this out on some projects.

nicksanta’s picture

You guys have done an awesome job! I had a little play on some low traffic sites and it seems to be working nicely! Next week I'll be upgrading to views 2.6, a site which has been suffering really severe performance issues due to the sheer volume of content being displayed views on every page. This will hopefully put a stop to that!!!

Great job again!

Scott Reynolds’s picture

I have just committed a caching plugin for Apache Solr Views (http://drupal.org/cvs?commit=224746). One thing I noticed that probably needs to be addressed, is that the standard caching plugins show up as options on the settings page. This is a problem because those caching plugins wont work with the Solr facet blocks.

Be nice to define the plugin with an additional param in hook_views_plugins().

'query type' => 'default',

I looked to find where that check would go and it looks like views_fetch_plugin_names() is the spot just need to figure out how to pass in the query type to it.

Scott Reynolds’s picture

hmm I think I have decided to retract my suggestion :-D. The standard caching plugins do work just fine with Apache Solr Views project, the facet blocks just don't work with the caching but with my cache plugin they do work.

Seems like its just a documentation problem for me, and not and issue for Views.

pavel.karoukin’s picture

Status: Needs review » Patch (to be ported)
FileSize
582 bytes

I am using panels and embed views into it. Noticed following: views cache do not respect atleast "Offset" parameter. Small patch attached to fix this, but I am pretty sure we need more generic code for this issue

Scott Reynolds’s picture

Status: Patch (to be ported) » Needs review

Wrong status

merlinofchaos’s picture

Status: Needs review » Fixed

Hmm. The amount of data that can be tweaked externally is going to make it very difficult to do caching when you do it like this. Besides, Panels has its own caching mechanism, you're much better off using that. Not accepting the patch.

pavel.karoukin’s picture

Case: I am allowing users create their own layout via Panels. And not all of them going to switch on Panels caching. That's why I need caching in more deep level like Views in this case.

merlinofchaos’s picture

The provided cache mechanisms are not expected to be appropriate for every scenario. The scenario you outline, sadly, is one of them. If the Panels caching mechanism doesn't wokr, then you may need to create a customized caching plugin that fits your scenario better.

pavel.karoukin’s picture

Ok. got you point. Can you point me somewhere where i can see good tutorial on creating plugins for views? I can use views_cache as example, but I'd like to place this plugin into my own module. So how I can hookup it?

merlinofchaos’s picture

The system is new enough that there are not yet tutorials. However, nodecomment.module (2.x line) and apache solr both have caching plugins already, so there may be examples you can get from there.

Scott Reynolds’s picture

Apache Solr Views has the caching plugin: http://drupal.org/project/apachesolr_views

Apache Solr module does not. (and ironically there is actually two Apache Solr modules..)

dawehner’s picture

Issue tags: +views 3.x roadmap

adding tag.

te-brian’s picture

Hey, a little late, but I wanted to thank you guys for the work your doing on this. The more we start to use memcached and other performance enhancing techniques the greater we appreciate thoughtful and configurable caching.

sbydrupal’s picture

Do we apply both Eaton's and Merlin's patches or, Merlin's patch is sufficient ? Thanks

dawehner’s picture

i think you need #11 and #14

sbydrupal’s picture

thanks dereine

dawehner’s picture

or you could just use the current version of views(dev :) ), i think they are both included there.

eaton’s picture

or you could just use the current version of views(dev :) ), i think they are both included there.

The caching feature is included in Views 2.6 -- no need to grab the bleeding edge. Just the cutting edge. ;-)

dawehner’s picture

let's say the working version , put thats true cutting edge :p.

See #503774: Cache Time based does not reset

sbydrupal’s picture

Case:

There is a simple "View" that lists node titles.

When the cache is enabled,

tried 2 options:

- query 1 min, html output 1 min
- query 1 min (no html output caching)

When a node is deleted, the view still is showing the Node title in the cache after 1 minute...
(I am not clicking any tabs on the browser to activate the cache after the node deletion,...)
(No query is made on the view during this period of testing cache)

what would be the best way to use views caching ? Node is deleted and it is still showing indefinitely ???

Thanks

merlinofchaos’s picture

sbydrupal: http://drupal.org/node/503774

Let's end this thread. Can we have any further issues with this patch as new issues?

sbydrupal’s picture

thanks Merlin.

Status: Fixed » Closed (fixed)
Issue tags: -views 3.x roadmap

Automatically closed -- issue fixed for 2 weeks with no activity.