Hello

I am almost finishing the site and, since Open Atrium implements so many things, in comparison with a standard Drupal installation, is a bit slow. So I wonder if you could give to me some tips to optimize the cache or whatever you consider. Thanks very much.

CommentFileSizeAuthor
#7 event_without_cache.png92.98 KBcandelas
#7 event_with_cache.png98.08 KBcandelas
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

cgove’s picture

I've spent about a week trying different cache solutions for my OA install.
Here is what I ended up with:

  • Redis (memory cache)
  • Boost (anonymous user file cache set to 1 day)
  • Expire (expire and re-crawl any updated pages)
  • Expire Panels (add panels to the expire module)
  • Panels Hash Cache (set and forget panel caching)
  • Entity cache (puts entities in memory cache)
  • Views Content Cache (Cache views and expire if contents change)
  • Session Cache (to make Boost work with modules that set anonymous user cookies)
  • Custom css aggregate (to make aggregated CSS work with IE, AdvAgg also does this but was breaking Panelizer for me)
  • DB Maintinence (to keep sql tables optimized)
  • Bootstrap Optimizer (Improves bootstrap load speed by preventing broken module checks)
  • XML Sitemap (I run a nightly wget/curl job to visit every page nightly and warm the Boost cache)
  • Also disable modules that do logging. I also switched from DBlog to syslog.

I tried AuthCache, but I couldn't get it to cache in a way that wouldn't confuse editors. If anyone has working experience with this, I'd love to hear about their success or failure.

Other good resources:

candelas’s picture

Thanks a lot!!!!!!!!!!!!!!!!
And in the views and panel panes, do you use the option to cache?

cgove’s picture

Yes, Views Content Cache is enabled per view. I rely on Panels Hash Cache to take care of things automatically in panels. If anyone has a better recommendation, I'd love to hear. I need to avoid any unexpected behavior for content editors using Panelizer.

candelas’s picture

Thanks. Next week, at the end, I have to start with that and anything that I learn I will put here :)

hefox’s picture

Status: Active » Fixed

Thanks cgove

candelas’s picture

Status: Fixed » Active

Hello

I didn't forget about this but I had to make other things, so I reopen to try to share the experience with cache. At the moment I am working with views cache.

To be able to use Views Content Cache I had to apply this patch in #9 #2054811: Organic Groups integration doesn't work in Drupal 7 because it was using a Organics Group function deprecated.

@cgove the module is very good but I can only configure content and not other entities... like users or messages. How did you do?

Also, I found https://www.drupal.org/project/qa_checklist that is similar to https://www.drupal.org/project/pasc , but I think that it is more complete, just in case @cgove you are interested to test it :)

I go to tests and configure the other modules and then I will report.

candelas’s picture

Also, I can't cache events because the "Add to calendar" widget gets broken as you can see in the images that I attach.
The html should be

<a title="" class="addthisevent-drop" href="xxx/event/samplig-tropical-sponges-and-polychaetes" id="atedrop1">Add to Calendar<span class="_url" style="display: none; ">xxx/event/samplig-tropical-sponges-and-polychaetes</span>

and is

<a title="Add to Calendar" class="addthisevent" href="xxx/event/samplig-tropical-sponges-and-polychaetes">Add to Calendar<span class="_url">xxx/event/samplig-tropical-sponges-and-polychaetes</span>

So I left events without caching until I find a solution. To reproduce:

  • make events view cache with content type Event and save.
  • Go to an event page and load twice. The first it is not cached and it works perfect, but the second it is cached and it doesn't get the right html (I imagine that it is because JavaScript no being applied).
candelas’s picture

Hello @cgove

I am configuring AuthCache. When you say

I tried AuthCache, but I couldn't get it to cache in a way that wouldn't confuse editors.

What do you mean? Thanks :)

cgove’s picture

Using AuthCache, after editing a panel layout with the IPE, the changes wouldn't be shown. I couldn't get it to clear the cache with panel modifications.

cgove’s picture

For users, I just use time based caching. I simply instruct users that changes to their information won't show up on the staff directory until the next day.

cgove’s picture

Ok, found an even cooler module than Views Content Cache. It even works for users!

https://www.drupal.org/project/autocache

candelas’s picture

Thanks for sharing. I have seen it, but since there are so many modules I didn't try it.
I've just finished with the cache and discovered that Hash Cache module doesn't work with Panelizer. If I use it, the page doesn't get refreshed (I tried Context and Full URL (including query strings)).

I also tried AuthCache but you need to update to Flags 7.x-3.6 #2474195: Update to Flag 7.x-3.6 and it doesn't support Organic Groups #2474327: Organic Groups / Open Atrium. The maintainer said:

At the moment I cannot give any advice on Authcache in combination with Organic Groups and/or Open Atrium. If there is a strong demand for that combination, then I'm willing to explore the options we have. However, because all this projects are rather complex, I will do this only if I get the opportunity to work together with experienced developers in that area (either maintainers or active contributors for one of the aforementioned projects).

So I started asking the feature in OG #2474473: Add Authcache support

Authcache doesn't support neither Boost. I discovered that with Redis you can cache also the full pages, so I was not using Boost.
$conf['cache_class_cache_page'] = 'Redis_Cache';

Now I have to start with the crossbrowser compatibility (Explorer8 for the public part), but I am very interested in your experience with Autocache :)

candelas’s picture

I've just discoverd that Panelizer doesn't work with Expire/Hash Cache if the pane is a view. With the others work.
Difficult to get Open Atrium Cache. But I am happy that I learned a lot about cache :)

cgove’s picture

Yea Authcache really takes over for everything authenticated and anonymous cache. Unfortunately the support for things OA relies on isn't really there yet. So in place of that I use Boost to cache anonymous content and these various modules to cache authenticated content. Autocache is very nice mostly because it requires no configuration whatsoever. I force all of my views to use it with views_cache_bully . To make it use autocache instead of time-based, I had to manually change line 79 of views_cache_bully.module to 'type' => 'autocache', and delete lines 80 and 81. I found no equivalent module for panels, so I have to change all of those in order to use it.

cgove’s picture

I've opened another issue for AddThisEvent breaking on a cached Event Detail view. https://www.drupal.org/node/2474963

candelas’s picture

Thanks for the tips. I mostly use Views Content Cache and I see that Views Content Cache has many patches that are waiting review. I go to test my OA view by view to see if there are any that are needed and report.

candelas’s picture

Adding related issues

candelas’s picture

mpotter’s picture

I should probably chime in here.

Caching is a very complex subject. People reading this thread should not see it as a "cure all" solution. Multiple cache layers can be very tricky to configure and can be very site specific.

Typically when I build Enterprise-level sites the most important caching layer I use is Memcache (or Redis, your choice, but I typically use Memcache). Some sort of memory cache like this is *critical* for ANY Drupal site where you have lots of logged-in users (e.g. Open Atrium).

Second caching layer is Varnish for anonymous users. I prefer Varnish over stuff like Boost.

Definitely disable any "PoormansCron" task in the Drupal cron system and also disable DBLog. Those are standard procedures for any Drupal prod site.

Open Atrium isn't any different from any other production-level Drupal 7 site. So there are a lot of resources available on the net for improving performance. With just the above items I mentioned we have a production site with 16,000 spaces, hundreds of users, with page load times around 2 seconds for both logged-in and anonymous users.

Obviously there are related topics on tuning your MySQL database, etc.

I rarely get deep into content-level caching, such as Views or Panels caching. That kind of caching can easily cause more trouble than it's worth if you are not careful with it. And those should definitely only be done once you have the basics listed above.

Be careful with *any* module that you add that isn't already part of Open Atrium. Those additional modules can cause obscure problems, and potentially expose content to the wrong users if they cache in the wrong way.

If you *do* end up with performance issues, it's very important to use proper debugging and profiling tools such as xhprof to determine the exact cause of the performance issue. We spend a lot of time with our clients on this and have learned that without REAL DATA about exactly what is causing the performance issue you can easily waste a lot of time and resources on the wrong problem. Just throwing more caching layers at the site is not usually the solution.

candelas’s picture

@mpotter thanks for your time and knowledge :)

I am learning about caching as it can be seen and for me it is very difficult.
So for what I understand, with Varnish and memcache/redis is enough, isn't it? Besides what you said about cron and DBLog.

I rarely get deep into content-level caching, such as Views or Panels caching. That kind of caching can easily cause more trouble than it's worth if you are not careful with it.

Indeed, I have spent days because of that. Like with the following error that made me spent hours this morning...

@gcove the module https://www.drupal.org/project/autocache , when you add/edit a content (no in other forms) and you have redis enabled, gives this error

PDOException: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '= 'user')' at line 2: SELECT DISTINCT ogm.entity_type AS entity_type, ogm.etid AS entity_id, users.uid AS revision_id, users.uid AS bundle FROM {og_membership} ogm INNER JOIN {users} users ON users.uid = ogm.etid WHERE (ogm.gid = :db_condition_placeholder_0) AND (ogm.entity_type = :db_condition_placeholder_1) AND (users. = :db_condition_placeholder_2) ; Array ( [:db_condition_placeholder_0] => 190 [:db_condition_placeholder_1] => user [:db_condition_placeholder_2] => user ) in field_sql_storage_field_storage_query() (line 632 of /xxx/modules/field/modules/field_sql_storage/field_sql_storage.module).

If you disable redis, then it works.

I go to install Varnish in my local server and ask the site admin in the center were I am building the OA site to do it too (I hope...). Thanks again for the help @mpotter.

mpotter’s picture

Keep in mind that Varnish only helps with anonymous user traffic.

Haven't seen the errors you mentioned in Memcache, so might try that instead of Redis if you can. There are instructions in the memcache Drupal module (you don't need to enable the Drupal module, it just provides an inc file that helps with config).

cgove’s picture

Just for the record, I'm not seeing that error with Autocache and Redis.

candelas’s picture

Thanks. First I go to install Varnish and see if I can get the admin in the server to install it (the server it is not use specially for sites). I have disable the module (I go to report there) Autocache and in next site I will try memcache, because I am late...

A last doubt (at the moment), what is it the configuration that I have to use for caching authenticated users? Is enough with memcache/redis? I have tried AuthCache, which looks very promising, but when I started to configure it, I had to make updates as in the Flag module or some errors and I decided that it was too much for this time.

Thanks

candelas’s picture

@cgove did you try to create/edit a node and save? If so, can you put your Redis configuration, please? :)
Edit: What version are you using in Redis and Autocache?

cgove’s picture

Yea, Varnish is a little better performance-wise than Boost, but the setup is much more difficult, especially when sharing the server with sites that don't use it.

cgove’s picture

On the topic of caching Views, we basically have to. Our imported Google calendar makes our calendar section take a full minute to load uncached. It is basically loading every event from the entire calendar, not just the month view. It would be nice if it dynamically loaded the calendar months, but that mode of FullCalendar module doesn't work when it is in a Panel. The related issue: https://www.drupal.org/node/2454337

cgove’s picture

I'm using recommended release 1.4 of Autocache and 2.8.17 of Redis server. Here is my config:

if (file_exists('/var/run/redis/redis-server.pid')) {
  $redis_up = TRUE;
}

if (!empty($redis_up)) {
	$conf['redis_client_interface'] = 'Predis';
	$conf['redis_client_host']      = '127.0.0.1';
	$conf['redis_client_port'] 		= '6379';
	$conf['lock_inc']               = 'sites/all/modules/redis/redis.lock.inc';
	$conf['path_inc']               = 'sites/all/modules/redis/redis.path.inc';
	$conf['cache_backends'][]       = 'sites/all/modules/redis/redis.autoload.inc';
	//$conf['cache_backends'][] 		= 'sites/all/modules/authcache/authcache.cache.inc';
	//$conf['cache_backends'][] 		= 'sites/all/modules/authcache/modules/authcache_builtin/authcache_builtin.cache.inc';
	//$conf['authcache_p13n_frontcontroller_path'] = 'authcache.php';
	$conf['cache_default_class']    = 'Redis_Cache';
	
	$conf['cache_prefix'] = array('default' => 'cms_');
	
	// Do not use Redis for cache_form (no performance difference).
	$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';
	// Use Redis for Drupal locks (semaphore).
	$conf['lock_inc'] = 'sites/all/modules/redis/redis.lock.inc';

	// Higher performance for smaller page counts.
	// High performance - no hook_boot(), no hook_exit(), ignores Drupal IP blacklists.
    $conf['page_cache_without_database'] = TRUE;
    $conf['page_cache_invoke_hooks'] = FALSE;
    // Explicitly set page_cache_maximum_age as database won't be available.
    $conf['page_cache_maximum_age'] = 86400;

	// Higher performance for larger page counts.
	//$conf['cache_class_cache_page'] = 'DrupalDatabaseCache';
}
candelas’s picture

@cgove, thanks. I have a similar configuration but my redis server is older, though in the server 2.8.4. I have seen a bug in my version with fields, so I will actualize my local server for next site and also install authcache to get faster pages for registered user.

Thanks a lot for all your help and sharing :)

candelas’s picture

@cgove, one last question: you say

Custom css aggregate (to make aggregated CSS work with IE, AdvAgg also does this but was breaking Panelizer for me)

Could you explain a little more what gets broken, please? Do you mean the pages tipus event, discussion etc? I go to make now the aggregation part to make the site public part. Thanks :)

candelas’s picture

@cgove in my case, AdvAgg was not working because I have a folder in drupal root with .htaccess deny all. When I removed, it started to work. Now I go to see if I can exclude the folder from the module search to find css and js.

katrien_w’s picture

Following this thread with a lot of interest. Have memcache and Varnish running but need something to speed up logged in users. I will try Autocache, thanks for sharing this.

dpoletto’s picture

@cgove: could you elaborate more about the "DB Maintenance (to keep sql tables optimized)" part?

Despite now I'm not directly interested on any web caching mechanism (It's a little bit too complex for me and I still don't know if the effort is needed at all in a little deployment) I would love to explore the layer over which the OA site is running...the (LAMP) system level...so will be nice to start with the "M" part: any DB Maintenance/Optimization practice (Assuming that DB Logging and Statistics are disabled for performance) to perform on the Server on a scheduled basis.

Thanks, Davide.

cgove’s picture

dpoletto’s picture

@cgove: Thanks! Davide.

Side Note: OptimizeDB (optimizedb) looks interesting too (it doesn't require the Xautoload module like DB Maintenance module).

drvdt’s picture

Thanks Mr. mpotter
VPS/Server: Perfect solution for Performance of Open Atrium (And other Drupal sites): Varnish + APC + Memcache.
Should not try others. And I think should be closed for newbie easier to choose.
Should have:
1. OptimizeDB for clear form_cache (very big after some weeks) and for optimize tables

cgove’s picture

I definitely agree that smaller subset Boost, Opcache, Redis (which is equivalent to Varnish/APC/Memcache) is the most essential part. The other caching options are not as absolutely necessary and take a lot more tweaking to ensure security and fresh content when it matters.

Switching from dblog to syslog and switching cron to run as a system job instead of on page load are huge easy wins for performance too.

Argus’s picture

Created a placeholder document page about caching. This needs work and could use some input.

@cgove: You provided most detailed info, could you summarize it?

Argus’s picture

I'm getting the same kind of error error as mentioned in #20 when editing and saving a node with the autocache module enabled.

PDOException: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '= 'user') AND(( ( EXISTS (SELECT na.nid AS nid FROM node_access na WHERE (( (n' at line 2: SELECT DISTINCT ogm.entity_type AS entity_type, ogm.etid AS entity_id, users.uid AS revision_id, users.uid AS bundle FROM {og_membership} ogm INNER JOIN {users} users ON users.uid = ogm.etid WHERE (ogm.gid = :db_condition_placeholder_0) AND (ogm.entity_type = :db_condition_placeholder_1) AND (users. = :db_condition_placeholder_2) AND(( ( EXISTS (SELECT na.nid AS nid FROM {node_access} na WHERE (( (na.gid IN (:db_condition_placeholder_3)) AND (na.realm = :db_condition_placeholder_4) )OR( (na.gid IN (:db_condition_placeholder_5)) AND (na.realm = :db_condition_placeholder_6) )OR( (na.gid IN (:db_condition_placeholder_7, :db_condition_placeholder_8, :db_condition_placeholder_9, :db_condition_placeholder_10, :db_condition_placeholder_11, :db_condition_placeholder_12, :db_condition_placeholder_13, :db_condition_placeholder_14)) AND (na.realm = :db_condition_placeholder_15) )OR( (na.gid IN (:db_condition_placeholder_16, :db_condition_placeholder_17, :db_condition_placeholder_18)) AND (na.realm = :db_condition_placeholder_19) ))AND (na.grant_view >= :db_condition_placeholder_20) AND (ogm.etid = na.nid) )) AND (ogm.entity_type = :db_condition_placeholder_21) )OR (ogm.entity_type <> :db_condition_placeholder_22) ); Array ( [:db_condition_placeholder_0] => 17 [:db_condition_placeholder_1] => user [:db_condition_placeholder_2] => user [:db_condition_placeholder_3] => 0 [:db_condition_placeholder_4] => all [:db_condition_placeholder_5] => 7 [:db_condition_placeholder_6] => oa_unpublished [:db_condition_placeholder_7] => 5 [:db_condition_placeholder_8] => 6 [:db_condition_placeholder_9] => 13 [:db_condition_placeholder_10] => 25 [:db_condition_placeholder_11] => 26 [:db_condition_placeholder_12] => 27 [:db_condition_placeholder_13] => 1100 [:db_condition_placeholder_14] => 1105 [:db_condition_placeholder_15] => oa_access [:db_condition_placeholder_16] => 24 [:db_condition_placeholder_17] => 12 [:db_condition_placeholder_18] => 4 [:db_condition_placeholder_19] => og_access:node [:db_condition_placeholder_20] => 1 [:db_condition_placeholder_21] => node [:db_condition_placeholder_22] => node ) in field_sql_storage_field_storage_query() (regel 652 van /srv/bindings/66369bfc5bb94d639b9e034c7d259835/code/modules/field/modules/field_sql_storage/field_sql_storage.module).

delacosta456’s picture

hi
Old and nice thread but would like to share my experience as an alternative when having less ressource either on local or online shared hosting.

Try (an alternative to memcahe, Varnish, etc..) ==> Filecache (the dev version that worked with enitycache) + Entitycahe modules with some little direction below

- on local (for single computer or on-prmise hosting)

*in setting.php configure cahe_form and cache_update to use DrupalDatabasecahe and Filecahe storage as the default.
*in setting.php change the location of where cache should me store : Different Partition of you drive or Different hard drive
*a more performance could be possible if using an SSD partition or separate SSD Disk
*configure correctly your cron with ulfimate_cron (module) + system job's
*AuthCache integrate well with Filecache module ( so if you configure it correctly, more performace)

- on shared hosting
same configuration as above except :
*mapping cache location to a directory outside drupal
**configure correctly your cron with ulfimate_cron (module) + hosting cron job's

People may try and give feedback