Problem/Motivation
Drupal.org has gotten quite slow over the time. Especially the D7 version. The more comments on a page, the slower the page load.
Proposed resolution
Comments can be cached near perfectly with render caching (https://www.drupal.org/project/render_cache) and a hash based approach:
- They need the modified timestamp of the comment
- They need the modified timestamp of the author
- They need a flag for is_viewer / is_author
DRUPAL_CACHE_PER_ROLE as base cache and thats it as the Cache ID is calculated at run-time.
This means that for normal issues the second time for most users all comments are cached.
Due to the hash changing, no expensive cache invalidation is needed.
Remaining tasks
- Check if entity module 7.x-1.5 is installed
- Check that instead of comment_view_multiple entity_view() can be used instead (that automatically enables render_cache module)
- Implement render_cache module on a sandbox / VM
- Make test suite pass
- Have a faster d.org :)
Deployment - window scheduled for Thursday, 24 July 2014 2-4pm PDT (21:00 UTC)
- Enable entity_modified 7.x-1.x-dev
- Enable render_cache 7.x-1.0
- Update drupalorg code
- cc all
Comment | File | Size | Author |
---|---|---|---|
#17 | performance_implement-2299547-17.patch | 2.81 KB | Fabianx |
Comments
Comment #1
drummI generally like this idea.
We have dev sites - http://drupal.org/node/1018084. These are good for regression testing, but are nowhere near representative of production performance. That's okay, we mostly want to check for regressions. At the moment, this is mostly manual testing on dev. We have BDD tests, https://www.drupal.org/project/doobie, but they are not easy to run against dev yet. And even BDD can miss caching bugs. Dev doesn't have memcache enabled, but we could spin up a memcache server for testing.
What we will want in this issue is deployment instructions. What versions of projects to install, and configuration changes.
Comment #2
Fabianx CreditAttribution: Fabianx commentedSure, can I get read-only access to any dev instance plus XHProf to check which code path is used to show comments.
If its just the normal comment_view_multiple the tasks will be:
- Install and enabled entity, render_cache (once new release is out - due today), entity_modified modules
- Also enable render_cache_comments
- Add to a custom module like dorg_performance something like:
And that should theoretically just work without knowing much about the internals.
I'll probably release a stable version of render_cache first (has been in production since almost a year now) and apply the RTBC patch for render_cache_comments (which I wrote), but this can be used to start playing around with this.
Memcache is not absolutely needed as cache_render bin is created by render_cache module, but DB could get full quite soon as there is no LRU obviously on DB Cache Backend ...
Though a drush cc all fixes that, its probably not a problem.
Also I suggest to keep the variables to disable at run-time and for dev-instances.
Comment #3
drummMoving to the project with the site's custom code. I think we should patch this into drupalorg itself, I'm not a fan of lots of custom modules. And the
variable_get()
kill switch is more convenient than enabling/disabling a module.Comment #4
drummLooks like you already have an account on devwww. Can you confirm ssh Fabianx@devwww.drupal.org works?
Comment #5
Fabianx CreditAttribution: Fabianx commentedYes, I I can still login to devwww.
Comment #6
Fabianx CreditAttribution: Fabianx commentedThe 1.0 version of render_cache and 1.1 version of entity_modified have been released.
I will now make a patch to drupalorg module and then all we need is testing this and this should be good to go ...
Comment #7
Fabianx CreditAttribution: Fabianx commentedAdding an obviously untested patch for drupalorg module.
Comment #8
Fabianx CreditAttribution: Fabianx commentedAnd a new patch, which also has update hooks and _dependencies() in the .install (if I got this right). Also adds to .info file.
Comment #9
drummThe drupalorg.install portion isn't needed. We'll enable the modules manually either ahead of, or after, the code deployment.
drupalorg_update_dependencies()
is effectively dead code since the D7 upgrade is done, and we aren't running a few hundredhook_update_N()
s.Comment #10
Fabianx CreditAttribution: Fabianx commentedOkay, cool. Thanks for the explanation.
Keep the change to the .info though?
Comment #11
drummYep, .info is good.
Comment #12
Fabianx CreditAttribution: Fabianx commentedAnd a new patch with suggested changes that I am gonna try on my dev instance now.
Comment #13
Fabianx CreditAttribution: Fabianx commentedTL;DR: 30% performance improvement on a page with just 44 comments ...
I have to do two changes to entity_modified and drupalorg to skip a rather extensive, but unnecessary user_load(), but besides that this is finished it seems:
Full Page
Just the callback showing the comments
Interdiff
Likely to get the new release for entity_modified out today with that little change ...
Comment #14
killes@www.drop.org CreditAttribution: killes@www.drop.org commentedMaybe we could re-enable the mailing of complete comments in notifications after this hast been deployed...
Comment #15
Fabianx CreditAttribution: Fabianx commentedCNW, I forgot the new_first and new flags that are set within classes and different markup.
Comment #16
Fabianx CreditAttribution: Fabianx commentedOkay, implemented is_new, first_new, in_preview and divs and divs_final properties.
I verified NEW functionality works perfectly now again.
Comment #17
Fabianx CreditAttribution: Fabianx commentedWithout whitespace change...
Comment #18
webchickThis should really be at least major.
Loading #2183231: Make ContentEntityDatabaseStorage generate static database schemas for content entities (486 comments) takes around 1.2 minutes on my machine (via Safari's network inspector). A 30% speed increase from that sounds like a great start!
Comment #19
Fabianx CreditAttribution: Fabianx commentedOn special request by webchick two screenshots of issues with 220 and 146 comments:
node/15365 - threaded; handbook
74% saved time
node/2280195 - issue
50% saved time
Comment #20
Fabianx CreditAttribution: Fabianx commentedI did some more testing and I still think this is ready. After more investigation I think I'll be able to remove the @todos ...
@drumm: Any chance you'll be able to review that?
I will open a new issue to track the new render_cache-7.x-2.x stuff that should theoretically allow advanced node, block and page caching - similar to Drupal 8 render_cache. This is a huge WIP still, so this will continue to serve as the stable version here, but putting that here, because it means my sandbox could be unstable and is not usable for testing the change once I deploy the experimental features test wise.
Comment #21
drummI've done a bit of preliminary review while on staff retreat and vacation, but not enough to really comment on substantively quite yet. Everything does look great as far as I can tell, so we went ahead and scheduled a deployment window, since we announce those ahead of time now.
I started a quick deployment section of the issue summary.
Comment #22
drummComment #23
drummComment #24
drummLet's disable that. There isn't much of a point in caching previews, especially if it alters the hash.
Can you explain why we might want to disable for threaded comments? I can't think of any drawbacks offhand. I expect
divs
to stay the same for any particular comment.Comment #25
Fabianx CreditAttribution: Fabianx commented#24: Thanks for the review!
- Yes, I will disable comment caching if the preview flag is set.
- I think for nested discussions the divs can change some, but on the other hand it is a valid variation and the speed ups are very nice there, too. I guess we will just need to monitor memcache cache growth to see the whole impact of this feature. I'll add a kill switch for nested divs - just in case.
Comment #26
drummComment #28
drummI went ahead and committed #17, with completely removing the
in_preview
-related code. In my testing, I didn't see caching being hit at all on comment preview.A killswitch for nested divs isn't a blocker to deployment, so proceeding with getting this on staging.
Comment #29
drummThis is running on https://git7site.devdrupal.org/ for testing.
Comment #30
drummNow deployed on Drupal.org. https://www.drupal.org/node/2183231 loads in 10-20 seconds for me now.
I'm tempted to mark this as fixed, I'm not sure we need a killswitch for nested divs.
Comment #31
Fabianx CreditAttribution: Fabianx commentedYes, this is fixed! Thanks so much!
The preview is never needed to be added to the hash as its displayed via comment_view() instead of entity_view(), which means that it is not run through the caching function.
Comment #32
tim.plunkettOpened #2310011: Comment render caching breaks comments with patches attached
Comment #35
Wim LeersOpened another regression: #2334899: Comment render caching should happen per timezone.