While working on #652246: Optimize theme('field') and use it for comment body, it was found that rdf module is slow on pages with many comments. From a 128ms page load without the RDF module, a page with 50 comments will take 147ms (+15%) when the RDF module is enabled. Most of the extra time is spent in rdf_preprocess_comment() which calls url() multiple times (150 in total per page). Ideally we should call url() once per comment, or even maybe cache its result in the comment object so we don't need to call url() at all.

Comments

catch’s picture

Subscribing.

catch’s picture

StatusFileSize
new90.67 KB

Here's some quick benchmarks on 50 comments:

With RDF:

Concurrency Level:      1
Time taken for tests:   200.119 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      142104000 bytes
HTML transferred:       141578000 bytes
Requests per second:    5.00 [#/sec] (mean)
Time per request:       200.119 [ms] (mean)
Time per request:       200.119 [ms] (mean, across all concurrent requests)
Transfer rate:          693.45 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   182  200  15.8    195     322
Waiting:      173  191  15.5    186     313
Total:        182  200  15.8    195     323

Without RDF:

Concurrency Level:      1
Time taken for tests:   176.428 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      119363000 bytes
HTML transferred:       118837000 bytes
Requests per second:    5.67 [#/sec] (mean)
Time per request:       176.428 [ms] (mean)
Time per request:       176.428 [ms] (mean, across all concurrent requests)
Transfer rate:          660.70 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   159  176  15.9    172     348
Waiting:      150  167  15.3    163     338
Total:        159  176  15.9    172     348

That works out about 12% of request time.

xdebug has it as 25% of total time, and additional overhead from other rdf functions too - it's likely some of the disparity is down to xdebug itself, but still that's a lot. Attached kcachegrind screenshot.

scor’s picture

Status: Active » Needs review
StatusFileSize
new1.85 KB

This patch introduces $variables['comment_url'] (in the same fashion as node_url) and saves a call to l() in template_preprocess_comment(). The comment_url value is then used in rdf_preprocess_comment(). In total we're saving 100 calls to url().

- rdf_preprocess_comment() still contains a call to url() for the relation to the node. I was not able to use the same technique as for the comment link because node_url is only present in hook_preprocess_node(), and not in hook_preprocess_comment(). There is also another call to url() for the link to the parent comment.
- rdf_preprocess_username() contains one call to url for the user profile page.
- rdf_rdfa_attributes() has to convert the date in iso8601 and relies on the PHP date() function, so there isn't much we can do here, unless we cache it in the db but I'm not sure if it'd be faster.

Random thoughts. Maybe rdf.module should implement its own uri caching. Could we build the comment_url without using url()? you can't possibly put a comment/2#comment-2 in path.module so we don't gain much by using url(). Am I missing another benefit url() brings us here? (beside the base_path which we can get for much cheaper anyways).

scor’s picture

StatusFileSize
new1.63 KB

quick and dirty implementation of a cache for url(), caching only those urls without query parameter. A quick benchmark shows ~1ms/117ms speed improvement on HEAD with rdf.module disabled, and 3.5ms/136ms (2.5%) with rdf.module enabled.

I measure 74 cache hits with rdf.module off, and 242 with rdf module on.

Status: Needs review » Needs work

The last submitted patch, 687712_url_caching_4.patch, failed testing.

scor’s picture

Status: Needs work » Needs review
Issue tags: +Performance, +RDF
StatusFileSize
new1.72 KB

only non absolute urls with no query parameter are statically cached (keeping the cache overhead as small as possible).

scor’s picture

StatusFileSize
new209.36 KB

I ran some more detailed benchmarks with 50 and 300 comments.

       |  50 w/o RDF | 300 w/0 RDF |  50 w/ RDF | 300 w/ RDF |
|HEAD  |     121.7ms |     592.7ms |    139.6ms |    714.8ms |
|patch |     121.7ms |     580.4ms |    137.2ms |    696.5ms |
|hits  |          52 |         302 |        214 |       1249 |
| %    |          0% |         -2% |      -1.7% |      -2.5% |
(negative % is faster)

The performance gain is 1.7% with rdf enabled on a 50 comments node page. when displaying 300 comments, more than 2% improvement can be measured for both rdf on and off.

What's killing us next is drupal_attributes() and its call to check_plain() which is called by theme_rdf_metadata() and theme_rdf_template_variable_wrapper().
This sanitize on output rule really slows us down here. There is no user input, and we could require for contrib to sanitize their attributes to save all these calls to drupal_attributes/check_plain.

catch’s picture

I opened #522716: drupal_attributes() calls check_plain() on all attributes indiscriminately a while back but moved it to Drupal 8 already due to API change :(. We could try moving it back though if it makes a big difference - but would need to heavily document the change so contrib doesn't open up XSS holes.

scor’s picture

StatusFileSize
new3.21 KB

trying a new approach: caching the values needed to render the RDFa of each comments, that is the results of url() and PHP date(). all cached comment attribute values are retrieved using cache_get_multiple(). (using the 'cache' bin for now for testing purposes). I've measured a 3.1% performance improvement on displaying a node with 50 comments, and 6% on a node with 300 comments.

scor’s picture

StatusFileSize
new5.03 KB

combining #3 and improved #9 allows us to spare one variable in the cache.

HEAD: 112ms
HEAD w/ RDF: 130.5
HEAD w/ RDF w/ patch: 123.4

this patch brings 5% improvement on a node with 50 comments.

catch’s picture

I'd be a bit worried about the cache_set() per comment on misses - that's a lot of inserts. Also a lot of cache entries on a site with multiple comments.

scor’s picture

@catch: do you favor a cache_set() per node instead? I thought of it but found it clunky to store comment data on a per node cache record. Or do you suggest to drop the persistent caching idea altogether? I'm not 100% happy with it either but it's the only way I found to optimize further rdf.module. We're relying on API functions (drupal_attributes, check_plain, theme layer) which are inherently slow, so I'm not sure where to look next :(

scor’s picture

that's a lot of inserts

Forcing caches misses, i.e. 1 cache_get() and 50 cached_set(), gives a page load of 153ms to be compared with 124ms on warm caches with this patch. 153ms is only when you fill the caches on the first page load. Once all these cache inserts have been done, it's only one big select with cache_get_multiple(). Note that the data being cached is not user specific (no timezone in the ISO 8601 date for example) and can be reused for anonymous and logged in users, not matter what their settings are. Each comment cache is [BLOB - 260B], to be multiplied by 50 when loading 50 comments.

scor’s picture

From catch in IRC:
[09:42] scor: OK that doesn't look too bad - I want to see what moshe thinks of this since he's been speaking out against adding more persistent caching recently.

Status: Needs review » Needs work

The last submitted patch, 687712_combined_10.patch, failed testing.

moshe weitzman’s picture

Thanks to scor for the pointer here. Yeah, I'm really displeased with the dozens of persistent caches that have gone into D7. Lets try not to add another here. Can we put what we need into $comment object and then sites will use entity cache module to speed up?

In general, we are in real trouble when we are afraid to call our own API functions. url() and drupal_attributes() are fundamental functions. One call per comment has to be acceptable or else we are headed down a bad path IMO.

catch’s picture

Yeah if we were to have these available from comment_load() then we could cache them in entitycache module (and that's the sort of thing which will make entitycache more of a candidate for core in D8 too). That'd mean no additional inserts with a stale cache, and less places to pull things from.

Would just be these three attributes right?

+    $comment_data['date'] = rdf_rdfa_attributes($comment->rdf_mapping['created'], $comment->created);
+    $comment_data['nid_uri'] = url('node/' . $comment->nid);
+    $comment_data['pid_uri'] = url('comment/' . $comment->pid, array('fragment' => 'comment-' . $comment->pid));

Again, this means you get a bit less luck if you want to mess around with them in the theme, but you can always overwrite in your own preprocess.

scor’s picture

Status: Needs work » Needs review

thanks for the useful directions moshe and catch. will work on adding these 3 variables in $comment. in the meantime, could I get a review of the patch #3? I will keep using this approach which leads to less calls to url(), comment_url will be provided in $comment and used directly in template_preprocess_comment().

moshe weitzman’s picture

#3 looks reasonable to me.

catch’s picture

#3 looks good except I think we need to add a comment there to state why we're not using l().

catch’s picture

Just posted #698992: Rationalize calls to check_plain() in l(), I'm not sure if that will fly, or have as much impact as this patch, but crossposting in case I can persuade scor to run benchmarks on the same data set as used here.

scor’s picture

related thought: we should have a way to pass the output of url() to l() and skip the call to url() in l(). Afaik it's not possible today, that's why we can't use l() in #3 and we have to hard code the HTML code for an A tag...

catch’s picture

We could add an array key to $options without breaking APIs, there's already a boolean for alias or not which is a similar sort of thing. Can't think of what to call it, but worth an issue even if it gets bumped to D8.

scor’s picture

StatusFileSize
new3.98 KB

simpler approach with no persistent caching, but instead we include the date attributes, parent node uri and parent comment uri in the $comment object ($comment->rdf_data), allowing the entity cache module to cache these values.

moshe weitzman’s picture

Shouldn't we introduce $comment->comment_url during comment_load() so any object can use it later in the render process? Doing it in preprocess_comment leaves a page building code to make urls by themselves.

scor’s picture

@moshe: yes, I'm working on a patch for a separate issue, since it is broader than rdf per se (we should probably do it on a per entity basis). see also related full url() support for comment path callbacks cachable with entitycache at #525622-34: 'as link' formatters need a generic way to build the url of an 'entity'

scor’s picture

StatusFileSize
new2.13 KB

Now that some of the aspects of this issue have been generalized and integrated in #699440: Add bundle support to entity_uri callback to remove performance overhead of forum_url_outbound_alter() (regression from D6), it makes this patch even more simple. It does not bring any performance impact in core, but makes it possible for entitycache to cache these values.

moshe weitzman’s picture

Status: Needs review » Reviewed & tested by the community
scor’s picture

StatusFileSize
new1.16 KB

Note that this patch does not depend on #69940: Streamlined eCommerce purchase for single item (site subscription) or any other patch and #29 can be committed any time. I've put together a quick'n dirty persistent cache in comment_load_multiple() for proof of concept only, with benchmarks below:
HEAD 136.5ms
patch 132.7ms
2.8% faster

Similar performance gain should be measured when the entitycache module is functional.

scor’s picture

StatusFileSize
new2.13 KB

rerolling #29

webchick’s picture

Status: Reviewed & tested by the community » Fixed

Yay, performance! This looks pretty straight-forward.

Committed to HEAD.

Status: Fixed » Closed (fixed)
Issue tags: -Performance, -RDF

Automatically closed -- issue fixed for 2 weeks with no activity.