Problem/Motivation

When many queries happen on the same tables the sheer volume of preg_replace() calls can easily save upwards of 100ms-200ms on approx 8000 sql calls by avoiding doing the preg_replace on the same table or field names.

More typical 10-30ms savings on a 600 sql query page.

Proposed resolution

Add a static to each method to memo-ize the preg_match results.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

joelpittet created an issue. See original summary.

joelpittet’s picture

Title: Add static cache to DatabaseConnection::escapeTable & DatabaseConnection::escapeField » Add static cache to DatabaseConnection ::escapeTable(), ::escapeField() and ::escapeAlias()
Status: Active » Needs review
FileSize
1.82 KB
joelpittet’s picture

Issue summary: View changes
joelpittet’s picture

Here's some xhprof with 500 line items in a commerce cart.

Function Name Calls Diff Calls
Diff%
Incl. Wall
Diff
(microsec)
IWall
Diff%
Incl. CPU Diff
(microsec)
ICpu
Diff%
Incl.
MemUse
Diff
(bytes)
IMemUse
Diff%
Incl.
PeakMemUse
Diff
(bytes)
IPeakMemUse
Diff%
Current Function
preg_replace -72,058 -100.0% -107,247 -3.2% -163,151 -4.4% -7,774,616 -52.1% -298,744 -2.0%
Exclusive Metrics Diff for Current Function -107,247 -100.0% -163,151 -100.0% -7,774,616 -100.0% -298,744 -100.0%
Parent functions
DatabaseConnection::escapeField -25,471 -35.3% -60,451 -56.4% -79,311 -48.6% -2,970,288 -38.2% -96,080 -32.2%
DatabaseConnection::escapeTable -26,890 -37.3% -49,682 -46.3% -73,758 -45.2% -2,500,472 -32.2% -187,016 -62.6%
DatabaseConnection::escapeAlias -19,698 -27.3% -31,088 -29.0% -45,644 -28.0% -2,323,520 -29.9% -16,336 -5.5%
filter_xss 0 0.0% 10,205 9.5% 12,013 7.4% 10,960 0.1% -608 -0.2%
joelpittet’s picture

Issue summary: View changes
mikeytown2’s picture

Looking over some cachegrind's of mine and I see right around 10ms for 3k calls for each of the escape methods (~30ms total). Is this similar to what you're experiencing?

joelpittet’s picture

Yes around 30ms

joelpittet’s picture

Issue summary: View changes

yeah last test I just did on a different site was only 9ms savings. Not a really big win here.

joelpittet’s picture

Issue summary: View changes

That table is messy in the IS.

It seems to keep coming up in my profiling even though it's not huge it seems pretty straight forward.

stefan.r’s picture

Version: 7.x-dev » 8.2.x-dev
Assigned: Unassigned » Fabianx

Shouldn't this be against 8.x first?

stefan.r’s picture

Assigned: Fabianx » Unassigned
daffie’s picture

This sounds like a very good idea to me.

dawehner’s picture

+++ b/includes/database/database.inc
@@ -919,11 +919,15 @@ abstract class DatabaseConnection extends PDO {
+    static $tables = array();

@@ -933,11 +937,15 @@ abstract class DatabaseConnection extends PDO {
+    static $fields = array();

@@ -948,11 +956,15 @@ abstract class DatabaseConnection extends PDO {
+    static $fields = array();

This doesn't have to be a static one. It would be totally enough to have a property on the object, caching the result.

daffie’s picture

I think that input from Crell would be wise.

joelpittet’s picture

@dawehner naming is a big hard;) Good suggestion though. We'd have to have a new property for each function because though similar the preg_match() regex is subtly different for aliases. I think I'd rather have it in context as a method static, is that ok?

joelpittet’s picture

The last submitted patch, 15: add_static_cache_to-2672088-15.patch, failed testing.

daffie’s picture

Status: Needs review » Needs work
Issue tags: -Needs subsystem maintainer review

The patch look good, but I have two remarks:

  1. +++ b/core/lib/Drupal/Core/Database/Connection.php
    @@ -925,7 +925,11 @@ public function schema() {
    +    static $databases = [];
    
    @@ -942,7 +946,11 @@ public function escapeDatabase($database) {
    +    static $tables = [];
    
    @@ -959,7 +967,11 @@ public function escapeTable($table) {
    +    static $fields = [];
    
    @@ -977,7 +989,11 @@ public function escapeField($field) {
    +    static $fields = [];
    

    @dawehner stated in comment #13 that the variables do not have to be static ones. Can you explain why you keep them static.

  2. +++ b/core/lib/Drupal/Core/Database/Connection.php
    @@ -977,7 +989,11 @@ public function escapeField($field) {
       public function escapeAlias($field) {
    -    return preg_replace('/[^A-Za-z0-9_]+/', '', $field);
    +    static $fields = [];
    +    if (!isset($fields[$field])) {
    +      $fields[$field] = preg_replace('/[^A-Za-z0-9_]+/', '', $field);
    +    }
    +    return $fields[$field];
    

    Can we rename the variable fields to aliases, because we already have a variable fields and its is holding an array of aliases.

joelpittet’s picture

Status: Needs work » Needs review

@daffie thanks for the review. Regarding point 1). I tried to answer that in #15

We'd have to have a new property for each function because though similar the preg_match() regex is subtly different for aliases. I think I'd rather have it in context as a method static, is that ok?

2) Probably out of scope and I thought the same thing but left it to avoid unnecessary changes. Though not against it.

daffie’s picture

@joelpittet: If you change the variable name $fields to $aliases in the method escapeAlias(), the patch will for me be good enough for RTBC. Thank you for your explanation in the previous comment.

joelpittet’s picture

Ok here's the variable name change from $fields to $aliases

daffie’s picture

Title: Add static cache to DatabaseConnection ::escapeTable(), ::escapeField() and ::escapeAlias() » Add static cache to DatabaseConnection ::escapeDatabase(), ::escapeTable(), ::escapeField() and ::escapeAlias()
Status: Needs review » Reviewed & tested by the community

I all looks good to me. But I am not 100% sure this is the right solution and I would like to ask one of our framework managers to be the committer for this issue.

xjm’s picture

Thanks @daffie.

Adding the rc deadline tag because I think this improvement could go into 8.2.x during its beta but not RC, if we do choose to commit it with its current implementation. (After the beta it can go straight into 8.3.x of course.)

catch’s picture

Status: Reviewed & tested by the community » Needs work

Since these are normal methods, we could use a regular protected property on the class - no need for an actual static here. I see above this was avoided because it would be three properties, that doesn't seem like a reason not to do it?

It would also be good to see xhprof profiling with a standard install - I imagine we'll see a small improvement because we hit the same cache tables often.

joelpittet’s picture

@catch one property per method(4) or one per regex difference(2). See why I did statics in #19

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

catch’s picture

@joelpittet one per regex difference seems best - also if a database/table/column happened to be named the same (albeit unlikely) we could re-use the sanitization.

I see the reasoning for the static, but the disadvantage outweighs the advantage for me.

joelpittet’s picture

Status: Needs work » Needs review
FileSize
2.22 KB

Thanks @catch, here's a first crack at naming those 2 member properties.

escapedNames() and escapedAliases() for the two regex'd patterns. Slight benefit that it catches names that are the same between table, database and field.

catch’s picture

Status: Needs review » Reviewed & tested by the community

Thanks! That looks a lot better. Moving back to RTBC.

  • xjm committed 4295f59 on 8.3.x
    Issue #2672088 by joelpittet, daffie, catch, dawehner, mikeytown2: Add...
xjm’s picture

Status: Reviewed & tested by the community » Fixed

A nice little performance improvement. I was going to assign it to @catch for signoff but he RTBCed the issue, so I've committed 4295f59 and pushed to 8.3.x. Thanks!

stefan.r’s picture

Version: 8.3.x-dev » 7.x-dev
Status: Fixed » Needs review
FileSize
2.48 KB

And now we can add this to 7.x as well :)

Status: Needs review » Needs work

The last submitted patch, 32: 2672088-32.patch, failed testing.

stefan.r’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 32: 2672088-32.patch, failed testing.

stefan.r’s picture

Status: Needs work » Needs review
FileSize
2.49 KB
joelpittet’s picture

Status: Needs review » Reviewed & tested by the community

Thanks RTBC #36, checked their is no database name escaping and the rest is identical to D8. The previous failure was PHP 5.4 short array syntax.

stefan.r’s picture

Assigned: Unassigned » Fabianx
Fabianx’s picture

Issue tags: +Pending Drupal 7 commit

Marking for commit.

Very nice change.

  • stefan.r committed df7b3fd on 7.x
    Issue #2672088 by joelpittet, stefan.r, daffie, catch, mikeytown2,...
stefan.r’s picture

Assigned: Fabianx » Unassigned
Status: Reviewed & tested by the community » Fixed
Issue tags: -rc deadline, -Pending Drupal 7 commit +Dublin2016

Re-reviewed with Fabianx, and committed and pushed to 7.x, thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.