Performance issue with cache expiration

Maybe it would be worth having a whitelist of node types that can be excluded from expiring?

@Spleshka I'm using this with a commerce setup. Using this a a guide for AuthCache:
https://www.drupal.org/node/2158615

I've got Memcache Storage + Redis (For when cache items which grow > 1MB) and DB storage for a few bins like cache_form.

It's PHP 5.5.9 with Zend OPcache v7.0.3 enabled and configured.

I'll have to get back to you on the node_update and node_delete as I was digging pretty deep with microtime calls in core to get at those specific times and sorted the results.

Also the thing I was testing the node_insert with was not a commerce product or page, it was a retail location node with address and geo fields on it. So one of the less complicated node types.

Here is the hook_node_update and hook_node_delete added to the issue summary.

Here's what I was using for a performance test with a breakpoint condition on the return statement of:

$hook == 'node_update' || $hook == 'node_insert' || $hook == 'node_delete'

So that we aren't seeing all the $hooks that get called.

Digging a bit further I have the following in ExpireNode::expire()

Which is just hunking different pieces out to see where the bottle neck is at:

Further down the rabbit hole, inside ExpireAPI::executeExpiration()

Which brings me into one call to authcache_builtin_expire_v2_expire_cache().

I think this may be an authcache issue and it boils down to number of roles I have and how many cache key variations occur. In my case there are 512 role choices because it takes every combination which produces enum_user_keys at 1024.

Then it times 4 keys to expire which does it 4K times... I guess for now I'll just turn off caching on a bunch of the roles I have.

Changing the number of roles from 11 to 5 had a huge differences because of the exponential growth of the choices.

2.2s down to 0.05s!

Maybe we should move this over to authcache module as I don't think expire can do much here?

Though I will likely still be looking at ways to temporarily disable expire module while some of my feed imports are running to avoid all the unnecessary expire calls. Maybe expire module could determine that it's in a batch process and collect the nodes and expire them at the end?

The configuration form would be a great place to start with documentation! I had roles dialed to 11 and that tapped my site's spine pretty good.

I finally wrote my own authcache_enum_role_combine. That took some time, OMG, and if I missed any there would just be a cache miss so those obscure combos or if someone goes creazy with the roles, they will get some slower page loads, which is what I got before when I turned off some of the roles.

It may not be perfect but may help someone trying to see another approach:

It was returning 1024 combos before with all enabled, now it's returning 39!

/**
 * Default enumeration method for authcache keys.
 *
 * Default method for enumerating possible combinations of roles. Each item is
 * an array with one or more role-ids forming a role-combination.
 *
 * This approach will build an array with 2^n elements. If you enable more
 * than a couple of roles (say 10) in authcache, then memory consumption and
 * computation time will rise quickly.
 *
 * In order to override this function with your own implementation, set the
 * variable authcache_enum_role_combine to the desired function, e.g. in
 * settings.php:
 *
 *   $conf['authcache_enum_role_combine'] = '_feature_cache_enum_default_role_combine';
 */
function _feature_cache_enum_default_role_combine() {
  module_load_include('inc', 'authcache_enum', 'authcache_enum.comb');

  $roles = authcache_get_roles();
  $choices = array();

  // Anonymous users do not have any authcache-key. Therefore there is no need
  // to include this role in the property-info.
  unset($roles[DRUPAL_ANONYMOUS_RID]);

  // The authenticated-user role is exclusive, only include it once. Do not
  // generate combinations including this role.
  if (isset($roles[DRUPAL_AUTHENTICATED_RID])) {
    $choices[] = array(DRUPAL_AUTHENTICATED_RID);
    unset($roles[DRUPAL_AUTHENTICATED_RID]);
  }
  
  // Reference.
  // $roles [
  //   1 => 'anonymous user',
  //   2 => 'authenticated user (without additional roles',
  //   3 => 'administrator',
  //   4 => 'Site Admin',
  //   9 => 'Editor',
  //   12 => 'B2B Commerce',
  //   8 => 'Sales Rep',
  //   5 => 'Partner - USA',
  //   10 => 'Partner - Canada',
  //   11 => 'Partner - International',
  //   7 => 'Staff',
  //   6 => 'Trainers',
  // ];

  $admin_roles = [
    3 => 'administrator',
    4 => 'Site Admin',
    9 => 'Editor',
  ];

  $customer_roles = [
    5 => 'Partner - USA',
    10 => 'Partner - Canada',
    11 => 'Partner - International',
  ];

  $staff_roles = [
    4 => 'Site Admin',
    9 => 'Editor',
    6 => 'Trainers',
    8 => 'Sales Rep',
  ];

  // Any combo of the staff roles + B2B Commerce.
  $choices = array_merge($choices, _feature_cache_enum_comb(array_keys($staff_roles) + [12], 4));

  // Admin + B2B Commerce.
  foreach (array_keys($admin_roles) as $rid) {
    $choices = array_merge($choices, _feature_cache_enum_comb([$rid, 12], 2));
  }

  // Other Staff roles + Staff and/or B2B Commerce.
  foreach (array_keys($staff_roles) as $rid) {
    $choices = array_merge($choices, _feature_cache_enum_comb([$rid, 7, 12], 3));
  }

  // Customer + B2B Commerce.
  foreach (array_keys($customer_roles) as $rid) {
    $choices = array_merge($choices, _feature_cache_enum_comb([$rid, 12], 2));
  }

  // Staff + B2B.
  $choices[] = [7, 12];

  // Combine remaining roles.
  $choices = array_merge($choices,  _feature_cache_enum_comb(array_keys($roles), 1));
  sort($choices);
  $choices = array_unique($choices, SORT_REGULAR);
  return $choices;
}

// Limit the combos.
function _feature_cache_enum_comb(array $set, $limit) {
  $result = [];
  for ($k = 1; $k <= $limit; $k++) {
    $result = array_merge($result, _authcache_enum_comb_k($set, $k));
  }

  return $result;
}

A little test script to see the combos:

// drush scr test.php
module_load_include('inc', 'authcache_enum', 'authcache_enum.comb');

$combos = _authcache_enum_default_role_combine();
foreach ($combos as $combo) {
  drush_print(implode(',', $combo));
}
drush_print('--------');
drush_print(count($combos));

$combos = _feature_cache_enum_default_role_combine();
foreach ($combos as $combo) {
  drush_print(implode(',', $combo));
}
drush_print('--------');
drush_print(count($combos));

One thing I was thinking that may be cool for an admin UI thing to help manage these at least as a start:
What about just a text area to write the combos in. People can just write in their most common combos, one per line and leave the single value ones as they could be auto added.

Then I could just dump in something like this:

3,12
4,6
4,9
4,12
5,12
6,12
7,12
8,12
8,7
9,6
9,7
9,8
9,12
10,12
11,12
4,6,8
4,7,12
4,9,6
4,9,8
6,7,12
8,7,12
9,6,8
9,7,12
4,9,6,8

Or if you want to get really fancy you could use a tag style autocomplete (thinking Chosen JS style) with add "combo button" aka phase 2.

May not be quite right... because I noticed quite a few of these notices after using my function:

Notice: Undefined offset: 38 in _authcache_enum_cartesian() (line 94 of contrib/authcache/modules/authcache_enum/authcache_enum.comb.inc).