After smashing into a 2.5GB+ memory limit, it seems like startTracking may be the problem. Am I right in thinking that this is trying to entity_load all 50,000 of my nodes at once? :)


 public function startTracking(array $indexes) {
    if (!$this->table) {
      return;
    }
    // We first clear the tracking table for all indexes, so we can just insert
    // all items again without any key conflicts.
    $this->stopTracking($indexes);

    // @TODO Is there a faster approach that can deal with the permutations?
    $entities = entity_load($this->getEntityType(), FALSE);

    $this->trackItemChange(array_keys($entities), $indexes);
  }

In a hurry, I ended up building the table with an INSERT INTO .. SELECT FROM node .. :/

Comments

das-peter’s picture

Am I right in thinking that this is trying to entity_load all 50,000 of my nodes at once?

According to the API doc you are right: https://api.drupal.org/api/drupal/includes!common.inc/function/entity_lo...

This issue is related to this one: #2114721: Using batch API to perform trackItemChange

das-peter’s picture

Issue summary: View changes

Wrong function

jason.fisher’s picture

Thanks for fixing that .. I realized a bit after posting but a poorly timed crash changed my 4AM plans. ;)

das-peter’s picture

Issue summary: View changes
Status: Active » Fixed
Related issues: +#2114721: Using batch API to perform trackItemChange

I've just committed an enhancement for the whole re-index / clear index methods.
The generation of the permutations is now done using the queue and cron. Further I've replaced entity_load() wherever possible by Entity Field Query which hopefully drastically reduces the required processing & memory consumption.
There are also new / adjusted drush commands:

  • sapi-gpq will process the items in the permutation queue.
  • sapi-gg uses the permutation queue now. If you abort it's execution you should be able to continue the processing later with sapi-gpq - or cron will do it for you.

For now I didn't adjust the method trackItemChange(). Let's see if this method really becomes such and amount of entities to process that we need to use an extra queue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.