Tuning the worker script

Problem/Motivation

The following code is found in the queue worker script.
There is a small performance problem in it.
The longer it runs, the more time it will spend validating that URL was processed or not due to the linear complexity of in_array function.

/**
   * @param array $results
   */
  protected function removeDuplicates(&$results) {
    if ($this->generatorSettings['remove_duplicates'] && !empty($results)) {
      $result = $results[key($results)];
      if (isset($result['meta']['path'])) {
        if (in_array($result['meta']['path'], $this->processedPaths)) {
          $results = [];
        }
        else {
          $this->processedPaths[] = $result['meta']['path'];
        }
      }
    }
  }

Proposed resolution

This should be an equivalent implementation for the method, that will handle the look-ups in constant time and will not degrade in performance when $this->processedPaths gets too long.

  /**
   * @param array $results
   */
  protected function removeDuplicates(&$results) {
    if ($this->generatorSettings['remove_duplicates'] && !empty($results)) {
      $result = $results[key($results)];
      if (isset($result['meta']['path'])) {
        $this->processedPaths[$result['meta']['path']] = TRUE;
        if (isset($this->processedPaths[$result['meta']['path']])) {
          $results = [];
        }
      }
    }
  }

Remaining tasks

Discussion, Patch?

User interface changes

None.

API changes

None.

Data model changes

None.

Release notes snippet

None.