When queueing up long-running tasks (10s-100s of thousands), adding a single queue item at a time is a performance loss. I've written and tested the patch and found the following:

Number of items createItem() createMany() [100 items at a time]
~146,500 2m 28s 58s

My particular use case also updates other tables during this process as well, but no matter how you slice it, the speedup is noticeable.

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

nvahalik created an issue. See original summary.

nvahalik’s picture

Heck. Just realized how out of date my branch is with dev.

Just in case I lose it, this is how I implemented things on my end (on a much earlier version of 7.x-1.x):

  public function createMany($items) {
    $query = db_insert('advancedqueue')
      ->fields(array(
        'name',
        'uid',
        'title',
        'data',
        'created',
        'status',
      ));

    foreach ($items as $data) {
      $query->values(
        array(
          'name' => $this->name,
          'uid' => is_array($data) && isset($data['uid']) ? $data['uid'] : $GLOBALS['user']->uid,
          'title' => is_array($data) && isset($data['title']) ? $data['title'] : t('Unnamed item'),
          'data' => serialize($data),
          // We cannot rely on REQUEST_TIME because many items might be created
          // by a single request which takes longer than 1 second.
          'created' => time(),
          'status' => ADVANCEDQUEUE_STATUS_QUEUED,
        )
      );
    }

    return (bool) $query->execute();
  }
nvahalik’s picture

StatusFileSize
new3.48 KB

This patch refactors the item creation by pulling it into a separate function so that it can be used in both the createUniqueItem() and createMany functions. It also pulls the grabbing of the schema and setting of the $title_max variables into the constructor so that it doesn't happen on every call.

Tests to follow, but I'm curious for any feedback on this approach before I spend more time on it.

nvahalik’s picture

Status: Active » Needs review
StatusFileSize
new3.49 KB
new927 bytes

Looks like I goofed on the name of the preparation function. Patch updated accordingly.

nvahalik’s picture

StatusFileSize
new3.6 KB
new750 bytes

Patch in #4 didn't work. #5 does! Was not calling values() was calling fields(), instead. This only created one at a time.

nvahalik’s picture

StatusFileSize
new3.6 KB
new750 bytes

Ack. Nevermind. Do this instead.

nvahalik’s picture

StatusFileSize
new3.6 KB
new746 bytes

Sheesh. Having problems, obviously. OK. Only create the query if we have something to do.

alexpott’s picture

Version: 7.x-1.x-dev » 8.x-1.x-dev

This sounds like a good idea and I don't think there is a corollary in the 8.x-1.x issue queue so going to move it there. Obvsiously the existing work will need to be completely redone.

alexpott’s picture

Title: Allow for the creation of multiple queue items at once » Fix \Drupal\advancedqueue\Plugin\AdvancedQueue\Backend\Database::enqueueJobs() to use transaction correctly
Category: Feature request » Bug report
Status: Needs review » Needs work

Actually we have this functionality in 8.x-1.x and we're trying to use transactions to do it properly but we've not done it right. Going to use this issue to fix it so I can credit @nvahalik for their original work here.

alexpott’s picture

Status: Needs work » Needs review

  • alexpott committed 2db57857 on 8.x-1.x
    Issue #2911972 by nvahalik, alexpott: Fix \Drupal\advancedqueue\Plugin\...
alexpott’s picture

Status: Needs review » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.