When queueing up long-running tasks (10s-100s of thousands), adding a single queue item at a time is a performance loss. I've written and tested the patch and found the following:
| Number of items | createItem() |
createMany() [100 items at a time] |
|---|---|---|
| ~146,500 | 2m 28s | 58s |
My particular use case also updates other tables during this process as well, but no matter how you slice it, the speedup is noticeable.
| Comment | File | Size | Author |
|---|---|---|---|
| #7 | 2911972-7-allow_for_the_creation_of.patch | 3.6 KB | nvahalik |
Issue fork advancedqueue-2911972
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
nvahalik commentedHeck. Just realized how out of date my branch is with dev.
Just in case I lose it, this is how I implemented things on my end (on a much earlier version of 7.x-1.x):
Comment #3
nvahalik commentedThis patch refactors the item creation by pulling it into a separate function so that it can be used in both the
createUniqueItem()andcreateManyfunctions. It also pulls the grabbing of the schema and setting of the$title_maxvariables into the constructor so that it doesn't happen on every call.Tests to follow, but I'm curious for any feedback on this approach before I spend more time on it.
Comment #4
nvahalik commentedLooks like I goofed on the name of the preparation function. Patch updated accordingly.
Comment #5
nvahalik commentedPatch in #4 didn't work. #5 does! Was not calling
values()was callingfields(), instead. This only created one at a time.Comment #6
nvahalik commentedAck. Nevermind. Do this instead.
Comment #7
nvahalik commentedSheesh. Having problems, obviously. OK. Only create the query if we have something to do.
Comment #8
alexpottThis sounds like a good idea and I don't think there is a corollary in the 8.x-1.x issue queue so going to move it there. Obvsiously the existing work will need to be completely redone.
Comment #9
alexpottActually we have this functionality in 8.x-1.x and we're trying to use transactions to do it properly but we've not done it right. Going to use this issue to fix it so I can credit @nvahalik for their original work here.
Comment #10
alexpottComment #13
alexpott