Problem/Motivation
1 entity = 1 item in the queue. Let's allow this to be batch so users can configure how to process as once and then we can leverage ::loadMultiple() for better performance.
Proposed resolution
Add a user-configured amount of entities to each queue item.
Remaining tasks
User interface changes
New form element to collect the entities_per_dataset setting on the sitemap configuration form.
API changes
None
Data model changes
New entities_per_dataset in simple_sitemap.settings configuration.
Issue fork simple_sitemap-3203715
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #3
alexpottI've run this on a site with 10,000 nodes. Here's the performance results with and without the patch.
With the patch - 500 entities per queue item
With the patch - 50 entities per queue item
Without the patch
Comment #4
alexpottComment #5
daniel.bosenSo, I have tested this back and forth, because I could not believe it :-) Without this patch my sitemap generation takes about 15 Minutes, with it, it takes 1m23s... (entities_per_dataset: 500). And the sitemaps are definitely identical, I checked multiple times.
Thanks, @alexpott, this will be saving a lot of electrons ...
I did not take a look at the implementation yet, so I will leave it at needs review, but functionality wise it works for me.
Comment #6
gbyteloadMultiple was on todo since I first introduced queuing in 3.x. So glad @alexpott is looking into these things.
I will be taking a chunk of my time soon to review all your MRs and probably alter/comment on a few small things, e.g the nomenclature above should be more abstract as Drupal entities are processed by a single plugin, whereas every plugin uses the queue.
Until then, thanks for your (and possibly your sponsor's) time.
Comment #8
gbyte@alexpott I have made some adjustments and pushed everything so I can merge more issues of yours. Let me know if you are unhappy, so we can address it.
Now that that's sorted, let's fix Bitcoin. :)
By the way, I just remembered, years ago, I met with some of you Thunder guys at Burda becasue of a Thunder submodule I was creating for Valiton. One of you said "Thanks for the simple_sitemap module, but we will be switching away to xmlsitemap as soon as it is available for D8." I find it amusing that Thunder is now helping out. :P
Comment #9
alexpottThe changes look good to me. I ummed and ahhed on queue vs dataset... if you like entities_per_queue_item then that's great! Thanks for the merge :)
Comment #10
daniel.bosenWell, such a bold and obviously wrong statement could have only been done by me :-)
Thanks for getting all these performance tickets in so quickly!