Problem/Motivation

1 entity = 1 item in the queue. Let's allow this to be batch so users can configure how to process as once and then we can leverage ::loadMultiple() for better performance.

Proposed resolution

Add a user-configured amount of entities to each queue item.

Remaining tasks

User interface changes

New form element to collect the entities_per_dataset setting on the sitemap configuration form.

API changes

None

Data model changes

New entities_per_dataset in simple_sitemap.settings configuration.

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

alexpott created an issue. See original summary.

alexpott’s picture

Status: Active » Needs review

I've run this on a site with 10,000 nodes. Here's the performance results with and without the patch.

With the patch - 500 entities per queue item

time vendor/bin/drush ssg -v --uri http://drupal8alt.test
>  [notice] 21 out of 21 total items have been processed. queries: 20278, memory: 380.5 MB, non-peak mem: 378.5 MB

________________________________________________________
Executed in    7.93 secs   fish           external
   usr time    4.93 secs  100.00 micros    4.93 secs
   sys time    0.52 secs  509.00 micros    0.52 secs

With the patch - 50 entities per queue item

time vendor/bin/drush ssg -v --uri http://drupal8alt.test
>  [notice] 201 out of 201 total items have been processed. queries: 21719, memory: 384.5 MB, non-peak mem: 384.5 MB

________________________________________________________
Executed in   10.18 secs   fish           external
   usr time    5.87 secs  138.00 micros    5.87 secs
   sys time    0.61 secs  650.00 micros    0.61 secs

Without the patch

time vendor/bin/drush ssg -v --uri http://drupal8alt.test
>  [notice] 5019 out of 10001 total items have been processed. queries: 50297, memory: 318.5 MB, non-peak mem: 316.5 MB
>  [notice] 9766 out of 10001 total items have been processed. queries: 47482, memory: 338.5 MB, non-peak mem: 336.5 MB
>  [notice] 10001 out of 10001 total items have been processed. queries: 2367, memory: 338.5 MB, non-peak mem: 326.5 MB

________________________________________________________
Executed in   21.25 secs   fish           external
   usr time   10.90 secs  115.00 micros   10.90 secs
   sys time    1.23 secs  650.00 micros    1.23 secs
alexpott’s picture

Issue summary: View changes
daniel.bosen’s picture

So, I have tested this back and forth, because I could not believe it :-) Without this patch my sitemap generation takes about 15 Minutes, with it, it takes 1m23s... (entities_per_dataset: 500). And the sitemaps are definitely identical, I checked multiple times.
Thanks, @alexpott, this will be saving a lot of electrons ...

I did not take a look at the implementation yet, so I will leave it at needs review, but functionality wise it works for me.

gbyte’s picture

Assigned: Unassigned » gbyte

loadMultiple was on todo since I first introduced queuing in 3.x. So glad @alexpott is looking into these things.

I will be taking a chunk of my time soon to review all your MRs and probably alter/comment on a few small things, e.g the nomenclature above should be more abstract as Drupal entities are processed by a single plugin, whereas every plugin uses the queue.

Until then, thanks for your (and possibly your sponsor's) time.

  • gbyte committed 6c857af on 8.x-3.x authored by alexpott
    Issue #3203715 by alexpott, gbyte: Process multiple entities with a...
gbyte’s picture

Assigned: gbyte » Unassigned
Status: Needs review » Fixed

@alexpott I have made some adjustments and pushed everything so I can merge more issues of yours. Let me know if you are unhappy, so we can address it.

Thanks, @alexpott, this will be saving a lot of electrons ...

Now that that's sorted, let's fix Bitcoin. :)

By the way, I just remembered, years ago, I met with some of you Thunder guys at Burda becasue of a Thunder submodule I was creating for Valiton. One of you said "Thanks for the simple_sitemap module, but we will be switching away to xmlsitemap as soon as it is available for D8." I find it amusing that Thunder is now helping out. :P

alexpott’s picture

The changes look good to me. I ummed and ahhed on queue vs dataset... if you like entities_per_queue_item then that's great! Thanks for the merge :)

daniel.bosen’s picture

"Thanks for the simple_sitemap module, but we will be switching away to xmlsitemap as soon as it is available for D8."

Well, such a bold and obviously wrong statement could have only been done by me :-)

Thanks for getting all these performance tickets in so quickly!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.