Problem/Motivation

When you have a huge numbers of queue's after full regenerate for simple_sitemap memory limit can be reached.
For our case it occurs for 300k+ items in queue.

SELECT count(*) FROM queue where name LIKE '%simple_sitemap%' 
======
count(*)
334491

Steps to reproduce

1. Generate the content to have a 300k+ records for queue
2. Run drush ssr to regenerate entities for sitemap
3. Run drush ssg

Proposed resolution

Load queue items in chunks

Remaining tasks

User interface changes

API changes

Data model changes

CommentFileSizeAuthor
#12 simple_sitemap-yield_memory-3231298-12.patch1.2 KBmaximpodorov
#2 3231298-1.patch1.19 KBdmitry.korhov

Issue fork simple_sitemap-3231298

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

dmitry.korhov created an issue. See original summary.

dmitry.korhov’s picture

dmitry.korhov
Primary language English
Location Poland, Warsaw
CreditAttribution: dmitry.korhov at EPAM Systems commented
FileSize
3231298-1.patch1.19 KB
dmitry.korhov’s picture

dmitry.korhov
Primary language English
Location Poland, Warsaw
CreditAttribution: dmitry.korhov at EPAM Systems commented
Title: Function yieldItem SimplesitemapQueue allocates too much memory » Function yieldItem in SimplesitemapQueue allocates too much memory

dmitry.korhov opened merge request !24

dmitry.korhov closed merge request !24

dmitry.korhov opened merge request !25

dmitry.korhov closed merge request !25

dmitry.korhov opened merge request !26

dmitry.korhov’s picture

dmitry.korhov
Primary language English
Location Poland, Warsaw
CreditAttribution: dmitry.korhov at EPAM Systems commented
Status: Active » Needs review

Status: Needs review » Needs work

The last submitted patch, 2: 3231298-1.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

gbyte’s picture

gbyte CreditAttribution: gbyte as a volunteer and at gbyte commented
Version: 8.x-3.10 » 4.x-dev
Category: Bug report » Task
Priority: Major » Normal

3.x is at feature freeze and 4.x has been deemed good enough for non-api-dependant use cases. Let's test the necessity of this in 4.x and implement if necessary.

maximpodorov’s picture

maximpodorov CreditAttribution: maximpodorov commented
Status: Needs work » Needs review
FileSize
simple_sitemap-yield_memory-3231298-12.patch1.2 KB

Re-rolled for 4.x-dev.

Berdir’s picture

Berdir
Primary language German
Location Switzerland
CreditAttribution: Berdir at MD Systems GmbH commented

are you sure the offset makes sense?

aren't finished queue items removed when completed successfully? According to \Drupal\simple_sitemap\Queue\QueueWorker::generate, they are.

that means you load 1-50, then skip 51-100, load 101-150, then skip 151-250 and so on.