Problem/Motivation

With a lot of entries in the xmlsitemap table, we have ~450,000, running drush xmlsitemap:regenerate hits memory errors on the latest 8.x-1.x-dev. I'm using Drush 9, enabled by this patch: https://www.drupal.org/project/xmlsitemap/issues/2968172#comment-12770930.

Here is the error and stack I hit, again this is not happening on alpha2, just the latest 8.x-1.x-dev

mmap() failed: [12] Cannot allocate memory
PHP Fatal error:  Out of memory (allocated 1041145856) (tried to allocate 20480 bytes) in /var/www/vendor/drush/drush/includes/preflight.inc on line 54
PHP Stack trace:
PHP   1. {main}() /var/www/vendor/drush/drush/drush:0
PHP   2. require() /var/www/vendor/drush/drush/drush:4
PHP   3. Drush\Runtime\Runtime->run($argv = *uninitialized*) /var/www/vendor/drush/drush/drush.php:66
PHP   4. Drush\Runtime\Runtime->doRun($argv = *uninitialized*) /var/www/vendor/drush/drush/src/Runtime/Runtime.php:41
PHP   5. Drush\Application->run($input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/drush/drush/src/Runtime/Runtime.php:112
PHP   6. Drush\Application->doRun($input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/symfony/console/Application.php:148
PHP   7. Drush\Application->doRunCommand($command = *uninitialized*, $input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/symfony/console/Application.php:248
PHP   8. Consolidation\AnnotatedCommand\AnnotatedCommand->run($input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/symfony/console/Application.php:964
PHP   9. Consolidation\AnnotatedCommand\AnnotatedCommand->execute($input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/symfony/console/Command/Command.php:251
PHP  10. Consolidation\AnnotatedCommand\CommandProcessor->process($output = *uninitialized*, $names = *uninitialized*, $commandCallback = *uninitialized*, $commandData = *uninitialized*) /var/www/vendor/consolidation/annotated-command/src/AnnotatedCommand.php:404
PHP  11. Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter($names = *uninitialized*, $commandCallback = *uninitialized*, $commandData = *uninitialized*) /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:150
PHP  12. Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback($commandCallback = *uninitialized*, $commandData = *uninitialized*) /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:181
PHP  13. call_user_func_array:{/var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:235}(*uninitialized*, *uninitialized*) /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:235
PHP  14. Drupal\xmlsitemap\Commands\XmlSitemapCommands->regenerate(*uninitialized*) /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:235
PHP  15. xmlsitemap_run_unprogressive_batch(*uninitialized*) /var/www/web/modules/contrib/xmlsitemap/src/Commands/XmlSitemapCommands.php:77
PHP  16. batch_process($redirect = *uninitialized*, $url = *uninitialized*, $redirect_callback = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/xmlsitemap.module:1481
PHP  17. _batch_process() /var/www/web/core/includes/form.inc:880
PHP  18. xmlsitemap_regenerate_batch_generate($smid = *uninitialized*, $context = *uninitialized*) /var/www/web/core/includes/batch.inc:294
PHP  19. Drupal\xmlsitemap\XmlSitemapGenerator->regenerateBatchGenerate($smid = *uninitialized*, $context = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/xmlsitemap.module:2473
PHP  20. Drupal\xmlsitemap\XmlSitemapGenerator->generatePage($sitemap = *uninitialized*, $page = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/src/XmlSitemapGenerator.php:320
PHP  21. Drupal\xmlsitemap\XmlSitemapWriter->generateXML() /var/www/web/modules/contrib/xmlsitemap/src/XmlSitemapGenerator.php:192
PHP  22. Drupal\xmlsitemap\XmlSitemapGenerator->generateChunk($sitemap = *uninitialized*, $writer = *uninitialized*, $chunk = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/src/XmlSitemapWriter.php:160
PHP  23. Drupal\Core\Url::fromUri($uri = *uninitialized*, $options = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/src/XmlSitemapGenerator.php:247
PHP  24. Drupal\Core\Url::fromInternalUri($uri_parts = *uninitialized*, $options = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Url.php:307
PHP  25. Drupal\Core\Path\PathValidator->getUrlIfValidWithoutAccessCheck($path = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Url.php:415
PHP  26. Drupal\Core\Path\PathValidator->getUrl($path = *uninitialized*, $access_check = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Path/PathValidator.php:89
PHP  27. Drupal\Core\Path\PathValidator->getPathAttributes($path = *uninitialized*, $request = *uninitialized*, $access_check = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Path/PathValidator.php:122
PHP  28. Drupal\Core\Routing\Router->match($pathinfo = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Path/PathValidator.php:163
PHP  29. Drupal\Core\Routing\Router->matchRequest($request = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Routing/Router.php:116
PHP  30. Drupal\Core\Routing\Router->applyRouteEnhancers($defaults = *uninitialized*, $request = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Routing/Router.php:130
PHP  31. Drupal\Core\Routing\Enhancer\ParamConversionEnhancer->enhance($defaults = *uninitialized*, $request = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Routing/Router.php:259
PHP  32. Drupal\Core\ParamConverter\ParamConverterManager->convert($defaults = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Routing/Enhancer/ParamConversionEnhancer.php:45
PHP  33. Drupal\Core\ParamConverter\EntityConverter->convert($value = *uninitialized*, $definition = *uninitialized*, $name = *uninitialized*, $defaults = *uninitialized*) /var/www/web/core/lib/Drupal/Core/ParamConverter/ParamConverterManager.php:100
PHP  34. Drupal\node\NodeStorage->load($id = *uninitialized*) /var/www/web/core/lib/Drupal/Core/ParamConverter/EntityConverter.php:103
PHP  35. Drupal\node\NodeStorage->loadMultiple($ids = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Entity/EntityStorageBase.php:249
PHP  36. Drupal\node\NodeStorage->doLoadMultiple($ids = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Entity/EntityStorageBase.php:279
PHP  37. Drupal\node\NodeStorage->getFromPersistentCache($ids = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php:356
PHP  38. Drupal\Core\Cache\DatabaseBackend->getMultiple($cids = *uninitialized*, $allow_invalid = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php:941
PHP  39. Drupal\Core\Cache\DatabaseBackend->prepareItem($cache = *uninitialized*, $allow_invalid = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Cache/DatabaseBackend.php:122
PHP  40. unserialize(*uninitialized*) /var/www/web/core/lib/Drupal/Core/Cache/DatabaseBackend.php:167
PHP  41. drush_shutdown() /var/www/vendor/drush/drush/includes/preflight.inc:0
PHP  42. error_get_last() /var/www/vendor/drush/drush/includes/preflight.inc:54

Proposed resolution

Investigate what the cause of the memory issues are. Perhaps introduce queuing or something to chunk the work?

Remaining tasks

  • Identify the memory issues
  • Implement a work around and/or bring down memory requirements

User interface changes

None

API changes

Mayyyybe

Data model changes

Mayyyybe

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

m4olivei created an issue. See original summary.

m4olivei’s picture

munish.kumar’s picture

Hi,

I am also facing the same problem when using drush xmlsitemap-regenerate it ends with memory errors. We have 300k entries in xmlsitemap tables. Any update on this how can I successfully regenerate xmlsitemap. We are using drush 8 version with xmlsitemap alpha3 branch.

Thanks,
Munish kumar

neclimdul’s picture

Queues is a cool idea but I don't know that addresses the root of the problem. I have a possible solution for this with a little bit deeper change. #3132913: Allow different uri protocols for more effecient url generation

Using that patch my site went from taking minutes and dying with memory errors or timeouts to finishing in seconds. I'm not sure of all the impacts that has for example if the expensive url generation was being used for some undocumented security reason or something that's not clear to me but I'm opened to reviews and feedback on it.

amateescu’s picture

Edit: commented in the wrong issue.

mstef’s picture

This issue appears to be the same as Simple Sitemap: #3170261: Entity static cache can cause memory exhaustion during sitemap regeneration

Clearing just the static cache of the target entity type doesn't seem to be enough (ie, #2857379: Out of memory when rebuild sitemap via drush ). The memory still climbs..

Here is the memory usage now (it ends around 1.4gb):

$ drush xmlsitemap:rebuild
>  [notice] Links cleared
>  [notice] Processed node 100 (100 of 54301): Memory 37.921936035156 MB.
>  [notice] Processed node 200 (200 of 54301): Memory 40.656723022461 MB.
>  [notice] Processed node 300 (300 of 54301): Memory 43.40404510498 MB.
>  [notice] Processed node 400 (400 of 54301): Memory 46.104507446289 MB.
>  [notice] Processed node 500 (500 of 54301): Memory 48.745574951172 MB.
>  [notice] Processed node 600 (600 of 54301): Memory 51.495986938477 MB.
>  [notice] Processed node 700 (700 of 54301): Memory 54.166763305664 MB.
>  [notice] Processed node 800 (800 of 54301): Memory 56.867172241211 MB.
>  [notice] Processed node 900 (900 of 54301): Memory 59.600448608398 MB.
>  [notice] Processed node 1000 (1000 of 54301): Memory 62.305335998535 MB.
>  [notice] Processed node 1100 (1100 of 54301): Memory 65.094062805176 MB.
>  [notice] Processed node 1200 (1200 of 54301): Memory 67.795234680176 MB.
>  [notice] Processed node 1300 (1300 of 54301): Memory 70.466773986816 MB.
>  [notice] Processed node 1400 (1400 of 54301): Memory 73.153129577637 MB.
>  [notice] Processed node 1500 (1500 of 54301): Memory 75.854301452637 MB.
>  [notice] Processed node 1600 (1600 of 54301): Memory 78.525840759277 MB.
>  [notice] Processed node 1700 (1700 of 54301): Memory 81.227012634277 MB.
>  [notice] Processed node 1800 (1800 of 54301): Memory 83.898551940918 MB.
>  [notice] Processed node 1900 (1900 of 54301): Memory 86.584907531738 MB.
>  [notice] Processed node 2000 (2000 of 54301): Memory 89.411079406738 MB.
>  [notice] Processed node 2100 (2100 of 54301): Memory 92.316993713379 MB.
>  [notice] Processed node 2200 (2200 of 54301): Memory 95.003349304199 MB.
>  [notice] Processed node 2300 (2300 of 54301): Memory 97.704521179199 MB.
>  [notice] Processed node 2400 (2400 of 54301): Memory 100.38063812256 MB.
>  [notice] Processed node 2500 (2500 of 54301): Memory 103.06699371338 MB.
>  [notice] Processed node 2600 (2600 of 54301): Memory 105.76816558838 MB.
>  [notice] Processed node 2700 (2700 of 54301): Memory 108.43970489502 MB.
>  [notice] Processed node 2800 (2800 of 54301): Memory 111.12606048584 MB.
>  [notice] Processed node 2900 (2900 of 54301): Memory 113.82723236084 MB.
>  [notice] Processed node 3000 (3000 of 54301): Memory 116.49877166748 MB.
>  [notice] Processed node 3100 (3100 of 54301): Memory 119.1851272583 MB.
>  [notice] Processed node 3200 (3200 of 54301): Memory 121.87148284912 MB.
>  [notice] Processed node 3300 (3300 of 54301): Memory 124.57265472412 MB.
>  [notice] Processed node 3400 (3400 of 54301): Memory 127.24419403076 MB.
>  [notice] Processed node 3500 (3500 of 54301): Memory 129.93054962158 MB.
>  [notice] Processed node 3600 (3600 of 54301): Memory 132.6169052124 MB.
>  [notice] Processed node 3700 (3700 of 54301): Memory 135.30326080322 MB.
>  [notice] Processed node 3800 (3800 of 54301): Memory 138.00443267822 MB.
>  [notice] Processed node 3900 (3900 of 54301): Memory 140.67597198486 MB.
>  [notice] Processed node 4000 (4000 of 54301): Memory 143.36232757568 MB.

As a simple test, within XmlSitemapGenerator::rebuildBatchFetch(), I added this to the end:

\Drupal::service('entity.memory_cache')->deleteAll();

And the result:

$ drush xmlsitemap:rebuild
>  [notice] Links cleared
>  [notice] Processed node 100 (100 of 54301): Memory 37.921936035156 MB.
>  [notice] Processed node 200 (200 of 54301): Memory 40.573471069336 MB.
>  [notice] Processed node 300 (300 of 54301): Memory 43.23363494873 MB.
>  [notice] Processed node 400 (400 of 54301): Memory 45.854751586914 MB.
>  [notice] Processed node 500 (500 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 600 (600 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 700 (700 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 800 (800 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 900 (900 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 1000 (1000 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 1100 (1100 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 1200 (1200 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 1300 (1300 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 1400 (1400 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 1500 (1500 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 1600 (1600 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 1700 (1700 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 1800 (1800 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 1900 (1900 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 2000 (2000 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 2100 (2100 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 2200 (2200 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 2300 (2300 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 2400 (2400 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 2500 (2500 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 2600 (2600 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 2700 (2700 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 2800 (2800 of 54301): Memory 48.392211914062 MB.
>  [notice] Processed node 2900 (2900 of 54301): Memory 48.392211914062 MB.
mstef’s picture

mstef’s picture

Status: Active » Needs review
FileSize
3.31 KB

Not sure this is the ideal place to execute the call.

maximpodorov’s picture

Patch #8 helps, but it's only for the "rebuild" phase. Some solution is required for "regenerate" phase which also loads entities as mentioned in #3132913: Allow different uri protocols for more effecient url generation.

maximpodorov’s picture

This patch adds cache clearing in the regeneration phase.

amateescu’s picture

Status: Needs review » Reviewed & tested by the community

Looks great to me :)

neclimdul’s picture

That's a simple approach. Have to do similar things with migrations to work around some cached.

I think this is still a non-issue with #3132913: Allow different uri protocols for more effecient url generation though. Since that issue forces xmlsitemap into using the efficient path through the URL generation it creates the links faster and bypasses these caches entirely removing the need to fight with them. Since this is basically a hack and that fixes the root of the issue it seems like that would be a better fix.

maximpodorov’s picture

For me, the solution from #3132913: Allow different uri protocols for more effecient url generation was not enough to build the sitemap having about 1 million links with reasonable memory consumption.

Dave Reid’s picture

So I'm trying to figure out why we need to reset the memory cache in \Drupal\xmlsitemap\XmlSitemapGenerator::rebuildBatchFetch() when we call \Drupal::entityTypeManager()->getStorage($entity_type_id)->resetCache(); from inside the process callback, added in #2857379: Out of memory when rebuild sitemap via drush .

Dave Reid’s picture

Aha, so the notes for \Drupal\Core\Entity\ContentEntityStorageBase::resetCache() say:

   * Content entities have both an in-memory static cache and a persistent
   * cache. Use this method to clear all caches. To clear just the in-memory
   * cache, use the 'entity.memory_cache' service.

I also note that if we don't pass in an actual list of IDs, all this method does is invalidate the memory cache, not actually delete from the cache. So our attempt to clear up memory in xmlsitemap_xmlsitemap_process_entity_links() wasn't actually working as expected. I updated the function to clear the entity memory cache, which removed the need to do so in rebuildBatchFetch() from the previous patch. Posting new version for review.

Dave Reid’s picture

I have tested this with some debugging to check the number of items in the entity memory cache when running the drush commands and confirm that before this, the number was never reset, but now it gets reset after each step of the batch operation.

  • Dave Reid authored 8b8e682 on 8.x-1.x
    Issue #2999456 by maximpodorov, Dave Reid, mstef: Fixed possible memory...

  • Dave Reid authored 13c4c9b on 2.x
    Issue #2999456 by maximpodorov, Dave Reid, mstef: Fixed possible memory...
Dave Reid’s picture

Status: Needs review » Fixed

Committed #15 to 8.x-1.x. Thanks all for your persistence with this issue and I learned the downsides of the resetCache() method on the entity storage class for the future!

  • Dave Reid authored 8b8e682 on 2.x
    Issue #2999456 by maximpodorov, Dave Reid, mstef: Fixed possible memory...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

ankondrat4’s picture

Hello.

I have a big project with more 150000 nodes and when I try run rebuild XML Sitemap and regenerates the cached files from UI, batch doesn't finish success.

Errors from logs:

[02-Feb-2023 07:32:33 Europe/Paris] PHP Fatal error:  Maximum execution time of 90 seconds exceeded in /mnt/www/html/docroot/core/lib/Drupal/Core/Entity/Query/Sql/Tables.php on line 263 request_id="v-fc6c2b6c-a2c2-11ed-b923-9b540a5d00ce"

[02-Feb-2023 08:20:02 Europe/Paris] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 1048576 bytes) in /mnt/www/html//docroot/core/lib/Drupal/Core/Entity/EntityStorageBase.php on line 221 request_id="v-a09a9254-a2c9-11ed-89a5-cbc98f92183c"

[02-Feb-2023 11:37:15 Europe/Paris] PHP Fatal error:  Maximum execution time of 120 seconds exceeded in /mnt/www/html//docroot/core/lib/Drupal/Core/Field/BaseFieldDefinition.php on line 606 request_id="v-fe405f0e-a2e4-11ed-beae-0faeab71a96d"

[02-Feb-2023 12:05:00 Europe/Paris] PHP Fatal error:  Maximum execution time of 150 seconds exceeded in /mnt/www/html//docroot/core/lib/Drupal/Core/Database/Query/Select.php on line 812 request_id="v-a094f848-a2e8-11ed-9135-ef7aaf4987cf"

Feb  2 11:23:06 [2244]: Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /mnt/www/html/docroot/core/lib/Drupal/Core/Database/Query/Condition.php on line 405
Feb  2 11:23:06 [2244]: Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 40960 bytes) in /mnt/www/html//vendor/composer/ClassLoader.php on line 571 in Consolidation\SiteProcess\ProcessBase->getOutputAsJson() (line 171 of /mnt/www/html//vendor/consolidation/site-process/src/ProcessBase.php).

I tried from console run drush xmlsitemap:regenerate but doesn't finish success too.
Errors:

In ProcessBase.php line 171:
                                                                                                                    
  Unable to decode output into JSON: Syntax error                                                                   
                                                                                                                    
  Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /mnt/www/html/  
  /docroot/core/lib/Drupal/Core/Database/Query/Condition.php on line 319                                    
                                                                                                                    
  Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 65536 bytes) in /mnt/www/html/  
  /vendor/symfony/http-kernel/Event/TerminateEvent.php on line 24    

Patch #15 has realized in last version of module, but it didn't resolve issues with memory as I see. Maybe someone has new ideas how fix this?

ankondrat4’s picture

Priority: Normal » Major
ankondrat4’s picture

Related issues: +#3194370: Memory limit