Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
Problem/Motivation
With a lot of entries in the xmlsitemap table, we have ~450,000, running drush xmlsitemap:regenerate hits memory errors on the latest 8.x-1.x-dev. I'm using Drush 9, enabled by this patch: https://www.drupal.org/project/xmlsitemap/issues/2968172#comment-12770930.
Here is the error and stack I hit, again this is not happening on alpha2, just the latest 8.x-1.x-dev
mmap() failed: [12] Cannot allocate memory PHP Fatal error: Out of memory (allocated 1041145856) (tried to allocate 20480 bytes) in /var/www/vendor/drush/drush/includes/preflight.inc on line 54 PHP Stack trace: PHP 1. {main}() /var/www/vendor/drush/drush/drush:0 PHP 2. require() /var/www/vendor/drush/drush/drush:4 PHP 3. Drush\Runtime\Runtime->run($argv = *uninitialized*) /var/www/vendor/drush/drush/drush.php:66 PHP 4. Drush\Runtime\Runtime->doRun($argv = *uninitialized*) /var/www/vendor/drush/drush/src/Runtime/Runtime.php:41 PHP 5. Drush\Application->run($input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/drush/drush/src/Runtime/Runtime.php:112 PHP 6. Drush\Application->doRun($input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/symfony/console/Application.php:148 PHP 7. Drush\Application->doRunCommand($command = *uninitialized*, $input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/symfony/console/Application.php:248 PHP 8. Consolidation\AnnotatedCommand\AnnotatedCommand->run($input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/symfony/console/Application.php:964 PHP 9. Consolidation\AnnotatedCommand\AnnotatedCommand->execute($input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/symfony/console/Command/Command.php:251 PHP 10. Consolidation\AnnotatedCommand\CommandProcessor->process($output = *uninitialized*, $names = *uninitialized*, $commandCallback = *uninitialized*, $commandData = *uninitialized*) /var/www/vendor/consolidation/annotated-command/src/AnnotatedCommand.php:404 PHP 11. Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter($names = *uninitialized*, $commandCallback = *uninitialized*, $commandData = *uninitialized*) /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:150 PHP 12. Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback($commandCallback = *uninitialized*, $commandData = *uninitialized*) /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:181 PHP 13. call_user_func_array:{/var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:235}(*uninitialized*, *uninitialized*) /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:235 PHP 14. Drupal\xmlsitemap\Commands\XmlSitemapCommands->regenerate(*uninitialized*) /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:235 PHP 15. xmlsitemap_run_unprogressive_batch(*uninitialized*) /var/www/web/modules/contrib/xmlsitemap/src/Commands/XmlSitemapCommands.php:77 PHP 16. batch_process($redirect = *uninitialized*, $url = *uninitialized*, $redirect_callback = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/xmlsitemap.module:1481 PHP 17. _batch_process() /var/www/web/core/includes/form.inc:880 PHP 18. xmlsitemap_regenerate_batch_generate($smid = *uninitialized*, $context = *uninitialized*) /var/www/web/core/includes/batch.inc:294 PHP 19. Drupal\xmlsitemap\XmlSitemapGenerator->regenerateBatchGenerate($smid = *uninitialized*, $context = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/xmlsitemap.module:2473 PHP 20. Drupal\xmlsitemap\XmlSitemapGenerator->generatePage($sitemap = *uninitialized*, $page = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/src/XmlSitemapGenerator.php:320 PHP 21. Drupal\xmlsitemap\XmlSitemapWriter->generateXML() /var/www/web/modules/contrib/xmlsitemap/src/XmlSitemapGenerator.php:192 PHP 22. Drupal\xmlsitemap\XmlSitemapGenerator->generateChunk($sitemap = *uninitialized*, $writer = *uninitialized*, $chunk = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/src/XmlSitemapWriter.php:160 PHP 23. Drupal\Core\Url::fromUri($uri = *uninitialized*, $options = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/src/XmlSitemapGenerator.php:247 PHP 24. Drupal\Core\Url::fromInternalUri($uri_parts = *uninitialized*, $options = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Url.php:307 PHP 25. Drupal\Core\Path\PathValidator->getUrlIfValidWithoutAccessCheck($path = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Url.php:415 PHP 26. Drupal\Core\Path\PathValidator->getUrl($path = *uninitialized*, $access_check = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Path/PathValidator.php:89 PHP 27. Drupal\Core\Path\PathValidator->getPathAttributes($path = *uninitialized*, $request = *uninitialized*, $access_check = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Path/PathValidator.php:122 PHP 28. Drupal\Core\Routing\Router->match($pathinfo = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Path/PathValidator.php:163 PHP 29. Drupal\Core\Routing\Router->matchRequest($request = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Routing/Router.php:116 PHP 30. Drupal\Core\Routing\Router->applyRouteEnhancers($defaults = *uninitialized*, $request = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Routing/Router.php:130 PHP 31. Drupal\Core\Routing\Enhancer\ParamConversionEnhancer->enhance($defaults = *uninitialized*, $request = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Routing/Router.php:259 PHP 32. Drupal\Core\ParamConverter\ParamConverterManager->convert($defaults = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Routing/Enhancer/ParamConversionEnhancer.php:45 PHP 33. Drupal\Core\ParamConverter\EntityConverter->convert($value = *uninitialized*, $definition = *uninitialized*, $name = *uninitialized*, $defaults = *uninitialized*) /var/www/web/core/lib/Drupal/Core/ParamConverter/ParamConverterManager.php:100 PHP 34. Drupal\node\NodeStorage->load($id = *uninitialized*) /var/www/web/core/lib/Drupal/Core/ParamConverter/EntityConverter.php:103 PHP 35. Drupal\node\NodeStorage->loadMultiple($ids = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Entity/EntityStorageBase.php:249 PHP 36. Drupal\node\NodeStorage->doLoadMultiple($ids = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Entity/EntityStorageBase.php:279 PHP 37. Drupal\node\NodeStorage->getFromPersistentCache($ids = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php:356 PHP 38. Drupal\Core\Cache\DatabaseBackend->getMultiple($cids = *uninitialized*, $allow_invalid = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php:941 PHP 39. Drupal\Core\Cache\DatabaseBackend->prepareItem($cache = *uninitialized*, $allow_invalid = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Cache/DatabaseBackend.php:122 PHP 40. unserialize(*uninitialized*) /var/www/web/core/lib/Drupal/Core/Cache/DatabaseBackend.php:167 PHP 41. drush_shutdown() /var/www/vendor/drush/drush/includes/preflight.inc:0 PHP 42. error_get_last() /var/www/vendor/drush/drush/includes/preflight.inc:54
Proposed resolution
Investigate what the cause of the memory issues are. Perhaps introduce queuing or something to chunk the work?
Remaining tasks
- Identify the memory issues
- Implement a work around and/or bring down memory requirements
User interface changes
None
API changes
Mayyyybe
Data model changes
Mayyyybe
Comment | File | Size | Author |
---|---|---|---|
#15 | 2999456-actually-clear-entity-memory-cache.patch | 3.96 KB | Dave Reid |
|
Comments
Comment #2
m4oliveiComment #3
munish.kumar CreditAttribution: munish.kumar commentedHi,
I am also facing the same problem when using drush xmlsitemap-regenerate it ends with memory errors. We have 300k entries in xmlsitemap tables. Any update on this how can I successfully regenerate xmlsitemap. We are using drush 8 version with xmlsitemap alpha3 branch.
Thanks,
Munish kumar
Comment #4
neclimdulQueues is a cool idea but I don't know that addresses the root of the problem. I have a possible solution for this with a little bit deeper change. #3132913: Allow different uri protocols for more effecient url generation
Using that patch my site went from taking minutes and dying with memory errors or timeouts to finishing in seconds. I'm not sure of all the impacts that has for example if the expensive url generation was being used for some undocumented security reason or something that's not clear to me but I'm opened to reviews and feedback on it.
Comment #5
amateescu CreditAttribution: amateescu as a volunteer commentedEdit: commented in the wrong issue.
Comment #6
mstef CreditAttribution: mstef at FFW commentedThis issue appears to be the same as Simple Sitemap: #3170261: Entity static cache can cause memory exhaustion during sitemap regeneration
Clearing just the static cache of the target entity type doesn't seem to be enough (ie, #2857379: Out of memory when rebuild sitemap via drush ). The memory still climbs..
Here is the memory usage now (it ends around 1.4gb):
As a simple test, within XmlSitemapGenerator::rebuildBatchFetch(), I added this to the end:
And the result:
Comment #7
mstef CreditAttribution: mstef at FFW commentedComment #8
mstef CreditAttribution: mstef at FFW commentedNot sure this is the ideal place to execute the call.
Comment #9
maximpodorov CreditAttribution: maximpodorov commentedPatch #8 helps, but it's only for the "rebuild" phase. Some solution is required for "regenerate" phase which also loads entities as mentioned in #3132913: Allow different uri protocols for more effecient url generation.
Comment #10
maximpodorov CreditAttribution: maximpodorov commentedThis patch adds cache clearing in the regeneration phase.
Comment #11
amateescu CreditAttribution: amateescu as a volunteer commentedLooks great to me :)
Comment #12
neclimdulThat's a simple approach. Have to do similar things with migrations to work around some cached.
I think this is still a non-issue with #3132913: Allow different uri protocols for more effecient url generation though. Since that issue forces xmlsitemap into using the efficient path through the URL generation it creates the links faster and bypasses these caches entirely removing the need to fight with them. Since this is basically a hack and that fixes the root of the issue it seems like that would be a better fix.
Comment #13
maximpodorov CreditAttribution: maximpodorov commentedFor me, the solution from #3132913: Allow different uri protocols for more effecient url generation was not enough to build the sitemap having about 1 million links with reasonable memory consumption.
Comment #14
Dave ReidSo I'm trying to figure out why we need to reset the memory cache in \Drupal\xmlsitemap\XmlSitemapGenerator::rebuildBatchFetch() when we call \Drupal::entityTypeManager()->getStorage($entity_type_id)->resetCache(); from inside the process callback, added in #2857379: Out of memory when rebuild sitemap via drush .
Comment #15
Dave ReidAha, so the notes for \Drupal\Core\Entity\ContentEntityStorageBase::resetCache() say:
I also note that if we don't pass in an actual list of IDs, all this method does is invalidate the memory cache, not actually delete from the cache. So our attempt to clear up memory in xmlsitemap_xmlsitemap_process_entity_links() wasn't actually working as expected. I updated the function to clear the entity memory cache, which removed the need to do so in rebuildBatchFetch() from the previous patch. Posting new version for review.
Comment #16
Dave ReidI have tested this with some debugging to check the number of items in the entity memory cache when running the drush commands and confirm that before this, the number was never reset, but now it gets reset after each step of the batch operation.
Comment #19
Dave ReidCommitted #15 to 8.x-1.x. Thanks all for your persistence with this issue and I learned the downsides of the resetCache() method on the entity storage class for the future!
Comment #22
ankondrat4 CreditAttribution: ankondrat4 commentedHello.
I have a big project with more 150000 nodes and when I try run rebuild XML Sitemap and regenerates the cached files from UI, batch doesn't finish success.
Errors from logs:
I tried from console run drush xmlsitemap:regenerate but doesn't finish success too.
Errors:
Patch #15 has realized in last version of module, but it didn't resolve issues with memory as I see. Maybe someone has new ideas how fix this?
Comment #23
ankondrat4 CreditAttribution: ankondrat4 commentedComment #24
ankondrat4 CreditAttribution: ankondrat4 commented