Problems running cron to build the sitemap on large sites

Couple questions:
1. What version of PHP are you using?
2. What's your PHP memory limit and Apache/PHP max execution time limits?
3. Do you see any odd errors at admin/reports/dblog? Also if any cron errors are there either.

Ok I think I have hit a similar issue while rebuilding the sitemap. My site has around 130k nodes. Initially I was getting "An error has occurred.Please continue to the error page An HTTP error 500 occurred." during the rebuild process. I was using mod_fcgid which I changed to prefork and now the rebuild gets stuck at different node numbers. When I set the batch size to 5 I got stuck around 600 nodes and when I set to 100 I got stuck at around 7800. There were no errors in watchdog.

The same problem. I have about 150K entries after rebuilding about 30K it crushes. I increased PHP memory from 128MB to 256MB and now it crushes at 70K. So I have only trancated version of sitemap.

An error occurred. /batch?id=15&op=do { "status": true, "percentage": 24, "message": "Remaining 4 of 5.\x3cbr/\x3eNow processing \x3cem\x3enode\x3c/em\x3e 553279 (33400 of 154301)." }
Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 16444555 bytes) in /home/______.com/public_html/includes/common.inc on line 626

After that I get a list of errors:

user error: Negative changecount value. Please report this to http://drupal.org/node/516928.
array ( 'type' => 'node', 'id' => '519881', 'subtype' => 'contact', 'loc' => 'node/519881', 'status' => '1', 'status_default' => '1', 'status_override' => 0, 'priority' => '0.5', 'priority_default' => '0.5', 'priority_override' => 0, 'changefreq' => 0, 'changecount' => -1, 'lastmod' => '1258105774', 'access' => true, 'language' => '', ) in /home/f______.com/public_html/modules/xmlsitemap/xmlsitemap.module on line 471.

The error was also in alpha1

Drupal 6.12
MySQL database 5.0.51a
PHP 5.2.6-1+lenny3
PHP memory limit 256M
PHP register globals Disabled
Web server Apache/2.2.9 (Debian) DAV/2 PHP/5.2.6-1+lenny3 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 OpenSSL/0.9.8g

The same problem was in 6.x-2.0-alpha1

Tested 6.x-2.0-unstable5. The same problem.

An error occurred. /batch?id=16&op=do { "status": true, "percentage": 58, "message": "Remaining 4 of 9.\x3cbr/\x3eNow processing \x3cem\x3exmlsitemap_node\x3c/em\x3e link 35100." }
Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 21912988 bytes) in /home/________.com/public_html/includes/common.inc on line 642

@webcons - I think you have hit a memory limit issue. How many nodes are you processing in one go?

Actually cron is running fine with 5000 nodes per run. Before it was 100.
So it looks that it will index all content, but the problem is I cannot rebuild sitemap.

So I have in yellow "The XML sitemap data is out of sync and needs to be completely rebuilt."
Then I press "rebuild" I get errors.
PHP memory limit is 256 MB

Ok (I think) I have a working solution for the "cannot rebuild sitemap problem". Please try this.

a) Disable and Uninstall XMLSitemap completely
b) Install XMLsitemap
c) Make sure that no content types are enabled for the sitemap
d) Rebuild the sitemap while no content types are enabled. This should completely run successfully
e) Enable contenttypes you wish to include in the sitemap
f) Set the number of items to be processed per cron run to be a small number and increase cron frequency if you want. I used 100 per cron run and cron every 1 min
g) Wait until all the nodes are processed
h) Verify that you can see your sitemap and subpages at the given sitemap URL

Anoop

Changing status back to active as there are a few people who have hit this issue

Hmm...I wonder if this is bad node data causing this.

I have more information - possibly related to this and hence posting it here instead of as a new issue. I have had cron timing out since yesterday after I had issued the rebuild and added the content types. I ran cron from command line and got this error

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 83 bytes) in /sites/all/modules/xmlsitemap/xmlsitemap.generate.inc on line 31

The code says

    $query = db_query("SELECT src, dst FROM {url_alias} WHERE language = '' ORDER BY pid");
    while ($alias = db_fetch_array($query)) {
      $aliases['all'][$alias['src']] = $alias['dst'];
    }

Isn't that a lot of strings to be loaded into the array for a site like mine which has around 150K nodes?

I have the module running fine (mostly) on this site where I have closely 200,000 nodes. I have not ventured to rebuild the site map after I had the problems though. So I cannot confirm whether the rebuild is working fine.

Anoop

Rebuilding should only be used if absolutely necessary. Ideally you shouldn't ever have to do it.

I think most of the problem is caused by pre-fetching of the url aliases. I'll add an option to have that disabled by default.

I'm pretty sure that has to deal with the pre-fetching of the URL aliases. Try the latest 6.x-2.x-dev and turn that option off.

subscribe
I really need to have this up and running....

subscribing

I have this same issue. I have about 75 links included in the sitemap and 128M of memory (dedicated virtual host). My error log says "Cron run exceeded the time limit and was aborted."

I unchecked the option for "Prefetch URL aliases during sitemap generation."

Got this cron error as soon as the xml sitemap module was installed, was ticking along fine before.

*subscribe*

As our site has grown this module has become unusable

I've created a more basic sitemap based on the code at

http://www.seo-expert-blog.com/tutorial/drupal-6-xml-sitemap-for-nodes

This builds very quickly - lists only node pages - but it works.

"As our site has grown this module has become unusable" is not at all helpful to me. What is wrong? What error messages do you get? Have you tried some of the performance tips recommended in the handbook pages and/or issue pages?

I see timeout or memory exhaustion errors - depending on the client I use to access the page.

I've looked around for solutions and tried various versions of this module - I'm not asking for help or complaining - just sharing my solution as personally I find it helpful when people post alternatives on issue queues.

It's still hard to say how we can address anything when we don't know what's at fault.

subscribing

I saw the following in my logs today: "Cron run exceeded the time limit and was aborted."

Cron had not run for two days and when I tried to re-run cron manually I see this: Fatal error: Call to undefined function lock_acquire() in /homepages/8/****/htdocs/Domains/****/sites/all/modules/xmlsitemap/xmlsitemap.module on line 1162

I am on D6 using 6.x-2.0-beta1 and I do not even have 5000 nodes.

Hi earnie,

This site is on Drupal 6.14.

BTW, the only way I can get cron to run is by rebuilding the site map.

One more thing, Cron runs every half hour as it should on my site, but once a day it fails. I have 'Minimum sitemap lifetime' set to one day as well. Which leads to me to believe that regenerating the sitemap is what is causing cron to fail or exceed the time limit/timeout.

@earnie: Thank you. I'm going to give Elysia Cron a look now.

earnie, that looks pretty confusing. How would that help me and how can I set it up? Can you please help?

Thank you!

In our case, this was the solution.

FYI.

The problem is because of this function:

_xmlsitemap_create_cache_files()

Someone wrote everything properly to generates chunks one by one, and then (probably that same) someone wrote the cache function that loops through ALL the chunks to generate the entire cache all at once. That's not appropriate. If I am querying chunk 3, why do you generate chunk 0, 1 and 2 first? And then go on and generate chunk 4, 5 and 6? Really! I only need chunk 3 right now. 8-P

The cache files function needs to get the chunk as a parameter and generate only that one chunk as required.

I think that the needs update flag needs to be used in a completely different way. In xmlsitemap_output(), do something like this:

if (variable_get('xmlsitemap_sitemap_needs_update', FALSE)) {
  xmlsitemap_erase_cache();
  variable_get('xmlsitemap_sitemap_needs_update', TRUE);
}

Then have a new check & create function pair with $chunk as a parameter:

  if (xmlsitemap_check_cache_file($chunk)) {
     xmlsitemap_create_cache_file($chunk);
  }

Now, if I were to rewrite that file, I would have ONE function to generate the filename of a given file. Right now, this is a HUGE mess. Maybe even create an array (an object?!?!?) with all the parameters and pass that around the different functions so we can generate things as required.

I'll be happy to show you what I can do (so it actually works for websites with 1 million nodes,) but you have to promise me that you will check in my patch (once it works which may not be the 1st iteration, but you know...)

Thank you.
Alexis Wilke

@Alexis Wilke: Please do us a favor and double-check which version you're using compared to this issue, because there is no function like that in 6.x-2.x.

Dave,

Thank you for reminding me! 8-) I have version 2 of another module and thought it was this one...

Anyway, I can see that the system was changed to make use of a batch. But I don't think a batch can work in a cron call. Instead, each step of the batch should be "manually" called one at a time on each cron run. That's the way I would do it.

Anyway, outside of the version 2.x looks a lot better, but as I commented on another issue, it's missing a few lines of code in regard to the proper contexts... and without those no XML sitemap is generated.

Update 1: the context is not necessary for the nodes to be added, however, you need to make sure that they are enabled (in Content types.) There is an issue in regard to creating a new XML sitemap & contexts... it is not necessary to have any context to create a first (basic/default) sitemap. However, it was not automatically created on the upgrade path.

Update 2: I can now see the "Default base URL" entry in the "Advanced settings". This is good for my first case problem (i.e. www. versus secure. — same website but I want the XML sitemap on the www. and never on the secure. pages.) This works. Now I still have a problem with sitemaps on other URLs that use the same site to show a few pages... but that is a rather special case. So at this point I'm good with the current 2.x version.

Thank you.
Alexis Wilke

I can confirm a combination of comment #10 and comment #25 worked for me. My site has about 15k articles and 10k taxonomy terms.

This is what I did -

- Disable and uninstalle XMLSitemap completely - this will clean up the XMLsitemap tables too
- Clear out all cached xmlsitemaps from sites/default/files/xmlsitemap
- Install XMLsitemap (I used 2x-beta)
- Disable pre-fetching of the url aliases (IMPORTANT! This is one of the main culprits if memory is a constraint)
- Make sure that no content types or taxonomies are enabled for the sitemap
- Rebuild the sitemap while no content types or taxonomy are enabled.
- Enable content types you wish to include in the sitemap
- Set the number of items to be processed per cron run to 5000 (or a smaller number if your cron times out)
- Run cron manually -> note that the XMLsitemap tables increase in size, and sitemap is generated in sites/default/files/xmlsitemap folder
- Repeat manual cron run until all the nodes are processed
- now enable taxonomies to be included
- repeat manual cron runs until all of it is indexed -> check index status in XMLsitemap settings page
- verify that you can see your sitemap and subpages at the given sitemap URL

Cheers,
Anshuman

It would be a good idea to test how much memory you have left in the the loop process and end the process if you're low in memory (or time too.) That way you avoid problems. Dunno whether that's possible to stop the loop without preventing nodes from making it to the XML sitemap on the next cron call.

Hello, I had the issue, then I solved restarting Apache. Now there's something else worrying me: links on xml sitemap have been rebuilt, but on configuration page I see only 1 page has been indexed? I can't understand what this means. In settings I've included all content types I want to be indexed, and the list there shows they've actually been indexed... so what's the meaning of "links 150" "Pages 1"?
Another scary fact is that the address mysite/sitemap.xml returns a white page... I remember I could see all xml links before...
In Google WebMaster Tools page no errors are reported concerning the xml sitemap.... I know these may be stupid questions but I can't understand, and I need an explanation! Thanks everybody for support!!!!

giozz,

(1) the word "Pages" is indeed confusing, I made a comment about that. It should be "Chunks" as defined in the XML sitemap documentation (i.e. how many XML sitemap files are required to handle your entire site.)

(2) if you are logged in on your website, you may not be able to see the XML sitemap because only anonymous users are supposed to do so.

Thank you for this explanation! Everything seems to work right... only thing is I'm still unable to see the xml links page, even as anonymous. I hope this ain't a problem....

giozz,

You may want to give it a try without that CSS style sheet support in case you had that turned on. But you should see it. Another problem could be that you somehow got a cached version somewhere along the way.

Thank you.
Alexis

ok... but what could I do to find/delete this cached version you refer to? .. I've tried to clean the cache with "css optimization" unchecked, but still no results... but maybe again tha's not what you meant... thanks again for your commitment!

I meant the style sheet reference in the sitemap.xml file.

I'm attaching a screenshot of the XML sitemap settings (under admin/settings/xmlsitemap/settings)

Notice the first checkbox? That's the one that you should try unchecked. Then make sure that the XML sitemap is regenerated at least once and try again to access it at /sitemap.xml

Thank you.
Alexis

Thanks for http://drupal.org/comment/reply/720878/3239180#comment-3239180

I've added a couple of arguments to url to handle count limits...
This way hopefully I'll get rid of all those memory/time/resource issues

function nu_sitemap() {

  if ( !is_numeric( arg(1) )  )  {
    echo 'please provide a valid integer as arg(1) ';
    exit;
  }

  if ( !is_numeric( arg(2) )  )  {
    echo 'please provide a valid integer as arg(2) ';
    exit;
  }

  $base = ($_SERVER['HTTPS'] ? 'https://' : 'http://') . $_SERVER['SERVER_NAME'] . base_path();
  $urls = array();

  $qry  = "SELECT ua.dst, n.changed ";
  $qry .= "FROM node n ";
  $qry .= "INNER JOIN url_alias  ua ON ua.src = CONCAT( 'node/', n.nid ) ";
  $qry .= "WHERE n.status =1 ";
  $qry .= "AND n.nid > " . arg(1) . " ";
  $qry .= "AND n.nid <= "  . arg(2) . " ";
  $qry .= "ORDER BY n.changed DESC ";

  $result =db_query($qry );

  while($r = db_fetch_object($result)) {
    if ($r->dst != 404 && $r->dst != 403) {
      $urls[$base . $r->dst] = $r->changed;
    }
  }
 
  header('Content-Type: text/xml');
  $xml =<<<EOF
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>$base</loc><changefreq>daily</changefreq></url>
EOF;
  foreach ($urls as $url => $t) {
    $xml .= '<url>';
    $xml .= '<loc>' . $url . '</loc>';
    $xml .= '<lastmod>' . date("Y-m-d", $t) . '</lastmod>';
    $xml .=  '</url>';
  }
  $xml .= '</urlset>';
  print $xml;exit();
}

Thanks again! now i can see it!!! Sorry for my sluggishness! :)

Have same this problem when the node number was over 150,000. Cron stop running because of this module. At the end, have to disable this module and generated sitemap manually with some sql function and put the file under the website root folder.

Hi all, My xmlsitemap did'nt update new node
My site is more than 500 k node, over 2500 new node per day.
Php 5.216, php limit 256M, Server Centos, run cron setting every 1 minute max 5000 node per run finish xmlsitmap in 8 hour 500k node.
after 2 days index increase over 7000 node but visible do not change
This appear like this http://baomuabanraovat.com/xmlsitemap.jpg
If i need new xmlsitemap must remove old xml sitemap node from module and run elisa cron over 9 hour
Please advice

Running 6.x-2.0-beta3 on a site with 111k nodes and attempting to build the initial xmlsitemap. Site is running Pressflow 6.20.97 with PHP 5.2.6

Preload path aliases is disabled.

xmlsitemap is configured to build a default, english, and spanish sitemap.

Initially tried via web interface, but a page reload timed out during the batch process and when I reloaded the page, while it appeared to continue on its merry way, the counter reached 400K+ of 111K after a full day so I stopped it. I had hoped the counter was just off by 3x due to the multiple languages.

I then tried via drush however it failed with the following:

drush xmlsitemap-rebuild
Drush command terminated abnormally due to an unrecoverable error.                                                                                           [error]
Error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 24 bytes) in
/includes/cache.inc, line 33

Yes, that is 1GB of RAM allocated.

In it's current state cron is running ok (only 100 nodes processed per run), but the sitemap is incomplete and there is the warning that the whole sitemap needs to be rebuilt.

I would appreciate suggestions on how to proceed.

To follow up on my earlier post, I tried running the rebuild process via the web interface and, rather than starting over at the beginning, it appeared to pick up where the earlier drush rebuild process has run out of RAM. I say this because it completed the rebuild very quickly.

All appears to be working now, including cron runs.

I'm working with a site running Drupal 6.19 and XML sitemap 6.x-2.0-beta2 that is obviously pushing some limits w/ more than 1 million nodes and custom links that need to be included in the generated sitemap. At present, I'm unable to complete a manual or scheduled cron. The client times out, and although I am not seeing any errors generated in the dblog, the process appears to eventually time out. I can see a number of .xml files generated in the sitemap cache directory, but eventually those cease to generate leaving the most recent file remaining empty. The next attempt to execute the cron results in an error "Cron has been running for more than an hour and is most likely stuck.", but a subsequent attempt runs, appearing to start over, but again times out as before.

I've unchecked the option to "Prefetch URL aliases during sitemap generation" and even attempted to reduce the "Maximum number of sitemap links to process at once" option to its minimum value of 5, but have not been able to complete a cron run or even rebuild the sitemap since increasing the number of node/links for inclusion.

Am I just pushing the limitations of this module too far, or are there other steps I can take to allow this to complete normally?

The cron process, from what I've seen, does not take the "Maximum number of sitemap links to process at once" in account. So it will try to generate the 1 million + extra links all at once. You obviously will exhaust your memory or time out before that happens.

I'm afraid that your best bet in this case is to write an external process that creates your sitemap.xml "off line" (not via Apache).

earnie,

And most of the code is there, but it's still not 100% correct yet... 8-)

Still, with 1 million pages, going through CRON to update those will slow down the site much more than creating a separate system (I think).

Earnie,

Agreed. The initial generation of the sitemap is a single occurrence in my case. Subsequent updates will be a substantially smaller fraction of the initial number. It would be nice to pay that cost up front once and only have a small hit for incremental updates. This actually reflects my earlier understanding of a rebuild operation vs. the continuing cron operations.

What is the status of this issue? I have not run into problems in later versions of xmlsitemap. Can we mark this as fixed?

I believe there won't be any support / fix for a problem identified in 6x going forward. Given the fact that D6 is already EOL. Hence i guess this issue can be closed.

Changing the status, if incorrect pls revert the status.