Xml stops working Uncaught exception 'XMLSitemapGenerationException' error

I get this in 7.2.x-dev too while trying to regenerate the sitemap via drush.
Worth mentioning this happened after a huge taxonomy vocabulary inclusion (about 4 million entries) and that it was working before with 20K links split in 4 pages (sitemap.xml?page=1 / sitemap.xml?page=2 / sitemap.xml?page=3 ...)

xmlsitemap-regenerate -v
Initialized Drupal 7.12 root directory at /var/www/html/xxx/d7                                                                                                         [notice]
Initialized Drupal site default at sites/default                                                                                                                          [notice]
WD xmlsitemap: XMLSitemapGenerationException: Unknown error occurred while writing to file                                                                             [error]
public://sites/default/files/xmlsitemap/g3XYqcXbSKPVBDODwnT6pq7oqhCFkPryj4vVqrl_Kfc/24.xml. in XMLSitemapWriter->endDocument() (line 157 of
/var/www/html/xxx/d7/sites/all/modules/xmlsitemap/xmlsitemap.xmlsitemap.inc).
WD php: XMLSitemapGenerationException: Unknown error occurred while writing to file                                                                                    [error]
public://sites/default/files/xmlsitemap/g3XYqcXbSKPVBDODwnT6pq7oqhCFkPryj4vVqrl_Kfc/24.xml. in XMLSitemapWriter->endDocument() (line 157 of
/var/www/html/xxx/d7/sites/all/modules/xmlsitemap/xmlsitemap.xmlsitemap.inc).
XMLSitemapGenerationException: Unknown error occurred while writing to file public://sites/default/files/xmlsitemap/g3XYqcXbSKPVBDODwnT6pq7oqhCFkPryj4vVqrl_Kfc/24.xml. in XMLSitemapWriter->endDocument() (line 157 of /var/www/html/xxx/d7/sites/all/modules/xmlsitemap/xmlsitemap.xmlsitemap.inc).
Drush command terminated abnormally due to an unrecoverable error.

Number of links in each sitemap page 5000.
Maximum number of sitemap links to process at once: 50 (not sure if this affects drush )
I'd say earnie is pretty close to answer it.

Please let me know

Yep...applied that...still running but got way far than previous runs. I'll let you know how it ends up :)

Well it went past that error it seems but it eventually died throwing this:

Fatal error: Maximum execution time of 240 seconds exceeded in /var/www/html/blottr/d7/sites/all/modules/xmlsitemap/xmlsitemap.xmlsitemap.inc on line 128
Drush command terminated abnormally due to an unrecoverable error.                                                                                                     [error]
Error: Maximum execution time of 240 seconds exceeded in /var/www/html/blottr/d7/sites/all/modules/xmlsitemap/xmlsitemap.xmlsitemap.inc, line 128

wich i think it's unrelated.

I'm gonna keep trying.

EDIT:
changed drupal_set_time_limit(240); to drupal_set_time_limit(0);
fingers crossed

drush> xmlsitemap-regenerate -v

Apparently it went fine generating 5004 xml chunks but then thrown this:

Initialized Drupal 7.12 root directory at /var/www/html/xxx/d7                                                                                                         [notice]
Initialized Drupal site default at sites/default                                                                                                                          [notice]
WD xmlsitemap: PDOException: SQLSTATE[HY000]: General error: 3 Error writing file '/dev/shm/MYn0q1Nn' (Errcode: 28): SELECT x.loc AS loc, x.lastmod AS lastmod,        [error]
x.changefreq AS changefreq, x.changecount AS changecount, x.priority AS priority, x.language AS language, x.access AS access, x.status AS status
FROM 
{xmlsitemap} x
WHERE  (x.access = :db_condition_placeholder_0) AND (x.status = :db_condition_placeholder_1) AND (language IN  (:db_condition_placeholder_2,
:db_condition_placeholder_3)) 
ORDER BY x.language DESC, x.loc ASC
LIMIT 5000 OFFSET 2520000; Array
(
    [:db_condition_placeholder_0] => 1
    [:db_condition_placeholder_1] => 1
    [:db_condition_placeholder_2] => en
    [:db_condition_placeholder_3] => und
)
 in xmlsitemap_generate_chunk() (line 164 of /var/www/html/xxx/d7/sites/all/modules/xmlsitemap/xmlsitemap.generate.inc).
WD php: PDOException: SQLSTATE[HY000]: General error: 3 Error writing file '/dev/shm/MYn0q1Nn' (Errcode: 28): SELECT x.loc AS loc, x.lastmod AS lastmod, x.changefreq  [error]
AS changefreq, x.changecount AS changecount, x.priority AS priority, x.language AS language, x.access AS access, x.status AS status
FROM 
{xmlsitemap} x
WHERE  (x.access = :db_condition_placeholder_0) AND (x.status = :db_condition_placeholder_1) AND (language IN  (:db_condition_placeholder_2,
:db_condition_placeholder_3)) 
ORDER BY x.language DESC, x.loc ASC
LIMIT 5000 OFFSET 2520000; Array
(
    [:db_condition_placeholder_0] => 1
    [:db_condition_placeholder_1] => 1
    [:db_condition_placeholder_2] => en
    [:db_condition_placeholder_3] => und
)
 in xmlsitemap_generate_chunk() (line 164 of /var/www/html/xxx/d7/sites/all/modules/xmlsitemap/xmlsitemap.generate.inc).
PDOException: SQLSTATE[HY000]: General error: 3 Error writing file '/dev/shm/MYn0q1Nn' (Errcode: 28): SELECT x.loc AS loc, x.lastmod AS lastmod, x.changefreq AS changefreq, x.changecount AS changecount, x.priority AS priority, x.language AS language, x.access AS access, x.status AS status
FROM 
{xmlsitemap} x
WHERE  (x.access = :db_condition_placeholder_0) AND (x.status = :db_condition_placeholder_1) AND (language IN  (:db_condition_placeholder_2, :db_condition_placeholder_3)) 
ORDER BY x.language DESC, x.loc ASC
LIMIT 5000 OFFSET 2520000; Array
(
    [:db_condition_placeholder_0] => 1
    [:db_condition_placeholder_1] => 1
    [:db_condition_placeholder_2] => en
    [:db_condition_placeholder_3] => und
)
 in xmlsitemap_generate_chunk() (line 164 of /var/www/html/xxx/d7/sites/all/modules/xmlsitemap/xmlsitemap.generate.inc).

Looks like it had a hard time writing to tmpfs wich looks like:

tmpfs                 15799720   2747728  13051992  18% /dev/shm

Question for earnie:
If i restart xmlsitemap-regenerate now what happens ? Is it smart enough to pick up where it left or it'll start all over from the beggining. I don't particulary want to read trough all the module's code but i'll gladly help sorting this module out as currently is kinda useless for big data / high volume websites.
Here's more info about my server just to clarify any memory hog questions:

top - 09:55:11 up 43 days, 13:16,  5 users,  load average: 3.06, 3.26, 2.65
Tasks:  79 total,  10 running,  67 sleeping,   2 stopped,   0 zombie
Cpu(s): 51.7%us,  5.5%sy,  1.5%ni, 37.7%id,  3.6%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  31549440k total, 26549560k used,  4999880k free,        0k buffers
Swap:        0k total,        0k used,        0k free, 16378360k cached

EDIT:
I got my answer...it seems like it's starting over. Maybe this can be a nice optimization ? Add an index of all sane xml chunks and only regenerate from there on. Or maybe add an option to partialy regenerate.

This fixed my problem that I was having. here is a patch.

Fixed up the paths so it will apply

Patch from #12 fixed my issue in 6.x-2.0-rc1. Sitemap generation is now way faster too.

Here is a version of the workaround patch in #12 for D7.

It fixed the issue for me.

I never said it did, and you might notice I didn't set it back to needs review for that very reason.

It is just there as a drupal 7 version of the workaround that was previously provided for drupal 6 for those who either don't know how to fix it better or don't have time to dig into it.

Patch against 7.x that removes the getStatus() method and associated code.

Same patch against 6.x.

Resubmit corrected 7.x patch.

*sigh*

It's apparently been a while since I did this... apologies.

And, once again, here's the 6.x patch in the proper format.

Fixing the status back.

Sorry, was obviously in a hurry while I was doing this, so I got my file order backwards when rolling the 6.x patch. It's not otherwise identical to #23, though, as the offsets are a few lines off in the 6.x patch due to other differences in that file between 6.x and 7.x.

To your other point, though, I'm not sure why you want to keep the flush() method around... That method's *sole* purpose is to set the status property based on the return value from the actual XMLWriter::flush() method. Since the only place that the status property is used is in getStatus, and we've already determined that the return from XMLWriter::flush() isn't useful to what we're doing, I'm not sure why it needs to be kept. If the getStatus() method is removed, then flush() is setting a property that will never get read.

I'm definitely missing something, then. The PHP docs for XMLWriter::flush() don't state anywhere that the flush() method *might* return FALSE... It just says that it simply returns either the buffer or the number of written bytes, never FALSE...?

Right, OK... Checking the $return from the parent::endDocument() is exactly what my patch does, so we're good there.

As to the flush() method, here is what it was before:

public function flush($empty = TRUE) {
  $return = parent::flush($empty);
  $this->status &= (bool) $return;
  return $return;
}

Notice that the *only* thing it does, other than calling parent::flush() is set the value of the status property, which we are no longer using. So, again, I see no reason to override it. Removing the method from the XMLSitemapWriter *sub*class won't remove the method from the class... it will just inherit the method from its parent, as all subclasses do. And, since it would no longer be doing anything that the inherited method wasn't, there's no reason to override it. Hence, I removed the method override in my patch... no longer necessary.

And, for the record, I checked the rest of the code, and the *only* place I see the flush() method being called directly is from the writeSitemapElement(), and it's calling it in the object context, so it will use the inherited method if there is no overriding method, just as it should.

So, based on all that, I still don't see what's wrong with the patch I supplied.

Sure thing, here ya go.

I wouldn't think so... I'm making an assumption here, but I would think that calling endDocument would actually force a flush of the buffer since you're closing out the XML document. Though, if you do decide to add a flush call, you don't need to call parent::flush(), rather $this->flush().

Just reporting in that patch from #34 seems to be working for me.

So, is it time to push this one to 7.x-dev?

Hi,

i had the same problem. It appears if the return value in the function flush is 0.
then (bool) $return will become to FALSE and will not change back again.

From the header in parent::flush

@return mixed If you opened the writer in memory, this function returns the generated XML buffer,
* Else, if using URI, this function will write the buffer and return the number of
* written bytes.

so the number of written bytes seems to be 0.

I just changed the function flush to the following lines and it works

/**
   * Override of XMLWriter::flush() to track file writing status.
   */
  public function flush($empty = TRUE) {
    $return = parent::flush($empty);
    if (isset($return)) {
      $this->status = TRUE;
    }
    else {
      $this->status = FALSE;
    }
    return $return;
  }

I'm experiencing the 'XMLSitemapGenerationException' error from tome to time with D6. The patch from #34 seems to fix the problem, including Cron runs through Drush. Thanks @mgriego!

Since there also are other confirmations that this patch is fine (#38), I'd like to set the status to RTBC, but I can't comment on #35-#37. I can just say that before applying the patch, cron dies and sitemap generation fails, and after applying the patch, cron finishes and sitemap generation succeeds (tested on a site with split sitemaps with 132,209 links).

Patch from #23 seems to solve the problem.

Add the comments containing the patches for review.

Just another confirmation that #34 fixes this on 6.x-2.0-rc2. Thanks!

#34 fixes this on 6.x-2.0-rc2. Thanks!

#23 fixed my problem with 7.x-2.0-rc2

Committed #23 to 7.x-2.x and #34 to 6.x-2.x. Sorry everyone for the wait.
http://drupalcode.org/project/xmlsitemap.git/commit/5c78eec
http://drupalcode.org/project/xmlsitemap.git/commit/d7f36f6

xmlsitemap-7.x-2.0 and xmlsitemap-7.x-2.x-dev both give me:
XMLSitemapGenerationException: Generated public://xmlsitemap/NXhscRe0440PFpI5dSznEVgmauL25KojD7u4e9aZwOM/1.xml resulted in an empty file. in XMLSitemapWriter->endDocument() (line 156 of /var/www/www.******.com/public/sites/all/modules/xmlsitemap/xmlsitemap.xmlsitemap.inc).

Running the cron gives me a connection reset.

Any idea how to get it working?

thnx

response to comment #51, check out this issue, looks like a different problem (with solution) : https://drupal.org/node/2220707

Solution on #52 fixed my problem

#52 pointing to "Error on sitemap generation: "An AJAX HTTP request terminated abnormally." #6 resolved this for me, except that in XML Sitemap 6.x-2.0, the offending line is on line 161 in the file xmlsitemap.xmlsitemap.inc.

I was getting blank white pages on "Rebuild links", and running the cron. I am running Drupal 6.30.