Google Webmaster Tools has a problem with the Sitemap - Invalid tag value for changefreq

Have you tried visiting it while logged out?

Do you mind if we know the link to the sitemap? You can also send it through my contact form if you'd prefer not to post it here.

One of my sites is under review from Google now. Hopefully it gets accepted. See my latest support request post explaining the situation.

I am facing the same issue. I have a bit more information about my problem. The problem tag is the changefreq tag. I have verified that this tag comes out as empty in some of the cases.

562 Invalid tag value
This XML tag has an invalid value. Please fix it and resubmit.
Parent tag: url
Tag: changefreq
Value:
Problem detected on: Sep 23, 2009

563 Invalid tag value
This XML tag has an invalid value. Please fix it and resubmit.
Parent tag: url
Tag: changefreq
Value:
Problem detected on: Sep 23, 2009

564 Invalid tag value
This XML tag has an invalid value. Please fix it and resubmit.
Parent tag: url
Tag: changefreq
Value:
Problem detected on: Sep 23, 2009

I checked out the database and did a group-by on changefreq. Most of the 14000 records had changefreq 0 except around 70 that had non-zero change freq.

The problem is with http://www.benzinga.com/sitemap-7.xml

That's really odd. Can you run this SQL on your database and post the result here?
SELECT * FROM xmlsitemap WHERE type = 'node' AND id = 16337;

The csv export of the row is as follows.

"16337","node","globenewswire_story","node/16337",,"1","1","0","1253718607","0.5","0","0","0"

Changefreq is 0

in xmlsitemap.inc,

    if ($link['changefreq']) {
      $link_output .= '<changefreq>' . xmlsitemap_get_changefreq($link['changefreq']) . '</changefreq>';
    }

In xmlsitemap.module

function xmlsitemap_get_changefreq($interval) {
  if ($interval <= 0 || !is_numeric($interval)) {
    return FALSE;
  }

  foreach (xmlsitemap_get_changefreq_options() as $value => $frequency) {
    if ($interval <= $value) {
      return $frequency;
    }
  }

  return 'never';
}

xmlsitemap_get_changefreq is coded to handle invalid inputs. So shouldn't the return value be checked before creating the changefreq tag

    if ($link['changefreq']) {
      $changefreq = xmlsitemap_get_changefreq($link['changefreq']);
      if ($changefreq) {
        $link_output .= '<changefreq>' . $changefreq . '</changefreq>';
      }
    }

I'm still trying to find out why it would pass this condition in the first place:

    if ($link['changefreq']) {
      $link_output .= '<changefreq>' . xmlsitemap_get_changefreq($link['changefreq']) . '</changefreq>';
    }

Thanks for reporting this.

Could this have anything to do with this issue with REQUEST_TIME?

http://drupal.org/node/437704

If REQUEST_TIME < lastmod we will get the symptom that we are getting now.

Is request_time the time stamp of the client or from the server? Would server and client in different time zones cause a problem?

I find strange that REQUEST_TIME is less than the last modification time. It could happen when the server time is updated after the passage from or to the daylight saving time, but I would expect the server Unix timestamp to be always referred to GMT.

I can think of no possible method for REQUEST_TIME and lastmod to be out of sync except for a modification to time at the server level.

I would think so too.
That is clearly out of control for XML sitemap, which cannot verify if the time is changed at server level. Checking the validity of the time obtained from the server seems a waste of time, IMO.

I have added a watchdog call to track the values of the variables when the error actually happens. I will post back with details and we can take it on from there. If it is a low freq event like somebody changing server time then we won't get anything soon. Couldn't the server time decay because of a weak battery?

    if ($link['changefreq']) {
      $changefreq = xmlsitemap_get_changefreq($link['changefreq']);
      if ($changefreq) {
        $link_output .= '<changefreq>' . $changefreq . '</changefreq>';
      } 
      else {
        watchdog('xmlsitemap',
            'changefreq-bug: loc: @loc, chunk: @chunk, request_time: @request_time, time: @time, lastmod: @lastmod, changefreq: @changefreq',
            array(
              '@loc' => $link['loc'],
              '@chunk' => $chunk,
              '@request_time' => REQUEST_TIME,
              '@time' => time(),
              '@lastmod' => $link['lastmod'],
              '@changefreq' => $link['changefreq'],
            ),
            WATCHDOG_NOTICE,
            l(t('View sitemap'), 'sitemap-' . $chunk . '.xml')
        ); 
      }
    }

Here are some of the watchdog entries I got after I added this message

changefreq-bug: loc: node/16955, chunk: 8, request_time: 1253812201, time: 1253812261, lastmod: 1253812259, changefreq: -29

changefreq-bug: loc: node/16954, chunk: 8, request_time: 1253812201, time: 1253812261, lastmod: 1253812258, changefreq: -28.5

changefreq-bug: loc: node/16953, chunk: 8, request_time: 1253812201, time: 1253812261, lastmod: 1253812257, changefreq: -28

changefreq-bug: loc: node/16952, chunk: 8, request_time: 1253812201, time: 1253812261, lastmod: 1253812256, changefreq: -27.5

changefreq-bug: loc: node/16949, chunk: 8, request_time: 1253811721, time: 1253811729, lastmod: 1253811727, changefreq: -3

changefreq-bug: loc: node/16948, chunk: 8, request_time: 1253811721, time: 1253811729, lastmod: 1253811726, changefreq: -2.5

This is the set of corresponding records exported from the xmlsitemap table.

"id","type","subtype","loc","language","access","status","status_override","lastmod","priority","priority_override","changefreq","changecount"
"16948","node","marketwire_story","node/16948",,"1","1","0","1253811726","0.5","0","0","0"
"16949","node","marketwire_story","node/16949",,"1","1","0","1253811727","0.5","0","0","0"
"16952","node","globenewswire_story","node/16952",,"1","1","0","1253812256","0.5","0","0","0"
"16953","node","marketwire_story","node/16953",,"1","1","0","1253812257","0.5","0","0","0"
"16954","node","marketwire_story","node/16954",,"1","1","0","1253812258","0.5","0","0","0"
"16955","node","marketwire_story","node/16955",,"1","1","0","1253812259","0.5","0","0","0"

All of these correspond to new nodes programmatically created during cron run

All of these correspond to new nodes programmatically created during cron run

Which module do you use to programmatically create nodes on cron tasks?

REQUEST_TIME isn't set in the session. It's initialized at the beginning of every page request by the server. It should always be updating. REQUEST_TIME is *the* replacement to use for time() in PHP 5+.

@Dave - The watchdog call is logging both REQUEST_TIME (label request_time) and time() (label time). They seem to be different.

@KiamLaLuno - It is a custom module. I checked out the code and it is creating nodes using

drupal_execute('story_node_form', $submit, (object) $node);

There is a difference of values between REQUEST_TIME and the value returned by time(); that is because REQUEST_TIME is the Unix timestamp recorded from the server when it receives the request Drupal is currently handling, while time() returns the Unix timestamp for the moment the function is called. The difference should be minimal, though.

It could be of any help to see how the object $node is initialized.

The watchdog logs are coming from site generation, and REQUEST_TIME is the beginning of the request, so it stays the same. Because the generation is potentially a long process, time() of course is going to increment since it counts seconds. Let's take the first one for #21 and list the timestamps in order to see what's going on:

Start of execution (REQUEST_TIME) of cron.php (when the sitemaps are regenerated):
1253812201

{xmlsitemap}.lastmod value (since this is a new node, this is when the node was created)
1253812259

Current time() at code execution in xmlsitemap_cron/etc:
1253812261

I see what's happening now: If these nodes are being created in other module's cron runs using time() and not REQUEST_TIME, they're going to end up with a negative changecount value when it comes to run xmlsitemap_cron(). Because I don't want to call time() up to n number of records in {xmlsitemap}, that's why I'm using REQUEST_TIME.
?>

So maybe what we should do is not try and figure out a changefreq value for new things since we won't be able to tell how quickly they will be changing since they were just created. I bet if we changed the code to the following, it would fix this problem. anoopjohn could you try this out?

    if ($link['changefreq'] && $link['changecount']) {
      $link_output .= '<changefreq>' . xmlsitemap_get_changefreq($link['changefreq']) . '</changefreq>';
    }

The downside to the solution in #27 is if you have a node you created a year ago and has not been changed or commented on since (changefreq = 0 and changecount = 0) should get an appropriate changefreq tag. Maybe we should just check if $link['lastmod'] > REQUEST_TIME and skip the changefreq for things created during the same cron run as sitemap regeneration. Everything else we should have accurate data to output a changefreq.

@Dave - Can't another module change the node during the same cron run leading to a non-zero changecount?

The problem is that Drupal 6 doesn't use REQUEST_TIME when initializing the properties of a node object that contains a Unix timestamp. In example, node_object_prepare() uses the following code:

  if (!isset($node->nid)) {
    foreach (array('status', 'promote', 'sticky') as $key) {
      $node->$key = in_array($key, $node_options);
    }
    global $user;
    $node->uid = $user->uid;
    $node->created = time();
  }

node_save() uses the following code:

  // Set some required fields:
  if (empty($node->created)) {
    $node->created = time();
  }
  // The changed timestamp is always updated for bookkeeping purposes (revisions, searching, ...)
  $node->changed = time();

  $node->timestamp = time();
  $node->format = isset($node->format) ? $node->format : FILTER_FORMAT_DEFAULT

Normally, the difference between REQUEST_TIME, and time() is minimal, but when the node is created during cron tasks (which can last more than a normal Drupal page request), the difference between the two values is bigger.
If Drupal 6 would use REQUEST_TIME instead of time() then the difference between the values would be null.

@anoopjohn: Changecount is only incremented when a comment is added or a new node revision is created, not necessary whenever a node is updated. So only if that's the case, then we have valid data for creating a changefreq value. Otherwise, we should exclude trying to output the changefreq value (for nodes only added on the same cron run).

Easiest solution is to check if the condition $link['lastmod'] > REQUEST_TIME && !$link['changecount'] is TRUE, it means the node was added during the same cron run, and let's skip output changefreq. Easy as that.

Or I guess we could also do:

    if ($link['lastmod']) {
      $link_output .= '<lastmod>' . gmdate(DATE_W3C, $link['lastmod']) . '</lastmod>';
      // If the link has a lastmod value, update the changefreq so that links
      // with a short changefreq but updated two years ago show decay.
      $link['changefreq'] = (abs(REQUEST_TIME - $link['lastmod']) + $link['changefreq']) / 2;
    }
    if ($link['changefreq']) {
      $link_output .= '<changefreq>' . xmlsitemap_get_changefreq($link['changefreq']) . '</changefreq>';
    }

Another solution would be to use the absolute value of the difference between $link['lastmod'], and REQUEST_TIME.

Sometimes I really think I should start my own IRC channel #drupal-xmlsitemap or #drupal-seo. :)

I'm going to test both solutions and see which I prefer. I'm leaning towards using abs().

I have set up the abs() based solution on the server. I don't think the watchdog call will ever be called again unless of course the changefreq can go negative in some other ways. I will post back if I get any more errors.

Committed the abs() solution to CVS. Please post back your results. Marking as fixed for now. Thanks everyone for helping get to the bottom of this!

Any news about the test? I have also problems submitting my sitemaps...

Google Webasters Tools did accept my www.apartmentinrio.net/sitemap.xml, and after one day they changed status to "error", next day they changed it back to "OK"... None of my two sitemaps for Portuguese and Spanish have never had the status "OK", just error... :(

Please find my attachment..

Anyway great module!

Thank you!

Isak

@prisonbreaker82 The fix is working perfectly fine on my server. You might want to look at the exact error message that the webmaster tools shows you.

Hi Anoopjohn,

My sitemaps are located at and anonymous users can read the content...
www.apartmentinrio.net/sitemap.xml
www.apartmentinrio.net/es/sitemap.xml
www.apartmentinrio.net/pt-br/sitemap.xml

ERROR FROM GOOGLE WEBMASTERS TOOLS;

Error: The last attempt at downloading the Sitemap failed. The details below are representative of the last successful download. Resubmit?

Go to URLSee Sitemap
Format URLs in Sitemap Submitted Downloaded
Sitemap Total: 41, Indexed: 1
Oct 8, 2009 – Error
Oct 8, 2009
Sitemap errors and warnings
Line Status Details
Errors -
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.
Problem detected on: Oct 8, 2009

I am using PHP 5.2.9 with XML sitemap 6.x-2.x-dev (2009-Oct-03) at Bluehost.com.

Thank you!

@prisonbreaker82: For future reference, don't re-open fixed issues unless you are certain you are experiencing the exact same issue.

"Error: The last attempt at downloading the Sitemap failed. The details below are representative of the last successful download."

That means that Google couldn't actually get the sitemap content, not that anything was wrong with the data. Reasons might include if your site was in offline mode. I can access all three of the linked sitemaps myself, so you might want to try resubmitting it in Webmaster Tools and if it still reports an error, open a new issue.

Sorry David! I am new to Drupal and how to report...

I will try to re-submit the sitemaps. I made so changes, and would like to ask you if this can maybe solve the error... look at my attachment. I created three different sites for each language and added the sitemaps.

@prisonbreaker82: From what I can tell, everything looks like its working fine.