I have about 15 unpublished pages that are still appearing in the sitemap resulting in Google crawl errors. I've confirmed that not only are the pages unpublished but they are not excluded from menus as well. I can specifically exclude the node from being in the sitemap and that works. But if a page is unpublished it shouldn't appear in the sitemap.

In XML Sitemap settings, if a node is not specifically excluded from the sitemap, it is definitely showing up as available/indexed/visible even if it's unpublished.

Comments

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented
Status: Active » Postponed (maintainer needs more info)

I can't confirm this. Do you have any node access modules running? Were these nodes published at one point, but the sitemap files haven't regenerated since you unpublished them?

Lloyd’s picture

Lloyd commented

I do have content access running and these nodes were published previously. However, I rebuilt the sitemap and the unpublished nodes still appeared. And I removed them from the sitemap by changing their XML setting to disabled, re-built the sitemap (they were not listed at that point), and then changed their setting back to enabled and despite still being unpublished they showed back up in the sitemap. Really odd.

So I then created a new page and did not publish it from the very beginning. Wasn't in the sitemap as expected. I published it, manually ran cron to regenerate the sitemap, and it showed up. Fine. I then went back and unpublished the page (which had no content access restrictions), ran cron again, and the page still showed up in the sitemap. I made sure that the sitemap wasn't being cached.

Anonymous’s picture

Anonymous (not verified) commented

I've confirmed that not only are the pages unpublished but they are not excluded from menus as well.

What do you mean by this?

Fine. I then went back and unpublished the page (which had no content access restrictions), ran cron again, and the page still showed up in the sitemap. I made sure that the sitemap wasn't being cached.

But was the page and it's access still cached for anonymous users?

Lloyd’s picture

Lloyd commented

Oh sorry, should have read pages are unpublished and they ARE excluded from the menus. And they aren't part of any taxonomy. So there's no reason they should still be in the sitemap.

No, the page is not part of any cache. It is specifically excluded from being cached.

Fire away any questions you have. This is weird behavior and I'm happy to troubleshoot as much as possible.

Anonymous’s picture

Anonymous (not verified) commented
I then went back and unpublished the page (which had no content access restrictions), ran cron again, and the page still showed up in the sitemap.

Did you admin/settings/performance->Clear cached data before running cron?

Lloyd’s picture

Lloyd commented

I went back and created a new test page and had the same, repeatable experience.

1) Created as unpublished, ran cron - not in sitemap
2) Changed to published, ran cron - in sitemap
3) Changed to unpublished, cleared cache, ran cron - still in sitemap

I'll check a little later and see if I can repeat this on one of my other Drupal sites.

tomsm’s picture

tomsm commented

I have just installed xml sitemap.
I have 3 sitemaps, one for each language: English, Dutch and French

The nodes that are unpublished also appear in the sitemaps.
They were already unpublished before I installed the XML sitemap module.

I use version 6.x-2.0-beta1 and Content Access 6.x-1.2

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

My original question was never answered: Anyone with this problem, do you have any node access or moderation modules enabled?

tomsm’s picture

tomsm commented

Yes, I have Content Access 1.2 enabled (node access). And I enabled CCK content permissions (cck field access).

Lloyd’s picture

Lloyd commented

I have Content Access enabled as well.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented
Assigned: Unassigned » dave reid
Status: Postponed (maintainer needs more info) » Active

Ok, awesome. I think I've narrowed it down to what's wrong. Since the module is still in beta this doesn't require an SA since technically this is information disclosure.

Lloyd’s picture

Lloyd commented

Fantastic. Thanks for the quick response.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented
Status: Active » Fixed

Committed a fix in http://drupal.org/cvs?commit=372196. It should be available in 6.x-2.x-dev release now if you can try it out. Please report back on success/failure.

Lloyd’s picture

Lloyd commented

Cool. I'll install and test it during a maintenance cycle this weekend.

tomsm’s picture

tomsm commented

I just tested the new dev. The unpublished nodes are not included in the sitemap. So it works.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

Great, thank you very much for testing and reporting back. Sorry for the bug it wasn't something easily caught since it required a non-core node access module. :(

shaundychko’s picture

shaundychko
Location Vancouver, Canada
commented

Thank you Dave, I ran into this issue using Taxonomy Access Control, and I can confirm that the 2.x-dev version fixed it. And I'm impressed that xmlsitemap also respects access control (private nodes are not listed in the sitemap even when the content type is enabled by default)!

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

finex’s picture

finex commented

I can confirm the -dev version has fixed the bug (reproducible in beta1)