Archive published pages that no other page links to

Problem/Motivation

HTTrack only finds pages that another page links to. Published pages that nothing links to, for example thank-you pages, campaign pages and old result pages, are left out of the archive. The Include URLs from sitemap.xml option helps only when the sitemap is complete, which it often is not: it depends on how the sitemap module is configured and when it was last generated.

In a full archive of a Drupal 11 site with 1,984 published nodes, 167 were not in the archive. About a dozen of them were real pages that no page links to. The sitemap had 600 URLs and lacked most of them. The others were nodes that are not meant to be shown as pages: 22 nodes that Rabbit Hole answers with "Page not found", 1 node that Rabbit Hole redirects, 103 contact persons shown inside other pages (whose Rabbit Hole setting is a matter of site configuration), and the nodes used as the 403 and 404 pages.

Proposed resolution

Add an Include all published content option to the HTTrack archiver, next to the sitemap option. When the crawl starts, add the URL of every node that is a page of its own to the start URLs:

1. Published nodes that the anonymous user may view, in every language, with the URL from $entity->toUrl() for each translation so that path aliases and language prefixes are right.
2. If the Rabbit Hole module is enabled, only nodes whose Rabbit Hole action is display_page. Use the rabbit_hole.behavior_invoker service (getRabbitHoleValuesForEntity()), so that both the bundle setting and an override on the node are respected, and only for entity types that Rabbit Hole handles (getPossibleEntityTypeKeys()). Rabbit Hole stays optional: check it with moduleExists('rabbit_hole'), and without it every published node that the anonymous user may view is included.
3. Leave out the nodes set as the 403 and 404 pages in system.site (page.403, page.404), and URLs that match an exclude pattern.
4. Move the URLs to the scheme and host of the archived URL, as is done for the sitemap.

Pages that are already linked are not fetched twice, since HTTrack handles every URL only once. Start URLs that answer with an error are not counted as broken links, since no page links to them.

Add earkiv-unlinked-pages.csv to the archive: the pages that were only archived because of this option, with the node ID, title and content type. It shows editors which pages nothing links to.

Remaining tasks

• Implement and test, with Rabbit Hole 1.x, Rabbit Hole 2.x and without Rabbit Hole.
• Review.

User interface changes

New option "Include all published content" for the HTTrack archiver, and a matching option for drush earkiv:create.

API changes

None.

Data model changes

None. The HTTrack archiver configuration gets the optional key published_content.

AI-Generated: Yes (Used Claude Code to analyse the archive and the Rabbit Hole settings, and to draft this issue.)