Add start URLs and sitemap.xml as crawl entry points

Problem/Motivation

The HTTrack archiver only finds pages that another page links to. Published pages that are only reachable from the site search, a listing that links directly to a file, or nowhere at all are left out of the archive without any warning.

On a Drupal 11 site with 1,984 published nodes, a full archive was missing 417 of them. 284 of the missing pages are listed in the sitemap.xml generated by Simple XML Sitemap, among them all 261 remittance pages: their listing links straight to the PDF of each response, so the pages themselves, and 32 documents that are only attached to them, never get crawled.

Proposed resolution

Add two settings to the HTTrack archiver:

• Include URLs from sitemap.xml (checkbox, off by default). Before the crawl, read sitemap.xml at the archived URL, follow a sitemap index if there is one, and use every listed URL as a start URL.
• Additional start URLs (textarea, one URL per line).

Write the start URLs to a file in the working directory and pass it to HTTrack with -%L, so the rest of the crawl works as before and the exclude patterns still apply.

Sitemaps often list the production host even on test and local environments (Simple XML Sitemap uses its configured base URL). Sitemap URLs on another host are therefore rewritten to the scheme and host of the archived URL, keeping the path and query.

Remaining tasks

• Implement and test.
• Review.

User interface changes

Two new settings for the HTTrack archiver: "Include URLs from sitemap.xml" and "Additional start URLs".

API changes

None.

Data model changes

None. The HTTrack archiver configuration gets the optional keys sitemap and start_urls.

AI-Generated: Yes (Used Claude Code to analyse a full archive, draft this issue and write the merge request.)