Problem/Motivation

Editors often write links as full URLs to the production host, for example https://www.example.com/sites/default/files/2023/08/report.pdf, or to other names of the same site, such as example.com without www or http://. HTTrack treats every other host name as another site:

  • When a test or local copy of the site is archived, all links to the production host stay absolute and lead to the live site instead of the archived page or file. Files that are only linked this way are not archived at all.
  • When production is archived, links to the other names of the site (without www, or with http) are left out in the same way.

In a full archive of a Drupal 11 site made from a local copy, 1,016 links pointed to the production host. 637 of them pointed to pages or files that were in the archive under the archived host; the others, mostly documents and old addresses that redirect, were not archived.

Proposed resolution

Add an Alternative host names setting to the HTTrack archiver: host names that belong to the same site, one per line. After the crawl, before broken links are handled and the archive is created:

  1. Find links (href and src) in the archived HTML pages that point to an alternative host name, with any scheme. Move them to the scheme and host of the archived URL, and leave out URLs that match an exclude pattern.
  2. Fetch the ones that are not yet in the archive in a short second HTTrack run with only those URLs, and merge the result and its URL log into the archive.
  3. Rewrite links to the alternative host names, and absolute links to the archived host, to relative links when the target is in the archive. Links to anything that is not in the archive are left as they are.

Broken links found in the second run end up in earkiv-broken-links.csv like the others.

Remaining tasks

  • Implement and test.
  • Review.

User interface changes

New setting "Alternative host names" for the HTTrack archiver, and a matching --alias-host option for drush earkiv:create.

API changes

None.

Data model changes

None. The HTTrack archiver configuration gets the optional key alternative_hosts.

AI-Generated: Yes (Used Claude Code to analyse archives, draft this issue and write the merge request.)

Issue fork earkiv-3636718

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

thomas alsén created an issue. See original summary.

thomas alsén’s picture

Status: Active » Fixed

Merged into 1.0.x!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.