Problem/Motivation

An archive is meant to be kept for a long time and handed over to an e-archive, but it does not describe itself:

  • Nothing in the archive says which URL was archived, when, with which settings (for example the exclude patterns) and with which versions of the module and HTTrack, or how many URLs and broken links it has. That is only in the Drupal database, and is lost when the job is deleted.
  • There is no checksum, so nobody can verify later that the archive or the files in it are unchanged.
  • The index.html in the root of the archive is HTTrack's own project page, with one link for every start URL (thousands with Include all published content), HTTrack's images and no link to the reports of the module. The start page of the site is somewhere like www.example.com/index/index.html.

Proposed resolution

  1. Manifest: write earkiv-manifest.json to the root of the archive, with the job ID, the archived URL, when the crawl started and finished, the versions of the module, Drupal and HTTrack, the archiver settings, the counts of URLs, files, bytes, broken links, unlinked pages and published pages with errors, the path of the start page, and the names of the report files.
  2. Checksums of the files: write earkiv-checksums.sha256 with the SHA-256 checksum of every file in the archive, in the format of sha256sum, so that sha256sum -c earkiv-checksums.sha256 verifies an extracted archive.
  3. Checksum of the archive: compute the SHA-256 checksum of the .tar.gz file when it has been created, store it on the archive entity in a new field, show it in the archive list, and write it next to the stored file as earkiv_….tar.gz.sha256, which is removed with the archive.
  4. Start page: replace HTTrack's index.html in the root with a page that says which site was archived and when, links to the start page of the archived site, and links to the manifest and the reports. Remove HTTrack's images backblue.gif and fade.gif. The texts go through the translation system, like the notice on error pages.

Each part is a separate commit in the merge request.

Remaining tasks

  • Implement and test.
  • Review.

User interface changes

The archive list shows the checksum of each archive.

API changes

None.

Data model changes

A new base field checksum on the archive entity, installed by a post update function.

AI-Generated: Yes (Used Claude Code to review the archives and draft this issue.)

Issue fork earkiv-3636938

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

thomas alsén created an issue. See original summary.

  • thomas alsén committed 8cda261a on 1.0.x
    feat: #3636938 Add a manifest, checksums and a start page to the archive
    
thomas alsén’s picture

Status: Active » Fixed

Merged into 1.0.x!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.