Problem/Motivation

A review of the module, with test runs against a Drupal 11 site, found five bugs that can give a broken or incomplete archive, or leave large files behind:

  1. A crawl that fails completely is marked as done. A host that cannot be reached gave an archive with 0 URLs, and a start page that returned 404 gave an archive with only that error page. Both were marked "Done" without a warning, since HTTrack exits with status 0 in both cases.
  2. A leftover HTTrack process gives an incomplete archive. If only the Drush process dies, HTTrack keeps running. When the job is resumed, a second HTTrack is started in the same directory. The result was an archive with 613 of 702 URLs and 188 dead internal links, marked "Done".
  3. drush entity:delete earkiv_archive leaves the archive behind. The archive file is only deleted in an overridden Archive::delete(), which the delete form calls, but not entity:delete or bulk deletion, which delete through the storage. The file entity, the file (often several GB) and the working directory in the temporary directory are left. The README and the message of earkiv:cancel recommend that command. The working directory is not removed either when a job ends with an error.
  4. The forms save the plugin settings unprocessed. The settings form and the form for a new archive do not call submitConfigurationForm() of the plugins. A storage directory with a trailing slash gave the file private://dir//earkiv_….tar.gz, which cannot be downloaded, not even by user 1. Exclude patterns are saved with \r\n and the number of connections as a string. The module has no configuration schema.
  5. Error pages with another file extension are deleted. An e-mail address entered as a page link (info@example.se) is saved by HTTrack as index.se, and a host without a scheme (www.example.com) as index.com. The module decides from the file extension whether an error page is HTML, so these error pages are deleted and the links lead to a missing file. That was 16 of 72 broken links in a full crawl.

Steps to reproduce

  1. Create an archive of a host that cannot be reached, or whose start page returns 404. The job ends as "Done".
  2. Kill only the Drush process during a crawl and resume the job. Two HTTrack processes write to the same directory.
  3. Run drush entity:delete earkiv_archive ID. The archive file is left in the storage directory.
  4. Set the storage directory to earkiv/ on the settings page and create an archive. The file cannot be downloaded.
  5. Add a link to info@example.se without mailto: to a page and create an archive. The link leads to a missing file.

Proposed resolution

  1. After the crawl, check the URL log. If no URL returned a successful response, or the start URL returned an error, fail the job (with the usual resume attempts) instead of creating the archive.
  2. Before HTTrack is started or continued, stop any HTTrack process that still runs in the working directory of the job.
  3. Delete the archive file, the working directory and any temporary archive in Archive::preDelete() or postDelete(), so that every way of deleting an archive removes them. Remove the working directory when a job ends with an error.
  4. Call submitConfigurationForm() of the plugins in the settings form and the archive form, trim slashes from the storage directory, and add a configuration schema.
  5. Decide whether an error page is HTML from the MIME type in the URL log instead of the file extension. Save the notice for error pages with another extension as an HTML file and point the links to it.

Each fix is a separate commit in the merge request.

Remaining tasks

  • Implement and test.
  • Review.

User interface changes

A failed crawl is shown as an error instead of "Done".

API changes

None.

Data model changes

A configuration schema for earkiv.settings.

AI-Generated: Yes (Used Claude Code to review and test the module and draft this issue.)

Issue fork earkiv-3636849

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

thomas alsén created an issue. See original summary.

  • thomas alsén committed 47a01140 on 1.0.x
    fix: #3636849 Fix five bugs that can give a broken or incomplete archive
    
thomas alsén’s picture

Status: Active » Fixed

Merged into 1.0.x!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.