Archiving is slow and the archive can be incomplete

Problem/Motivation

Archiving a medium-sized Drupal 11 site (about 2,000 published nodes and 9,500 files) with the HTTrack archiver takes several hours, and the resulting archive is incomplete. Measured in DDEV against the site itself:

1. HTTrack fetches about one URL per second, regardless of how fast the server is. Cached pages answer in 10 to 30 ms, even with 16 parallel requests. By default HTTrack checks the content type of every link without a file extension (every page on a Drupal site) and waits for the answer before handling the next link. Changing the number of connections or disabling the security limits has no effect on its own.
2. Files are missing from the archive. The archive is built with --userdef-cmd "tar --append ... \$0", one tar call per file. After a 60 minute crawl the tar file held 1,040 entries while HTTrack had fetched 4,652 URLs. $0 is not quoted, so file names with spaces are lost.
3. Appending to the tar file gets slower for every file. tar --append reads all existing headers to find the end of the archive. For 6,000 files this takes 23 s compared with 32 ms for a single tar -cf, and the cost grows with the square of the number of files.
4. Drush uses 100% of a CPU core while waiting for HTTrack and gzip (while ($process->isRunning()) without a pause).
5. Jobs can get stuck and block all later jobs. If the process is killed, the job stays in RUNNING_ARCHIVE and jobInProgress() returns TRUE forever. A FileException in the storage step is logged but leaves the job in RUNNING_STORE. Two runs started at the same time (for example from cron) can process the same job.
6. Resume starts over. A resumed job creates a new empty tar file and runs --mirror again instead of continuing from the HTTrack cache.
7. No traceability and no clean-up. --do-not-log removes the HTTrack log, and the working directory in the temporary directory is never deleted.
8. No way to exclude URLs. On the test site one sortable table (7 columns, 2 directions, 150 pages) produced 4,966 URLs and 70% of the crawl time, while every variant contains the same rows.

Proposed resolution

• Run HTTrack with --check-type=0 -%N0 --disable-security-limits and a configurable number of connections. In tests this made the crawl about four times faster with the same content.
• Remove --userdef-cmd and create the .tar.gz in a single tar call after the crawl, leaving out the HTTrack cache and unfinished *.tmp files but keeping hts-log.txt and hts-cache/new.txt.
• Wait for processes with a pause instead of a busy loop.
• Hold a lock while processing. A job still marked as running when the lock is free was abandoned and is resumed. Mark storage failures for resume.
• Resume an interrupted crawl with --continue.
• Add --timeout and --retries so stalled connections are abandoned.
• Delete the working directory once the archive is created.
• Add an "Exclude URLs" setting to the HTTrack archiver for crawler traps such as sortable tables and faceted search.

Remaining tasks

• Review.
• Test on a production-sized site.

User interface changes

Two new settings for the HTTrack archiver: "Simultaneous connections" and "Exclude URLs".

API changes

None. EArkivCommands and HTTrackCommands get the lock service as a new constructor argument.

Data model changes

None. The HTTrack archiver configuration gets the optional keys connections and exclude.

AI-Generated: Yes (Used Claude Code to measure the performance problems, draft this issue and write the merge request.)