Problem/Motivation

Archiving a site takes from minutes to hours, but there is little feedback while it runs:

  • drush earkiv:process shows a spinner with "Crawling..." and updates the number of crawled URLs once a minute. While the archive is compressed it only shows "Compressing...".
  • The archive list at /admin/content/earkiv only shows the status, for example "Creating archive". It does not show how far the job has come or whether it is still alive. Jobs usually run from cron, where nobody sees the Drush output.
  • When a job is done, nothing records how many URLs were crawled, how many links were broken or how long it took.

This is listed in the README as "Think about how we can improve feedback during the archiving run".

Proposed resolution

  • Progress record. While a job runs, store its progress in the State API, every 10 seconds and at each phase change: phase (crawling, handling error pages, compressing, storing, done, failed), crawled URLs, start URLs, broken links, archive size, start time and last update. No schema change is needed. The record is deleted with the archive.
  • Drush during the run. Update the progress line every 5 seconds, for example "Crawling: 1,234 URLs (1,697 start URLs), 12 min, 1.7 URLs/s", and show the growing archive size while compressing.
  • New drush earkiv:status command. A table of the jobs with status, phase, URLs, broken links, size, running time and last update, with a warning when a running job has not been updated for 5 minutes.
  • Archive list. Show the progress below the status of a running job, for example "Creating archive: 1,234 URLs, 12 min", and a summary for a finished job, for example "Done: 7,174 URLs, 72 broken links, 1 h 24 min". Reload the list every 30 seconds while a job is running.
  • Log. Write a summary to the Drupal log when an archive is done.

Remaining tasks

  • Implement and test.
  • Review.
  • Remove the item from the README to-do list.

User interface changes

The archive list shows progress and a summary in the status column and reloads while a job is running. New Drush command earkiv:status.

API changes

None.

Data model changes

None. Progress is stored in the State API under one key per archive.

AI-Generated: Yes (Used Claude Code to draft this issue and write the merge request.)

Issue fork earkiv-3636586

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

thomas alsén created an issue. See original summary.

thomas alsén’s picture

Status: Active » Fixed

Merged into 1.0.x!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.