Problem/Motivation

HTTrack runs in one thread and waits 100 ms between its rounds over the connections, reading only a little from each. It fetches about 0.1 URLs per second per connection, however fast the server answers, so the number of connections decides how long the crawl takes. Measured on a site with about 7,400 URLs and 2.9 GB:

Connections Requests per second Time
8 1.5 83 min
32 3 38 min
64 6.5 19 min
128 13 10 min

The result was the same with 64 and 128 connections. Today:

  • The module allows at most 32 connections, and higher values are lowered to 32 without a warning.
  • Only the form field has a minimum and maximum. From drush earkiv:create, values such as 0, -1 and abc are saved.
  • The field does not say what load a number of connections gives, which makes it hard to choose a value for a production server.
  • --connection-per-second is set to five times the number of connections but does not limit anything in practice.

Proposed resolution

  • Raise the maximum to 128.
  • Validate the number in validateConfigurationForm() as a whole number from 1 to 128, so that drush gets the same check as the form.
  • If a stored configuration is outside the range, for example after a configuration import, use the nearest allowed value and show and log a warning instead of changing it silently.
  • Describe the load in the field description and the README, with a recommendation for production servers: start with 16–32 outside office hours and watch the server load before going higher, and use higher values against an internal address or a copy of the site.
  • Remove --connection-per-second, and correct the comment about what limits the speed.

Remaining tasks

  • Review.

User interface changes

"Simultaneous connections" accepts up to 128 and has a longer description with a recommendation.

API changes

None. HTTrackArchiver::MAX_CONNECTIONS changes from 32 to 128.

Data model changes

None.

AI-Generated: Yes (Used Claude Code to measure the crawl and draft this issue.)

Issue fork earkiv-3636844

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

thomas alsén created an issue. See original summary.

  • thomas alsén committed 497e9162 on 1.0.x
    feat: #3636844 Allow up to 128 connections and recommend a value for...
thomas alsén’s picture

Status: Active » Fixed

Merged into 1.0.x!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.