Problem/Motivation
HTTrack runs in one thread and waits 100 ms between its rounds over the connections, reading only a little from each. It fetches about 0.1 URLs per second per connection, however fast the server answers, so the number of connections decides how long the crawl takes. Measured on a site with about 7,400 URLs and 2.9 GB:
| Connections | Requests per second | Time |
|---|---|---|
| 8 | 1.5 | 83 min |
| 32 | 3 | 38 min |
| 64 | 6.5 | 19 min |
| 128 | 13 | 10 min |
The result was the same with 64 and 128 connections. Today:
- The module allows at most 32 connections, and higher values are lowered to 32 without a warning.
- Only the form field has a minimum and maximum. From
drush earkiv:create, values such as0,-1andabcare saved. - The field does not say what load a number of connections gives, which makes it hard to choose a value for a production server.
--connection-per-secondis set to five times the number of connections but does not limit anything in practice.
Proposed resolution
- Raise the maximum to 128.
- Validate the number in
validateConfigurationForm()as a whole number from 1 to 128, so that drush gets the same check as the form. - If a stored configuration is outside the range, for example after a configuration import, use the nearest allowed value and show and log a warning instead of changing it silently.
- Describe the load in the field description and the README, with a recommendation for production servers: start with 16–32 outside office hours and watch the server load before going higher, and use higher values against an internal address or a copy of the site.
- Remove
--connection-per-second, and correct the comment about what limits the speed.
Remaining tasks
- Review.
User interface changes
"Simultaneous connections" accepts up to 128 and has a longer description with a recommendation.
API changes
None. HTTrackArchiver::MAX_CONNECTIONS changes from 32 to 128.
Data model changes
None.
AI-Generated: Yes (Used Claude Code to measure the crawl and draft this issue.)
Issue fork earkiv-3636844
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #4
thomas alsén commentedMerged into 1.0.x!