A few tests with concurrent requests via GuzzleHttp\Pool show notable performances improvments.
MR is following.

Issue fork sitemap_url_checker-3594593

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

nicolasgraph created an issue. See original summary.

nicolasgraph opened merge request !5

nicolasgraph’s picture

nicolasgraph
Primary language French
Location Strasbourg
commented
Status: Active » Needs work

Speeds up the URL checker by issuing HTTP requests concurrently instead of one at a time, and exposes the concurrency level as configuration.

Performance changes

  • Concurrent URL checking. SitemapUrlCheckerForm::processLinks() previously checked each sitemap URL with a sequential blocking HEAD request (checkUrl()), so total runtime was the sum of every request's latency. Replaced with a GuzzleHttp\Pool (checkUrls()) that runs up to N requests in parallel. Per-request options (timeout, connect_timeout, http_errors => FALSE, allow_redirects) and the existing behaviour — failed requests reported as 500 and logged — are preserved.
  • Concurrent sub-sitemap loading. For a sitemap index, extractLinks() previously fetched each nested sitemap sequentially. Nested sitemaps are now downloaded in parallel via a new loadXmlMultiple() pool; extractUrlsFromSitemap() was refactored to operate on an already-parsed \SimpleXMLElement (load vs. parse responsibilities separated).
  • Batch chunk aligned to concurrency. The Batch API chunk_size was tied to the configured concurrency so each batch step feeds the pool a full wave (it was hardcoded to 5, which would have throttled the pool).

New configuration

  • Added a settings form (SettingsForm, ConfigFormBase) at /admin/config/search/sitemap-urlchecker/settings with a single "Maximum concurrent requests" field (default 10, range 1–50), gated by administer site configuration.
  • Backing config sitemap_url_checker.settings:concurrency with config/install default and config/schema definition.
  • Added the configure: key to *.info.yml and a menu link, so the module exposes a Configure link.

MR saved as draft for now as it needs more tests on my side.
Observed sitemap checking time is 4min30s for 370 URLs with a concurrency set to 10, against 8min45s for the current module version.

nicolasgraph’s picture

nicolasgraph
Primary language French
Location Strasbourg
commented
Assigned: nicolasgraph » Unassigned
Status: Needs work » Needs review

Work as expected on one of our websites.

cmarrufo’s picture

cmarrufo commented

Hi @nicolasgraph,

I find this very interesting.

I will test it in one of our site that has a very amount of content (around 13.5k of urls hehe).

Thank you very much!!

  • cmarrufo committed a00bd56f on issue--3594593-performance 
    Issue #3594593: Add checker settings and improve checker performance

  • cmarrufo committed 9bc88cf2 on issue--3594593-performance 
    Issue #3594593: Add checker settings and improve checker performance

  • cmarrufo committed bf6b139e on 1.0.x 
    Issue #3594593: Add checker settings and improve checker performance

By...
cmarrufo’s picture

cmarrufo commented
Status: Needs review » Reviewed & tested by the community
cmarrufo’s picture

cmarrufo commented
Status: Reviewed & tested by the community » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

cmarrufo closed merge request !5

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.