pa:check bulk mode does not time out cleanly on large media libraries

Problem/Motivation

Running drush pa:check against a large media library, especially over a remote stream wrapper like s3fs, does not time out cleanly. Memory grows monotonically, remote reads have no upper bound on duration, the smalot parser can spin on malformed PDFs, image-heavy scans can decompress 50–100× and trigger uncatchable memory_limit fatals, and a sustained run of read failures (e.g. backend outage) looks identical to a healthy run. Field reproduction on a ~10K-PDF library locked the host VM hard enough to require a manual reboot after only ~750 items.

The bulk path needs several related fixes — bounded I/O and memory, structured failure classification, an outage-aware circuit breaker, queue-mode and incremental-resumption support, and machine-readable output — coordinated as one change.

Resolution

Safety nets

• Chunked iteration with memory reclaim. 50-item chunks; entity cache, static cache, and GC reset between chunks. Memory plateaus across long runs.
• Bounded I/O. 30s socket timeout around every read in PdfParserService. 60s pcntl_alarm wall-clock guard around Parser::parseContent().
• Three-way failure classification. _missing_file (orphan), _parse_error (permanent), _io_error (transient). A 5s read-elapsed threshold separates orphan 404s from real backend timeouts.
• Transient-only circuit breaker. Only _io_error counts toward the breaker; permanent failures and missing files don't trip it. Configurable via --max-consecutive-failures=N (default 10) and pdfa11y.settings:max_consecutive_io_failures.
• \Throwable catches in the analyzer's per-plugin and parser's outer paths so a PHP Error in one plugin is isolated to one STATUS_ERROR row instead of crashing the loop.

Bulk fitness

• Raw-bytes reuse. PdfParserService caches the most-recent file's bytes; HeadingStructureCheck's ObjStm fallback consults the cache before re-reading.
• Transactional storeResults(). DELETE + INSERTs in one transaction; parallel readers never see an empty window.
• Sentinel-aware --missing-only. A fid counts as checked when any row's check_id is not _io_error. Permanent sentinels (_too_large, _image_payload_too_large, _parse_error, _subprocess_failed, _missing_file) are definitive; only transient _io_error rows retry. Plus --since-fid=N and --status=published|unpublished|any.
• Queue mode. pa:check --queue enqueues into pdfa11y_check; drush queue:run pdfa11y_check drains one short-lived worker per item. Recommended pattern for full-library backfills.
• --skip-missing-files + pdfa11y.settings:skip_missing_files for runs where the orphan set is already characterized.

Out-of-memory guard

Field testing surfaced image-heavy PDFs whose stream decompression exceeds any reasonable memory budget. Three-part defense:

• Image-payload pre-flight. New PdfPreflightService walks the PDF's cross-reference table with bounded fseek/fread windows (~1–2 MB resident) and estimates decompressed image-payload weight via per-filter multipliers (DCTDecode/JPXDecode ×10, CCITTFaxDecode/FlateDecode/LZWDecode ×8, JBIG2Decode ×5). Files over threshold record _image_payload_too_large without invoking smalot. Configurable via --max-image-bytes=N / pdfa11y.settings:max_image_bytes; default 50 MB.
• File-size backstop. Coarse cap on raw source-file size for outliers and PDFs the preflight can't introspect. Configurable via --max-filesize=N / pdfa11y.settings:max_filesize; default 10 MB. Records _too_large.
• Subprocess isolation (two-layer fork-safety). Each parse runs in a forked child (pcntl_fork() + stream_socket_pair()); the parent owns all DB writes. Layer 1: the child sets memory_limit = parent_limit + subprocess_memory_headroom (default +128 MB) so PHP shutdown can run a register_shutdown_function() handler that SIGKILLs the child before any PDO destructor can write COM_QUIT to the inherited MySQL socket. Layer 2: on detected child failure the parent writes the _subprocess_failed sentinel via a freshly-opened DB connection so any in-flight corruption can't propagate. Configurable via --no-subprocess / pdfa11y.settings:use_subprocess_isolation and pdfa11y.settings:subprocess_memory_headroom; transparent no-op without pcntl.

Operator ergonomics

• --format=summary|table|csv|json|quiet. summary (one line per file) is the new bulk default; table remains the single-mid default. CSV/JSON accumulate and flush at end of run.
• In-memory result context. AccessibilityCheckResult gains optional readonly fid/mid/uri plus withContext() so CSV/JSON output identifies the file without joining file_managed.
• Categorized summary line with explicit buckets: checked, with-issues, missing files, too large, image-payload too large, subprocess failures, permanent parse errors, transient I/O errors, skipped-via-flag.
• README "Bulk operations" section. Documents the queue pattern, failure classification, breaker semantics, the calibration table for the OOM-guard thresholds, the subprocess-isolation two-layer model with tuning guidance, and the recommended chunked --missing-only operator flow.

Field test verification

Validated across three rounds on a real ~8,150-PDF s3fs-backed library on an Acquia ODE (PHP 8.3, 512 M CLI). Final round-3 result, one --limit=1500 chunk: 1,478 items checked to a clean summary line, 1 _subprocess_failed row (parent's MySQL connection survived, batch continued), 10× drop in subprocess-failure rate vs. round 2 (11 → 1). No "MySQL server has gone away" or "Packets out of order" cascade. Observed throughput ~25–30 items/min on this tier; full 10K-PDF coverage in 6–7 chunks of ~50–60 min each.

Deferred to follow-up issues

• uri column on pdfa11y_results — schema migration deferred to a follow-up.
• Distinct editor-facing message for encrypted PDFs (a FAILURE_ENCRYPTED kind with specific remediation copy).
• memory_get_peak_usage(true) in the bulk summary line for ops visibility on long runs.
• Admin report improvements: expose all stored sentinel + plugin data with drill-down filters.

Remaining tasks

• None.

User interface changes

New pa:check options: --max-consecutive-failures, --missing-only, --since-fid, --status, --queue, --format, --skip-missing-files, --max-filesize, --max-image-bytes, --no-subprocess. New default bulk output is one line per file; single-mid output unchanged. New configuration keys: max_consecutive_io_failures, skip_missing_files, max_filesize, max_image_bytes, use_subprocess_isolation, subprocess_memory_headroom.

API changes

AccessibilityCheckResult gains optional readonly fid/mid/uri and withContext(). PdfParserService gains public FAILURE_* constants, getLastFailureKind(), getRawBytes()/clearRawBytes(). Pdfa11yAnalyzer::analyze() gains an optional ?int $mid parameter; the class adds analyzeIsolated(), canFork(), warmAnalysisCaches(), resolveSubprocessMemoryLimit(), parseMemoryLimitToBytes(), storeResultsOnFreshConnection(), writeResults(), plus tooLargeResult() / imagePayloadTooLargeResult() / subprocessFailedResult() factories. New service pdfa11y.preflight (PdfPreflightService) with getImageStreamBytes(string $fileUri): ?int. New QueueWorker plugin pdfa11y_check. AccessibilityCheckInterface is unchanged. New result check_id sentinels: _missing_file, _io_error, _too_large, _image_payload_too_large, _subprocess_failed (plus existing _parse_error).

Data model changes

None. pdfa11y_results schema unchanged.

Environment

• Reproduces on pdfa11y 1.0.6, Drupal 10 / 11.
• Most severe with remote stream wrappers (s3fs); local-filesystem libraries hit the memory and parser-spin issues but not the read-timeout issue.
• Manifests on libraries large enough that one process can't complete the full run before exhausting the container.