Add a dunning framework: past_due state, scheduled retries, grace period, and lifecycle notifications

Related: #3619074: Succeeded charge records are terminal, so payer-initiated reversals (SEPA refunds, Bacs indemnity claims, chargebacks) have nowhere to land covers the case this framework must NOT handle like an ordinary failure: a payer-initiated reversal of an already-succeeded charge. Re-charging after a SEPA refund or a chargeback is prohibited or hostile on most schemes, so when that issue lands, the retry scheduler needs to suppress automatic retries for reversed charges — recovery routes through the notification events and the instrument-replacement flow instead. Worth keeping in mind while designing the retry/exhaustion machinery so the suppression hook has somewhere to attach.

Test-coverage note from the #3618739: Add a charge-based connector mode with a local billing cycle engine review: CycleWorker's failure paths are untested — the cron end-to-end test covers only the happy path, not the worker dropping deleted-subscription or malformed queue items, nor exceptions bubbling for requeue. Since this issue reworks what happens when charges fail, those tests should be the first commits here so current behavior is pinned down before it changes.

Design for this, resolving the open decisions in the summary and folding in #3 (reversal suppression) and #4 (worker tests first):

Parallel state field; status stays boolean

The "tri-state or parallel field" question, decided: status keeps meaning exactly one thing — does this subscription confer access — which stays genuinely two-valued, because past_due retains roles. Every existing consumer (the single-active invariant, the role machinery, due-queries, login redirect, every connector) reads it that way today and none of them needs to change. Dunning state is parallel, on the subscription: billing_state (current / past_due), past_due_since, next_retry, and retry_attempts, installed by update hook. The failed charge record remains the per-payment audit trail; the subscription carries the per-episode dunning bookkeeping.

Access and recovery are two independent tracks

This resolves the "after final failure or grace expiry" ambiguity by decoupling what grace and retries each govern:

• Grace governs access. At past_due_since + grace_period_days, a cron sweep (inline, like the expiry sweep) sets status = 0; the #3616929: Role revocation depends on a live plan lookup, so users can keep paid roles after expiry or cancellation snapshots revoke the roles. That is suspension.
• Retries govern recovery. Configurable offsets (default 3, 7, 14 days from the initial failure) re-attempt the charge on schedule — before or after suspension alike. Success at any attempt reactivates: status = 1, billing_state back to current, bookkeeping cleared, roles re-granted by the existing presave.
• Exhaustion ends the schedule, nothing else. A subscription can sit suspended-and-exhausted indefinitely; recovery then routes through instrument replacement plus a public retryCharge() — the failed → submitted transition #3618739: Add a charge-based connector mode with a local billing cycle engine reserved for this, and the hook a portal "Pay now" action will use.

Retries are new remote attempts on the same charge record

A retry transitions the failed charge back to submitted and re-submits through the connector, with a new attempt counter in the charge context alongside the record's charge_uuid: connectors derive a per-attempt idempotency key from the pair, since reusing the original key would make most processors dedupe the retry into a no-op. Due retries run through a dedicated queue worker with per-item isolation, mirroring the cycle queue.

Managed retries: an attribute key

managed_retries: TRUE on the connector attribute (the supported_modes pattern; defaults FALSE). The framework then schedules nothing: failures still enter past_due and grace still governs access, but the connector's own machinery (e.g. GoCardless Success+) owns the attempts and reports each outcome via resolveCharge(), which feeds the same state transitions.

A DunningManager service, shared by both rails

Dunning is not engine-internal — self-scheduling connectors need it too — so it lives in a subscription_manager.dunning service. The cycle engine calls it when a charge fails; cron calls its retry and suspension sweeps; and self-scheduling connectors' webhook handlers call it directly: markPastDue(), markRecovered(), suspend(). That is the concrete target for mapping Stripe's past_due/unpaid (and the remote-status normalization noted in the #3616779: GoCardless connector module (bank debit: Bacs, SEPA, ACH, PAD) comments).

Notifications: events first, mails optional

Four dispatched events — entered past_due, retry failed (carrying the attempt number and an exhausted flag), suspended, recovered — so sites attach event subscribers. (ECA reacts through its own event plugins, so ECA integration would be a small optional sub-module in a separate issue; the events themselves are the API and need no change for it.) Bare-bones mails ship in-module behind a default-off setting, linking the customer to the manage-billing page, where instrument replacement lives once a connector implements it.

Reversal suppression (#3)

Retry scheduling happens only in the failure-entry path (enterDunning(), called when a charge attempt fails). The #3619074: Succeeded charge records are terminal, so payer-initiated reversals (SEPA refunds, Bacs indemnity claims, chargebacks) have nowhere to land reversed state never calls it, so reversed charges get notification-and-replacement recovery with no automatic re-charging — the attachment point #3 asked for, by construction rather than by flag.

Settings, scope, tests

Settings: retry_offsets (sequence of day offsets), grace_period_days, send_mails, with schema, admin-form section, and config-defaults update hook. Past-due indication in the portal: a notice on the billing-history page and the state exposed in the my-subscription API response. Per #4, the first commits pin CycleWorker's current failure-path behavior (deleted and malformed items dropped, exceptions bubbling for requeue) before any of it is reworked. Kernel tests: grace timing and role retention through past_due, retry sequencing and per-attempt idempotency context, exhaustion, reactivation on late success, managed-retries deferral, suspension and revocation, the webhook-facing service methods, and the mail toggle.

Implemented in the MR, per the design in #5, in ten commits.

Worker tests first (#4): CycleWorkerTest pins the queue worker's current failure paths — unusable items dropped, exceptions bubbling, a poison item not blocking the queue — before anything else changes.

Data model and settings. status stays boolean and keeps governing access; the subscription gains parallel dunning bookkeeping — billing_state (current/past_due), past_due_since, next_retry, retry_attempts — via update 10021. Settings gain dunning.retry_offsets (days from the initial failure, default 3/7/14), dunning.grace_period_days (default 21) and dunning.send_mails (default off) via update 10022, with a Dunning section on the settings form. Connectors may declare managed_retries on the attribute or annotation, and the charge context now carries an attempt counter alongside charge_uuid so connectors derive a per-attempt idempotency key (documented on ChargeBasedConnectorInterface::charge()).

DunningManager (subscription_manager.dunning) owns the lifecycle for both rails. The engine reports every failure to recordFailure(): the first enters dunning (PAST_DUE), later ones raise RETRY_FAILED with the attempt number and an exhausted flag, and the next retry is scheduled from the offsets — none for managed-retries connectors. This is the only path that schedules retries, so the #3619074: Succeeded charge records are terminal, so payer-initiated reversals (SEPA refunds, Bacs indemnity claims, chargebacks) have nowhere to land reversed state simply never calls it (the suppression point from #3, by construction). Every success calls markRecovered(): back to current, reactivated if suspended, RECOVERED. Cron enqueues due retries into a dedicated queue worker and runs the suspension sweep: grace expiry sets status off (the #3616929: Role revocation depends on a live plan lookup, so users can keep paid roles after expiry or cancellation snapshots revoke roles; SUSPENDED) while the retry schedule continues, so a late success reactivates. markPastDue(), markRecovered() and suspend() are public for self-scheduling connectors' webhook handlers — the Stripe past_due/unpaid mapping target.

CycleEngine::retryCharge() re-submits a failed charge as a new remote attempt (record back to submitted, next attempt number in the context); the outcome flows through the normal result path. It is the dunning retry path and the hook for a future instrument-replacement or "pay now" action. Found and fixed along the way: while a subscription is past due its anchor stays in the past, so the cycle used to re-enqueue it every cron run and open a second charge for the same period; the cycle now stands down during dunning.

Notifications. Four dispatched events (past due, retry failed, suspended, recovered) for subscribers; the optional built-in mails ship behind the default-off setting, linking to the manage-billing page. The billing-history page shows a past-due notice; the my-subscription API already exposes billing_state.

Money-adjacent hardening from a coverage audit. Two things the extra tests caught: duplicate retry queue items (cron running twice before the queue drained) produced a second, off-schedule attempt against the customer's instrument — the worker now re-checks that a retry is due, which with beginRetry() clearing next_retry in flight also guards overlapping attempts; and SubscriptionChargeEntity::getAmount() now returns a canonical two-decimal string, since the decimal field drops trailing zeros on load and a retry was forwarding '15.5' where the original sent '15.50'. One policy decision made in review: plan changes are refused while a subscription is past due (an upgrade would submit a fresh proration charge against a failing instrument, and either direction would swap the plan under a failed charge) — the customer fixes payment first.

Tests. DunningTest (fourteen scenarios: entering dunning with roles retained, retry sequencing with attempt numbers and exhaustion, recovery on retry success with anchor advance, grace expiry suspension and late-success reactivation, managed-retries deferral with webhook recovery, the webhook-facing methods, mails following the setting, retry refusing non-failed charges, duplicate queue items retrying once, asynchronous retries resolved either way, a pending initial charge failing by webhook, retries re-submitting the original amount regardless of intervening usage changes, an empty retry schedule, the worker's drop paths, a retry whose connector call throws, retry refused for a non-charge-based connector, every mail's wording and the no-address skip, the settings form normalizing the schedule, the newest of several failed charges being retried, and a failed proration pausing period billing until recovery) plus CycleWorkerTest. Suite is at 128 kernel tests / 1687 assertions; updates 10021–10022 verified on a live site.

Change record drafted.