Incident model: allow human intervention on dead-lettered tokens instead of always failing the instance

Concrete kernel-level implementation plan, building on the dead-letter primitives already shipped (token STATUS_ERROR, the per-token attempts counter, WorkflowEngine::advanceQueued() and handleAdvanceFailure()).

Design decision

The instance stays RUNNING while incidents are open; "in incident" is driven by open incident records, not a hard status that freezes healthy branches. Other branches keep advancing, so advance()'s isRunning() guard is unchanged. A derived STATUS_INCIDENT may be added for display only.

New state

• Optional ProcessInstanceInterface::STATUS_INCIDENT (display/derived).
• Token STATUS_ERROR and the attempts counter already exist (audit #13).

New kernel entity orchestra_incident

Execution state, so kernel-owned (its own content entity, not fields on ProcessInstance): instance ref, token ref, node_id, message, attempts, status (open|resolved), resolution (retried|skipped|resumed|cancelled|failed), resolved_by, created/changed. Deleting an instance deletes its incidents, mirroring InstanceCleanup.

Engine: branch handleAdvanceFailure at the cutoff

if (attempts at limit) {
  token->setStatus(STATUS_ERROR)->save();
  policy === 'fail' ? failInstance(...) : raiseIncident(token, e);
  return; // swallow
}

raiseIncident() creates the incident row and emits an audit event, but does NOT call cancelLiveTokens(), so the other branches keep running. Policy comes from an orchestra.settings key on_unrecoverable_failure (incident|fail, default incident), overridable per node.

checkCompletion respects open incidents

Completion now requires zero live tokens AND zero open incidents, so an instance with a stuck branch does not wrongly complete; it halts awaiting resolution.

Recovery API (new WorkflowEngineInterface methods)

Operator-driven, behind a new "resolve orchestra incidents" permission:

retryIncident(incident)   // attempts=0, token ACTIVE, enqueue(token), resolve(retried)
resumeIncident(incident, variables) // setVariable() each, then retry
skipIncident(incident)    // produce successors WITHOUT running the task, resolve(skipped)
cancelIncident(incident)  // cancel() the token subtree, resolve(cancelled), checkCompletion()
failFromIncident(incident)// explicit failInstance() (today's behavior, now a choice)

All reuse existing primitives (enqueue, setVariable, cancel, failInstance) except skip.

The one refactor: skipNode

advanceNode() is join then task then split. Extract the successor-production tail into produceSuccessors(); advanceNode() calls it after the task, skipIncident() calls it directly (in the same transaction advance() uses). Split/flow-condition/fork logic is unchanged.

Supporting

• Audit: new orchestra.incident channel (raised, retried, skipped, resumed, cancelled, failed) via the existing emitter; the orchestra_audit_trail bridge picks it up.
• UI/Views: an incidents view in orchestra_views (tenant-scoped by ViewsTenantScope already); per-incident action buttons in orchestra_ui; optionally surface incidents in the inbox.
• Companion: promote MAX_ADVANCE_ATTEMPTS to per-node config (max_attempts, backoff).
• Documentation: a new docs/incidents.md plus mkdocs nav, a roadmap entry, and cross-links from docs on timers/distributed execution; update the French (and other) translations accordingly.

Migration and BC

New orchestra_incident table and a new permission. Pre-1.0 ships no update hook (fresh installs get it; live sites via the throwaway entity-update); post-1.0 it would ship a real update hook. Adding methods to the @api WorkflowEngineInterface is safe for consumers but breaks a third-party decorator implementing it; note in the changelog if it lands post-1.0.