A release-candidate audit of the recovery layer (after #3608146 merged) found concurrency and recovery defects that #3608146 did not address, since it was a behavior-preserving refactor. They share a theme: status flips and recovery paths not guarded against concurrent actors, and recovery sweeps and teardown that do not account for every state. Grouped here because they were found together and several are fixed by the same work; individual fixes may land as separate merge requests.

High

  • Incident actions have no concurrency claim. retryIncident, skipIncident and cancelIncident resolve the incident and flip the token with an unguarded load-then-save, and openIncidentToken checks only the caller's in-memory isOpen(). Two concurrent operator actions (or a double-click) both proceed, so the branch downstream of a skipped or retried node runs twice: an exactly-once violation. Fix: claim the incident open-to-resolved transition and the token status transition with a guarded write, keeping the entity save so the inbox work-item hook still fires.
  • TimeoutSweep double-fires when the re-read deadline is NULL. fireInner claims the timeout (guarded deadline clear) only when the re-read deadline is not NULL. A prior fire of an until- or anchor-based timeout leaves the token parked with a NULL deadline, so a second concurrent fire (operator "fire now" racing cron) skips the claim and runs the action again, duplicating a notify or spawn, and on the unclaimed path can resurrect a consumed token and strand the instance. Fix: return when the re-read deadline is NULL.

Medium

  • fireTimer does every flip as unguarded load-then-save inside its transaction. No claim protects it, so cron and "fire now" can double-fire a timer, run an action against an already-answered task (snapshot-stale read), and a recurring re-arm can resurrect a cancelled timer. The same class the refactor removed elsewhere; the timer path was left out.
  • Migration cannot see tokens created after planning. applyMigration iterates the tokens captured at plan time; a token created between planning and the version bump is never validated or rewritten, so a version bump that renamed its node strands it and its next advance fails the instance. Fix: re-query live tokens inside each instance transaction.
  • retryIncident and resumeIncident are not atomic. Unlike skip, cancel and fail, they do the re-queue and the incident resolve as two autocommit writes. A crash between them leaves an open incident on a run whose work finished, which blocks completion and both recovery sweeps forever.
  • recoverStalledInstances head-of-batch starvation. An instance stalled at a non-timeout join keeps matching the ordered, range-limited finder every run, so with more than a batch of them the instances behind are never visited. Needs a cursor or an exclusion for already-diagnosed instances.
  • cancelInstance and failInstance do not serialize against an in-flight advance. A worker past its isRunning() check finishes its node after the teardown commits, leaving a terminal instance carrying live PARKED or ACTIVE tokens and phantom inbox work items that no sweep cleans up.
  • Open incidents survive instance teardown. A FAILED instance can keep an open incident; operator actions on it then re-activate a token or create successors on a terminal instance. Teardown should resolve the incidents, or the incident actions should refuse a non-running instance.

All verified against the code by the audit. See #3608215 (decomposition): the shared StatusTransitions service and an IncidentManager fix the incident and timer flips by construction, so some of these are best landed with that work rather than as isolated patches.

Issue fork orchestra-3608219

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

mably created an issue. See original summary.

mably’s picture

Status: Active » Fixed

All eight findings (RC-H1, RC-H2, and M1 through M6) are fixed and merged to 1.x via the recovery-layer decomposition in #3608215: Decompose WorkflowEngine into StatusTransitions, InstanceRecovery, IncidentManager and JoinCoordinator collaborators, and enforce the transition primitive (MR !273).

  • Every contended status flip now goes through one shared guarded primitive (StatusTransitions).
  • RC-H2: the timeout sweep no longer double-fires a cleared-deadline timeout.
  • M1: the timer fire path is claimed and its flips guarded.
  • RC-H1, M3, M6: the incident actions claim the incident atomically, retry and resume are transactional, and teardown resolves open incidents.
  • M2: migration re-derives live tokens instead of trusting the stale plan.
  • M4: the stalled-recovery sweep uses a cursor, so wedged instances no longer starve the batch.
  • M5: a cron backstop cancels live tokens orphaned on a terminal instance.

Each fix has kernel test coverage. The further structural decomposition (the thin-facade extraction) continues in #3608236: Extract a core execution service so the recovery and incident logic can move out and WorkflowEngine becomes a thin facade. Marking fixed.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.