Run a subprocess child on a remote Orchestra, and give both subprocess nodes one answer for a child's ending

Problem/Motivation

A subprocess node runs its child on the local engine. SubprocessTask::launchChild() calls the engine's own start(), and the child records its caller in parent_token, an entity reference to a local token row, so the resume that follows a child can only ever fire for a child in the same database.

Driving another site's engine is already solved in the project: OrchestraClientInterface is transport-agnostic, orchestra_server_api serves it over OAuth, and orchestra_client speaks it to a remote. What was missing is a subprocess node that uses that contract as the child's transport, together with the machinery a child on another site needs and a local one does not: something to notice it has ended, something to end it when the parent moves on, and an answer for an ending nobody is waiting for.

That last one turned out not to be about remote children at all. A child that failed or was canceled, on a node naming no state variable, left its parent parked for ever on a child that was already over, with nothing anywhere saying why. That was true of local children too.

Proposed resolution

1. A remote Orchestra is a configuration entity. A site holds one per engine it calls: a host, the consumer identity to authenticate as, the Key holding that consumer's secret, how long a call to it is worth waiting for, and whether it is the one that code naming no remote reaches. They are managed under Configuration, Workflow, Orchestra, taken out of service and back from that list, and one a workflow still calls cannot be deleted. The base URL must be https, except a loopback host or one an operator allowed in settings.php, and that is re-checked at request time so a configuration import cannot slip past the form.

2. A remote_subprocess node, in a submodule of its own. It names a remote and the workflow to run there, carries the same input and output maps as the local node, and parks its parent until the child ends. The two node types share those maps, the re-launch settings and the rule for what an ending gives the parent, through two traits, so they cannot answer the same question differently.

3. A cron sweep asks each remote about the run its parent recorded. Bounded at fifty parents a run, because each of them is an HTTP call, with a cursor so every parent comes round, and inside the engine's batch context so the resumes it makes are left to the queue worker rather than run to their ends inside the cron request. A parent that moved on without its child has that child canceled over there. A remote that stops answering is asked on the site's own advance budget, and then the parent is handed to an operator rather than waiting on a site that is never going to answer.

4. A failed child is launched again while the node's budget lasts, through the same relaunch_attempts and relaunch_backoff settings the local node carries. An operator resolving an incident with Retry hands that budget back whole, on both node types: it bounds what the engine does unattended, and a retry is attended.

5. A child that ends with nowhere to send the run raises an incident. Named a state variable, a failed or canceled child routes as it always did. With none there is nothing to route on, and advancing the parent would carry the run past work that never happened, so the branch stops. Stopping silently left a run that could never move again with nothing saying why, so it stops with an incident and an operator decides. This is the one change a site already running local subprocess nodes will notice.

6. Cancellation across the hop. cancelInstance() on the client contract, and POST /orchestra-api/instance/{instance_id}/cancel behind the same OAuth guard as the rest, so a parent that ends can end the run it started elsewhere.

7. Two flow-condition operators, set and not_set, which ask whether anything wrote a variable rather than whether its value is truthy. A child that returned nothing and a child that returned 0 are not the same thing, and empty cannot tell them apart.

What differs from a local child

The two sites commit separately. A parent that rolls back after the start returned leaves that child standing on the other site, which is why the node records what it started before parking: a retried advance finds the record and does not pay for the child twice.

A synchronous remote has drained the child inside the start call, so the node reads the result and advances in the same request, exactly as a local synchronous child does. And nothing on this site hears anything on its own: the sweep is what notices an ending, ends an orphan, and raises what needs raising.

Remaining tasks

The other half of the feature is #3623524: signed event notices, so a caller hears about its child instead of asking; the handshake that registers where to send them; a list-definitions operation; the initiator carried across the hop; the caller recorded on an instance; and an idempotency key on the start.

User interface changes

A remotes listing under the Orchestra configuration, with add, edit, enable, disable and delete, the delete page naming the workflows that still call one and refusing while any do. The new node type appears in the modeler with its own settings, and the state-variable field on both subprocess nodes says what leaving it empty now means.

API changes

OrchestraClientInterface::cancelInstance(), and ProcessControlInterface::executeInBatch(), which runs a pass over many runs as one batch so that operations inside it leave their work to the queue worker rather than draining each run in the request: the engine's own cron sweeps use it, and so must an extension sweeping runs of its own.

SubprocessTaskInterface now asks one question, getResumeVariables(), in place of the two it was assembled from, and gains getRelaunchDeadline(); collectVariables(), getVariablesForTerminalChild() and relaunchBackoff() leave it. Two traits are public alongside it, SubprocessMappingTrait and RelaunchConfigTrait. RemoteInterface and RemoteClientFactoryInterface are new, for code that names which Orchestra it means.

Data model changes

One new configuration entity, orchestra_remote, replacing the single-remote settings object that a post-update carries over. What a parked parent waits on is an internal instance variable holding the remote and the child's id there, never a bare id, so the runtime schema is otherwise untouched; the variable table gains an index on its name column, which is how the sweep finds those records without reading the whole table.

AI-Generated: Yes (Claude Code wrote the code on this issue and this summary. I reviewed both before posting.)