Tracing and Debugging Workflows

Overview

Maestro gives you three levels of insight into a running or finished workflow process:

• Trace (core): a table view of every task, process variable and linked entity in a process, with tools to correct them.
• Visual Trace (Maestro ReactFlow Trace Viewer): the process drawn on top of its workflow diagram, showing which tasks ran, their status and which routes were taken.
• Trace Archive: before and after snapshots of process variables and entity identifiers for every task, shown in the Visual Trace as "what changed during this step".

All three require the administer maestro templates permission.

Opening a trace

The Outstanding Tasks and All Active Tasks views link to the trace for each process. You can also go directly to /maestro/trace/PROCESS_ID.

The Trace page

The Trace page shows the process number, whether it is Active or Completed, and three tables:

• Tasks: every queue entry with its queue ID, task, status, completed date, who completed it and whether it is archived.
• Process variables: every variable and its current value.
• Linked entities: the entity identifiers attached to the process, with their unique ID, entity type, bundle and entity ID.

To fix a stuck or broken process, tick Change on a row and edit a task's status, its archive state or a variable's value, or delete a task, then click Save changes. Delete process removes the whole process. Use these with care on production processes.

When the Visual Trace is enabled, a View Visual Trace button appears at the top of the Trace page.

Visual Trace

Enable the Maestro ReactFlow Trace Viewer sub-module (it requires the Maestro ReactFlow Template Builder):

drush en maestro_reactflow_trace_viewer

Then click View Visual Trace on any Trace page, or go to /maestro/trace/PROCESS_ID/visual. The Visual Trace is read only: it draws the process's template exactly as it looks in the ReactFlow editor and overlays the process's progress.

Reading the diagram

Each task is filled with the colour of its latest status in the selected loop:

Lines show the path the process took:

• Thick blue lines are routes that were taken. Taken FALSE branches stay red and are drawn thick.
• Grey dashed lines are routes that have not been taken (yet).
• Highlighted (yellow green) lines are regeneration routes: the step that sent the process back around into its next loop.

Hover over a task to see its full name, status, task ID, type and stage. Hover over a line to see whether that route was taken.

Loops

When a workflow loops back (for example, a rejected form that goes back to the initiator), Maestro regenerates the tasks for another pass. The Visual Trace groups each pass into a loop: Loop 1, Loop 2 and so on. Use the loop selector to switch between them. The latest loop is marked (current) and is shown by default.

Step details

Click a task to open its details in the side panel: the task name, status, started and completed dates, and who completed it. Tasks that have not run yet say so.

When Maestro Trace Archive is enabled, the side panel also shows Changes During This Step: the process variables and entity identifiers before and after the task ran, with added, updated and removed values highlighted. This makes it easy to see, for example, exactly which variable an AI Task or Tool Task wrote. A task that is still active shows no snapshot yet.

Auto-refresh

Turn on Auto-refresh to watch a process live. Choose to refresh every 5, 10, 30 or 60 seconds. The viewer shows when it last updated and retries if a refresh fails. Auto-refresh stops by itself once the process is complete. Your auto-refresh choice is remembered in your browser.

On phones and narrow screens, the side panel becomes a bottom sheet that opens when you tap a task or the Details button.

Maestro Trace Archive

The Maestro Trace Archive module (maestro_trace_archive, Maestro 4.5.1 or above) captures a snapshot of a process's variables and entity identifiers immediately before and immediately after every task executes. The Visual Trace uses these snapshots for its "Changes During This Step" view.

drush en maestro_trace_archive

Retention

Snapshots are taken for every task in every process, so they can grow your database quickly. The module adds a Maestro Trace Archive section to the Maestro engine settings (/admin/config/workflow/maestro):

• Delete trace snapshots older than a number and a unit. The default is 1 week.
• Set the number to 0 to keep snapshots forever. Trace snapshots can use a great deal of database space if kept for long periods.

Old snapshots are deleted on cron, in batches, so a large backlog is worked through over several cron runs. When a process is deleted, its snapshots are deleted with it.

Other debugging tips

• Test without side effects. Use the AI Task's Test Mode (per task or site wide) to run a workflow without calling AI. See Maestro AI Task.
• Check the logs. AI Tasks log under maestro_ai_task (turn on "Log the response from AI?" to see each answer) and Tool Tasks log failures under maestro_tool_task, including the user the tool ran as.
• Cancelled versus Aborted. On AI Tasks, Cancelled means a real "no" answer and Aborted means there was no usable answer. Look for red tasks in the Visual Trace and check the step details and the log.
• Run the orchestrator by hand. drush maestro:orchestrate runs one orchestrator pass. Add --user to test as a specific account.
• Validate the template. The validity check in either template builder lists configuration problems before they reach production.
• Development Mode in the Maestro engine settings resets the entity cache during orchestrator runs if you suspect stale cached data.