# Troubleshooting workflows

> Symptom by symptom: a failed, paused, held or timed-out run, a scheduled run that did not start, a workflow marked Needs attention, a refused webhook, and a node whose app is off.

Source: https://orbitalaip.com/docs/workflows/troubleshooting

## Where to look first

- In the editor, the status chip beside the toolbar names the run's state.

- The **Console** opens for a completed or failed run, with an **Error** block at the top.

- **Run History** in the toolbar lists runs newest first. Expand a row to read each node's status, duration and error, then click **View on canvas** to paint that run onto the graph. See [Run History](https://orbitalaip.com/docs/workflows/runs-and-versions#run-history).

- On the Workflows list, the status button reads **All clean** or the count of failures over seven days. It opens a status panel, where **Failures · 7d** names the workflows responsible, or counts them when there are more than two.

- A failed run also sends a notification titled with the workflow's name followed by **run failed**, with the error as its body. It files under **Workflow runs** in **Settings** → **Notifications**; see [Notifications](https://orbitalaip.com/docs/alerts-and-notifications/notifications).

## A run failed

The run's error lists each node that failed or was skipped, for example **2 node(s) failed:** followed by each node's id and message. The nodes that depended on the failure read **Skipped: upstream node failed**.

1. **Read the error on the node**
   
   Open **Run History**, expand the run and read the text under the node's id. **View on canvas** shows the same run on the graph.

2. **Check the inputs**
   
   The common causes are an empty required input, a symbol that does not resolve, an empty watchlist, a date range with no data, or a field path that matches nothing.

3. **Check credits and limits**
   
   An error beginning **Monthly AI credits exhausted** means the workspace's [credits](https://orbitalaip.com/docs/glossary#credits), included and purchased, ran out during the run with no provider key of its own connected; see [Credits and billing](https://orbitalaip.com/docs/workspace-and-account/credits-and-billing). If the workspace balance cannot cover a typical run, the run is refused before anything is spent. **Not enough credits** then shows **Estimated cost**, **Your balance** and **Short by**, with **Add credits**. A workflow with no history is never refused on cost. A run refused for another reason shows **Not run** with the reason, such as the plan's request limit.

4. **Run again**
   
   Fix the node, click **Save**, then click **Run**. A paused run shows **Resume** instead, which retries the version the run started with; see the next section.

What the rest of the run did follows **On node failure**; see [Run and node timeouts](https://orbitalaip.com/docs/workflows/runs-and-versions#run-and-node-timeouts).

An error beginning **Execution outcome is unknown after the worker lease was lost** means the process running the workflow stopped mid-run. Check what its nodes already did, such as alerts sent or watchlists written, before you run it again. Some runs read **Run reaped: executor heartbeat went stale (process died or stalled).** instead.

## A run is paused or awaiting approval

A paused run comes from **Pause on first failure**. The canvas shows a banner reading **Run paused at** the node's id, and **Resume** replaces **Run** in the toolbar. **Resume** reruns the failed node from the version the run started with and reuses the outputs of every node before it, so those are not paid for twice. Use it when the cause lay outside the workflow, such as data, credits or a provider. A change to the node's settings reaches only a new run: save it, reopen the workflow and click **Run**.

If you left the editor:

1. **Open Run History**
   
   Click **Run History** in the toolbar.

2. **Load the paused run**
   
   Expand it and click **View on canvas**.

3. **Resume it**
   
   Click **Resume** in the toolbar.

**Resume** answers **Could not resume run** when the run cannot continue, for example because the workflow was archived or deleted, or a stop was already requested. Wait a few seconds and try again, or start a new run.

A run held for approval waits under **Needs your decision** in **Automations**; see [Run policy](https://orbitalaip.com/docs/workflows/schedules#run-policy).

## A scheduled run was skipped

A scheduled run that did not start usually met the schedule's **Run policy** (overlap, **Cooldown**, the hourly or daily cap). The row's **Last outcome** says what happened to the latest occurrence, such as **Rate limited** for a cap or the cooldown; see [Run policy](https://orbitalaip.com/docs/workflows/schedules#run-policy). An occurrence the policy drops is never a run: it does not appear in **Run History** and does not count as a failure.

An exchange-timed schedule does not run on weekends or exchange holidays, and **XNYS market hours** also skips any time outside the regular session; see [Timing](https://orbitalaip.com/docs/workflows/schedules#timing).

An occurrence that Orbital reaches more than 5 minutes late is skipped. The row shows **Needs attention** with **Skipped stale occurrence** and the time, until a later occurrence runs on time.

A schedule can also stop itself. That happens when its settings are invalid, or when the workflow's declared inputs changed since the schedule was saved. Orbital disables it, marks its row **Needs attention** with the reason, and sends a notification titled **Automation stopped** and its name. Edit the schedule, fix what the reason names, and save it.

> [!NOTE] Skipped is not failed
> A run with the status **skipped** declined to do any work, most often because a node's app is off; see [Run statuses](https://orbitalaip.com/docs/workflows/runs-and-versions#run-statuses).

## A workflow needs attention

The Workflows list has a **Needs attention** tab. A workflow is in it when a run failed in the last seven days or its latest run failed. A workflow installed from a template that has never run is in it too. It leaves the tab once a week has passed since the failure and the latest run did not fail. The **HEALTH** column sums up the last 12 runs, for example **1 run · 1 failed** in red, so a failure older than a week can still show there.

The same words appear on an automation's row in **Automations**. A red **Needs attention** badge means the automation cannot start runs, and its status reads **Unavailable**, or **Paused** when Orbital stopped a schedule. An amber badge is a warning, and the automation keeps running. The reason sits under it, such as **The data provider is limiting checks. Orbital will retry automatically.** or **This signal source is no longer available. Choose another source to resume.** Provider limits clear on their own; a missing source needs a new one.

## A run timed out

When a run passes its **Run timeout (minutes)**, it fails with the error **Run exceeded wall-clock timeout of** the seconds it had. The **Console** adds that some nodes were skipped to surface the timeout cleanly, and each node not reached reads **Skipped: run exceeded wall-clock timeout**. When one node passes its **Node timeout (minutes)**, that node fails with an error naming the node and the seconds it had. See [Run and node timeouts](https://orbitalaip.com/docs/workflows/runs-and-versions#run-and-node-timeouts) for the defaults and for what **On node failure** does next.

Raise a budget only when the work is expected to take that long. Fewer symbols, or a smaller model in a **Custom LLM Prompt** or **Synthesizer** node, usually serves better. Moving work behind a **Run workflow** node does not add time: the child gets only what is left of that node's budget. A changed budget applies to the next run, never to one in flight.

## A webhook is refused

The webhook endpoint answers the sender before any run starts. The replies to a refused request include:

- **Not found**: the token in the URL is unknown, or the webhook is paused or removed.

- **Unauthorized**: the request failed the authentication check.

- **Trigger payload is too large.**: the body is over 1 MB.

- **Temporarily unavailable, please retry**: Orbital could not queue the delivery.

Other refusals read **Malformed trigger payload.** or **The trigger delivery key was already used with different content**.

A request that is accepted but produces no run was dropped by the **Run policy**, for example by **Maximum per hour**, or held for approval, exactly as a schedule is. It may also have been treated as a duplicate. With **Provider delivery ID**, a repeated id is one delivery; with **Same-body window**, identical bodies inside the window are one delivery. Runs that did start appear in **Run History** with **webhook** beside the status. The setup is on [Webhooks and signals](https://orbitalaip.com/docs/workflows/webhooks-and-signals).

## A node is missing because its app is off

These nodes exist only while their app is on in the project: **Get Watchlist**, **List Watchlists** and **Watchlist Write** need Watchlists; **List Strategies**, **Get Strategy** and **Get Backtest Run** need Backtesting. With the app off, the node is absent from the **Nodes** panel.

Running a workflow that holds the node executes nothing. The run ends skipped. The gated node reads **The Watchlists app is turned off in this project**, or the same sentence for Backtesting, and every other node reads **Skipped:** followed by it. **Resume** on a paused run declines the same way.

Turn the app on for the project under **Settings** → **Apps**, then reopen the workflow; see [Switch an app on or off in a project](https://orbitalaip.com/docs/workspace-and-account/apps#switch-an-app-on-or-off-in-a-project).
