Attempting to solve silent failures: how we're detecting runs n8n marks green but broke

[Michael_Frostbutter]

I've seen this question asked over and over and over again, so I decided to put some real work into figuring out a solution. The problem in one line: turn on Continue On Fail, or wire a node's error output nowhere, and a node that throws no longer stops the workflow. The run keeps going, finishes, and n8n records it as success. It is the failure class that worries me most in n8n, because the usual signal (a red, failed execution) never fires. A run can quietly stop doing its job and nothing pages you.

I have been trying to actually detect these runs, not just warn that they are possible. The goal is plain: surface a broken run from your own monitoring, before it is a client calling to say their workflow stopped working. To get there I went through the two things n8n actually gives you, the execution data and the OpenTelemetry traces, looking for anything that stays true when the status lies. This is a writeup of that process: the thesis, the approach, and the results on real workflows. It is a working experiment, not finished science, and the honest open questions are at the end. I would genuinely like to hear how the rest of you catch these.

The problem: green, but broken

Here is a small, realistic workflow. It fetches orders over HTTP, validates and normalizes them, filters to the ones still needing fulfillment, and queues those. On a healthy run it pulls 100 orders, keeps 50, passes them on. Every node green, and it should be.

Now the orders API returns a 503. The HTTP node has Continue On Fail enabled (a very common choice, so one bad upstream call does not nuke the whole run). Watch what n8n reports:

Succeeded in 98ms. Every node has a green check. The HTTP node swallowed the 503 and passed an error item downstream; the validator found nothing valid and emitted zero; the last two nodes had no input and never ran. Zero orders queued. To the execution log, and to any dashboard that reads execution status, nothing here is worth looking at. The integration has "just stopped working" with no trail.

Why the status can't catch it

The obvious idea is to read the node's status. It does not work. n8n's native OpenTelemetry exporter marks a Continue On Fail node's span OK, because from the engine's point of view the node did continue successfully. The status lies at every layer: execution status, node status, span status all say fine.

The truth survives in exactly two places, and both live in the run's data, not its status:

• The demoted error. The error is still there, moved into the node's normal output as an item. n8n records it inconsistently by source, which is the annoying part.
• The item counts. A node that normally emits N items and now emits 0 (or far fewer) shows it in the per-node input/output counts the OTel spans already carry, with no extra round-trip.

The error shapes I have had to normalize so far, mapping the same concept across three different recordings:

• HTTP / Axios failure: json.error is an object with a .status (e.g. 503). Structured, easy.
• Thrown Code node error: json.error is a bare string. Same idea, no structure.
• Dropped work with no error at all: nothing in json.error; the only evidence is the item count going N to 0 (or 500 to 5). Absence is the signal.

So the detector ignores status entirely and reads output shape instead.

The approach: two detectors, split by cost

The dashboard receives n8n's OpenTelemetry traces on an embedded OTLP receiver and, on ingest, enriches each run with per-node health.

• Demoted error (costs one fetch). For each node, union the three places n8n may have recorded an error (execution status, a run-level error object, and the item-level json.error that Continue On Fail pushes into normal output). Normalize the inconsistent shapes above to one (type, summary, http_status). This needs one run-data fetch per execution, done once and cached.
• Low output (free). Read n8n.node.items.output and .input straight off the span. No round-trip. A zero or a sharp drop is the signal, but only when it is anomalous for that specific node, which is the whole problem.

A run is flagged a silent failure when its top-level status is success and yet a node resolves to ERROR or LOW. The clearest way to see it is the same node on two runs. Here is the healthy one in AgeniusDesk's Observe view: Get Orders reports status OK and 100 items out, the whole span reading the way it should.

Now the broken run. n8n still called Get Orders OK; the detector reads the 503 out of the output and marks it health ERROR, with the output collapsed:

Deciding which zero is actually a failure

A zero output is not automatically a problem. Plenty of nodes return nothing on a perfectly healthy run: a poller checking for new signups and usually finding none, a filter that legitimately matches nothing this cycle. Alerting on every zero is a firehose, and a noisy alarm gets muted and becomes worthless. The design rule is precision over recall: when unsure, stay quiet.

Each node is classified against its own recent history:

• Cold start (not enough history): never fire. We do not know this node's normal yet.
• Intermittent / dormant (zeros are common for this node): a zero is within normal. Stay quiet.
• Steady producer (reliably emits data): the only class that can fire. If it had input and returned zero, that is a silent empty. If it returned far below its normal band (say under 10% of its median), that is a magnitude drop, which catches "200 rows became 3" that a zero-only rule would miss.

Report the origin, not the cascade

When a node returns zero, every node downstream also sees nothing. Firing on all of them turns one root cause into fifteen alerts. Two rules keep it to one:

• For a zero: n8n does not run a node that receives no input, so downstream nodes simply do not execute. Only the node that actually went N to 0 fires.
• For a drop: downstream nodes do run, each with a reduced-but-nonzero count, so each independently looks low. The detector keeps a per-node input history alongside output history: a node whose input is itself anomalously low just carried an upstream drop through (a victim) and is suppressed. The origin is the node whose input held normal while its output collapsed.

The lead-pull workflow shows the drop path. A query drifted and the API returned 5 rows instead of its usual 500. n8n, again, calls it a success with every node green:

Fetch Leads is flagged as the origin (health LOW, out 5, "far below this node's normal volume"); the three downstream nodes that faithfully passed 5 through are suppressed:

Making it loud everywhere

Detection is worthless if it only lives in a trace viewer most operators never open. A detected silent failure gets written into the same errors pipeline that already drives the Executions and Errors feed, so it surfaces in every view as its own distinct class rather than lumped in with ordinary errors. The Overview picks up a dedicated stat card, and the analytics page carries a matching tile, so a "green but broken" count is on the home screen, not buried:

Where this lives, and the honest limits

This is built into AgeniusDesk, a self-hostable n8n observability dashboard I am building. It is open source (MIT), and the full writeup, including the classifier config knobs and their conservative defaults, is here:

• Repo: github.com/Mfrostbutter/ageniusdesk-ce
• Full silent-failure writeup: docs/architecture/silent-failure-detection.md

The rough edges I already know about, because I would rather name them than pretend:

• Enrichment lag. Span-only anomalies (the LOW/drop path) surface the moment spans land. The demoted-error path needs a run-data fetch, so a Continue On Fail HTTP error can lag the execution by up to about a minute before it appears.
• History poisoning. A node that fails often enough teaches the classifier that being empty is normal for it, and it stops firing. Correct precision behavior, but a chronically half-broken node can go quiet.
• Dead man's switch. A node that was supposed to run and did not run at all (never produced a span) is not detected yet. That is the natural next detector.

The actual question

How do you catch these today? Error Workflow trigger, a Set node that asserts item counts, an external check on the destination, something smarter? And where do you think my classifier is wrong, especially the "which zero is a real failure" call, since that is the part I am least sure about. Counter-examples are the most useful thing you can throw at me.

[Ayomikun27]

Really solid writeup Michael, this is the most thorough take on silent failures I’ve seen here.

The item count anomaly detection is smart. Most people just slap a Code node after critical steps to throw on zero items crude but it forces a real red execution at least.

On history poisoning it is worth considering a hard floor per node type regardless of history. A “fetch orders” node should never learn that zero is normal.

The dead man’s switch gap is honestly the scariest one. No spans means no detection at all. External heartbeat monitoring is the only real fix there nothing inside n8n can catch it.

And the cascade suppression logic is the part most homegrown solutions get wrong. Usually ends up as 15 alerts for one problem, so that’s a nice touch.

[Michael_Frostbutter]

@Ayomikun27

Thank you, this is genuinely one of the most useful replies I could have hoped for. You went straight at the parts I am least settled on.

On the Code-node-throw approach: it works, and honestly it is the right move for a step you know is critical. The thing I wanted was to get that signal without editing every workflow, and without changing what the run does. A throw is in-band, it converts the silent failure into a hard red execution, which is great for alerting but it also stops the run, so you cannot both continue-on-fail and assert. The detector is out-of-band, it just watches. So I think of them as complementary: put the hard assertion where the cost of being wrong is high, and let the passive detector be the safety net for the hundred steps you never got around to annotating.

The hard floor idea is a good one, and I think it is the same lever that fixes history poisoning, which is why I like it. One refinement: the signal is not really the node type. An HTTP Request node is a "fetch orders" in one workflow and a legitimately-sometimes-empty poller in the next, so a global per-type floor punishes the poller. What the user knows and history cannot is the node's role. So the clean version is a declared "must-produce" flag on the node: mark fetch-orders must-produce, and it never learns that zero is normal, no matter how many empty runs it sees. Same mechanism, and it doubles as the escape hatch for anywhere history infers wrong.

The dead man's switch is the one that scares me too, and you are right that nothing inside n8n catches a run that never happened. External heartbeat is the only real fix for that case, full stop. There is a middle case worth separating out, though: the workflow did run, but a node that should have produced a span did not (a branch got skipped, a node upstream returned empty so the node never executed). That one you can catch without leaving n8n, by diffing the workflow's node set against the spans that actually landed for that execution, expected node X, saw no span for it. So I have started thinking of it as two layers: external heartbeat for "the workflow never fired," and a definition-versus-trace diff for "a node went missing inside a run that did fire."

On cascade suppression, glad that one landed. You nailed why it matters: 15 alerts for one root cause and people mute the whole thing within a week. The rule that keeps it to one is keeping a per-node input history alongside the output history, a node whose input is itself anomalously low just carried an upstream drop through, so it is a victim and gets suppressed; the origin is the node whose input held normal while its output collapsed.

The reason I am treating this as a priority rather than a curiosity: these workflows are going into production and companies are starting to rely on them, and a green run has to actually mean green. Related to your heartbeat point, I am also writing up an n8n feature request to get the continued error onto the OpenTelemetry span itself, since right now n8n holds the typed error and then exports the span as OK, which is the root of half of this. Fixing it at the source would make everyone's detection easier, not just mine.

Two things I would genuinely like your take on: how are you handling the heartbeat side today, cron-based external ping or something heavier? And would a per-node "must-produce" flag be the right shape for the floor, or were you picturing it more automatic than that?

Cheers,

Michael

[Michael_Frostbutter]

An update, I spent all day yesterday digging into this more and it’s moved my thinking. Here is where it landed and what changed.

The shape got clearer. What I was treating as one problem is really four failure classes, and each wants its own layer:

• A node runs but returns less than it should (a swallowed 503, a query that drifts to 5 rows instead of 500). Item-count and output-shape anomaly, with a declared must-produce flag so a node whose role is to always return rows never learns that zero is normal.
• A node continues past an error but the run stays green. More on this below, it turned out to be the deep one.
• A node that should have run did not (a branch skipped, an upstream node returned empty). Caught by diffing the workflow’s declared nodes against the spans that actually landed for that execution.
• The workflow never fired at all. External heartbeat, nothing inside n8n sees this one.

And one rule holds it together so a single root cause is a single alert: keep input history next to output history, so a node whose own input was low is flagged as a victim carrying an upstream drop rather than the origin.

The thing I got wrong. The continued-error class… I first assumed I could read it off the span by inspecting output. I built that, then took it apart, because it’s unsound [the error-field ambiguity + stale forwarded markers; only the catch inside the node knows].

Where the real fix is. So I stopped inferring downstream. n8n holds the typed error one step before it demotes it into the output and exports the span as OK. Record it in core once, every consumer reads it. Built a PoC, validated live against n8n → OTLP → Jaeger, working on getting it upstream so it helps everyone’s detection, not just mine.

For me, green has to actually mean green in production. A good part of the detector’s complexity exists only because the platform drops a signal it already has.