One trap on the missing-runs layer: if the heartbeat is a node inside the workflow, it dies exactly when the workflow does not run, so it never warns you. The ping has to come from something outside that expects it on a schedule and alerts on its absence, not a node that only fires on a successful run.
The three-tier framework in this thread – hard fails, green-but-nothing, missed runs – is the right mental model. One more layer worth adding if you run workflows for multiple clients from a single n8n instance: error routing by client.
The problem with one shared Slack channel for all error alerts: ownership is ambiguous (who owns the 3am alert when it could belong to any of five clients?), the volume from one active client buries alerts from quieter ones, and if clients are ever given access to that channel they can see each other’s workflow names.
The pattern I use:
Put a Set node at the very top of every workflow and set a sticky field – something like client_id = “acme_roofing” or client_id = “westside_clinic”. This becomes the single source of truth for which client the workflow belongs to.
In your Error Workflow, read that field from the error data and route accordingly: if client_id is “acme_roofing”, send to #alerts-acme; if “westside_clinic”, send to #alerts-westside or their account manager’s email.
The Set node also acts as a workflow header: anyone who opens the workflow immediately knows what client and process it handles. Worth adding even if you never segment the alert channels – it saves a lot of head-scratching when someone unfamiliar has to debug a 3am failure.
For smaller setups that don’t need separate channels, the same idea still helps: prefix every alert with the client name, “[Acme Roofing] appointment reminder failed.” That alone makes it easy to filter in Slack and assign triage ownership in the morning.
Working on implementing silent failure detection into AGD, I’ll be running some extensive testing today on my solution for this and I will report my findings back here.
So far so good but I’m going to be pushing this today: So my proposed solution here:
We want to catch these silent failures from n8n’s OpenTelemetry data, outside the workflows, instead of adding a guard to every node inside them. n8n already emits a span per node, so one watcher can sit over all your workflows at once and decide whether a run actually did its work, separate from the success or error n8n reports.
The key thing we found is that you can’t trust the status at any level. A Continue-On-Fail node comes back “success” on the execution, the node, and the span. So instead of reading status, we read the output itself: the error that got pushed down into the normal data, and how many items each node actually produced.
That’s enough to catch the three ways a run looks green but isn’t: a node that errored and kept going, a node that quietly returned nothing when it normally returns plenty, and a workflow that never fired at all. The only part that needs care is the empty case, since plenty of nodes return zero legitimately, so we judge each node against its own history instead of treating zero as broken.
How the warning system decides what failed
Three signals, and none of them is the status field.
First, a swallowed error. If a node errored but the run still came back green, that’s a silent failure. We only count it when the execution itself reported success, so a run that n8n already flags as failed doesn’t get double-counted as silent.
Second, a node that stopped producing. This is the one that takes judgment, because zero items is normal for a lot of nodes. So we don’t treat zero as bad on its own. We look at each node’s own recent history and sort it into a few groups:
- Nodes that reliably produce, almost never empty. If one of these returns nothing, or far less than it usually does, we flag it.
- Nodes that are empty often by design, like a poller or a filter that usually matches nothing. Zero is normal for them, so we leave them alone.
- Nodes we haven’t seen enough runs of yet. We wait until there’s enough history instead of guessing.
When we’re not sure, we stay quiet. A warning system that cries wolf gets turned off, so we would rather miss a rare edge case than flag a healthy run.
There’s one more piece to the empty case. If a node had no input, its empty output came from upstream, not from that node. So we point at the first node that actually broke, not the downstream ones that just passed the emptiness along.
Third, a workflow that never ran. There’s no data to read in that case, so it’s a separate check on whether each workflow ran when it was supposed to.