OpenTelemetry: represent Continue-On-Fail node errors with OTel error semantics (they currently export as OK)

[Michael_Frostbutter]

The idea is:

n8n's native OpenTelemetry is good. When a node throws and onError is set to continue, the node span already comes through as Error with the exception recorded. Thank you for that.

There is one path it does not cover. When a node handles the error itself per item (it calls continueOnFail(), writes the error into its output items, and returns without throwing), the engine never sets a node-level error, so the span is exported as status = OK with no error information. An HTTP Request node set to continue on a failed call is the common example: the call returns 503, the node puts the error in the item, the run stays green, and the span says OK. The only trace of the failure is inside the output item JSON, which the span does not carry.

The ask is to bring that path in line with the throw path, additively:

1. Set the node span status to Error when the node continued a per-item error, the same as the throw path already does, while the parent execution span stays OK. Some consumers count any error-status span as a failed run, so this is worth offering as opt-in (a config flag) if that is a concern. The attributes below carry the signal regardless of status.

2. Attach the error using stable OTel attributes, so any backend reads it with no custom code. The important one is error.type, which is stable and low cardinality:

error.type = "503"          # the HTTP status as a string, or the error class, or \_OTHER

When the error carries an HTTP status, also expose it in its native form:

http.response.status_code = 503     # int, standard HTTP semconv

Full detail where available uses the stable exception attributes: exception.type, exception.message, exception.stacktrace. Note that the exceptions-on-spans convention is deprecated in favor of exceptions-in-logs, and exception.escaped is deprecated, so I would leave the exact vehicle (span event or a correlated log record) to your read of current OTel guidance. The ask is the standard attribute names, not a specific method.

None of this needs output-shape parsing. n8n holds the typed error at the point it decides to continue, one step before it demotes it into the node output.

3. Populate n8n's existing continuation attributes so a consumer can tell a continued error from a clean run without inspecting output. n8n.continuation.reason and n8n.node.termination_reason already exist in the constants; documenting and setting them for this case is enough:

n8n.continuation.reason = "continueOnFail" | "errorOutput"

4. Document and guarantee the item-count attributes. n8n.node.items.input and n8n.node.items.output are already emitted. Committing to them as a stable, documented convention on every node span makes volume-based checks first-class (input greater than zero, output at or near zero, is a strong drop signal).

Out of scope on purpose: this is not asking n8n to judge whether output is semantically wrong or degraded (correct shape, correct count, wrong data). That is user-specific and belongs in downstream assertions. The ask is limited to facts n8n already holds at emit time.

My use case:

I build observability on top of n8n's OpenTelemetry traces and needed to catch runs that finish green but silently did nothing, which is the most dangerous failure class because the usual signal (a failed execution) never fires.

A common, sensible setup: an HTTP Request node set to continue on fail, so one bad upstream call does not fail the whole workflow. The upstream returns a 503. The node writes the error into its output item and continues. The run is reported as success, every node green.

Here is the same HTTP node on two runs, as exported today by native OTel.

Healthy run:

span: Get Orders

  status: OK

  n8n.node.name         = Get Orders

  n8n.node.type         = n8n-nodes-base.httpRequest

  n8n.node.items.input  = 1

  n8n.node.items.output = 100

Continue-on-fail run, upstream returned 503:

span: Get Orders

  status: OK                       (the node errored per item; the span says OK)

  n8n.node.name         = Get Orders

  n8n.node.type         = n8n-nodes-base.httpRequest

  n8n.node.items.input  = 1

  n8n.node.items.output = 1         (an error item, not a result)

  (no error.type, no message, no status code anywhere on the span)

The item counts are useful and already there. The error itself never reaches the span, so a backend cannot tell these two runs apart from the span alone. Note this is specifically the per-item path. If the same node threw instead, native OTel would already mark this span Error, which is exactly the behavior I am asking for here too.

I think it would be beneficial to add this because:

For the per-item path, the trace disagrees with reality: a node errored, and the span says OK. That forces every downstream consumer to fetch and parse node output to recover the error, and the error is shaped inconsistently by source (an HTTP failure is an object with a status field, a thrown Code error is a bare string), so each backend reinvents the same fragile parsing.

Moving it into the telemetry using OTel's standard error.type and exception attributes fixes this once, at the source, for everyone. It lights up in any OTel backend (Grafana, Honeycomb, Datadog, self-hosted collectors), not just custom tooling, and it costs almost nothing because the typed error is already in hand at the moment of continuation. It also makes the two continued-error paths (throw and per-item) consistent, which they are not today.

Any resources to support this?

• Full writeup with real traces and screenshots: ageniusailabs.com/blog/silent-but-deadly
• Technical detail, including the inconsistent error shapes across node types: silent-failure-detection.md
• OTel error.type (stable, the primary attribute this asks for): opentelemetry.io/docs/specs/semconv/registry/attributes/error
• OTel HTTP span attributes (http.response.status_code): opentelemetry.io/docs/specs/semconv/http/http-spans
• OTel exception semantics (now via logs; spans form deprecated): opentelemetry.io/docs/specs/semconv/exceptions/exceptions-logs

(This surfaced while building an open-source n8n observability dashboard that reads these traces.)

Are you willing to work on this?

Yes, happy to help. I can share sanitized traces, test any change against a live n8n to OTLP pipeline, and contribute a PR. If someone points me at where the per-item continueOnFail result is finalized in workflow-execute.ts relative to where the node span status is set, I can scope it.

[Michael_Frostbutter]

Quick update. I spent yesterday digging into this pretty deep, and I want to correct my own framing and share where I have landed, because the direction changed.

What I got wrong. My first ask assumed the OTel module could detect a continued per-item error by looking at the node's output. It cannot do that soundly. Two problems: a field named error in legitimate output (a Set node writing json.error = "already processed", or an API body with an error key) is identical to a swallowed error message; and the error markers ride on the items themselves and get copied as items flow, so a pass-through node (If / Merge) that forwards an item carrying a prior node's error looks like it failed. The one fact that actually distinguishes a continued error, that a catch fired and the node chose to continue, only exists inside the node at the moment it swallows. Restricting to the typed item.error slot is safe but misses nodes that write into json.error instead, and the HTTP Request node does exactly that. So output inspection is either complete-but-wrong or safe-but-incomplete.

Answering my own question from the first post (where the per-item result is finalized versus where the span status is set): the node span keys its status on taskData.error, which the per-item path never sets. The continued error is finalized after that, in the node and engine: declarative nodes at the continueOnFail branch in routing-node.ts, the HTTP Request node at its own continue branches, and the engine normalizes item errors later in workflow-execute.ts. None of those record a per-node "this run continued an error" fact, which is the gap.

The direction I am taking. Add a small, sanitized signal emitted where the swallow happens, carried on taskData using the same mechanism the codebase already uses for execution hints. A node calls something like addContinuedError(error, itemIndex) (mirroring addExecutionHints); it is sanitized at record time into low-cardinality metadata (a validated HTTP status, else the error class, else a fallback) and folded into a bounded rollup on taskData. The OTel module then reads that instead of guessing from output. Same span result I originally asked for (error.type, http.response.status_code, a continuation reason, parent execution span stays OK), but sound this time, and the editor UI or any other consumer can read the same signal.

Proof of concept. I built it end to end and validated it live against Jaeger. Before: an HTTP Request node continues past a real 503, the run is success, the span is OK with no error info. After: the same run, the span is Error with error.type = 503 and the HTTP status attached, while the parent workflow span stays OK. Screenshots:

Before I send a PR. I saw in CONTRIBUTING that changes under packages/core should be checked with the team first, so I am flagging here rather than dropping a PR cold. A few design calls I would want your read on:

• Signal home: its own field on ITaskData, or folded into the existing hints channel with a new hint type?
• Redaction and exception detail: the signal is non-PII by construction so it needs no scrubbing, but should the redaction strategies still pass it through explicitly, or drop it under full redaction? And is a fully sanitized signal (no span exception event) fine, or would you want a redaction-governed message so backends still get exception.message?
• Node ergonomics: the HTTP Request node has three separate continue-on-fail exits, which is easy to under-instrument by hand. Would a small combined helper that both records the continued error and builds the output item be welcome, so adoption is one call per exit?
• Rollout: I would split it as a core PR (the signal, generic and consumer-free) followed by the OTel consumer PR stacked on top. Does that match how you would want to review it?

Happy to shape it to whatever fits the codebase. I can share the branch, sanitized traces, and test any change against a live n8n to OTLP pipeline. Thanks for reading a long update.

[Imad]

Thanks for the very detailed write-up and the examples!

I shared this in the dedicated internal Slack channel so the right people can take a look.

[Michael_Frostbutter]

@Imad

Appreciate you getting it in front of the right channel, that is exactly what I was hoping for.

Where it stands on my end: both halves are built and green. The core signal (recording the continued error at the point the node swallows it, sanitized and bounded on task data) and the OpenTelemetry consumer that reads it. Typecheck, lint, and tests are clean, and I validated the whole path live against a Jaeger pipeline. Same continue-on-fail 503: the node span goes from OK to Error with the status attached, while the parent execution span stays OK.

Whenever the right person surfaces, I can go whichever way suits them. Open a PR so they have something concrete to react to, or keep it a design discussion first if they want a say in where the signal should live before any code lands. I can share the branch and the sanitized traces either way.

No rush on my side. Thanks again.

Michael

[Michael_Frostbutter]

Another update, and a bigger one. I spent today building the real thing and then stress-testing my own coverage claim instead of trusting it. Two things changed: where the signal gets collected, and an honest map of what it actually reaches.

Collect once at the engine, not node by node

My last post described a node calling addContinuedError(...) at each swallow site. That works, but it does not scale: n8n has hundreds of nodes that continue past errors, and hand-instrumenting each is a large, drift-prone diff. So I moved the collection into the engine.

The key realization is that there is already a shared marker. When a declarative (routing) node continues on fail, routing-node.ts pushes { json: {}, error }, where error is the typed NodeApiError / NodeOperationError on the item's typed error slot, not free-form output. That has been true for a long time. So instead of touching each node, workflow-execute.ts scans a node's output items once after it runs, folds any typed item.error into a small sanitized rollup on taskData, and the net change to routing-node.ts is zero. Every routing node is covered with no per-node code, current and future.

The signal is the same sanitized, low-cardinality rollup as before (a validated HTTP status, else the error class, else a fallback; a count; the first error), and the OTel module reads that rather than inspecting output. The typed-slot rule is what keeps it sound: the scan ignores a loose json.error string, so a Set node writing json.error or an API body with an error key never trips it.

What it actually covers, measured

Here is the part I want to be straight about, because it is easy to overclaim. I checked the emit shape across packages/nodes-base rather than assume. Of the roughly 550 files that touch continueOnFail:

• Routing / declarative nodes (about 100), plus a couple of programmatic ones (JWT, MySQL): covered automatically. They attach the typed marker, so the engine folds them for free.
• About 64 programmatic nodes: not covered yet. They continue past an item by writing a loose json.error string into the output instead of the typed slot, and the scan deliberately ignores loose content. HTTP Request is the highest-traffic case in this group, so it is the one node I hand-instrument in the first PR.

So the honest summary is: this gives every routing node the signal for free and establishes the typed-marker contract, and the remaining programmatic nodes opt in by attaching the marker. It is not "the whole catalog lights up at once." I would rather say that up front than have it found later.

A wrinkle worth flagging

Bringing the loose-emitter nodes in is additive, not a rewrite: you keep json.error for anyone referencing it in an expression, and also attach the typed error sibling. The one catch is that the engine already has a step that rewrites a continued item's json to { error: message } whenever a typed error is present. For the \~64 nodes whose loose output is exactly { error: message } that is byte-identical, so the migration is purely mechanical. For a small handful (Airtable, MySQL, Webflow) whose output is richer, that rewrite would flatten it, so those want a tiny preparatory change to make the rewrite non-destructive first. I mapped all of this so the follow-up is a sized, low-risk sweep rather than a guess.

Validated live again, on the exact build

I rebuilt the engine-level version plus the OTel consumer stacked on top, ran it against a live n8n to OTLP to Jaeger pipeline, and confirmed the new path end to end. The important proof this round is a declarative node with zero hand instrumentation: it continues past a real error, the engine scan reads its marker, and its span comes through as Error with n8n.continuation.reason = continueOnFail, while the parent workflow span stays OK. The HTTP Request path still fires alongside it. The run itself reports success, which is exactly the silent failure, now visible per node.

How the PR is shaping up

• PR 1 (core): the engine collector and the typed-marker contract, plus the one hand-instrumented node (HTTP Request). Generic and consumer-free. Green on the full suites.
• PR 2 (stacked): the OTel module reads taskData and sets the span status and attributes. No change to the signal shape.
• PR 3 (follow-up, scoped): the small non-destructive fix to the rewrite step, then the mechanical marker addition across the \~64 loose-emitter nodes, shipped in batches by node family and guarded by the existing per-node tests.

Still your call before I open the core PR

CONTRIBUTING says to check with the team before changes under packages/core, so I am still holding. Two questions I would genuinely defer to you on:

• Signal home: its own field on ITaskData, or folded into the existing hints channel with a new hint type?
• Scan scope: the scan folds a typed error on any output branch. Should items on an intentional error output (onError: continueErrorOutput) count as continued errors here, or should the scan be scoped to the regular output only? I have an argument either way and would rather match your intent than pick for you.

Happy to share the branch and test any change against the live pipeline. Thanks for reading another long one.

[Michael_Frostbutter]

Testing is going well

[Michael_Frostbutter]

Errors are now populating in OTel

[Michael_Frostbutter]

Update: the core fix has been reviewed and hardened, and it is ready to open

Since the last post I put the core signal through a hard review, tightened what it caught, and re-validated the whole thing. It is built, green, and sitting ready.

What the fix does, in one line. Record the continued error once, in the engine, from the typed marker nodes already attach when they continue past an item. One read in the one place every node’s output flows through covers every declarative node with no per-node code, so the signal scales to the whole catalog instead of a handful of hand-wired nodes. The HTTP Request node is the one case I instrument by hand, because it writes its error as a plain string rather than the typed marker.

What the review changed. Two things worth naming:

• A node set to send failures to its error output on purpose was getting counted as a silent failure. That is wrong, those items are visible by design. I scoped the collector to the swallow-in-place case, so intentional error routing stays clean and only the genuinely hidden failures get recorded.
• When an item traces back to more than one input, the record now points at the earliest failing input rather than whichever happened to sit first in the list.

Neither changes the shape of the signal. They stop it from firing on things that are not actually silent.

Validation. Full test suites green on core, nodes-base, and the OTel module, typecheck clean, plus new unit tests for the collector and its edges. Re-ran the live check: a continue-on-fail node eats a 503, the run reports success, the node span flips from OK to Error with the status attached, and the parent workflow span stays OK. Same result end to end through n8n to OTLP to a trace backend.

The one flag. Testing a workflow that trips three failure modes at once surfaced a clean boundary. The engine signal catches everything that runs in the main process. A Code node that continues past an error does not surface through it, because Code runs in the task runner, a separate process the engine collector cannot see. That residual is exactly why this belongs in core rather than in each consumer: fix it once at the source and every backend gets it, including the paths a heuristic cannot reach safely.

The question. Both halves are built, reviewed, and validated, and I am holding them unopened: the core signal on its own (generic, no consumer), and the OTel module reading it, stacked on top. CONTRIBUTING.md asks to check with the team before changes under packages/core, which is why I have not opened anything yet.

So, @Imad is the team open to me opening the PR? And if so, would you rather I open the core signal by itself first, so it can be reviewed generically, and stack the OTel consumer after, or send both together? Happy to point the note to maintainers at whoever is the right owner to review it.