Production Reliability Shield: Duplicate Detection + Semantic Checks

Built with n8n
1

Catch the webhook failures that still look green.

n8n says the workflow ran. The webhook returned 200. But did your CRM actually update? Did the order get fulfilled once, or twice? Did the email go out to zero recipients?

The Production Reliability Shield catches the failures that don’t raise errors — duplicate deliveries, near-duplicate events, and empty-success runs where your action returned nothing useful.

This is a free demo version. It uses n8n staticData as its store — no database required. staticData is saved with the workflow and persists between executions of an active workflow (it is not cleared when n8n restarts). The demo stores only hashes, timestamps, counters, and a redacted comparison string — never raw payloads. It demonstrates the full detection pipeline so you can evaluate the approach.

Download the workflow: Production Reliability Shield Lite (n8n) — duplicate detection, fuzzy near-duplicate quarantine, staleness + semantic checks. Free demo, no database required. Pro version: https://mdebrand.gumroad.com/l/production-reliability-shield-pro · GitHub — open the gist, click “Raw”, save the JSON file, then import it into n8n (Workflows → Import from File).

What this template catches:

  • Exact duplicates — same webhook event delivered more than once (retries, re-queues, network timeouts)
  • Fuzzy near-duplicates — events that are nearly identical but got a new delivery ID — quarantined for review before they reach your CRM, billing, or orders
  • Stale events — events older than your configured window that should be discarded
  • Empty-success runs — your action ran, n8n returned green, but the downstream output had zero items or missing required fields

How it works (5-minute quick-start):

  1. Import the workflow (gist link above) into n8n
  2. Click “Execute Workflow” using the Manual Trigger — the Demo Data node runs one pre-loaded scenario per execution (it starts with fresh, which returns decision: "process")
  3. Try the other single-run scenarios — open the Demo Data node and change the example variable, then execute again: "stale" returns decision: "stale" (event 130 minutes old), "empty_success" returns decision: "semantic_failure" (action output was empty)

To see duplicate blocking and fuzzy quarantine, use the webhook. These need the event store to persist between runs, and n8n only saves workflow staticData between executions of an active workflow — manual editor runs don’t save it. So: activate the workflow and send a POST request to the webhook URL. Note: event_timestamp must be the current time — events older than 60 minutes (the default stale_after_minutes) are rejected as stale. Copy-paste test (fills in the current UTC time for you):

curl -X POST <your-webhook-url> -H 'Content-Type: application/json' \
-d '{"stage":"pre_action","event_id":"evt_001","source":"myapp","entity_type":"order","entity_id":"ORD-12345","action_name":"process_order","event_timestamp":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","payload":{"amount":99.00,"currency":"USD"}}'

The first call returns decision: "process". Send the same command again — the second call returns decision: "duplicate".

What each result means:

Decision Meaning
process Fresh event — proceed with your action
duplicate Already seen — skip
quarantine Fuzzy near-duplicate — review before acting
stale Too old — discard
semantic_failure Action ran but output looks wrong

Scope of this demo (honest limitations):

  • No concurrency safety — staticData is not safe under simultaneous executions. Two concurrent requests can both see is_new=true. Demo/evaluation use only.
  • staticData persistence — n8n saves staticData between successful executions of an active workflow; it is NOT cleared on n8n restart. Stored entries expire only via the bounded-store eviction (oldest evicted above 500 keys). Stored per event: idempotency key (hash), payload hash, timestamps, delivery count, and a redacted fuzzy-comparison string (no raw payloads). To wipe the store, deactivate and re-import the workflow.
  • No ordering guard — sequence-based out-of-order detection requires persistent state
  • No quarantine table — suspicious events are flagged but not stored for review
  • No external alerts or audit export — a database-backed deployment would keep an audit record in tables; that’s not part of this demo
  • Semantic checks are demo-only — this demo includes a semantic check (that’s how it flags empty-success runs), but a persistent two-phase pattern (pre_action + post_action with database finalization and stored verdicts) isn’t in this demo

This is an evaluation template. It shows what the detection approach looks like and lets you test the decision logic with your own payloads. The detection approach was validated against a real n8n + Postgres deployment (20/20 scenarios); this Lite workflow’s webhook path was also live-tested before release (fresh → process, identical re-send → duplicate, near-identical → quarantine, genuinely different → process).

Nodes used:

  • Webhook Trigger
  • Manual Trigger
  • Code (6 nodes — demo data, config, canonicalize, engine, semantic check, format)
  • Switch (decision routing)
  • Respond to Webhook

No community nodes. No external dependencies.

2 Likes
2

Using staticData as a lightweight store for deduplication is a solid approach - no external DB overhead, and it works well for moderate traffic. One thing worth flagging for production use: if the n8n process restarts (not just deactivate/reactivate the workflow), staticData gets cleared. So for high-stakes dedup scenarios, it’s worth adding a periodic snapshot to Redis, a DB, or even a Google Sheet as a fallback.

The semantic check layer on top of the hash is a nice touch - catches near-duplicates that strict hashing would miss. Useful pattern for webhook-heavy pipelines.

1 Like
5

Thanks — and glad the semantic layer landed, that’s the part that catches the “green but wrong” runs strict hashing misses. One clarification on the restart point: workflow staticData is actually persisted to the database, not held in memory — n8n writes it back to the workflow record after each successful execution of an active workflow, so a normal process restart keeps it. The real caveats are narrower: it only saves on active/production executions (manual editor runs don’t persist), a crash mid-execution loses that run’s update, and it isn’t shared across instances in queue/multi-main setups. So your instinct is right for high-stakes dedup — once you need cross-instance or crash-durable state, an external store (Redis, or a DB table keyed on the idempotency hash) is the correct move; I’d treat staticData as the zero-dependency default rather than the end state. Appreciate you digging in.