Some failures never turn an execution red. An upstream form field gets renamed and an IF node starts sending every item down the false branch. Someone duplicates a sheet to tidy it, the team moves to the copy, and the Sheets node keeps writing to the original. A form tool’s webhook gets switched off, so the workflow is never triggered. The executions list shows Succeeded, or shows nothing at all, because nothing failed.
A check that counts records catches these, where error alerts cannot:
- Once a day, read yesterday’s records at the source: form responses, orders, labelled emails.
- Read the same window at each destination: sheet rows, CRM records, Slack messages.
- Match the two sides on a stable ID, such as an order number or a response ID. Do not match on names or row position.
- If a destination is short, send one message that lists the missing IDs and where they are missing.
In n8n that can be a Schedule Trigger, one read node per side and a Compare Datasets node keyed on the ID; its “In A only Branch” output is the missing list. Keep the check in its own workflow, so it still runs when the main one is broken.
Two details decide whether the alert is trusted:
- Give the time window a margin. A record created at 23:59 and written at 00:01 looks missing unless the destination window runs a little past midnight.
- Keep the check stateless. Read both sides on every run instead of storing what you saw last time, so there is nothing to drift.
If anyone wants it, I have a tested version for orders and invoices, built on synthetic data, and can post it in Built with n8n.
Good write-up. One thing I’d add: the count check is a scheduled workflow too, so it can go quiet the same way. If the instance is down at the scheduled time, or someone deactivates it while tidying up, you get no message, and that looks exactly like a good day.
The cheap fix is to never let it stay silent. On a clean day have it send a short “all matched, 0 missing” message, so a missing message becomes the alert. If you want it fully separate, check its last execution time from something that doesn’t run on the same instance, since a check on the box it’s checking goes down with the box.
Thanks, agreed. The daily “0 missing” message is the part I’d add first.
One catch if you build it in n8n; on a clean day the In A only output is empty, and nodes after an empty output don’t run. So the zero message can’t hang off that branch; it has to come from a step that runs on every execution.
Good catch. I’d fix it by turning on Always Output Data on the Compare Datasets node. On a clean day the “In A only” branch then sends out one empty item instead of nothing, so the next step runs. Put an IF after it that checks whether the ID field exists. True goes to the “missing IDs” message, false goes to “all matched, 0 missing”.
With Always Output Data on, the item count is never 0, so count the items that have an ID, not the items.
Disclosure: we make RunLookout, which runs this kind of check from outside the server for client n8n setups. It flags missed runs and green runs where a chosen step returned 0 items, with no changes to the workflows. If you’d try it on one server, we’d like your honest take on what it misses. app.runlookout.workers.dev
That works, thanks. I checked the n8n source: Always Output Data only adds the empty item when the node’s first output comes back empty, and In A only is the first output of Compare Datasets. So it covers rows missing at the destination, but not In B only, if someone also wants to flag rows that exist only at the destination.
I’d also add one more check after the IDs match: compare the fields that actually matter for the downstream process
Just because a record exists in both systems doesn’t mean it’s correct — the status, owner, or amount can still be off. I’d split missing records and mismatched records in the daily report, since they usually point to completely different failure modes
Good find on In B only. For that side I’d count instead of branching: put a Code node (Run Once for All Items) on the In A only branch, which now runs every time, and read $('Compare Datasets').all(1).length for the In B only count. Then one message can report both numbers, missing at the destination and extra at the destination, even on a clean day.
Agreed, and Compare Datasets already splits these for you. Records whose ID matches but whose other fields differ go to the Different output, the third one, so the count is $(‘Compare Datasets’).all(2).length, next to the missing and extra counts.
Set “When There Are Differences” to “Include Both Versions” and each item lists the fields that differ, with both values. Put timestamps and other fields that don’t matter in “Fields to Skip Comparing”, or a changed updated_at alone puts a record in Different.
One message with both numbers is cleaner, agreed. Two things I found when I checked it against the source:
- In B only is the fourth output (In A only, Same, Different, In B only), so it’s $(‘Compare Datasets’).all(3). all(1) returns the Same branch, which on a normal day is most of your records.
- On a clean day the In A only branch carries the one empty item from Always Output Data, so $input.all().length is 1, not 0. Count only the items that have the ID, e.g. $input.all().filter(i => i.json.order_id).length.
You’re right, thanks for checking the source. all(1) is Same, so my count would have reported nearly every record. For anyone copying this later: In A only is all(0), Same all(1), Different all(2), In B only all(3).