Hilario, here is the material, in the four parts you asked for.
One note before anything else: the workflow I am describing is a reduced rebuild, not the production export. 8 nodes, no credentials, no real endpoint, no keys, no client data, fake copy and fake recipients. It imports into a clean n8n as-is, and I tested that. Everything that matters to the failure is identical to production: node order, the error-output wiring on the send, the SQL that advances the state, the cron. Everything else is gone.
I am telling you this up front rather than letting you assume you have the real thing. Tell me where to send it, a PM on this forum or an email address, and it goes out the same day.
1. What the workflow does
A cron-driven outbound cadence engine. Every 30 minutes during business hours it:
- SELECTs the rows whose next touch is due (
next_touch_at <= NOW());
- decides, per row, whether to send the next touch or end the cadence;
- POSTs the message to a self-hosted messaging gateway;
- UPDATEs the row: increments
touch_count, pushes next_touch_at forward by the cadence spacing.
Step 4 is irreversible in practice. Once next_touch_at moves forward, no later run will ever retry that touch. The row is not due anymore.
The whole design rests on one assumption: that step 3 succeeding means the person received a message. That assumption is false, and the workflow has no way to notice.
2. The incident
Time window: 31 July to 3 August 2026, one tenant’s messaging instance, production.
The messaging instance disconnected on 31 July. Nobody noticed, because nothing said so.
The engine kept running on schedule for three days against the dead instance. Over that window, 13 rows had their touch advanced for messages that were never delivered. Those 13 people are, from the database’s point of view, people who received touches 2, 3 and 4 of the cadence. They received nothing. They are now further along a sequence they never entered, and some of them reached the end of the cadence and were marked done: closed out, never to be contacted again by the engine.
Five of those executions carried an HTTP 500 Connection Closed from the gateway.
Here is the part that matters to you:
- The executions were green. Status
success.
- The n8n execution detail showed “error on node: (none)”.
- Nothing in the run distinguished those three days from a normal week. No log line, no failed node, no alert.
The cause was a two-property mistake on the HTTP node: onError: "continueRegularOutput" plus alwaysOutputData: true. So when the gateway errored, the item left through output 0, the success output, walked straight into the UPDATE, and advanced the touch. The error never became an error.
What I fixed, and what the fix does not cover
I changed the send node to onError: "continueErrorOutput" and wired output 1 into a Code node that does return [] and deliberately writes nothing to the database. The row stays due, and the next cron run retries it. I removed alwaysOutputData at the same time, because with a real error output it would only inject an empty item into the success branch.
That closes the case where the gateway answers with an error.
It does not close the case that actually worries me, and it is the reason I answered your post. This gateway returns HTTP 201 the moment it accepts the message into its own internal queue, not when the message is delivered. So:
- gateway accepted, recipient number invalid: 201, nothing delivered, touch burned;
- gateway accepted, then its session dropped before flushing the queue: 201, nothing delivered, touch burned;
- gateway accepted, message silently dropped downstream: 201, nothing delivered, touch burned.
In all three the workflow is correct, the execution is green, the error output never fires, and the database records a touch that did not happen. There is nothing in the HTTP response to inspect, because the response is honest. It says “accepted”, and it was accepted. The only evidence of delivery lives out of band, in a delivery-status callback that arrives seconds to minutes later, on a different connection, long after the UPDATE has already run.
3. What I expected
- Touch 2 sent to 13 people between 31 July and 3 August.
- Any failure to send leaves the row due, so the next run retries it.
- A run that could not deliver anything is visibly distinguishable from a run that delivered everything.
4. What actually happened
- 0 of 13 messages delivered.
- All 13 rows advanced as if delivered; several ran the cadence out and closed.
- Every run green; no error on any node; three days before a human noticed, and they noticed from the absence of replies, not from any signal the system produced.
Where I will push on your framing
You said the point is to decide whether an action should proceed before it happens, then verify whether the outcome matched. That is the right shape for this, and it is why I answered you rather than skipping another audit offer. But it splits into two very different problems, and I think only one of them is actually hard.
The pre-action gate is the easy half. “Is the messaging instance connected right now? Is this recipient reachable? Has this person already been touched in the last N hours?” All of that is checkable before the send, and the reason it is missing from my engine is that nobody wrote it, not that it is hard to write. If your pilot delivers me a gate, I can build that gate myself in an afternoon. It is worth something, but not much.
The post-action verification is the hard half, and it is the one I would pay for. The problem is not knowing what I expected. I know exactly what I expected. The problem is that the moment I learn the truth is after the irreversible write already happened, on a different channel, and the workflow that did the writing is long gone. Solving that means one of:
- holding the state advance until the delivery callback confirms, which means the engine stops being one linear execution and becomes a two-phase commit across two workflows, with its own new failure mode (callback never arrives, row pinned open forever); or
- letting the write happen and reconciling afterwards, a compensating transaction, which needs an answer for “the touch was burned, now what: re-send and risk a duplicate, or skip and lose the person”.
Either way the interesting output is not a report. It is a decision about an irreversible write under an unresolved outcome.
So, being as blunt as you asked me to be:
- If what comes back is a document telling me the 13 rows failed and I should add error handling, that has no value to me. I already know, and I already fixed the half a document can see.
- If it flags the green execution as suspicious using only what is inside the n8n execution record, I will want to know the false positive rate before I believe it. Most of my green executions are genuinely fine, and an alarm that cries wolf gets muted within a week, which leaves me worse off than no alarm at all.
- If it can express “this send was accepted but not confirmed, so do not advance the state yet”, and survive the callback never arriving, that is a product, and I would run it in production.
Tell me which of those you are actually building and I will scope the pilot to it. I would rather you aim at the hard half and fail than hand me a clean report about the easy half.
One practical note on scope: I have three more failure shapes in the same family. Naive dedup producing duplicate sends, an empty SELECT emitting a single {success:true} item that becomes a ghost record and travels the whole pipeline, and a scheduler resolving its cron in the wrong timezone and firing outside the allowed window. All three are silent and all three end green. I left them out on purpose so the pilot stays on one case. Say the word if you want them.