When a workflow quietly stopped in your own business, who noticed first?

The ones that send invoices, route leads or message customers. They can stop without any error while the dashboard still looks fine.

Who spotted it first when that last happened to you, you or a customer?

In my case I try not to let the customer be the first one who notices :sweat_smile:

I use a central error handling workflow in n8n that catches failed workflows classifies the severity and sends an alert with the workflow and execution details.

Silent failures are the more interesting problem though.
Aworkflow can execute successfully while the actual business outcome never happens.
Monitoring those needs a different approach than just catching execution errors.

do your clients pay you separately for that alerting, or is it part of the build price?

Nobody noticed, and that is the answer I have. I have no customers, so there was no one to notice, and I found both of mine by accident while doing something else. I deleted a scheduled task, deployed, and the deleted task was still sitting in the running schedule, which is how I found the scheduler had been two weeks stale. The other one I found because I took a screenshot to show someone and saw a tool row that said nothing where a refused call should have been. Neither was caught by a check I had written on purpose. When it was a client’s workflow, did you tell them it had stopped, or fix it quietly and say nothing?

I currently treat basic error handling as part of the build.
If I deliver a production workflow i don’t really consider it complete without proper error handling and alerts.

Ongoing monitoring and support is a different story though.
I’d treat that as part of a recurring hosting/support package rather than the initial build price.

For the silent ones on your clients’ workflows, what do you use to catch them today, and does it cost you anything?

Nobody noticed in my case either, and I only found out because I went looking.

I pointed a read-only monitor at my own instance a couple of weeks ago and found 28 consecutive failures in one workflow, across about 24 hours, three days before I looked at it. They had been sitting in my own execution list correctly marked as errors the whole time. n8n had done its job perfectly. The cause was not in n8n at all we had migrated the app those workflows call from EC2 to Vercel, and the checklist had a line for DNS and nothing for “the thing that calls it”.

@Aghassi to your question, because it is the uncomfortable one: I told them. Not immediately I fixed it first, then said what had happened and how long it had been broken. What I could not answer was “how would we know next time”, and that was the part that actually cost me trust, not the outage itself.

@StefanGwiggner the split you describe is the one I have landed on too, error handling inside the build and monitoring in the recurring line. The part I have found genuinely hard is that a recurring line for monitoring has to be renewed on the strength of nothing having happened. When it works there is no incident to point at, so the client is paying for an absence. The only answer I have found is to send something regular that says what was checked and what it found, even when the honest answer is nothing otherwise the quiet looks like the service has stopped rather than the problem having stopped.

The read only monitor you pointed at your instance, is that a tool you pay for or something you built yourself?

That’s actually the part I’m still working on.

My central error workflow handles technical execution failures but a true silent failure is different because n8n may report a successful execution while the expected business outcome never happened.

For those I think the monitoring has to be tied to the expected outcome rather than just the workflow status.
For example:
Did the invoice actually get created
did the lead reach the CRM
did the customer message actually get sent
…

I don’t have one universal solution for that yet because the health check depends heavily on what the workflow is supposed to accomplish.

Do any of your clients pay you for that monthly hosting/support package today, and roughly how much a month?

The per workflow part is real, the check has to know what that workflow was supposed to accomplish. What does generalise is the shape of the check, and there are two of them worth keeping apart.

The first catches absence: did the invoice get created, did the lead reach the CRM. You query the target system on a schedule, independent of the execution log, and compare what landed against what the source offered. That’s the two sided version Aghassi describes, and it’s what stops a quiet day from looking like a broken one.

The second catches the case the first one misses: the invoice got created, the right number of rows moved, and the values in them are wrong. A mapping drifted, a model got swapped, an upstream field changed meaning but not shape. For that, pick one or two cases per critical workflow where you know the correct output ahead of time, a test invoice you can recompute by hand, a known lookup, and run exactly that case through the live production workflow on a schedule, separate from real traffic. You’re not checking whether it ran or how much it moved, you’re checking whether the answer still matches the one you wrote down beforehand.

The second one also helps with the renewal problem Loordhu raised. A monthly note that says your test invoice came back correct 30 times out of 30 carries a number that could have been wrong, so it reads as a check that ran, not as silence.

Not yet.

I’m currently transitioning from projects for myself, friends and family into working with external clients so I don’t have a real monthly support price backed by paying clients yet.
I just started a month ago.

The model I’m planning is a one-time implementation fee plus a recurring hosting-monitoring-support package.

I’d rather not throw out a number before I’ve actually validated it with real customers though :grinning_face_with_smiling_eyes:

The layer everyone in this thread is relying on to catch the loud failures can itself fail quietly. I went into the n8n source to check exactly how, and then watched each case happen on my own instance, so the numbers below come from my own execution list rather than from theory.

Three mechanisms, none of which show up anywhere in the editor.

An error workflow can be inactive while still being set as the error workflow on every workflow that points at it. n8n keeps the setting, the dropdown still shows the name, and when a failure happens it writes one line to the server log, Calling Error Workflow for "X". Workflow "Y" is not active and cannot be executed. That line is the entire notification. Same shape if the error workflow is active but has no Error Trigger node in it: one log line, no alert.

Crashed executions are the worse case, and @Daniel_Goh has a thread open on exactly this shape right now. When the process dies mid execution there is no run data and no error thrown, so nothing fires at failure time. The execution gets amended to crashed at the next startup, and the error workflow runs then. Not when it failed. At the next boot. If the container stays down for six hours, the alert is six hours late, and if it never comes back cleanly there is no alert at all.

So on @StefanGwiggner’s point that a production workflow is not complete without proper error handling, I would now put it the other way round: error handling is the part I trust least, because it is the only part of the stack whose failure mode is silence by design. Whatever watches an n8n instance has to sit outside it and read it back through the API. A watcher living inside the instance sits in the blast radius of the thing it is meant to report. When I switched my own error workflow off as a test, the native trace was that one log line. The outside monitor flagged it within five minutes.

@Loordhu_Jeyakumar on the renewal problem, @Adam13y’s answer is the right one and I can put numbers on it, because this is exactly how mine reports. Over the last five days it has recorded 1,409 heartbeat checks against my instance and opened six alerts. Two of those were unplanned: on Sunday my instance went quiet twice, each caught from a missed heartbeat, and both alerts closed on their own when the heartbeats resumed, one after 4 minutes and one after 29. The cause turned out to be my own laptop going to sleep, which is mostly a lesson about where not to run n8n. But that is the monthly line a client can renew on. Not “no incidents this month”, but 1,409 checks, two outages, both caught, both closed, here are the timestamps.

To answer your question directly, Vladislav: I built it rather than bought it. What I found when I looked wants a node installed inside every workflow you want watched, which puts the watcher back inside the blast radius, and means the workflows you forgot to instrument are exactly the ones you hear nothing about.

@StefanGwiggner one question back on the recurring package you are putting together, because the answer decides what the monitoring layer has to look like: are you planning to host your clients’ instances yourself, or to watch instances the clients own? Watching your own fleet and watching someone else’s turn out to be different problems, and I am curious which side you are building for.

Thanks, the inactive error workflow that still shows in the dropdown is a nasty one. Is anyone besides you running your monitor yet, and paying for it?

No. Just me so far, it has been watching my own instance for about a week. That’s the honest reason I’m in threads like this, I’m trying to find out if anyone would pay for this and for what, before I build any further.

Out of curiosity, you’ve asked everyone in this thread who pays for what. Are you building something in this space too?

@Vladislav_Vladislav built it myself. It’s small: it reads the n8n API from outside the instance, read-only, and watches for runs that failed, runs that should have happened and didn’t, and the instance itself going quiet. And to answer the question you asked Benzy before you ask me: no, nobody else is running it yet, and nobody is paying for it. Same place he described. If you run instances for clients and want to see what it says about one of them, I’d be glad to point it at it, DM me.

@Adam13y @Benzy on the renewal problem, the known-answer case is the part I didn’t have. Everything I send today is the honest-absence kind. My actual numbers since 12 Sep: 28,308 health checks against my own instance, 5 failed, and one error alert, the same sub-node failure 36 times between 11 and 18 Sep, which was real and was mine to fix. Those are numbers that could have been wrong, but none of them says a single invoice came out right.

One thing that cuts against my own setup, and fits Benzy’s point about where the watcher sits: on 25 Sep it raised a 50-second “instance down”, and in the same minute it failed to reach its own mail provider. So the likelier story is that my monitor lost its network, not that n8n went down. Outside the blast radius is right, but outside means you inherit a second blast radius of your own, and the report has to say which side blinked.

@Benzy Sorry for the late reply, I was offline for a couple of days.

Right now I’m leaning towards hosting and managing the client instances ourselves. Probably one production instance per client, with monitoring, backups and error handling included in the recurring service.

For clients who already have their own n8n setup I would still offer monitoring, but like you said, that becomes a different problem. If we host it we can monitor pretty much the whole stack. On their own infrastructure we obviously depend on what access they give us.

Your “watcher outside the blast radius” point really stuck with me btw. I already use internal error handling, but this discussion convinced me that relying only on that is probably not enough.

I’m still figuring out exactly what the recurring package will look like, so this thread has actually been really useful for me. :slightly_smiling_face:

And thanks for asking!

Makes sense, and hosting them yourself is the easier side to monitor. How many client instances are you planning to start with? If it would help while you put the package together, I can point my watcher at one of them for free and you can see what it would have told you.