Has your agent ever claimed it did something that didn't actually happen?

Feedback General
, , , , , ,
1

Not an error. Not a crash. The agent says “email sent” or “refund processed”, the trace looks clean, no exception anywhere — and the thing never happened. Hit this a few times and it bothers me that every observability tool I’ve tried reports it as a success, because from the trace’s point of view it is one.

Two questions for anyone running agents or workflows in production:

Has this happened to you?

If yes — how did you find out? Customer complaint, or did something catch it ?

Genuinely curious whether this is common or whether I’ve just built things badly.

2

Hi Aditya,

This is a common issue known as Silent Agent Failure. It happens when the LLM assumes a tool succeeded based on text output, or a tool fails silently without raising an error code.

Here is how to prevent and catch it:

  1. Deterministic Verification: Add a step right after the agent to independently verify the result via API/Database (e.g., check Stripe to ensure the refund is actually logged).

  2. Require Execution Proofs: Force tools to return strict JSON with specific IDs (transaction_id, message_id). If the ID is missing, treat the step as a failure.

  3. Strict Sub-workflow Errors: Ensure tool sub-workflows throw explicit errors (On Error: Stop Workflow) on non-200 responses so failures aren’t hidden behind clean text.

  4. Audit Cron Jobs: Run periodic checks to match requested actions against actual backend logs to flag mismatches early.

3

Thanks — the deterministic verification point is the one I keep landing on too.

Going back to my actual question though: have you had one of these in production yourself? I’m less interested in the prevention patterns than in what it looked like when it happened — what the run reported, and how long before anyone noticed.

Asking because everyone I talk to agrees it’s a problem in principle, and almost nobody has the story.

4

Fair question! To be transparent, I haven’t personally lost money to a silent agent failure in my own live production setup yet—mostly because we were warned early on to set up verification steps.

However, a colleague in the automation space experienced this exact scenario with a support agent issuing refunds:

  • The Setup: The agent triggered a tool to call Stripe for partial refunds.

  • What the run reported: The n8n execution trace was completely green (200 OK), and the LLM confidently replied: “Issued a $20 refund.”

  • What actually happened: The sub-workflow hit an invalid currency code error, but the API tool caught the error gracefully and returned { "success": false } inside a 200 OK status. The LLM saw the text response, ignored the false flag, and hallucinated that the action succeeded.

  • How long until noticed? 3 days later, when the customer followed up asking where their money was.

So while it’s a simulated story based on industry patterns, the 200 OK error-masking issue in n8n is very real!

5

Yes, it’s definitely happened to me, and I don’t think it’s a sign you’ve built things badly. The trace only knows the steps ran. It has no idea whether the thing you wanted actually happened in the real world.

The first time, I found out from a user. Since then I’ve done two things:

  1. Check the result after every important action. For example, look up the refund or email by the ID the tool returned. If it isn’t there, throw an error with a Stop and Error node.
  2. Run a daily reconciliation workflow. It compares what the agent says it did with what the external system shows, and alerts on anything that doesn’t match.

Also worth checking: in some cases the agent never called the tool at all and just said it did. The execution log shows this right away.

Curious whether yours were mostly the agent hallucinating the action, or the API accepting the request and failing later?

6

This is the exact distinction that led me to build UAEP:

EXECUTION SUCCESS ≠ VERIFIED DESTINATION REALITY

A clean trace establishes what the workflow recorded. It does not necessarily establish that the authorised destination objective became true.

I would also be careful about treating a returned transaction_id or message_id as automatic proof. An authentic identifier can still belong to the wrong resource, version, execution or environment. A provider can accept an operation without the final destination satisfying the objective.

UAEP separates:

• the agent or workflow claim;
• the executor or tool result;
• independently observed destination reality;
• whether the evidence applies to the exact objective, execution, resource and authority;
• whether recovery actually closes and reverifies the original objective.

The result becomes VERIFIED only when the required destination facts are established. If an action may have committed but its response was lost—or the evidence is stale, incomplete or mismatched—the truthful result may remain UNKNOWN / NOT_VERIFIED rather than being forced into SUCCESS or FAILURE.

UAEP is not just a theory. Its current implementation is complete and internally qualified within its declared evaluation boundary. It has been exercised against false executor success, wrong destinations, stale and wrong-resource evidence, partial effects, ambiguous commits, unsafe retries, conflicting observations and recovery that ran without closing the original objective.

Within those bounded qualified demonstrations:

FALSE VERIFIED = 0

That is bounded evidence, not a claim that UAEP cannot fail. I have opened the implementation to a controlled falsification challenge:

For an n8n workflow, the key design question is:

What destination-connected observer can establish the required state independently of both the AI Agent node and the tool path that produced the success claim?