Hey all — I kept running into a subtle failure mode with n8n AI agents: they’d look like they answered from the knowledge base, but they’d actually skipped retrieval and answered from the model’s memory. Same with routing — an agent quietly calling the wrong tool or specialist sub-workflow, and you don’t notice until a client does.
So I built a small evaluation harness and I’m sharing the free version. It runs scenarios against an agent and reports pass/fail on two things: whether it routed to the correct tool / sub-workflow, and — when retrieval was required — whether it actually retrieved (the check passes only if a retrieval call was observed, the retrieved payload contained a known fact, and the final answer used that same fact; miss any one and it fails).
What’s in the free version: 10 scenarios covering routing, tool selection, sub-workflow routing, and the 3-part retrieval check — each with a positive and a negative case (the negatives fail on purpose, so you can trust the checks are real). It runs on plain Node.js — no Docker, no API keys — against a deterministic mock agent. (The free version is mock-only; wiring it to your own live agent’s webhook is in the Pro version.) Ships with a tiny unit-test file and a README.
Get it (free): GitHub - mad-ton/t005-agent-eval-harness-lite: Free Lite eval harness for n8n AI agents — pass/fail checks for correct routing and honest retrieval. Node.js, no Docker, no keys. · GitHub — clone it, then run node src/run.js --dir scenarios/lite. You’ll see 4 passing + 6 failing scenarios (the failures are by design, so you can trust the checks). Adapt the scenario files to your own agent’s expected routes/tools/facts.
Who it’s for: anyone shipping n8n AI agents (freelancers/agencies especially) who wants a pass/fail reliability check before handing an agent to a client. Who it’s not for: it’s not a hallucination filter and it doesn’t read the model’s mind — it only asserts observable behavior (was retrieval called, did the answer use the retrieved fact). That honesty is deliberate.
Limitations (so nobody’s surprised): the retrieval check matches the exact fact token in the answer, so a correct answer that paraphrases and drops the token will fail — by design, because asserting semantic equivalence would need an LLM judge. And this complements n8n’s built-in Evaluations; it just ships the routing/retrieval scenario packs ready-made.
Feedback very welcome — especially from folks running multi-agent or tool-heavy workflows. If you hit a routing/retrieval failure mode I didn’t cover, tell me and I’ll add a scenario for it.
(There’s also a production version for teams that need a larger scenario pack, editable assertions, regression history, and a reference thin-router template — happy to point to it if useful, but the free version above stands on its own.)