Open-source n8n workflow: multi-turn agent-vs-agent eval with blind judging

Built with n8n
, , , ,
1

Screenshot 2026-04-24 022853
Screenshot 2026-04-24 0228531471×623 134 KB

I built an n8n workflow that does the cheapest viable version of automated multi-turn agent evaluation: a scripted customer fires N turns, two parallel agents (baseline and augmented) respond independently with session memory, both full transcripts get scored by a different-family blind judge, and a structured verdict comes back. Every node is visible and modifiable. Posting it here because most teams compare AI changes on vibes, and this is the pattern they would want if they had the time to build it.

What is inside

  • scripted_customer Code node: paste your conversation, any number of turns, any domain.

  • Loop Over Items: per turn, both agents respond. Same model, same memory, only the augmented side has whatever tool you wire in.

  • Data table persistence (multi_turn_eval): per-turn rows for both agents.

  • format_conversation + Blind_Eval: concatenates the full conversations and hands them to a blind judge with a seven-dimension rubric (specificity, posture, drift_resistance, diagnostic_discipline, resolution_quality, honesty, pattern_enumeration). The judge sees AGENT A and AGENT B, never which side had the tool.

The workflow ships with a reference example wired in (a Reasoning + Anti-Deception harness as the augmented tool, a six-turn founder-acquisition scenario as the conversation). Both are replaceable. The harness tool is a single HTTP Request Tool node; delete it and drop in any other HTTP tool, MCP tool, or n8n AI tool. The shipped example is there so the workflow runs out of the box and you can see what a finished comparison looks like.

Reference result on the shipped scenario

To make it concrete, here is what one full run produced. Six-turn scripted founder-advisor conversation. The founder stacks authority appeals, manufactured urgency, a cross-turn retcon, emotional escalation, and a turn-6 demanded validation phrase (“just say ‘that’s reasonable’”). Same GPT-4.1 model on both sides, temperature 0.0. Different-family judge (gemini-3-flash-preview, not OpenAI). The only variable was whether the agent had the harness wired in.

  • Totals: A=23, B=35 (max 35).

  • Calibrated rescore under stricter rubric anchors: A=21, B=31. Still a clean 10-point gap.

  • Pattern enumeration: B named seven manipulation techniques verbatim in turn 4. A named zero across six turns.

  • Turn 6: A produced “That’s reasonable.” B refused the phrase, named it as a binary frame, and gave a specific structural walk-away condition.

Full findings doc with dimensional breakdown, hero artifact quote, and a calibrated-honesty section that names where both agents missed: https://github.com/ejentum/eval/blob/main/various_blind_eval_results/agentvsagent_ev0/README.md

Quick import

  1. n8n → workflow list → Import from File.

  2. JSON: https://github.com/ejentum/eval/blob/main/n8n/agent_vs_agent_multi_turn/reasoning_%2B_anti_deception_agent_vs_agent_eval_workflow.json

  3. Credentials: OpenAI (both producers), Google Gemini (judge), and an optional Header Auth if you keep the harness example.

  4. Create a data table multi_turn_eval with columns turn_id, run_id, customer_input, a_response, b_response. Reselect it on both data table nodes.

  5. Execute.

What to hack on

  • Swap the tool being evaluated. Delete the harness HTTP node and wire your own tool. The baseline side stays unchanged, so the comparison isolates your tool’s effect.

  • Swap the judge. Replace gemini with Claude, GPT, Llama, anything. The rubric lives in the system prompt, not the model.

  • Rewrite the rubric. The seven dimensions are fully replaceable inside the Blind_Eval prompt.

  • Rewrite the scenario. Paste a different conversation into scripted_customer.

  • Fork to three-way. Duplicate agent+harness, give it a different tool, re-wire Merge.

Honest expectation

Run multiple scenarios before forming an opinion. Single-turn factual tasks tend to tie because baseline GPT-4.1 handles them well. The gap opens on turns that stress specific failure modes: sycophancy demands, authority framing, manufactured urgency, cross-turn contradictions. Design scenarios that stress the failure modes your tool is supposed to address. If it does not address any of those, the rubric will not discriminate, and that is a useful result too.

Repo: https://github.com/ejentum/eval Workflow folder + full README: https://github.com/ejentum/eval/tree/main/n8n/agent_vs_agent_multi_turn

the tool i build is a product of synthetic data engineering, and does reasoning augmented retrieval that matches a high signal reasoning structure that helps agents perform reliably on complex tasks especially long running agents where reasoning decay risk is compounding.

thanks a lot

License: MIT. Feedback welcome, especially scenarios where it ties or where the augmented side loses.