Sure. Person A asks the agent a question, it uses RAG to extract information, then Person A can evaluate the truthfulness of the answer from the agent. Admin can go through all evaluations, review and rerank answers.
Got you. I think you could do most of that with n8n, although getting data from past execs into your dataset might be a bit painful (and it’s something we’re planning on working on)
- You add the question and retrieved information as highlighted data, so that it’s easily readable when viewing a past exec
- Person A reviews the past execs, using the thumbs up/down buttons
- You extract the highlighted data from execs marked with a thumbs down into a Google Sheet or CSV
- The admin reranks them in a separate column
Thank you for this. Is there a template or something you can point me to, that I can repurpose?
Not really I’m afraid, since most of the process I just described takes place outside of the editor.
I really hope that the Google sheet integration is just a fast way to get started quickly and will not stay as the only way to manage the evaluation data. And that there will be a place to manage the input and output within n8n itself.
Somewhere built into n8n would be much nicer for me and people that I have talked to about this, even if setting the data inside of the trigger node its self.
Since I build a beta version of this myself, I still feel really strongly that managing the data inside of n8n is doable and user friendly.
Also, i know it’s not what the n8n team is focusing on right now, but I need to keep saying we want testing for more than just AI workflows. Since i made that presentation I have spoken to a lot of people who told me they really want and need this in n8n.
I am very excited for this feature to be coming natively to n8n. I hope it gets to where the community is expecting
3 curtidas
Liam, you’re absolutely right that Google Sheets is just a MVP way of getting this feature out and learning how people use it. We have plans to implement data tables in n8n and they would be an obvious data source for evaluations to use. Viewing the inputs/outputs in evaluation results is also on the roadmap (sooner than the data tables, actually).
Out of curiosity, what don’t you like about the Google Sheets solution? Is it having to switch between two apps, or more than that?
Switching between apps is inconvenient, but it’s moreso the dependency on one specific external app without another option.
I’m glad that the evaluation system was moved to a node approach from how it was before, so thank you guys for taking the communities feedback on that
My biggest thing would just be to also build this for “more traditional” logic/unit testing too so we can evaluate/test more than just AI
I’m hoping the output won’t be locked to the number type for very long, even for testing/evaluating AI I would see Boolean being much more useful (for me) so I can consider runs a pass or a fail
Also it’s exciting to see some sort of table tool coming to n8n. Hopefully something that has some SQL functionalities? Is this upcoming feature talked about anywhere yet?
1 curtida
That’s very nice @sirdavidoff! I have requested the access.
By the way, the n8n team read my mind xD
Some days ago I published a free integrated tool at n8nhackers.com. This tool generates a report scoring for any existing workflow after adding your n8n.
It only analyzes json itself. Obviously, it’s an alternative approach to be lightweight and avoid n8n executions.
You can know more at
Will try it out.
Working on making something, so once that’s done, I can use this feature to test 
As @liam mentioned in that reply, I would really love this to have more trigger options than just Google Sheets. I am starting to develop testing methods on our self-hosted n8n instance for an enterprise application. Our instance does not have a Google service account or access to an existing shared drive, but I am currently working with the enterprise team to help set up something. The feature looks very useful and helpful to the goals of testing these non-deterministic workflows!
That’s good to hear. 
Just a couple points for feedback from my side:
- I think evaluations shouldn’t be geared just towards AI. I think this should be developed out into an actual test framework for n8n workflows. That could be super powerful.
- Definitely needs more options than google sheets - at least have all the major databases / data stores as an option (MySQL, MSSQL, Postgres, Supabase)
1 curtida
Thanks for the feedback, @EricT - both points make sense.
It would be great to have other options in addition to Google Sheets like postgresql or ideally any structured database.
Well i just discover that running Evaluations costs workflows on the cloud hosting - this is not ‘free’ like testing a normal workflow. 
I blew thew 500 executions on my limited cloud hosted account pretty quick. Yikes!
Ya this is a huge buzzkill for me.
come on now….. is this where you want to make a paywall stand?
I’m about to go brush the dust off my old evaluator - at least it’s not killing my execution usage.
I hope this decision is not final.