I am trying to use Evaluation to test the output of my agent. I already successfully played with evaluation metrics. Everything went well until I tried to use it again today.
When I go to “evaluation” tab I get the message
Limit reached. Your plan includes custom metrics for one workflow only. Upgrade for unlimited use or delete the workflow with existing evaluation runs.
From there, it’s possible for me to run the evaluation, but impossible to see the results in the tab. I always get the message above and cannot see anything. I tried to archive my previous workflow since I was just playing with the feature but nothing changes.
From my understanding, I need to get an enterprise plan but I am just self-hosting it and I would really need this feature but I cannot spend that kind of money.
My question: is this feature totally locked up behind the highest plan (not mentioned in the documentation or pricing page btw) or is there something I am missing ?
In newer versions of n8n, soft delete is enabled, meaning you can’t directly delete workflows. And archiving a workflow means hiding it, not totally removing it. Go to the ‘Overview’ page where you see all workflows and toggle on ‘See archived workflows’ option from filters(refer to image). Afterwards delete all nodes in that workflow and save. You should be able to use evaluation for your new workflow.
That might solve the problem but what if I want to run evaluation (without custom metrics) in two different workflows ? The problem here I can have one and only one evaluation workflow in my entire instance.
Only one active evaluation at a time allowed in your plan.
Limit reached. Your plan includes custom metrics for one workflow only. Upgrade for unlimited use or delete the workflow with existing evaluation runs.
@Mookie_Lian
If this is the case, I think the error message is not very clear. I ran into the same issue yesterday and was confused because my setMetric nodes in workflow 1 didn’t contain any custom metrics, yet I received the same error message as OP in the evaluations tab in workflow 2. Maybe a better error message would be:
Limit reached. Your plan includes evaluations for one workflow only. Upgrade for unlimited use or delete the workflow with existing evaluation runs.
