Is it possible to mock AI Agent tool

Questions
,
1

Hello,

My question is related to mocking AI Agent tools.

Here is my scenario:

I want to test my workflow, with some different test scenarios.

These scenarios are json based objects that contains mocking data for the tool calling (the tool name, and the mock response).

example:

{
   "test_id": "111",
   "setup": {
       "user": {
          "name": "adam"
       }
       "tool_mocking": [{
          "tool_name": "greeting-generator",
          "tool_response": "welcome"
       }]
   }
   "expected": {
      "tools_executed": ["greeting-generator"],
      "response": "welcome adam"
   }
}

I’m not sure how to imlpement this in n8n. I was thinking of 2 ways:

  1. set a “Mock Node” tool before each AI Agent tool, that will check if we are in a test execution. if so, call the mock node instead the ai agent tool node
  2. set one global “Mock Node” tool, that every agent will be connected to. This tool will check if the enivornment is “mock enabled” and will check the json’s input for a valid mock definition. If it finds it will be invoked.

There are different issues with both ideas:

  1. The name of the tool will be different then the actual tool (Mock Node vs greeting-generator).
  2. Relying on the model to invoke the mock can cause a non-determenistic behaviour that sometimes might call the real tools instead of the mock tools.

How would you suggest me to handle it?

P.s

As a feature request I can suggest the following:

Adding a new feature to the AI Agent Tool that will be connected to a “Code Node” / or any node that can return output easily. Whenever such the tool will be invoked, the AI Agent Tool will systematically check if the current execution is “mock-enabled” and if so it won’t activate the agent but the mock.

Hope it is clear

Information on your n8n setup

  • n8n version: 1.109.0
  • Running n8n via (Docker, npm, n8n cloud, desktop app): n8n cloud
1 Like
2

One way to do it:

I messaged the mode with the mode set to PROD first, then I changed the mode to MOCK and messaged the model again:

image
image792Ă—300 9.27 KB

3

Thanks for you answer, This does achieve the goal but have several drawbacks:

  1. maintainence - every tool will require to have a “duplicated” tool.
  2. not determenistic - since these are different nodes, you can rely that the production node tool will be called.
4

There is no such thing as deterministic when it comes to LLMs. It is always a matter of interpretation of the combination of your instructions along with LLMs training data as well as the intensity of solar flares and whether the Mercury is in retrograde or not :slight_smile:

Also remember the “fast, good, cheap” rule :slight_smile:

5

That’s a really clear suggestion, thank you! I like the idea of having the AI Agent Tool smart-check whether execution is mock-enabled before running. It would definitely make testing smoother and avoid unnecessary activations. Appreciate you laying it out so clearly.

1 Like
6

I agree with you that try to acheive on truth (determenistic) with LLM is difficult. With that said I think we still need to aim to get determensitc results as much as we can. Especially for evalution and test methods for workflow.

In this case, I’m not looking for determenistic within the LLM itself, but just within the tool node that wraps the LLM invokation.

Mocking is common approach in SWE and testing strategies, and having it will allow scaling n8n workflows into production grade.

1 Like
7

I totally agree with you and this could make a good FR here.

8

It seems that there’s a forgotten FR about this, this one.

I also agree and see the value on pinning tools output, maybe we can resurrect it with the power of the community? :flexed_biceps: