Question/Feature: Debug n8n Tool Agent (OpenAI vs Local LLM Interchange)

Questions
1

Describe the problem/error/question

Hi, firstly I think the Tools Agent works very well with OpenAI - kudos. Now, I’m developing a testing environment where OpenAI is interchanged with a locally hosted LLM (LM Studio in particular). While the local LLM API works, the tools are not being called. I suspect there is some problem in the way the JSON is handled, and here is where I am stuck. The n8n Tool Agent doesn’t seem to have a way to log OpenAI’s handling/incorporation of the JSON input. Also, there appear parameters that I cannot edit. How can I get at this information and modify if I have to. Below is the log config of the API request for the local LLM. Does anything appear to be the culprit? Thanks

[2024-08-13 14:22:53.637] [INFO] [LM STUDIO SERVER] Verbose server logs are ENABLED
[2024-08-13 14:22:53.641] [INFO] [LM STUDIO SERVER] Success! HTTP server listening on port 1234
[2024-08-13 14:22:53.642] [INFO] [LM STUDIO SERVER] Supported endpoints:
[2024-08-13 14:22:53.642] [INFO] [LM STUDIO SERVER] ->	GET  http://localhost:1234/v1/models
[2024-08-13 14:22:53.642] [INFO] [LM STUDIO SERVER] ->	POST http://localhost:1234/v1/chat/completions
[2024-08-13 14:22:53.642] [INFO] [LM STUDIO SERVER] ->	POST http://localhost:1234/v1/completions
[2024-08-13 14:22:53.643] [INFO] [LM STUDIO SERVER] ->	POST http://localhost:1234/v1/embeddings     <------------ NEW!
[2024-08-13 14:22:53.643] [INFO] [LM STUDIO SERVER] Model loaded: lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
[2024-08-13 14:22:53.643] [INFO] [LM STUDIO SERVER] Logs are saved into C:\tmp\lmstudio-server-log.txt
[2024-08-13 14:23:01.626] [INFO] [LM STUDIO SERVER] Processing queued request...
[2024-08-13 14:23:01.626] [INFO] Received POST request to /v1/chat/completions with body: {
  "model": "lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF",
  "temperature": 1,
  "top_p": 1,
  "frequency_penalty": 0,
  "presence_penalty": 0,
  "n": 1,
  "stream": true,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_property_details",
        "description": "Call this tool to get the following property data:\n*address\n*postcode \n*type of the property \n\nQuery with an array of Property IDs.",
        "parameters": {
          "type": "object",
          "properties": {
            "input": {
              "type": "string"
            }
          },
          "additionalProperties": false,
          "$schema": "http://json-schema.org/draft-07/schema#"
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "get_tenant_details",
        "description": "Call this tool to obtain the following tenant data: \n* tenancy terms \n* rent amount and \n* notes\n\nQuery with an array of tenant IDs.",
        "parameters": {
          "type": "object",
          "properties": {
            "input": {
              "type": "string"
            }
          },
          "additionalProperties": false,
          "$schema": "http://json-schema.org/draft-07/schema#"
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "format_final_response",
        "description": "Always use this tool for the final output to the user. It validates the output so only use it when you are sure the output is final.",
        "parameters": {
          "type": "object",
          "properties": {
            "output": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "tenant_id": {
                    "type": "string"
                  },
                  "tenant_name": {
                    "type": "string"
                  },
                  "property_id": {
                    "type": "string"
                  },
                  "property_postcode": {
                    "type": "string"
                  },
                  "action required": {
                    "type": "string"
                  },
                  "details": {
                    "type": "string"
                  },
                  "date": {
                    "type": "string"
                  }
                },
                "additionalProperties": false
              }
            }
          },
          "additionalProperties": false,
          "$schema": "http://json-schema.org/draft-07/schema#"
        }
      }
    }
  ],
  "response_format": {
    "type": "json_object"
  },
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful AI assistant - a Tools Agent. \n\nYour task is to reconcile rent payments with the uploaded bank statement and alert only if there are any actions to be taken in regards to the tenants.\n* Identify and flag any tenants who have have missed their rent according to the month. Late payments which are within a few days of the due date are acceptable and should not be flagged.\n* Identify and flag if any tenants have not paid the correct ammount due, either less or more.\n* Identify and flag any tenants who are finishing their rentals within the time period of the current statement.\n* Identify and flag any remaining fees which are due and have not been paid from any tenant in the last month of their rental.\n\nIf the bank statement show incomplete months due to cut off, it is ok to assume the payment is pending and not actually missing.\n\nAnswer as quickly and concisely as possible. Do not elaborate too much!\n\nIMPORTANT: Always call `format_final_response` to format your final response!"
    },
    {
      "role": "user",
      "content": "Reconcile the Bank Statement for 2024-03-01 to 2024-05-01:\n|date|reference|money in|money out|\n|-|-|-|-|\n|2024-03-01|Rent from Tenant A|1200.00||\n|2024-03-03|Office Supplies||150.50|\n|2024-03-05|Rent from Tenant B|950.00||\n|2024-03-07|Electricity Bill||200.00|\n|2024-03-09|Payment to Supplier 1||400.00|\n|2024-03-11|Rent from Tenant C|1300.00||\n|2024-03-13|Internet Bill||80.00|\n|2024-03-15|Payment to Supplier 2||350.00|\n|2024-03-17|Maintenance Expense||500.00|\n|2024-03-19|Rent from Tenant D|1000.00||\n|2024-03-21|Payment to Cleaning Service||250.00|\n|2024-03-23|Rent from Tenant E|1100.00||\n|2024-03-25|Office Rent||800.00|\n|2024-03-27|Rent from Tenant F|900.00||\n|2024-03-29|Payment to Supplier 3||450.00|\n|2024-03-31|Office Supplies||120.00|\n|2024-04-01|Rent from Tenant A|1200.00||\n|2024-04-03|Office Supplies||160.00|\n|2024-04-05|Rent from Tenant B|950.00||\n|2024-04-07|Water Bill||90.00|\n|2024-04-09|Payment to Supplier 1||370.00|\n|2024-04-11|Rent from Tenant C|1300.00||\n|2024-04-13|Internet Bill||80.00|\n|2024-04-15|Payment to Supplier 3||420.00|\n|2024-04-17|Maintenance Expense||510.00|\n|2024-04-19|Rent from Tenant D|1000.00||\n|2024-04-21|Payment to Cleaning Service||260.00|\n|2024-04-23|Rent from Tenant E|1100.00||\n|2024-04-25|Office Rent||820.00|\n|2024-04-27|Rent from Tenant F|900.00||\n|2024-04-29|Payment to Supplier 2||460.00|\n|2024-04-30|Office Supplies||130.00|\n|2024-05-01|Rent from Tenant A|1200.00||\n\nFormat your final output to conform with the following JSON Schema instance markdown codeblock:\n```json\n{\"type\":\"object\",\"properties\":{\"output\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"tenant_id\":{\"type\":\"string\"},\"tenant_name\":{\"type\":\"string\"},\"property_id\":{\"type\":\"string\"},\"property_postcode\":{\"type\":\"string\"},\"action_required\":{\"type\":\"string\"},\"details\":{\"type\":\"string\"},\"date\":{\"type\":\"string\"}},\"additionalProperties\":false}}},\"additionalProperties\":false,\"$schema\":\"http://json-schema.org/draft-07/schema#\"}\n```\n"
    }
  ]
}

What is the error message (if any)?

Please share your workflow

(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)

Share the output returned by the last node

Information on your n8n setup

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
2

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
5

Hey @Jonathan_Sherman,

From what I understand the ability to use a tool depends on how smart the model you are using is so it could be that a different model will be a better option. That being said we did have a bug in 1.54.0 with tools not working so it could be that an update to one of todays 1.54.x releses or using 1.55.0 will help with this one.

6

@Jon thanks for the reply. Now it fails frequently with OpenAI as well, with the error being

tool input did not match expected schema

I am instructing the model to pass in an array of keys, since the tools are spreadsheets to return values from. However, if you look at the tool definitions I included, there is an “input” parameter and the type is set to string. Is this correct/

7

Hi, I’m still thinking it is not simply an issue of the model being too simple. There are a couple things I’ve come across that I’d like to get n8n’s feedback on. The main idea is that perhaps n8n implementations are too high-level and should allow users to add any custom field/parameter they see fit.

See the code snippet below for the first example. It allows better definition of the function, requires specific usage, and allows for enumeration of specific items. I’d imagine this removes a certain potential for error.

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                    "format": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "The temperature unit to use. Infer this from the users location.",
                    },
                },
                "required": ["location", "format"],
            },
        },
    },
]

The second example is with LM Studio/Ollama (both fail). LM Studio, for instance, expects input schema to consist of ‘system’, ‘user’ and ‘assistant’ fields. n8n offers no ‘assistant’ field, and perhaps this is an issue for tool-calling. Additionally, some parameters are automatically set by n8n (such as streaming: True) which I would like to set as False but cannot.

I know there is the langchain code node, but the documentation and examples are sparse, and for some reason it can only be used on self-hosted (why?).

If I am wrong about anything, or you know some solution I’ve missed, please let me know. Otherwise, what to do? Langchain code node is one.

8

Also, further evidence suggesting it’s not the “model’s fault” is that n8n ReAct and Plan and Execute Agents do succeed in calling tools. It is the Tools agent that fails, 100% of the time.

And in the ReAct logs, I can see that the agent calls the tools when it communicates to itself in the role of ‘assistant’

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.