File upload through chat - wrong behavior when passed to Agent?

Questions
1

Describe the problem/error/question

I just created a simple workflow consisting of a Chat Trigger and an AI Agent (just for testing). In the Chat Trigger, I enabled the toggle “Allow File Uploads”, and connected the node to the AI Agent node. See the screenshot:

Flow_Screenshot
Flow_Screenshot1138×770 58.6 KB

In the Agent node, I specified “Source for Prompt” as “Connected Chat Trigger Node”, which pre-fills the value for “Prompt (User Message)” with “{{ $json.chatInput }}”.

Now something strange is going on: When I send a chat message together with an uploaded file and look at the json of the execution, I can see, that even though in the input JSON the “chatInput” is just a string, and the prompt is defined to just use that $json.chatInput, the image is passed to the prompt anyways, resulting in an error “Bad request - please check your parameters
400 JSON format issue: expected string at field messages.content got instead array”.

This is really unintuitive. Passing the image to the LLM should not happen automatically, if it is not specified in the user prompt field.

Question: Is this correct, or am I doing something wrong?

Agent_Execution_Screenshot
Agent_Execution_Screenshot1920×1089 97.1 KB

Please share your workflow

Information on your n8n setup

  • n8n version: 1.88.0 (docker)
  • Database (default: SQLite): PostreSQL
2

Hi @nsky,

It looks like the model that you chose is not an openai model, its from meta. You need to choose a model like 4o that is made by open ai.

Here’s a workflow.

3

Hi @rbreen

thank you for your reply! Yes, I am using another LLM provider, which has an OpenAI compatible API.

But anyways, it should not depend on the model used whether the image is passed into the prompt or not. I am just wondering, why n8n passes the image to the LLM without me having this specified anywhere? It just says it should pass $json.chatInput, which according to the trigger output json contains just a string. But somehow, the image ends up in the LLM prompt as well… :smiley:

4

Try passing it through a set instead. I tested this model, and it doesnt look like its getting the image now.

Best,

Robert

5

Thank you, this looks like a workaround. However, I still think this behavior is rather a bug, because it is unexpected and feels wrong. Not sure if I should open an issue on GitHub :grinning:

Edit: Nevermind, I found that the Agent node has an option “automatically pass through binary images”…