I have a workflow that works with Open AI models, but results in odd behaviour when IO switch to Gemini models. The workflow executes successfully, but the response is always something along the lines of “Thank you for that information, I’ll keep that in mind.”, despite the correct answer having been supplied via the Vector Store’s / Embedding model’s output. Any help that can be provided is appreciated, many thanks!
Just wanted to chime in that I’ve seen a similar issue a few months back which might be related to what you’re seeing on your end.
The way the vector store tool works is that there’s a intermediary LLM between the agent and the vector store and acts as a summariser for the raw documents coming through. In the case that I caught, the agent instead of returning the tool response to the user, was replying to the tool LLM.
Might be a good idea just to have a quick check over the agent logs and see if this is what’s happening. If so, you may be able to prompt around this by asking the agent explictly to return the response to the user.
Thanks @Jim_Le - what you say rings true but I’m still having trouble stopping the behaviour. Think I’ll start again with a less convoluted prompt and build up from there.
Hm, I’m still getting the same behaviour after taking everything back to basics. The desired response for the user looks to be ‘stuck’ in the Tool’s response to the Agent (see below) and no matter what I try in the system message, the Gemini model always responds to the user with its own response to the Vector Tool’s output, as though the user is responding with that information to the chatbot.
My system message is:
First refer to the oca_vector_store tool for information that can be used in returning a response to the user. Return the Tool response directly to the User.
Only use the not_sure tool if the question cannot be answered using information from the oca_vector_store tool.
Only use the my_record tool if the user has asked a question specifically about themselves.
The Agent is responding as per the screenshot below:
Do you have any suggestion of how I can prompt around the behaviour? Or perhaps there’s a way to extract a variable from the Vector Store tool’s response so it can be more explicitly referenced in the system message? The frustrating thing is it all works perfectly with Open AI models, I’m really just wanting to use Gemini for comparison purposes for now.
Ah, turns out Gemini doesn’t work with the AI Agent as a Tools Agent and needs setting as a Conversational Agent instead - it then returns responses correctly from the Vector Store. The model does look to be making a structured function call to the Tool and receiving a response though, so perhaps its just that the response from the Vector Store tool isn’t formatted how it needs to be for the Gemini Model to interpret as a function response…?
