I have been working on a workflow using an AI agent to assess project proposals. I have a PDF of project guidelines that I want the agent to use (along with other things). I’m using Supabase to store the guidelines as a vector store and adding the SupabaseVectorStore tool to the agent node with the EmbeddingsOpenAI on that. In experimenting I ran the agent a couple times with the documents in the store. Then I deleted the documents from the store because I wanted to add some fail-safes in the prompt for the agent to return “cannot assess project” if there are no documents in the store. However, the agent keeps returning assessments and not the “cannot assess project” placeholder.
The prompt reads “if no documents of file_type “project_guidelines” exist in the vector store return ‘cannot assess project’ and go no further” at the very beginning of the prompt.
When looking at the logs it looks like the agent is returning nothing from the store as it should, but in the EmbeddingsOpenAI tool for the SupabaseVectorStore node it’s returning chunks which I’m assuming are from memory or something? How can I wipe the memory of that tool? Or is this stuff now part of OpenAI “knowledge”? Do I need to be more hyper-specific in my prompt?
It almost always comes down to fiddling with the system prompt. As for memory - you either want it or you don’t want it, you cannot both like the idea of conversational memory and dislike that the agent uses it to produce answers
I’m not sure I follow on the memory idea. Like, what if in the future I change the guidelines? In that case I absolutely don’t want the agent to pull from memory the old guidelines and use those along with the new ones to assess a project. I’m trying to get the agent to not just make stuff up.
The prompt states that if the vector store returns nothing then reply “whatever”. In the logs the vector store is returning an [{}] as its output. Do I need to specify not to use EmbeddingsOpenAI memory when the VectorStore is empty?
I do, yeah. I tried disconnecting it, but the agent is still bypassing the instruction of returning a “cannot assess” message and making up guidelines for the assessment and then telling me that it’s using guidelines in my a little “show your work” section I have.
I find that when a project is first run through, it’ll return the “cannot assess” state, but if I run it a second time it starts making stuff up.
Sorry I don’t know then, for me removing the memory have always helped to make the agent forget previous conversations - if the vector db returns nothing and there is no memory and there is an instruction to always only rely on the information received from the tool X - I just don’t know where the hallucinations can come from.
Yeah, I don’t get it either. The EmbeddingsOpenAI node connected the the SupabaseVectorStore node is the one returning ghost vectors even though there are no rows in the Supabase DB table that I’m pulling from. I don’t know how to make that stop.
Also, thank you for your help. I hope I’m not coming off as short in my replies. I’m a little annoyed at the agent behavior, but not at you for helping.
Yeah, interesting behaviour. What we could do is if you share that extraction workflow here and I will try to run in myself against my supabase and see what I get in my n8n.
What do you mean “extraction workflow”? Like, everything previous to the agent node? Not exactly sure I can do that since this is for my company and not sure I’d be allowed to share that info. But, maybe I could make a duplicate workflow and set up a different DB in Supabase and see if the hallucinations continue there.
Totally hear you—isolating where an agent starts to hallucinate can be maddening, especially once you add a vector store to the mix. What we’ve noticed in larger n8n + LangChain deployments is that the memory layer often leaks “prior expectations” back into the prompt if you’re not aggressively filtering on relevance AND age.
One pattern that’s helped us calm the agent down is a two-step retrieval gate:
Fetch the top-k chunks from the vector DB as usual but score them again with a cheap cross-encoder that measures semantic overlap with the current question. Anything below a 0.3 cosine threshold gets dropped so it can’t pollute the context.
Log the final memory payload size and pass that metric back to n8n via the sendMessage node. If the payload balloons above, say, 1.5 kB we short-circuit the run and surface a “cannot assess—insufficient context” message rather than letting the agent freestyle.
Because the scoring + control logic happens inside a tiny Sub-workflow, you still get retries, metrics, and dead-letter routing for free; the agent only sees what’s truly relevant. Pair that with a Store JSON node that snapshots every retrieval batch and you can quickly diff “good” vs “bad” runs to spot rogue embeddings.
Curious—have you tried re-ranking your vector hits or tracking how many tokens the memory chunks add to the prompt on each pass? Would love to hear if that stabilizes your assessments.
Thanks for the reply! That went mostly over my head. I am relatively new to using AI agents and Vector stores so am not really up to speed on the lingo. I will read a bit more to better understand what you’re saying. I have been trying to use metadata tags in the documents and use those in the prompts. Also, I just found that if I add an actual “are their documents in the vector store?” question and store the answer to the question the returned JSON object of the answer that the Agent is returning that it doesn’t hallucinate where it once was.
