Review: Vector Store Tool (+ Langchain Code Alternative!)

Jim_Le · July 15, 2024, 11:52am

In version 1.49, n8n finally included the Vector Store Tool as part of the officially supported tools for AI Agent nodes. Hurrah!

I spent a few hours familiarising myself with the tool and wanted to share some thoughts on how I’d use it going forward.

Note:
These views are my totally my own. I enjoy talking about n8n and AI and if you do too, check out my other posts in the forum!

What is the Vector Store Tool?

As with other n8n’s AI agent tools, the Vector Store Tool extends the knowledge fetching capabilities of your agents and offers a new standard for connecting to your vector store of choice without the additional complexity of introducing sub-workflows.

To start using this tool,

Find it in the “tools” options for your AI agent.
Connect your vector store and to that, connect your embedding model.
Connect an additional language model (LLM) to the vector store.

How it works

The Vector Store Tool performs very much like the Vector Store Retriever but a key difference is that rather than returning raw results, the tool uses an additional LLM to return an “answer” to the agent’s query.

Pros

For most knowledge retrieval use-cases, this works great and introduces what is effectively a multi-agent workflow by having two LLMs talk to each other.
Very easy to understand and put together - as mentioned previously, no need for a sub-workflow tool to use the vector store nodes. Great choice by the n8n team with UI choices to keep these tools similar to their original counterparts.

Cons

In my particular use-case - which I acknowledge may not be how others typically use vector stores - I noticed that prompt instructions from the root agent may not be fully transferred to the tool’s LLM prompt, which may miss some key details you wanted to pick out from the retreived information.
Additionally, if you use the metadata as part of the response and not only for filtering, then you may lose this information in the tool’s LLM response as well.
It’s not possible to modify the generic prompt instructions of the tool’s LLM. This means a lack of post-retrieval processing before sending the tool’s response and possibly a fix for the above.

Verdict: Nice!

A great addition to the AI agent’s arsenal and looking forward to more!

Should now be the standard way AI Agents (with tools) get access to vector store resources.
Suitable for a majority of vector store retrieval use-cases without additional modification.
It may be difficult to get exact wording/extracts from retrieved documents as the result is “summarised” twice (tool’s LLM > agent’s LLM).
If you use vector store metadata for more than just filtering, you may have trouble getting this data into the response.

Ok, Let’s Go Custom Langchain Code!

So, obviously with hitting the above limitations, I had to investigate how to go about overwriting the tool’s LLM prompt. The scenario is for my recipe library in my Qdrant Vectorstore where I store the recipe ID, categories and URL in the metadata and would like to pull these back out in the tool’s response. Here’s what I came up with…

1. Build your own Vector Store Tool to Override Tool’s LLM Prompt

After trying a few things, the only way I found to do this was to use a custom Langchain Code Node implementation:

A “supply data” sub-node using Langchain Code node which replicates the vector tool node shape, basically one-to-one.
Inside, it uses a createRetrievalChain which connects to the vector store and LLM.
Since we’re building the entire chain from scratch, we can now override the system prompt with SystemMessagePromptTemplate.
Finally, returning our chain’s “invoke” wrapped in a DynamicTool allows this to be used by our Agent. Don’t forget to set the name and description!

So we kinda just rebuilt the Vector Stool Tool but this worked pretty well in my opinion. By customising the tool’s prompt, I could get the recipe attributes I needed.

More control over tool response to surface information that is needed.
Agent and tool prompts need to be aligned. More risk of breaking due to upgrades.

2. Build your own Vector Store Tool Without LLM

After implementing the above, it got me thinking if the tool really needed the LLM in my scenario. So I tried an alternative implementation without the LLM and instead with the tool returns the vector store documents directly.

Similar to the above implementation but does not use the createRetrievalChain and so no need for the LLM.
The tool simply performs a similaritySearch() and returns them as the tool’s response.
Wrapped in DynamicTool as before.

After testing, the response were much more detailed for my particular use-case. Additionally, I also found possibility for post-retrieval processing which may help if you’re worried about token count.

Keep all prompts in one place for easier maintenance.
Higher token count for agent.

Conclusion

I think both custom implementations are equally fine depending on certain use-cases but I found in mine, the extra LLM summary/answer was redundant and I’d prefer to just return the actual documents to the agent. If the new vector store tool isn’t doing for you, definitely give these a try!

Thanks!
Jim
If you enjoy posts on n8n and AI, check out my other posts in the forum!
Follow me on LinkedIn or Twitter/X.

miko · August 2, 2024, 5:01pm

hi, it is possible that Search Filter on qdrant vector store (retrieve) , Search Filter wont work connected to “Vector Store Without Chain”?

Jim_Le · August 2, 2024, 5:43pm

Yes, you’ll need to upgrade to release 1.50.0+ where search filter option was added to Qdrant vector store node.

miko · August 2, 2024, 6:00pm

Yes i have it, but filter is not working when is connected to that longchain code.

Jim_Le · August 3, 2024, 9:04am

Interesting… I tested again with all 3 examples in this post and it does seem to be working for me at least. Maybe you’ve discovered a bug?

If you are able to share your workflow and maybe some sample data (ie. how are you defining metadata?), that would help greatly!

miko · August 4, 2024, 12:31am

Hi, i just add a metadata in the collection on qdrant, and use filter like this.

{“should”:[{“key”:“metadata.CaseID”,“match”:{“text”:“SOMETHING1”}},{“key”:“metadata.CaseID”,“match”:{“text”:“SOMETHING2”}}]}

when the Qdrant Vector Store node is connected to Vector Store Retriever, filter works ok. but if it is connected to Vector Store Without Chain it dont.

I just got the collection with this simply information:

Jim_Le · August 4, 2024, 9:31am

Thanks @miko for the example.
This is very similar to what I used to test so I’m not really sure why our results are different.

From what you pasted, I think “match” only excepts “value”, “any” and “except” as a child key (docs) so “text” might be invalid?

Can you try either:

{
  "should": [
    {"key": "metadata.CaseID", "match": { "value": "SOMETHING1" } },
    {"key": "metadata.CaseID", "match": { "value": "SOMETHING2"} }
  ]
}

or using the “any” case:

{
  "must": [
      {
          "key": "metadata.CaseID",
          "match": {
              "any": [ "SOMETHING1", "SOMETHING2" ]
          }
      }
    ]
}

Zotek · October 3, 2024, 2:20am

Second build work perfectly for my use case. but Im surprise that this is not part of the node options. Thank you!

Alexandre_Leitao · October 31, 2024, 10:20am

Sorry but hijacking this lol… Ok… the whole stuff works… but when I get to the vector store (qdrant) I want to use the filter with the project_id that the agent got from another tool.

I’m trying to figure out for myself for 2 days how can I pass something to that filter field but the ony thing it sees is the chat node cant see anything the agent node produced… How would you approach that? my workflow is basically like yours chat, agent, one tool to retrieve the project one to go to the vector store that uses a qdrant vector store with open ai embedding and model

Alexandre_Leitao · October 31, 2024, 5:19pm

Found in another topic. Added another agent before the main agent to recover the ID. its one more call to AI API but then I can reference it in the vector store filter

Jim_Le · October 31, 2024, 6:01pm

Hey @Alexandre_Leitao

It’s a limitation of the Vector Store Tool that you’re unable to change this value from within the agent node itself but yes, it is generally possible that the agent can use values from tools for its other tools.

Here’s a quick template to demonstrate using the Langchain Code node.

In chat, ask “I’m looking for a spicy asian recipe”
The agent will use the cookbook tool to find available cookbooks and retrieve their IDs.
The agent will use the cookbook recipe tool using the most appropriate cookbook ID along with the user’s query.

genieyuan · December 8, 2024, 3:53pm

@Jim_Le Does it mean that we can connect to another vector store that is not in the list? Nowadays, almost all database has their own vector capabilities. Just trying to understand how I might be able to use that instead of whatever n8n provided. Thanks

Jim_Le · December 9, 2024, 12:11pm

Does it mean that we can connect to another vector store that is not in the list?

To use a non-langchain supported vector store, yes you can through the use of langchain’s DynamicStructuredTool and provided

You’re on a self-hosted version of n8n
you have some way to connect to your vector store via Javascript (REST, RPC, GraphQL etc).

Also note, If your vector store is supported by Langchain but not exposed in n8n, you might want to submit a Feature Request to the team.

genieyuan · December 9, 2024, 2:55pm

Thanks Jim. Yes, the vector store is Couchbase (if you heard of it). it is supported by langchain, but not exposed by n8n. request made. Do you know how soon they will action this?

the workaround will be using DynamicStructuredTool only?

Jim_Le · December 9, 2024, 3:00pm

Do you know how soon they will action this?

Given how many other requests there are, not very soon I was expect but good to put it on the radar that people are asking for this stuff!

the workaround will be using DynamicStructuredTool only?

Yes. I’ve done a similar exercise for Redis. You can check out the tutorial here -Using Redis VectorStore Search in n8n

frknbr · December 31, 2024, 4:20pm

Hello,
Thanks it is interesting. I would like to try your custom approach.
However I am unable to find the module you are using as your Vector Store with Chain.
It seems there is non available
Cheers,