I’m new to n8n and automation in general, and I’m running into an item duplication issue. I’m trying to create a Pinecone vector database for my courses. My n8n workflow pulls course data from Supabase (5 courses), generates embeddings with OpenAI, and stores them in Pinecone. Instead of 5 items, I’m getting 93 in Pinecone (as shown in the screenshot). The duplication seems to be happening between Supabase and OpenAI where the item count jumps from 5 to 15. To make things worse, the AI agent is simply responding with “I don’t know.”
Any help would be greatly appreciated!
Thanks for posting here and welcome to the community!
Unfortunately working with vector databases is not as straight forward as with traditional databases. The duplication stems from Pinecone being a completely different system than Supabase. Supabase is a postgres database-like vector store whereas Pinecone is built to cater towards the collections-approach.
I think it would be quite a difficult task to build a workflow that “transfers” vector store data from one to the other adhering to the right structures and configurations. You would be better off to isolate your vector store embeddings from your source data.
Is there a reason why you can’t “feed” pinecone directly from your source data (i.e. human-readible text) ?
