Pinecone Namespace Not Updating Per File When Looping in n8n (Chunks Inherit First Namespace.
issue is that all files are upsert in same namespace, however when i traced the Namespace variable, it is defined correctly for each chunk, but pinecone node stores them all in the first namespace variable read in frist chunk.
Hi @Mahmoud_Tarek - That’s how the Pinecone Vector Store node works and is similar to how the Pinecone SDKs operate as well. This is because it’s batching a bunch of records to upsert to a Pinecone namespace at once for efficiency. You can read more about namespaces here.
Can you share your workflow here and more about what you’re trying to accomplish?
You could set up your workflow to break the data into the namespace groups before sending it to Pinecone Vector Store node. For instance, if your incoming data has a user id field that maps to namespace, filter that data first by user id then upsert all that data to Pinecone with that user id set as namespace. Repeat for other user ids.
Thank you @jennapederson. I tried to use a switch node to separate chunks based on their namespaces so the pinecone run reset after first batch ( namespace 1 ) and it correctly stored the second batch in their correct namespace ( namespace 2 )
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.