Pinecone Integration - Document ID definition to use Upsert instead of Insert Operation

Questions
1

I’m using the “Pinecone Vector Store” to load data.

After inserting data for the first time in Pinecone, when I insert the newly updated data again, I will have duplicated it in Pinecone.

I checked the Pinecone has the Upsert operation, based on the ID:

Question: Is there a way to define the Document ID to be inserted on Pinecone?

So we could take advantage of this upsert feature and avoid duplicated data!

An alternative solution should be to remove all index data and insert all data again, but I would prefer to avoid it.

I really appreciate any help you can provide.

  • n8n version: 1.36.4
  • Database (default: SQLite): postgres:13
  • n8n EXECUTIONS_PROCESS setting (default: own, main): default
  • Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
  • Operating system: Ubuntu
2

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
3

Hi @jonaswebdev

Thanks for posting here and welcome to the community! :raised_hands:

Not sure if I understand your question correctly…
But do you mean this feature:

image

In the Pinecone Node set the Operation Mode to “Insert Documents” and then select the Pinecone Index as “By ID”

4

Hi @ria thank you for your response and welcome message :slight_smile:

You mentioned a feature to define the Pinecone Index by ID instead of choosing from a list. Unfortunately, it isn’t related to my question.

I’m trying to figure out a way to avoid duplicating the data from Pinecone DB when I insert data after the first time that I load data there. Example:

  • 1st insertion - All OK, no duplicated data.
  • 2nd/3rd/etc. insertion - Duplicated data…

I found a way to clear a Pinecone Index by namespace before inserting the new data and partially solve my question, see:
Screen Shot 2024-04-30 at 01.22.41

It isn’t the perfect solution, but it will help me.

The ideal solution is to define the document ID to be inserted in Pinecode. This would take advantage of Pinecode’s upsert feature and avoid data duplication.

Thank you