Duplicated data created and sent to pinecone index

[Tatsu]

Hi n8n experts!

question

I’m trying to process the following flow;

• created Q & A contents from the email text by GPT-4o and send them to Pinecone node
• Pinecone node embeds those data by using GPT-4o, connected “Default Data Loader” as Document and send the outputs to Pinecone Index.
  But Pinecone node creates multiple same data then send them to Index. I guess I can not use this Index because can not keep the accuracy for data manipulation.
  It would be highly appreciated should you kindly advise how to modify them.
  
  

Information on your n8n setup

• n8n version: 1.103.2
• Operating system: Mac

[alexhuy]

@Tatsu

I’m trying to process the following flow;

• created Q & A contents from the email text by GPT-4o and send them to Pinecone node
• Pinecone node embeds those data by using GPT-4o, connected “Default Data Loader” as Document and send the outputs to Pinecone Index.
  But Pinecone node creates multiple same data then send them to Index. I guess I can not use this Index because can not keep the accuracy for data manipulation.
  It would be highly appreciated should you kindly advise how to modify them.

Hello,
To address the issue you’re encountering, You should consider adding a Set node (or equivalent preprocessing step) to normalize your input data before passing it to the Pinecone node. This will help avoid duplication and maintain consistency across your vector index.

[Tatsu]

Even got one advice, but not solved yet.