Inserting multiple PDF documents into Pinecone

[CIFSAI1969]

Hello!

I’ve been trying to insert about 30 PDF files into my Pinecone vector database and have sometimes succeeded but recently it always ends with an running out of memory error. I’ve even tried splitting up into two seperate workflows to see if that solves the issue but it didn’t. I also tried splitting the files into batches of 2,3,5,10 and they all resulted in running out of memory.

Here are the two workflows:

[n8n]

It looks like your topic is missing some important information. Could you provide the following if applicable.

• n8n version:
• Database (default: SQLite):
• n8n EXECUTIONS_PROCESS setting (default: own, main):
• Running n8n via (Docker, npm, n8n cloud, desktop app):
• Operating system:

[CIFSAI1969]

n8n version: 1.61.0
running n8n via cloud

[ria]

Hi @CIFSAI1969

Thanks for posting here and welcome to the community!

Can you share the error that you are getting exactly? Do you have any limitations on your pinecone account?

[CIFSAI1969]

Hi!

It’s on the Pinecone node and says the following:

Execution stopped at this node

n8n may have run out of memory while running this execution. More context and tips on how to avoid this in the docs

Error details

Other info

n8n version

1.61.0 (Cloud)

Time

04/10/2024, 15.13.48

Error cause

{
  "level": "error",
  "tags": {}
}

As for the limitations, I’m not sure what you’re referring to, my account is a starter account.

Thanks for the help in advance!

[ria]

Hi @CIFSAI1969

You can try to set a Set node at the end of your Pinecone Node to reduce the amount of items that is returned in each loop.

Something like this:

[CIFSAI1969]

Hi ria!

Thanks for the suggestion! I tried it out and now it fails even faster after 5 documents with the same error

Any other suggestions?