RAG - multiple files into vector database from cloud storage

Questions
1

Hi all,

I am hosting n8n on a vps and have set up the standard rag flow, using embeddings and a pinecone vector store.

Content is PDFS, held in an AWS S3 bucket. The challenge I face is being able to download and insert say, 100 PDFs into the vector database. I have no problem getting one but for many, perhaps I need a javascript node?

If anyone has figured this out, whether Gdrive, Onedrive, S3, it’d be a tremendous help!

2

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
3

Hi @boscocho,

Welcome to the community :tada:

Could you tell us a bit more about your workflow and what your struggles are with inserting multiple PDFs? Are you seeing any errors?

You can share your workflow by copy & pasting the workflow JSON between two (```) too.

4

Hi there, thanks! I need to know how to set up and configure the cloud storage so it downloads many or loops through files in my S3 bucket and inserts into Pinecone pls:

image
image1525×648 102 KB