Workflow crashing mid-execution : RAG pipeline with Qdrant

Questions
1

I am building a RAG pipeline to ingest and embed large oncology guidelines using Qdrant. Since I could not ingest all of the files at the same time because of n8n’s memory limit, I first divided these files into 38 batches and stored them in my Google Drive. I then looped over each of them before chunking and embedding them.

However, my problem is that my workflow keeps crashing unexpectedly. In the latest execution, the workflow went on for almost 10 minutes during which I think a certain number of chunks were indeed embedded as I could see the field Points (approx) getting updated on Qdrant. However, the workflow crashed after that. Is this because of the time out limit from n8n’s side since I am using n8n cloud?

Is there something you can recommend me to do instead?

Please share your workflow

Share the output returned by the last node

Information on your n8n setup

  • n8n version: 1.121.3
  • Database (default: SQLite): no database
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app): n8n cloud
  • Operating system: MacOS Tahoe 26.1
2

Hi Sudarshan,

Welcome to the community.

I am not a Qdrant user but have experienced similar issues with large documents. What I would suggest is making your Qdrant update section a sub-workflow so that it is executed for a single file each time. Call the sub-workflow within your file loop, passing it the extracted file content. In the sub-workflow you could also use a Splitter node to chunk the input data (instead of using the default data loader sub-node) from the file before submitting to the vector database - this is a common pattern with PGVector (I don;t know if this is needed with Qdrant but could be worth a try if the Qdrant node is timing out).

Hope this is useful.

Regards

Simon