I am having issues with a workflow I created, where the insertion of documents via the Qdrant Vector Store is becoming increasingly slow, meaning I cannot insert the data I want to Qdrant.
Currently I am trying to insert 100,000 tickets of data into Qdrant using the Qdrant Vector Store node. As embedding I set openAI’s embedding-small model, and for the default data loader I setup the data and metadata and wired the data.
This data is being programatically retrieved and cleaned up before being passed onto the vector store, even sorting it into a JSON with pageContent and Metadata fields. Since we are talking about 100,000 tickets, what I did is basically split it in batches of 100 items. I have tried many different combinations of batch splitting and what not, but the issue is that after 200-300 inserts, the data loader begins slowing down considerably, to the point that a batch of 100 tickets can take up to 5-10 minutes to insert. I have never managed to even load the first 1000 items completely.
Could someone help me in understanding where perhaps I am going wrong with this? I understand that maybe this is not the most efficient way of inserting data into qdrant, since its going over every item one by one and processing it.
I was looking with chatGPT as to possible solutions and it suggested to use the batch upload endpoint from qdrant, but the issue here is that in order to do so I would have to first embed the data before passing it into that batch endpoint, but the embedding openai node is a subnode, not a standalone node, so I cant wire it into the workflow.
So @gabz_am you’re trying to insert 100,000 tickets into Qdrant using n8n, and it’s crawling slowly. Yeah, I get it I ran into something similar before. Here’s what’s could’ve the problem:
OpenAI node is embedding each ticket sequentially which could be a reason for the sluggishness
Even with batches of 100, n8n is still making tons of API calls, and each one adds overhead
Qdrant’s Default Node Isn’t Built for Bulk
Solution:
Use OpenAI’s API directly to embed all your tickets in big batches. Then, push the pre embedded vectors straight into Qdrant using its batch endpoint.
Let me know if this helps and mark as the solution to help others
With regards to what you said, my question would be: How can I use the embeddings-small-3 as a standalone node? Currently when I search it in the list of nodes, it only appears as a subnode, which means I would need to have the qdrant vector store node in use.
Im very new to this, so im sure there are a few mistakes (as can be seen by the format for qdrant node)
Essentially, its like this:
(Node 2) First I set a pagecount to iterate 1110 times through the loop (probably not the most efficient right now, but im just testing since I cant get it to work at the moment):
(Node 4 and 5) Authenticate to our ticketing tool system and extract the data I need
(Node 6 - Parse Data) Simply clean up the data returned, since it doesn’t come out in a good format. I basically split the extract into 100 json items, each being one ticket of information.
(Node 7 - Classify Data) I structure the data in such a way that I get a return of 100 items, each item being a pageContent (which is basically the key information) and metadata. Example output below (sorry for blurring, have sensitive data):
(Node 8) Embeddings HTTP request via OpenAI. I think its working fine, but its definitely slow now. In my previous worknode where it was a subnode, the embedding was rather quick for batches of 50 inserts, the data loader was just slow:
