I have looked through other nested batch queries and some of the responses have helped but I am still having issues. My scenario is, I have 6 items (1000’s in reality), I want to first split the 6 into a batch of 3 and then process those 3 items in a batch size of one, then get the next 3 and again process them in a batch size of one.
Only the first batch of 3 gets processed. I would like the outer batch to run, get the first 3 items, pass it to the inner batch, which would then process each one of those 3 items. When the inner batch has processed the first 3 items, the loop should return to the outer batch, fetch the next 3 items and the inner batch would again process these 3 items individually.
Here is the workflow code:
Any help or tips on making the flow more efficient would be greatly appreciated.
what are you trying to achieve here, is this posting to a HTTP node, as you can do some batch sizes in there are wait times between requests, maybe that might be of use?
I am getting product data (1000’s) from an API and trying to create them in the product table in my SQL DB. If I process all at once, it does work but it gets quite slow so I wanted to do it in smaller batches.
What bit gets slow the SQL Process or n8n? Rather than making a batch then another batch it may be better to just create one batch of say 100 and see how that goes and tweak and as needed.
How do you have n8n deployed as well some of it could come down to hardware limitations.
From what I could see, the SQL insert process gets slower. The machine n8n is running on is very well spec’ed so I don’t think it’s a resource issue. If you see my workflow above, it has just 6 items but not sure how else to do it so that all items get processed but in smaller batches and not in one go.
I am using MS SQL. Sorry, we can ignore the speed issue. It was the save execution progress option that was making it slow. Thank you very much for your help.
