As n8n is using Bull Queue, there is no rate limiter configuration we found regarding the same, so please let us know what is the size of jobs that queue can hold and is by default all the jobs are in waiting state
Hey @aashish.jain,
Welcome to the community 
This will be something that @krynble may know the answer to, I would be surprised if we had a limit set so the limitation could be down to hardware.
Have you ran into an issue when using n8n in queue mode?
@Jon As part of scaling in queue mode, we are figuring out all the aspects so we just want to know that how queue is handling all the jobs/request, is there any capacity and how waiting and active states are driving in the queue.
Hello @aashish.jain
Sorry for the delay in replying back to you.
As can be seen here we do not set any rate limits to Bull, simply instantiating the queue with default values.
Unfortunately I cannot tell you exactly what are the default rate limiting values (if any) when no value is set. I believe it’d be unlimited, basically capped by the provided hardware.
Also as far as I can tell, all jobs are created with waiting status and then change to active during execution. No other status are used since n8n handles the failures internally.
I hope this clarifies your question and if need, feel free to ask and I can try to dig deeper.
Thanks for the response @krynble, I have asked this question because we have seen that if we executes thousands of workflow executions and If we open the executions listing page from the UI then UI will stuck, becomes unresponsive and we have checked API calls as well, they are fetching active and waiting jobs from the queue and at that time we have seen spikes in queue /redis, So is it expected behaviour or how we can manage our queue/redis properly
Sorry about the delay - it is not an expected / intended behavior and we should investigate.
We did some load testing but never with thousands of executions simultaneously so it might be worth this test.
On a side node, is n8n behaving fine other than the UI issues? How many concurrent executions do you usually have?
This information helps us throttle benchmarks for load testing.
Hi @krynble Thanks for updating.
We have executed 11k, 20k executions simultaneously.
Yes other than UI, we haven’t found anything. It is working as expected
Hey @aashish.jain sorry for the delay, we have been running some benchmarks and so far what I have found:
- Queue does not seem to have a size limit - i had more than 30k executions pending in queue, they were being processes by the workers. You are probably limited by the memory in your Redis only, this could be a limiting factor.
- If your queue is getting too long, I can recommend you to either add more workers or increase the concurrency for the running workers.
When starting a worker, such as with n8n start worker you can set --concurrency=100 to change the default value of 10 to a higher number.
I had success increasing this to 100 without issues, but naturally this could vary depending on the hardware you are using, the type of workflow you are running, etc. I tried increasing this to 200 and n8n would crash due to the JS event loop being too busy.
2 Likes
Could you please share some hardware specs on which you have tested that setting?
I am on concurrency=30 with 3 workers and about 5k-10k in queue.
My setup is on 8C, 12GB Instance and throws following error on the workers:
<--- Last few GCs --->
[6:0x7f93c35363c0] 765605 ms: Scavenge (reduce) 2045.7 (2082.9) -> 2044.9 (2083.1) MB, 15.5 / 0.0 ms (average mu = 0.176, current mu = 0.143) allocation failure
[6:0x7f93c35363c0] 765629 ms: Scavenge (reduce) 2045.8 (2083.1) -> 2045.1 (2083.1) MB, 17.7 / 0.0 ms (average mu = 0.176, current mu = 0.143) allocation failure
[6:0x7f93c35363c0] 765639 ms: Scavenge (reduce) 2045.9 (2083.1) -> 2045.2 (2083.4) MB, 7.4 / 0.0 ms (average mu = 0.176, current mu = 0.143) allocation failure
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory