As n8n is using Bull Queue, there is no rate limiter configuration we found regarding the same, so please let us know what is the size of jobs that queue can hold and is by default all the jobs are in waiting state
Welcome to the community
This will be something that @krynble may know the answer to, I would be surprised if we had a limit set so the limitation could be down to hardware.
Have you ran into an issue when using n8n in queue mode?
@jon As part of scaling in queue mode, we are figuring out all the aspects so we just want to know that how queue is handling all the jobs/request, is there any capacity and how waiting and active states are driving in the queue.
@krynble Please help here.
Sorry for the delay in replying back to you.
As can be seen here we do not set any rate limits to Bull, simply instantiating the queue with default values.
Unfortunately I cannot tell you exactly what are the default rate limiting values (if any) when no value is set. I believe it’d be unlimited, basically capped by the provided hardware.
Also as far as I can tell, all jobs are created with
waiting status and then change to
active during execution. No other status are used since n8n handles the failures internally.
I hope this clarifies your question and if need, feel free to ask and I can try to dig deeper.
Thanks for the response @krynble, I have asked this question because we have seen that if we executes thousands of workflow executions and If we open the executions listing page from the UI then UI will stuck, becomes unresponsive and we have checked API calls as well, they are fetching active and waiting jobs from the queue and at that time we have seen spikes in queue /redis, So is it expected behaviour or how we can manage our queue/redis properly
@krynble Could you please highlight on the above question.
Sorry about the delay - it is not an expected / intended behavior and we should investigate.
We did some load testing but never with thousands of executions simultaneously so it might be worth this test.
On a side node, is n8n behaving fine other than the UI issues? How many concurrent executions do you usually have?
This information helps us throttle benchmarks for load testing.
Hi @krynble Thanks for updating.
We have executed 11k, 20k executions simultaneously.
Yes other than UI, we haven’t found anything. It is working as expected