Execute workflow in queue is slow

[yukyo]

Describe the problem/error/question

I have a queue instance with a worker who is configured to have a concurrency of 30.
My understanding of concurrency is this worker can run up to 30 simultaneous tasks at the same time.

In my daily usage, I’m 100% happy with the performance and how good it is working.
Recently, I had to add a database query operation that require to run repetitive tasks (4 actions) to compare data.

At the beginning I built it using the Loop node, but 15k entries to run was super slow because one of the 4 actions requires running an API to download the data, the second action uses a JavaScript to extract a specific object from the array (nothing big), JavaScript executes in 7ms and last step is an IF to check is there are changes in the data.

So I built this with the idea ok 30 concurrency is like let’s say 20 actions every 3–4 seconds, that would make it in less an hour to complete the 15k entries.

But the reality is it runs aprox 10 tasks per minute… this is an image from one minute:

As you can see, most of the time, the “executions of the workflow” are 1-3 sec each one. I don’t understand why it’s just creating a few per minute… at this rate it would need a day or two to complete.

Any ideas? Should I try increasing the concurrency? Add another worker to the instance? I still don’t understand what is the difference between a worker with 30 concurrency or 10 workers with 3 concurrencies each…

My worker has access up to 32vcpu and 32GB ram. I had set up a timeout for the main workflow of 1 hour, and it hit the hour, this is the usage of the worker for that hour:

I did expect a very high usage for like 30–45 minutes, maybe 5 or 6gb ram usage and cpu but to perform this execution task at the earliest…

Thanks.

Information on your n8n setup

• n8n version: 1.16.0
• Database (default: SQLite): Postgresql
• n8n EXECUTIONS_PROCESS setting (default: own, main): queue
• Running n8n via (Docker, npm, n8n cloud, desktop app): docker image + railway

[yukyo]

I added 4 extra workers to my instance with 30 concurrences each one, so right now I have a total of 5 workers with 150 concurrence total… and the “speed” is still the same… The problem seems to be the “worker” that is running the main workflow that uses the “execute” is waiting until each workflow completes until and then starts the next so this means can’t speed up the process by adding more workers…

Any ideas how to make the main workflow holder to do not wait for each execution until it completes, and keep adding more and more executions to the pool, so other workers can pick them up faster?

[Jon]

Hey @yukyo,

You got it if one workflow is running a sub workflow it will wait for each occurance to complete before processing the next one. You could get around this by using a webhook workflow and set it to respond instantly then use the http request node to call the webhook and pass through what you need but if you do this you won’t be able to use the responses later in the workflow so you would need to make sure your error handling in the workflows is solid.