Scale n8n self hosted

[guilhermemena]

Hello guys,

We have a self hosted n8n running in aws ecs and we have some troubles to scale trying to execute the following case:

Webhook node → wait → http request

Our desire is to receive a webhook, wait some time and do a http request but the following troubles appears:

1 – With queue mode

We have set the concurrency to 5 but the wait node block other jobs to run, so if we send 10 requests to the webhook node, it will add only 5 jobs with the wait of 30s, and the other 5 remaining jobs doesn’t run until the first five completed, so it’s much longer than 30s, the last 5 jobs run with at least 1min instead 0f 30s.

Is there a way to wait node not block the executions of other jobs?

2 – without queue mode and execution process as “own”

As queue mode has blocking by concurrency number with wait, we have tried without queue mode but the performance it’s pretty bad, each request that we send to webhook node adds 100mb in memory usage, so we can’t scale this because of price

Our scenario is that we will receive thousands of webhook calls that can have wait nodes with minutes to hours and we want to be able to handle that

[MutedJam]

Hi @guilhermemena, welcome to the community!

Perhaps @krynble can suggest the best approach using queue mode here?

[BramKn]

Hi @guilhermemena

My first question would be, why are you using these wait nodes? Is there also a way of pushing these to a table to handle later for example?

I have had an issue with a lot of webhooks coming in and needing time to process before the next webhooks could be processed. No wait nodes, but simply a bit of processing that takes a few seconds.
I solved that with RabbitMQ, simply pushing the requests to the queue for it to then process one by one.
You can set how many process at the same time with the RabbitMQ trigger.
This way your n8n worker queue doesn’t get filled up and other stuff keeps running and also webhooks don’t stay waiting there for a long time.