We have a self hosted n8n running in aws ecs and we have some troubles to scale trying to execute the following case:
Webhook node → wait → http request
Our desire is to receive a webhook, wait some time and do a http request but the following troubles appears:
1 – With queue mode
We have set the concurrency to 5 but the wait node block other jobs to run, so if we send 10 requests to the webhook node, it will add only 5 jobs with the wait of 30s, and the other 5 remaining jobs doesn’t run until the first five completed, so it’s much longer than 30s, the last 5 jobs run with at least 1min instead 0f 30s.
Is there a way to wait node not block the executions of other jobs?
2 – without queue mode and execution process as “own”
As queue mode has blocking by concurrency number with wait, we have tried without queue mode but the performance it’s pretty bad, each request that we send to webhook node adds 100mb in memory usage, so we can’t scale this because of price
Our scenario is that we will receive thousands of webhook calls that can have wait nodes with minutes to hours and we want to be able to handle that