We have a lot of new developers on our n8n server. With the increased users and larger data queries, I’m looking for improvements for scale. I’m currently running n8n on windows via node with a postgress DB on the same server.
I want to move to dockers and queue mode so I can increase the worker count.
Will that help reduce the impact of a large data workflow, allowing the other workers to continue well one gets hung up? right now the whole server gets delayed or will have a javascript heap out of memory.
Thanks
Jason
Hi @jbenway
Right now, your n8n setup is like a one-person shop where the same person answers the phone, takes the order, and cooks the food. If a massive, complicated order comes in, that person gets overwhelmed, the phone stops being answered, and the whole shop grinds to a halt until the order is finished or the person collapses from exhaustion.
Moving to “Queue Mode” is like hiring a manager and a team of chefs. The manager (the Main process) only handles the phone and the schedule, while the chefs (the Workers) do the actual cooking in the back. This means that even if one chef is struggling with a huge order, the manager can still talk to customers, and the other chefs can keep pumping out smaller orders without any delay.
This also fixes your memory crashes. Instead of one giant bucket of memory that everyone shares, each chef gets their own dedicated workspace. If a specific workflow is so large that it crashes a worker, it only kills that one “chef.” The rest of the system stays online, and the crashed worker can be automatically restarted without affecting any other users.
In short, while this setup is a bit more complex to build because you have to add a tool called Redis to coordinate the work, it is the only way to support a growing team. It transforms your server from a fragile single point of failure into a professional system that can grow as your data and user count increase.
Yes it will help. But is not a silver bullet.
With queue mode you indeed have more options to spread the load so you can fix it with that. It might not be as simple as just enabling it though also because u mention memory issues.
Hi @jbenway!
Linux + Docker is the answer.
Yes, queue mode will help with your specific problem, which is one heavy workflow blocking everything else for your other developers. The restaurant analogy above is right: right now a single big data query ties up the one process and everyone else waits. Queue mode lets you add workers so a heavy job occupies one worker while the others keep serving.
A couple of specifics for your setup. Moving off Windows-node to Docker is worth it on its own, the Docker path is the supported, predictable one and queue mode is much easier to run there. Keep Postgres, but consider moving it off the same server as n8n once you scale workers, because if the DB and the workers compete for the same CPU and memory you just move the bottleneck. Start with two or three workers and watch the resource use rather than over-provisioning.
One thing queue mode does not fix: a single workflow that pulls a huge dataset into memory in one execution will still strain one worker. So alongside the move, look at whether that large-data workflow can process in batches or push the heavy query down into Postgres rather than loading everything into n8n. Queue mode stops it blocking others, batching stops it straining the worker it lands on. What does the heavy workflow actually do, a big query then processing, or large file handling?
Thank you everyone for your input.
I sounds like I’m on the right track to scale our environment.
I’m not directly working with all our devs or involved in the data they are sending through n8n. I’ve been watching the n8n logs hoping to see something that will tell me which workflows using the most resources and cause to be busy (disconnect), but I haven’t found it yet.
Sounds like I might have my docker environment to get this migration going next week. wish me luck!
Jason
Good luck @jbenway
If you found your solution here, please mark the best answer as a solution to support the community. Best regards
Hi @jbenway
Reading through your description, I’m essentially in the same situation as you: a growing number of developers and workflows, plus some “heavy” jobs that can put noticeable pressure on a single n8n instance. I’m also currently running n8n on a single machine (with Postgres on the same box), and I’ve seen how one large data workflow can slow everything down or even trigger JavaScript heap out‑of‑memory when it tries to do too much in one go.
From my understanding and experiments so far, moving to Docker + queue mode does help with exactly the concern you mentioned:
-
The main process focuses on webhooks, triggers and scheduling.
-
One or more workers do the actual execution in separate Node.js processes.
So if a heavy workflow misbehaves on one worker, it mainly affects that worker, while the main instance and the other workers can keep running. That’s already a big improvement compared to a single process where UI, triggers and execution all live together.
That said, I wouldn’t treat it as a silver bullet. Queue mode won’t automatically fix workflows that try to load or process huge datasets in one execution. You still need to look at:
-
How much data a single run keeps in memory.
-
Whether you can batch or stream work instead of doing everything at once.
-
Reasonable concurrency per worker so you don’t overload your database or downstream systems.
My own plan is:
-
Move to Docker with 1 main + a couple of workers on the same server to get isolation and clearer CPU/memory limits per process.
-
Start using queue mode so heavy executions are pushed to workers instead of blocking the main instance.
-
Gradually adjust workflow design and concurrency once I see how the new setup behaves under real load.
So in short: yes, Docker + queue mode should reduce the blast radius of a single large workflow and make the system feel much more stable, as long as you also take the opportunity to rethink how the heaviest workflows handle data.