But this doc doesn’t explain in details what must be taken in consideration in scalable setups like I want to implement.
From what I can understand:
Database/PostgreSQL must worry more about storage (SSD storage) for the needed storage of data your workflows generate.
Redis, must worry more about memory, since this is a volatile database.
N8N main instance, since we are delegating all the work for the worker instances, and webhook calls to the webhook instance, my guess is that the main instance does not need to have much memory.
N8N worker instance (with the sidecar n8n-taskrunner), here I think is the most resource intensive piece of the scalable architecture, so my guess is that it would need some CPU (for the taskrunner to run javascript/python), and memory for the workflows.
N8N webhook instance, this instance might be the least needing of resources, even less than the main instance, since it will only be available in the load balance to receive webhook payloads and distribute for the workers to do the resource intensive job.
Am I following the right idea on the resource separation for this architecture? I wish someone more experienced with this kind of setup could give me tips on how to distribute resources.
P.S: My company intends to scale N8N for multiple clients, so we are in an early phase of preparing the environment for that, it’s going to be a lot of try and error, but I’m asking here in the N8N community forum first to try to reduce the error part haha.
Looks about right. But don’t underestimate the main instance. It handles all UI and scheduled triggers for example so you should give it enough resources.
It all depends on workload though.
For example event based workflows don’t use too much resources. Where sync flows with larger data sets can use quite a lot.
As u are talking about multiple clients on your instance, please make sure to also carefully read the license.
I didn’t think before about the scheduled triggers, so only the main instance handles then?
And regarding the UI, what I’ve noticed when working with large workflows, with many nodes being loaded on screen on my browser, is some slowness, but I thought this was a client-side problem, and not server-side, because my browser had to load all the UI elements in memory on my computer. So, having a main instance with more resources would help with the UI part as well?
But in this case, depending on the workload I would need more or less workers in the pool of n8n workers right? Some auto-scaling strategy could help in this case I think, when the workers are filled up I could automate the instantiation of a new worker in the pool.
We are using the self-hosted community edition for now, and when I say ‘multiple clients’, it is not in the sense of multiple users accessing the n8n instance, but in the sense of my n8n workflows receiving webhooks from multiple clients of my company. Is it the point you brought regarding the n8n licensing?
I would highly recommend using the benchmarking tool to test and tweak your setup.
You can also have a look at this video of how to use the benchmarking tool and what generally impacts performance to get you started
Then lastly do understand it all depends on your workflow automations and how data heavy the processing of these would be. n8n workflows have a limitation of how many workflows it can execute per second for a single instance (220 as per the documentation). n8n is also based on nodejs in the backend and might not be suitable for heavy data processing tasks, easily done with other products built with Java, Rust, etc which is more performant for heavy loads. This might just mean you need scale up and build more efficient workflows
The benchmarking tool can simulate some different workflow scenarios, however performance will depend on actual real world workflows
Then some other things to consider, is that it is not only CPU and RAM which will have an impact, but the DB connection pool management as well as the type of HDD where SSD is of course a lot faster.
This will help a lot, since here in my company we usually use k6 to load test our products and I was unaware that n8n offers benchmarking tools with k6 for this use.