Hello everyone! Who had experience in creating highly loaded (relatively, of course) processes on n8n? What is the best way to implement this?
My scenario:
There is a point (webhook node) for receiving data, which works somewhat in a short period of time. This data comes from the Gupshup service, which sends several types of requests with different payloads to one point. These requests contain data from instant messengers (for example, WhatsApp), which reflect various events (for example: the message itself, status read, etc.).
When messages arrive in a large stream, then our server, on which n8n is installed, stops responding to incoming requests.
Current server configuration: 4 cores, 8 gigabytes of RAM.
Now the server “falls” if you try to send a few dozen messages.
What needs to be done to prevent this from happening?
Depends on the mode.
If you run in “main” mode you have to be aware that it will just use one CPU. So the only thing you could do there is to get a faster CPU.
If you use “own” mode it will use all CPUs but will be much slower to respond and use much more RAM. So what you can do there is to increase the RAM.
Generally would be best to set up n8n in a scaling way with a queue. You can find the documentation about it here:
Great to hear! So I assume you did run before in “own” mode. Btw. if that is not enough anymore also the above mentioned scaling mode is almost as fast and allows you to easily add more workers if you need more resources.