K8s horizontal scaling and non-http triggers

Questions
1

Hello everyone,

I’m planning for a bunch of workflows to run in a kubernetes cluster. All of them are triggered by kafka events. We usually have a preference for scaling up using K8s horizontal scaling, but I understand this would imply each and every N8n instance would be listening for the same event and get triggered, therefore duplicating executions.

I was wondering if it would be reasonable to use queue mode to still leverage on K8s HPA, for example by keeping one single instance of the N8n main process, and setting HPA just for worker nodes, so that they would spawn automatically basing on CPU/memory consumption. My understanding is that triggers would still remain on main instance, avoiding duplication of executions.

Would that be a sensible approach? Any more reasonable ways of using kafka triggers and k8s HPA? Or should I just avoid HPA and manually add workers?

Thank you very much

2

Hi @PrinceValium, welcome to the community!

@krynble would be the best person to answer questions around scaling n8n, maybe he can help out with this one?

3

Hey @PrinceValium sorry for the delay - I’ve somehow lost this thread in my notifications.

The approach you’ve mentioned is 100% correct, you need to have a single main process listening to your Kafka queue and multiple workers to do the hard job.

One thing to note is that the main n8n process will have a “listener” bound to the execution, waiting for each worker to finish. This means that if the volume of work is too large your main process might be overwhelmed. it’s hard to give an estimate of how it would behave since it has to do with the duration of your workflows and concurrency.

4

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.