N8N sequential executions and Kubernetes

Questions
1

I have read:

My current setup is all workflows in main process and kubernetes scaling n8n to two instances with load balancer. The workflow is triggered by webhook and processes alerts. Workflow requires jira: querying, modifying and creating tickets.

Would i have any advantage switching to queue? The documentation does not seem very clear about how the “main” mode works. You can run all in the same process but have separate threads. Quite frankly, one of the reasons that prompted me to switch to main is because i expected the executions to be sequential. This would avoid racing conditions and would lower the load on JIRA.

This question comes in the sequence of an issue i had with the Bash module where i am getting intermittent errors which seem to stem from parallel executions. I have created an issue:

and i have also noticed that some people asked for sequential execution as well:

TLDR;

  • Does n8n support sequential execution?
  • Given i use currently main mode for workflow execution of a webhook wokflow, will i have any advantage in switching to queue?

Many thanks in advance :slight_smile:

4 Likes
3

Hey @bmb very interesting questions you’ve brought.

So, let’s divide the answer in separates parts:

Your current deployment

Using kubernetes to scale the number of n8n instances with a load balancer is fine but can have problems in the future IF you decide to use any trigger that is not based on an HTTP Request (a cron, interval or similar, for instance).

This will happen because you now have multiple n8n instances doing the same thing, which is triggering an execution once some time has passed. You would have duplication of work.

This is one of the reasons we have separated the workers and webhooks to separate processes, so these can be scaled.

TL;DR: right now it won’t cause trouble if all your workflows are initiated from HTTP Requests.

Improvements by switching to queue

This would make your n8n deployment more future-proof, separating webhook and worker processes into separate containers and adding a message broker (Redis, in our case) to handle the traffic.

Sequential executions

No, n8n cannot guarantee sequential executions at all. We run on Node.js (javascript) and we use the advantages of Promises and the “false concurrency” that Javascript provides.

I say false concurrency because actually Javascript is single threaded, so in essence only 1 execution is happening at a time, but Javascript’s event loop jumps between different executions, therefore, they are not sequential.

Using main mode

If you’re not using queue mode, then main or own are the 2 possible execution methods. When using main, it means the executions all happen inside a single process, which is the n8n process. This comes with a major drawback: uses only 1 CPU core at a time, makes this process bigger (by concentrating resources) and makes your executions compete for CPU time in this single threaded environment. To counter this, you have blazing fast speed as everything happens in a single place, and if latency is sensitive, this could be the way to go.

On the other hand, when using own each execution happens in a totally separate process. This means all the incurred costs related to forking a new process (this is an intense OS operation) but it is more isolated therefore more secure (i.e. less prone to crashes), can use multiple cores from your machine and scales better with the hardware you have.

I hope these points made it clearer for you! If anything is not clear, please feel free to ask!

3 Likes
4

Thanks for the comprehensive response :slight_smile:
Are there any plans to support sequential execution? Some workflows need to modify tickets and other fields and a parallel execution can cause some trouble where a field failt to be updated properly or is overwritten.

Racing conditions will also be a norm which is not great.

On another note and i forgot to ask: since the database keeps track of workflows and sequence numbers for each workflow, do you see any potential issues with having multiple instances sharing the same database? Just curious

Best regards

1 Like
5

I don’t see any plans for sequential executions, but one important thing to note is that inside a single workflow executions (from start to finish), everything happens in sequence.

So what I mean is that if you have a workflow with nodes A → B → C → D, then each of these nodes will be executed in sequence. What I mean by concurrent executions is that you can have this same flow (A->B->C->D) happening twice in parallel for different input data.

So if the operations you are doing with the Jira tickets are all in a single workflow, then you don’t have to worry.

Regarding the shared database, this is not a problem. We use auto incrementing database numbers, so it simply acts as a multi-client database. You should have no issues there.

1 Like
6

Hey @bmb,

Did the above information help? Let us know if you still need help :slight_smile:

7

Hi guys, this is being a challenge here, but not related necessarily with the deployment method.

We have a workflow that receives webhooks from GitHub, process and store the information on Notion. However, GitHub sends many hooks really quick, so it’s a problem in the workflow to not being able to process them sequentially.

In general, it would be great to have an optional mutex on n8n to avoid the same workflow to have two executions in parallel.

1 Like
8

This feature request is documented already:

1 Like