N8N Queue mode load testing

I’m working on running N8N in Kubernetes, with 1 main container and 3 queue worker containers.

While I struggle with getting everything in K8s configured (I’m still pretty new to it), I was wondering if anyone had a good workflow (or idea for workflow) to test performance across the distributed workers. That is, I’d like to fire off “Workflow 1” with 3 queue workers, get a processing time, and then scale it up to 4 or 5 workers and see a shorter time.

Right now, I’m using a function node to create 1000 simple items, break those into batches, and send them off to a sub-Workflow which just adds a new field to each item. I use the Respond to Webhook node after the batching is done to get an overall time benchmark.

Does that seem reasonable? Are there better ways to construct workflows to take advantage of parallel processing across the workers? Any advice for constructing a solid test workflow would be gratefully received.

1 Like

Hi @Ian_Walls, I hope you’re well?

Just wanted to let you know this has already been flagged to @krynble internally who’d be best placed to provide more in-depth insights into the queue mode :slight_smile:

1 Like

Hey @Ian_Walls

This question you raised is something we are also working on internally to crunch some numbers.

There is an internal project for this and we intend to publish the results as soon as it is finished.

On a quick glimpse, our goal is to better understand how n8n will behave according to different hardware specs and types of workloads - IO intensive, CPU intensive, data intensive, etc.

I don’t have any details to share right now as we’re working on other projects but stay connected to our blog and social channels for updates.

1 Like

So, after some testing, I modified my workflow to fire off HTTP Requests, rather than included Workflows. The ‘subtask’ workflow is webhook triggered, and I have some initial numbers that make sense to me:

With 5000 records, in batches of 100, and 4 workers with concurrency 10 each:

  • ~40s to process with ‘respond on completion’ option for the subtask Webhook node
  • ~8s to process with ‘respond immediately’, and another 3-5 seconds for all the subtasks to wrap up

The next trick I want to figure out is how to get the main process to notify me of the results of the complete run while still having the subtask’s Webhook node respond immediately. I’m guessing I should have the subtask fire off an HTTP request somewhere on completion, but that listener would need to wait for all the worker processes to return either success or failure, and then act. Does anyone have any ideas how to set something like this up?

I don’t know if it will finally lead to a proper solution for you, but the wait node with “on webhook call” combined with some kind of callback structure could be interesting.

@Ian_Walls If possible can you share your n8n queue mode K8 YAML files? I am putting the same together and having confusions regarding deployment of worker pods.

We (to be more exact @kik00) are actually doing some load testing ourselves right now. Is almost done but currently still WIP. You can find the repository here:

1 Like

The Deployment (adjust resourcing and replicas as appropriate):

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: n8n-main-deployment
  labels: &labels
    app: n8n
    component: main-deployment
spec:
  replicas: 1
  selector:
    matchLabels: *labels
  template:
    metadata:
      labels: *labels
    spec:
      containers:
      - name: n8n-main
        image: n8nio/n8n
        imagePullPolicy: Always
        ports:
        - name: http-metrics
          containerPort: 5678
        envFrom:
        - secretRef:
            name: n8n-secrets
        - secretRef:
            name: n8n-main-config
        livenessProbe:
          httpGet:
            path: /healthz
            port: 5678
        readinessProbe:
          httpGet:
            path: /healthz
            port: 5678
        resources:
          limits:
            cpu: "1.0"
            memory: "1024Mi"
          requests:
            cpu: "0.5"
            memory: 512Mi


---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: n8n-worker-deployment
  labels: &labels
    app: n8n
    component: worker-deployment
spec:
  replicas: 3
  selector:
    matchLabels: *labels
  template:
    metadata:
      labels: *labels
    spec:
      containers:
      - name: n8n-worker
        image: n8nio/n8n
        imagePullPolicy: Always
        args: ["n8n", "worker"]
        ports:
        - name: http-metrics
          containerPort: 5678
        envFrom:
        - secretRef:
            name: n8n-secrets
        - secretRef:
            name: n8n-worker-config
        resources:
          limits:
            cpu: "1.0"
            memory: "1024Mi"
          requests:
            cpu: "0.5"
            memory: 512Mi

the “Main” node configurations:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: n8n-main-config
  labels:
    app: n8n
    component: secrets
stringData:
  N8N_DISABLE_PRODUCTION_MAIN_PROCESS: "false"
  N8N_SKIP_WEBHOOK_DEREGISTRATION_SHUTDOWN: "false"
  EXECUTIONS_MODE: "queue"

The Worker node configurations:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: n8n-worker-config
  labels:
    app: n8n
    component: secrets
stringData:
  N8N_DISABLE_PRODUCTION_MAIN_PROCESS: 'false'
  N8N_SKIP_WEBHOOK_DEREGISTRATION_SHUTDOWN: 'true'
  EXECUTIONS_MODE: "regular"

I’m skipping sharing the entirety of the shared configurations, as that will vary based on your circumstances, but you’ll want to define

QUEUE_BULL_REDIS_HOST
QUEUE_BULL_REDIS_PORT
QUEUE_BULL_REDIS_DB
QUEUE_BULL_REDIS_TIMEOUT_THRESHOLD
QUEUE_RECOVERY_INTERVAL

as appropriate for your Redis deployment.

3 Likes

@Ian_Walls Thank you.

Is there a way to run n8n cli in the K8 cluster? I am trying to execute a workflow using n8n execute --file but in queue mode on K8 cluster. Is this possible?

hmmm, that I don’t know… probably with kubectl exec on the main node, but I haven’t tried. I’m only using N8N as a web service at this point.

I was just thinking, Wouldn’t it be best to horizontally auto scale worker pods based on concurrency. What do you guys think? Can we achieve this? Like example set --concurrency=2 and an auto scale policy to pre-emptively add a pod when one process already in progress?

There is no difference between the worker, main and webhook docker images of n8n; the only change is related to how you start the container.

The default is the all-in-one main. This main container should not run more than once, since it’s responsible for triggering executions (i.e. cron-based tasks) and having multiple instances running would cause duplication of work.

You can certainly scale workers and webhooks to accomodate larger workloads.

More details can be found here: Configuring queue mode - n8n Documentation

Hey everyone, I have some updates about the benchmark tests that we were conducting.

@Ian_Walls in regards to your question about checking when the execution actually ends, I don’t think there is a way to easily get this using n8n only. If you are using webhook processes, n8n’s main process is actually completely unaware of the executions.

When running in scaled mode, to reduce the load on the main process, when set up with Webhooks + worker they talk to each other directly, not notifying the main process at all.

Sync is done via Reds + database.

@Gowthaman_Prabhu the correct way to scale this would be to actually scale based on the number of waiting jobs, which is something you can obtain as shown in the docs here: bull/REFERENCE.md at develop · OptimalBits/bull · GitHub

You can then scale your worker count based on this, depending on your requirements.

1 Like