Queue Mode: Main Process & Worker Processes on Same Machine

[jkf]

Hey, I’m curious if the following is possible for running n8n in queue mode:

• A single machine, configured via Docker
• The main n8n process running
• Several worker processes running as well.

Redis and Postgres would be on separate machines.

In most of the guides for scaling n8n, the setup involves utilizing separate machines for workers and main.

But instead of that, is it possible to run multiple instances of n8n on same machine with one as main and several others as workers?

My thinking is that if I have a 4 vCPU vm, I could have one main process and 3 worker processes. Is this possible?

Thanks.

[Wouter_Nigrini]

Technically yes, all you need to make sure is that each worker has a unique/different port. Lemme see if I can put together a docker-compose

If you’re using docker swarm or kubernetes, it’ll prob be easier to spin up more workers.

[Wouter_Nigrini]

Example docker-compose.yml:

services:
  n8n-queue-db:
    image: postgres:16.1
    container_name: n8n-queue-db
    restart: always
    environment:
      - POSTGRES_DB=n8n
      - POSTGRES_PASSWORD=n8n
      - POSTGRES_USER=n8n
      # - PGDATA=/var/lib/postgresql/data/pgdata
    volumes:
      - /var/containers/volumes/n8n-queue/postgres-data:/var/lib/postgresql/data

  n8n-redis:
    image: redis:7-alpine
    container_name: n8n-redis
    restart: always
    volumes:
      - /var/containers/volumes/n8n-queue/redis-data:/data

  n8n-main:
    image: docker.n8n.io/n8nio/n8n:latest
    container_name: n8n-main
    restart: always
    depends_on:
      - n8n-queue-db
      - n8n-redis
    volumes:
      - /var/containers/volumes/n8n-queue/app-data:/home/node/.n8n
    ports:
      - 5678:5678
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_HOST=n8n-queue-db
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_SCHEMA=n8n
      - DB_POSTGRESDB_PASSWORD=n8n
      - N8N_RUNNERS_ENABLED=true
      - N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true
      # Database connection settings
      - DB_POSTGRESDB_POOL_SIZE=5
      - DB_POSTGRESDB_CONNECTION_TIMEOUT=30000
      # Queue mode configuration
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=n8n-redis
      - QUEUE_BULL_REDIS_PORT=6379
      - QUEUE_BULL_REDIS_DB=0
      - OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS=true

  n8n-worker-1:
    image: docker.n8n.io/n8nio/n8n:latest
    container_name: n8n-worker-1
    restart: always
    command: worker
    depends_on:
      - n8n-queue-db
      - n8n-redis
    volumes:
      - /var/containers/volumes/n8n-queue/app-data:/home/node/.n8n
    ports:
      - 5679:5678
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_HOST=n8n-queue-db
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_SCHEMA=n8n
      - DB_POSTGRESDB_PASSWORD=n8n
      - N8N_RUNNERS_ENABLED=true
      - N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true
      # Database connection settings
      - DB_POSTGRESDB_POOL_SIZE=3
      - DB_POSTGRESDB_CONNECTION_TIMEOUT=30000
      # Queue mode configuration
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=n8n-redis
      - QUEUE_BULL_REDIS_PORT=6379
      - QUEUE_BULL_REDIS_DB=0
      - OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS=true

  n8n-worker-2:
    image: docker.n8n.io/n8nio/n8n:latest
    container_name: n8n-worker-2
    restart: always
    command: worker
    depends_on:
      - n8n-queue-db
      - n8n-redis
    volumes:
      - /var/containers/volumes/n8n-queue/app-data:/home/node/.n8n
    ports:
      - 5680:5678
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_HOST=n8n-queue-db
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_SCHEMA=n8n
      - DB_POSTGRESDB_PASSWORD=n8n
      - N8N_RUNNERS_ENABLED=true
      - N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true
      # Database connection settings
      - DB_POSTGRESDB_POOL_SIZE=3
      - DB_POSTGRESDB_CONNECTION_TIMEOUT=30000
      # Queue mode configuration
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=n8n-redis
      - QUEUE_BULL_REDIS_PORT=6379
      - QUEUE_BULL_REDIS_DB=0
      - OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS=true

networks:
  default:
    name: base_nginx
    external: true

Load being shared between workers

[jkf]

Perfect, that’s just what I needed to know, thanks @Wouter_Nigrini

[Wouter_Nigrini]

Ok great. Please mark my answer as the solution if this helped