Deploy a local cluster with scaling solution and secure public access

Tips & Tricks
, , ,
1

:wave: Good day, community!

Recently, I’ve been diving deep into scaling solutions for n8n, and I came up with a Docker Compose script to run in queue mode. I know many of you in the community are using self-hosted setups, so this should be super helpful!

:small_blue_diamond: Scaling Made Easy

You can add or remove worker nodes on the fly by simply adjusting the replicas count in your Docker Compose setup.

:bulb: Handling External Access

If you’re self-hosting, you might need to expose your deployment to the outside world for things like webhooks.

But here’s the catch—if you’re running this on your laptop, maintaining a public IP while traveling is a hassle.

:rocket: Enter ngrok!

I found a simple yet powerful solution by integrating ngrok, a globally distributed reverse proxy that secures, protects, and accelerates your applications.

This means seamless external connectivity, no matter where you are!

:wrench: How to Get Started

Before jumping in, here’s what you need:
:one: Sign up for ngrok and reserve a domain.
:two: Update the WEBHOOK_URL and the url command in the ngrok service within the docker-compose file.

:sparkles: Pro Tip

If you’re on ngrok’s free plan, use:

  • https://localhost:5678 for development.
  • The ngrok-provided domain only when external connectivity is required (e.g., adding credentials, authenticating Google Service Accounts).

:closed_lock_with_key: Security First!

:small_orange_diamond: Replace the N8N_ENCRYPTION_KEY in your compose file!
:small_blue_diamond: The value must remain consistent across all instances.

:e-mail: Need to Send Emails?

Just add the N8N SMTP variables in the environment section of your services.

:memo: Docker Compose File

version: '3.8'
services:
  n8n:
    image: n8nio/n8n:latest
    restart: always
    ports:
      - "5678:5678"
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=db
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=n8n_pass
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=redis
      - N8N_HOST=localhost
      - N8N_PROTOCOL=https
      - WEBHOOK_URL=https://domain.url #place your ngrok domain here, use this url for the google service account concent screen redirect urls. 
      - N8N_ENCRYPTION_KEY= encryption_key #place your encrypton key here. longer the better but not too much
      - N8N_RUNNERS_ENABLED=true
      - OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS=true
      - N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true
      - N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true
      - N8N_ONBOARDING_FLOW_DISABLED=true
    volumes:
      - n8n_data:/home/node/.n8n

  worker:
    image: n8nio/n8n:latest
    restart: always
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=db
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=n8n_pass
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=redis
      - N8N_HOST=localhost
      - N8N_PROTOCOL=http
      - N8N_ENCRYPTION_KEY= encryption_key #place your encrypton key here. longer the better but not too much
      - N8N_RUNNERS_ENABLED=true
      - OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS=true
      - N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true
      - N8N_ONBOARDING_FLOW_DISABLED=true
    command: worker
    deploy:
      replicas: 2
  webhook_processor: 
    image: docker.n8n.io/n8nio/n8n
    restart: always
    # ports:
    #   - 5679:5678
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=db
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=n8n_pass
      - EXECUTIONS_MODE=queue
      - N8N_ENCRYPTION_KEY= encryption_key #place your encrypton key here. longer the better but not too much
      - QUEUE_BULL_REDIS_HOST=redis
      - N8N_RUNNERS_ENABLED=true
      - OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS=true
      - N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true
      - N8N_ONBOARDING_FLOW_DISABLED=true
    command: webhook

  db:
    image: postgres:15-alpine
    restart: always
    environment:
      POSTGRES_USER: n8n
      POSTGRES_PASSWORD: n8n_pass
      POSTGRES_DB: n8n
    volumes:
      - db_data:/var/lib/postgresql/data

  ngrok:
    image: ngrok/ngrok:latest
    command:
      - "http"
      - "http://host.docker.internal:5678" 
      - "--url=https://domain.url"  #place your ngrok domain here
    environment:
      NGROK_AUTHTOKEN: authtoken #place your ngrok auth token here
    ports:
      - 4040:4040

  redis:
    image: redis:7-alpine
    restart: always

volumes:
  n8n_data:
  db_data:
  traefik_data:

:raised_hands: A Bit About Me

I’ve been building automation solutions with n8n for over three years, and I run DevKrypt Solutions (devkrypt.com)—a startup focused on low-code automation with n8n and internal tool solutions with Windmill software.

:bulb: Need help with your workflows, deployments, or automation strategies?
Let’s connect! I’d love to help you develop, consult and scale your solutions effortlessly. :rocket: (@kavishka)