Problems with the N8N queue configuration

Questions
#1

I’m using a lot of webhooks and sometimes the N8N crashes, because of that I would like to test the queue mode to see if it will solve the problem.

already read the documentation Configuring queue mode - n8n Documentation

and I tried using a github config example n8n/docker/compose/withPostgresAndRedis at master · st-ax/n8n · GitHub

But unfortunately it didn’t work, the N8N even opens, but it’s unstable, when trying to save a workflow or something like that, an error 502 appears

I’m going to share all the configuration I’m doing from scratch and I ask for your help in case I’m missing something or if I did something wrong.

sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

.docker-compose.yml

version: '3'

services:
  traefik:
    image: "traefik"
    restart: always
    command:
      - "--api=true"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.mytlschallenge.acme.tlschallenge=true"
      - "--certificatesresolvers.mytlschallenge.acme.email=${SSL_EMAIL}"
      - "--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ${DATA_FOLDER}/letsencrypt:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock:ro

  postgres:
    image: postgres
    restart: always
    environment:
      - POSTGRES_USER
      - POSTGRES_PASSWORD
      - POSTGRES_DB
      - POSTGRES_NON_ROOT_USER
      - POSTGRES_NON_ROOT_PASSWORD
    volumes:
      - ./init-data.sh:/docker-entrypoint-initdb.d/init-data.sh

  redis:
    image: redis
    restart: always
    command: redis-server --requirepass oqSxRwNyOJXZh94nB92OVvo7Z0D
    ports:
      - 6379:6379
    volumes:
      - ./redis-data:/var/lib/redis
      - ./redis.conf:/usr/local/etc/redis/redis.conf
    environment:
     - REDIS_REPLICATION_MODE=master

  n8n:
    image: n8nio/n8n
    restart: always
    ports:
      - "127.0.0.1:5678:5678"
    labels:
      - traefik.enable=true
      - traefik.http.routers.n8n.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`)
      - traefik.http.routers.n8n.tls=true
      - traefik.http.routers.n8n.entrypoints=web,websecure
      - traefik.http.routers.n8n.tls.certresolver=mytlschallenge
      - traefik.http.middlewares.n8n.headers.SSLRedirect=true
      - traefik.http.middlewares.n8n.headers.STSSeconds=315360000
      - traefik.http.middlewares.n8n.headers.browserXSSFilter=true
      - traefik.http.middlewares.n8n.headers.contentTypeNosniff=true
      - traefik.http.middlewares.n8n.headers.forceSTSHeader=true
      - traefik.http.middlewares.n8n.headers.SSLHost=${DOMAIN_NAME}
      - traefik.http.middlewares.n8n.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.n8n.headers.STSPreload=true
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=${POSTGRES_DB}
      - DB_POSTGRESDB_USER=${POSTGRES_NON_ROOT_USER}
      - DB_POSTGRESDB_PASSWORD=${POSTGRES_NON_ROOT_PASSWORD}
      - N8N_BASIC_AUTH_ACTIVE=true
      - N8N_BASIC_AUTH_USER
      - N8N_BASIC_AUTH_PASSWORD
      - N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - N8N_ENCRYPTION_KEY=uKCbpQ3cFxltXyhidigoyHlNhZJ0b56M5YLYl1uxuKJT
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=redis
      # - QUEUE_BULL_REDIS_PORT=6379
      - WEBHOOK_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
      - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
    depends_on:
      - postgres
      - redis
    volumes:
      - ${DATA_FOLDER}/.n8n:/home/node/.n8n
    # Wait 5 seconds to start n8n to make sure that PostgreSQL is ready
    # when n8n tries to connect to it
    command: /bin/sh -c "sleep 5; n8n start"

  n8n-queue:
      image: n8nio/n8n
      restart: always
      environment:
        - DB_TYPE=postgresdb
        - DB_POSTGRESDB_HOST=postgres
        - DB_POSTGRESDB_PORT=5432
        - DB_POSTGRESDB_DATABASE=${POSTGRES_DB}
        - DB_POSTGRESDB_USER=${POSTGRES_NON_ROOT_USER}
        - DB_POSTGRESDB_PASSWORD=${POSTGRES_NON_ROOT_PASSWORD}
        - N8N_BASIC_AUTH_ACTIVE=true
        - N8N_BASIC_AUTH_USER
        - N8N_BASIC_AUTH_PASSWORD
        - N8N_ENCRYPTION_KEY=uKCbpQ3cFxltXyhidigoyHlNhZJ0b56M5YLYl1uxuKJT
        - EXECUTIONS_MODE=queue
        - QUEUE_BULL_REDIS_HOST=redis
        # - QUEUE_BULL_REDIS_PORT=6379
        - N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
        - WEBHOOK_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
        - NODE_FUNCTION_ALLOW_BUILTIN=*
        - NODE_FUNCTION_ALLOW_EXTERNAL=*
        - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
        - N8N_LOG_LEVEL=debug  # error, warning, info, verbose, debug
      cpus: 3
      ports:
        - 5679:5678
      depends_on:
        - postgres
        - redis
        - n8n
      volumes:
        - ~/.n8n:/home/node/.n8n
      command: /bin/sh -c "n8n worker"

  n8n-wh:
        image: n8nio/n8n
        restart: always
        environment:
          - DB_TYPE=postgresdb
          - DB_POSTGRESDB_HOST=postgres
          - DB_POSTGRESDB_PORT=5432
          - DB_POSTGRESDB_DATABASE=${POSTGRES_DB}
          - DB_POSTGRESDB_USER=${POSTGRES_NON_ROOT_USER}
          - DB_POSTGRESDB_PASSWORD=${POSTGRES_NON_ROOT_PASSWORD}
          - N8N_BASIC_AUTH_ACTIVE=true
          - N8N_BASIC_AUTH_USER
          - N8N_BASIC_AUTH_PASSWORD
          - N8N_ENCRYPTION_KEY=uKCbpQ3cFxltXyhidigoyHlNhZJ0b56M5YLYl1uxuKJT
          - EXECUTIONS_MODE=queue
          - QUEUE_BULL_REDIS_HOST=redis
          # - QUEUE_BULL_REDIS_PORT=6379
          - N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
          - WEBHOOK_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
          - NODE_FUNCTION_ALLOW_BUILTIN=*
          - NODE_FUNCTION_ALLOW_EXTERNAL=*
          - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
          - N8N_LOG_LEVEL=debug  # error, warning, info, verbose, debug
        cpus: 3
        ports:
          - 5680:5678
        networks:
          - default
        depends_on:
          - postgres
          - redis
          - n8n
          - n8n-queue
        volumes:
          - ~/.n8n:/home/node/.n8n 
        command: /bin/sh -c "n8n webhook"

.ENV

POSTGRES_USER=changeUser
POSTGRES_PASSWORD=changePassword
POSTGRES_DB=n8n

POSTGRES_NON_ROOT_USER=changeUser
POSTGRES_NON_ROOT_PASSWORD=changePassword

N8N_BASIC_AUTH_USER=changeUser
N8N_BASIC_AUTH_PASSWORD=changePassword

DATA_FOLDER=/root/n8n/
DOMAIN_NAME=example.com
SUBDOMAIN=n8n
N8N_BASIC_AUTH_USER=user
N8N_BASIC_AUTH_PASSWORD=password
GENERIC_TIMEZONE=Europe/Berlin
[email protected]

init-data.sh

#!/bin/bash
set -e;


if [ -n "${POSTGRES_NON_ROOT_USER:-}" ] && [ -n "${POSTGRES_NON_ROOT_PASSWORD:-}" ]; then
	psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
		CREATE USER ${POSTGRES_NON_ROOT_USER} WITH PASSWORD '${POSTGRES_NON_ROOT_PASSWORD}';
		GRANT ALL PRIVILEGES ON DATABASE ${POSTGRES_DB} TO ${POSTGRES_NON_ROOT_USER};
	EOSQL
else
	echo "SETUP INFO: No Environment variables given!"
fi

mkdir /root/n8n/
sudo docker-compose up -d
sudo docker-compose stop

if there is any other configuration that works and can help me I would be very grateful

#2

Oh, that’s a tough one for me :frowning:

On first sight I couldn’t spot what might be wrong here, maybe @krynble can take a look when he has a bit of time?

1 Like
#3

thanks a lot, if @krynble an take a look when possible in his spare time I would be very grateful, or if anyone has a setup that works in docker or docker-compose with webhook that works could help me a lot

#4

Could someone share some valid configuration that uses?

#5

Hey @Rodrigo_Barbosa,

The first thing I would try as you getting a bad gateway error would be to check the docker logs to see which instance is possibly causing it. I would also maybe start by removing the n8n queue and n8n webhook instances for now and see if it works with just the main instance for now.

From there it would be a case of adding each part to ensure it is working, I am not entirely sure the webhook instance would actually work in that configuration. I am no expert when it comes to Traefik but I can’t see where the routing is to send the webhook traffic to that node.

1 Like
#6

I believe you have have a misconfiguration with Traefik on your setup.

You’re running n8n with the flag N8N_PROTOCOL=HTTPS

This means that n8n will be encrypting the data between itself (n8n) and Traefik, who has to decrypt the data and re-encrypt it, which seems a possible cause for the issues you are having.

From the configuration alone I cannot tell the reason why it fails, but I would suggest you run traefik with n8n alone (no webhook or worker processes) just to make sure that the configuration between traefik and n8n is fine.

N8N Scaling problems - Cannot Get