Help setting up queue mode

I’m having difficulties setting up N8N to run in queue mode.

Following the docs, first it says to “Set encryption key” like:
export N8N_ENCRYPTION_KEY=<main_instance_encryption_key>

So in docker-compose.yml, I did:

  • N8N_ENCRYPTION_KEY=<main_instance_encryption_key>

Then I tried entering in my own encryption key, like:

  • N8N_ENCRYPTION_KEY=fioengo8585h
    and it didn’t work, I couldn’t access N8N from the web, it said bad gateway.

That didn’t work, so I did this, which I found here in the forum:

  • N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}

When I start it with docker compose up -d, it said "
WARN[0000] The “N8N_ENCRYPTION_KEY” variable is not set. Defaulting to a blank string."

I could access N8N though, so I moved forward.

Then I added EXECUTIONS_MODE=queue

Then I started Redis per the docs with:
docker run --name some-redis -p 6379:6379 -d redis

Then I went back in to docker-compose.yml and added:
QUEUE_BULL_REDIS_HOST=localhost
QUEUE_BULL_REDIS_PORT=6379

localhost didn’t work, so I tried:
QUEUE_BULL_REDIS_HOST=some-redis and
QUEUE_BULL_REDIS_HOST=redis

Then I start the worker with:
docker run --name n8n-queue -p 5679:5678 docker.n8n.io/n8nio/n8n worker

And it said it’s unable to connect to Redis.

I’m not sure what to do now? I followed the directions to the best of my ability.

Here is my docker-compose.yml:

version: "3"

services:
  traefik:
    image: "traefik"
    restart: always
    command:
      - "--api=true"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.mytlschallenge.acme.tlschallenge=true"
      - "--certificatesresolvers.mytlschallenge.acme.email=${SSL_EMAIL}"
      - "--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ${DATA_FOLDER}/letsencrypt:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /local-files:/files
      - /root:/files

  n8n:
    image: n8nio/n8n:1.21.1
    restart: always
    ports:
      - "127.0.0.1:5678:5678"
    labels:
      - traefik.enable=true
      - traefik.http.routers.n8n.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`)
      - traefik.http.routers.n8n.tls=true
      - traefik.http.routers.n8n.entrypoints=web,websecure
      - traefik.http.routers.n8n.tls.certresolver=mytlschallenge
      - traefik.http.middlewares.n8n.headers.SSLRedirect=true
      - traefik.http.middlewares.n8n.headers.STSSeconds=315360000
      - traefik.http.middlewares.n8n.headers.browserXSSFilter=true
      - traefik.http.middlewares.n8n.headers.contentTypeNosniff=true
      - traefik.http.middlewares.n8n.headers.forceSTSHeader=true
      - traefik.http.middlewares.n8n.headers.SSLHost=${DOMAIN_NAME}
      - traefik.http.middlewares.n8n.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.n8n.headers.STSPreload=true
      - traefik.http.routers.n8n.middlewares=n8n@docker
    environment:
      - N8N_BASIC_AUTH_ACTIVE=true
      - N8N_BASIC_AUTH_USER
      - N8N_BASIC_AUTH_PASSWORD
      - N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - NODE_ENV=production
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_DATABASE=postgres  
      - DB_POSTGRESDB_HOST=postgres_n8n
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_USER=postgres
      - DB_POSTGRESDB_PASSWORD=XXXXXX
      - WEBHOOK_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
      - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
      - EXECUTIONS_DATA_PRUNE=true
      - EXECUTIONS_DATA_MAX_AGE=72
      - EXECUTIONS_DATA_SAVE_ON_ERROR=all
      - EXECUTIONS_DATA_SAVE_ON_SUCCESS=none
      - EXECUTIONS_DATA_SAVE_ON_PROGRESS=false
      - EXECUTIONS_DATA_SAVE_MANUAL_EXECUTIONS=true
      - N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=some-redis
      - QUEUE_BULL_REDIS_PORT=6379
    volumes:
      - ${DATA_FOLDER}/.n8n:/home/node/.n8n
      - /local-files:/files
      - /home:/files
      - /root/n8n:/files
      - /root:/files
      - /root:/Dropbox
      - ./n8n_exports:/n8n_exports
  
  postgres_n8n:
    image: postgres
    restart: unless-stopped
    environment:
      - POSTGRES_PASSWORD=XXXXX
    volumes:
      - postgresN8N-volume:/var/lib/postgresql/data
      
volumes:
  n8n_data:  
  postgresN8N-volume:
    external: true


It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:

Hi @mmac, I am sorry you are having trouble.

N8N_ENCRYPTION_KEY=fioengo8585h
and it didn’t work, I couldn’t access N8N from the web, it said bad gateway.

First things first, I don’t think the N8N_ENCRYPTION_KEY is related to a possible bad gateway error. If your encryption key is wrong, n8n would instead be unable to read existing credentials (or identify itself against the enterprise license server).

That didn’t work, so I did this, which I found here in the forum:
N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}
When I start it with docker compose up -d, it said “WARN[0000] The “N8N_ENCRYPTION_KEY” variable is not set. Defaulting to a blank string.”

This tells docker to use interpolation rather than a hard coded environment variable. I would suggest avoiding this, at least when getting started and instead provide the actual values directly in your docker compose file.

Now, with that out of the way, the bad gateway error would instead typically be thrown by a load balancer or reverse proxy. Is this docker compose setup the only application running on your server? Or is there a chance you already have an application like caddy or nginx running on your server? In this case you won’t need traefik from your docker compose file.

Then I started Redis per the docs with:
docker run --name some-redis -p 6379:6379 -d redis

Then I went back in to docker-compose.yml and added:
QUEUE_BULL_REDIS_HOST=localhost
QUEUE_BULL_REDIS_PORT=6379

localhost didn’t work, so I tried:
QUEUE_BULL_REDIS_HOST=some-redis and
QUEUE_BULL_REDIS_HOST=redis

This will fail because your docker compose setup lives on a separate docker network than your redis instance started outside of your docker compose file. You probably want to add redis to your docker compose file instead so all containers share the same network.

An example docker-compose.yml file starting n8n in queue mode with all required services (n8n main, instance, n8n worker instance, Redis, and a Postgres database) could look like so:

services:
  postgres:
    image: postgres:15
    restart: unless-stopped
    environment:
      - POSTGRES_USER=n8n
      - POSTGRES_PASSWORD=n8n
      - POSTGRES_DB=n8n
    volumes:
      - ./db_data:/var/lib/postgresql/data
    ports:
      - 5432:5432
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -h localhost -U n8n -d n8n']
      interval: 5s
      timeout: 5s
      retries: 10
  redis:
    image: redis:6-alpine
    restart: unless-stopped
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
      interval: 5s
      timeout: 5s
      retries: 10
  n8n_main:
    image: n8nio/n8n:1.21.1
    restart: unless-stopped
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=n8n
      - QUEUE_BULL_REDIS_HOST=redis
      - N8N_ENCRYPTION_KEY=foobarbaz
      - EXECUTIONS_MODE=queue
    ports:
      - 5678:5678
    volumes:
      - ./n8n_data:/home/node/.n8n
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
  n8n_worker:
    image: n8nio/n8n:1.21.1
    restart: unless-stopped
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=n8n
      - QUEUE_BULL_REDIS_HOST=redis
      - N8N_ENCRYPTION_KEY=foobarbaz
      - EXECUTIONS_MODE=queue
    command: worker
    volumes:
      - ./n8n_data:/home/node/.n8n
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
      n8n_main:
        condition: service_started

No additional containers and no .env file are needed in this case, just two empty folder n8n_data and db_data in the same directory.

You can then start n8n using docker compose up, after a short wait you should see the “Editor is now accessible via:” message and can open n8n on http://localhost:5678:

1 Like

Ok, thank you for the example file. I edited my docker-compose.yml as follows:

version: "3"

services:
  traefik:
    image: "traefik"
    restart: always
    command:
      - "--api=true"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.mytlschallenge.acme.tlschallenge=true"
      - "--certificatesresolvers.mytlschallenge.acme.email=${SSL_EMAIL}"
      - "--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ${DATA_FOLDER}/letsencrypt:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /local-files:/files
      - /root:/files

  n8n:
    image: n8nio/n8n:1.23.0
    restart: always
    ports:
      - "127.0.0.1:5678:5678"
    labels:
      - traefik.enable=true
      - traefik.http.routers.n8n.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`)
      - traefik.http.routers.n8n.tls=true
      - traefik.http.routers.n8n.entrypoints=web,websecure
      - traefik.http.routers.n8n.tls.certresolver=mytlschallenge
      - traefik.http.middlewares.n8n.headers.SSLRedirect=true
      - traefik.http.middlewares.n8n.headers.STSSeconds=315360000
      - traefik.http.middlewares.n8n.headers.browserXSSFilter=true
      - traefik.http.middlewares.n8n.headers.contentTypeNosniff=true
      - traefik.http.middlewares.n8n.headers.forceSTSHeader=true
      - traefik.http.middlewares.n8n.headers.SSLHost=${DOMAIN_NAME}
      - traefik.http.middlewares.n8n.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.n8n.headers.STSPreload=true
      - traefik.http.routers.n8n.middlewares=n8n@docker
    environment:
      - N8N_BASIC_AUTH_ACTIVE=true
      - N8N_BASIC_AUTH_USER
      - N8N_BASIC_AUTH_PASSWORD
      - N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - NODE_ENV=production
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_DATABASE=postgres  
      - DB_POSTGRESDB_HOST=postgres_n8n
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_USER=postgres
      - DB_POSTGRESDB_PASSWORD=XXXXXX
      - WEBHOOK_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
      - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
      - EXECUTIONS_DATA_PRUNE=true
      - EXECUTIONS_DATA_MAX_AGE=72
      - EXECUTIONS_DATA_SAVE_ON_ERROR=all
      - EXECUTIONS_DATA_SAVE_ON_SUCCESS=none
      - EXECUTIONS_DATA_SAVE_ON_PROGRESS=false
      - EXECUTIONS_DATA_SAVE_MANUAL_EXECUTIONS=true
      - N8N_ENCRYPTION_KEY=424242
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=redis
    volumes:
      - ${DATA_FOLDER}/.n8n:/home/node/.n8n
      - /local-files:/files
      - /home:/files
      - /root/n8n:/files
      - /root:/files
      - /root:/Dropbox
      - ./n8n_exports:/n8n_exports
      - ./n8n_data:/home/node/.n8n
  
  postgres_n8n:
    image: postgres
    restart: always
    environment:
      - POSTGRES_PASSWORD=XXXXXX
    volumes:
      - postgresN8N-volume:/var/lib/postgresql/data
  
  redis:
    image: redis
    restart: unless-stopped
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
      interval: 5s
      timeout: 5s
      retries: 10

  n8n_worker_001:
    image: n8nio/n8n:1.23.0
    restart: unless-stopped
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=postgres
      - DB_POSTGRESDB_USER=postgres
      - DB_POSTGRESDB_PASSWORD=XXXXXX
      - QUEUE_BULL_REDIS_HOST=redis
      - N8N_ENCRYPTION_KEY=424242
      - EXECUTIONS_MODE=queue
    command: worker
    volumes:
      - ./n8n_data:/home/node/.n8n

  n8n_worker_002:
    image: n8nio/n8n:1.23.0
    restart: unless-stopped
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=postgres
      - DB_POSTGRESDB_USER=postgres
      - DB_POSTGRESDB_PASSWORD=XXXXXX
      - QUEUE_BULL_REDIS_HOST=redis
      - N8N_ENCRYPTION_KEY=424242
      - EXECUTIONS_MODE=queue
    command: worker
    volumes:
      - ./n8n_data:/home/node/.n8n
      
volumes:
  n8n_data:  
  postgresN8N-volume:
    external: true

The containers all started running and I could access the UI. But, when processes started running, they never finished, and never actually did anything. By that I mean, if the first step in a workflow was to query a MySQL database, that query was never sent. They all showed they were running in the “Executions” page, but nothing was actually happening. I waited about 30 minutes and they were all still “running” but nothing actually being done - these are workflows that typically take about 10 seconds to run.

I also noticed that the button to “Stop” a workflow on the Executions page only had “View”, there was no “Stop” anymore.

So it seems like the main process is recognizing that a workflow should run/is running, but is failing to actually send it to a worker?

Hm, this sounds odd indeed. Can you enable debug logging for both your main and worker instances and share the server log output when starting such an execution?

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.