PoC mode queue: Scaling n8n Workers and Problem whith Webhook
Hello, good morning, my name is Pablo and I require a little help to carry out this PoC = proof of concept: n8n scaling.
Configuration.
n8n version: 1.72.1
Database (default: SQLite): Postgres & Redis
n8n EXECUTIONS_PROCESS setting (default: own, main): own / queue
Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
Operating system: Ubuntu 24.04.1 LTS x86_64
I am creating 8n8 network on docker, db redis, postgres, worker and webhook without problems
The problem occurs when I apply the option to the n8n-main process: N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true
In this way, the n8n-main does not respond to a webhook request and as far as I know, I do not have to place the n8n-main in the load balancer, but rather I have to point it to the webhook stream.
When I activate N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true and activate the webhooks, it is as if they do not reach the webhhook endpoint, how do I know? Why when activating
N8N_DISABLE_PRODUCTION_MAIN_PROCESS=false I can reach the n8n-main webhook, which would be the main one, but not the other webhooks1 and webhook2, and the webhooks show records as if they had executed the workflows at the same time, but no records are seen in the workers .
With the workers, I have no problem, it scales well and responds individually when I have N8N_DISABLE_PRODUCTION_MAIN_PROCESS=false in the main n8n-main process.
But when I set N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true the workers do not respond because obviously the webhooks in the n8n-main process do not work and the scaled webhooks are not responding to the requests, it seems to me that the error is in my nginx configuration
I attach commands executed one by one and their environment variables.
# Create Docker network for containers
docker network create n8n
# Redis - Background Queuing Service
docker run -d --name n8n-redis --network=n8n -p 6379:6379 redis
# PostgreSQL - Primary Database
docker run -d --name n8n-postgres --network=n8n -e POSTGRES_PASSWORD=mypasswd -p 5432:5432 postgres
# n8n Main Container – Interface and API
docker run --name n8n-main -e N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true --env-file .n8n/main.env.prod --env-file .n8n/redis.env.prod --env-file .n8n/postgres.env.prod --network=n8n -p 5678:5678 -v ./certs:/etc/nginx/certs n8nio/n8n
#n8n-main notes: we pass the environment variable “-e
N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true", so if we share the configurations of main.env.prod, not all containers will start with this variable activated, for example Workers and Webhooks, I don’t know if this will be correct but I have it in mind and configured like this.
# Workers - Background processing
docker run --name n8n-worker1 --env-file .n8n/worker1.env.prod --env-file .n8n/postgres.env.prod --network=n8n -p 5575:5678 --entrypoint /bin/sh n8nio/n8n -c "n8n worker --concurrency=5"
docker run --name n8n-worker2 --env-file .n8n/worker2.env.prod --env-file .n8n/postgres.env.prod --network=n8n -p 5474:5678 --entrypoint /bin/sh n8nio/n8n -c "n8n worker --concurrency=5"
docker run --name n8n-worker3 --env-file .n8n/worker3.env.prod --env-file .n8n/postgres.env.prod --network=n8n -p 5373:5678 --entrypoint /bin/sh n8nio/n8n -c "n8n worker --concurrency=5"
# Webhooks - Reception of external events
docker run --name n8n-webhook1 --env-file .n8n/webhook1.env.prod --env-file .n8n/postgres.env.prod -p 5677:5678 --network=n8n -v ./certs:/etc/nginx/certs --entrypoint /bin/sh n8nio/n8n -c "n8n webhook"
docker run --name n8n-webhook2 --env-file .n8n/webhook2.env.prod --env-file .n8n/postgres.env.prod -p 5676:5678 --network=n8n -v ./certs:/etc/nginx/certs --entrypoint /bin/sh n8nio/n8n -c "n8n webhook"
NOTE webhooks:
-p 5677:5678 host:contenedor = Como ya se encuentra ocupado el puerto 5678 con el editor de n8n en la red n8n, se debe asignar otro puerto a los webhook en host para no colisionar.
# Nginx Proxy – Routing and Security
docker run -d --name n8n-nginx-proxy --network=n8n -p 80:80 -p 81:81 -p 443:443 -v ./nginx/n8n.conf:/etc/nginx/conf.d/default.conf -v ./certs:/etc/nginx/certs -v ./nginx/conf.d/include:/etc/nginx/conf.d/include -v ./nginx/logs:/data/logs --env-file .n8n/nginx-proxy.env.prod nginx
As a note, I don’t think I have the nginx configuration right, since it is the first time that I am trying to implement a webhook stream, so I would ask for your collaboration to help me solve this problem and achieve this n8n Scaling PoC.
Maybe I need to configure some parameters in the webhook1.env.prod and webhook2.env.prod in the endpoints
On the other hand, the nginx server is not allowing me to correctly enter the nginx administration port 81, that is, I can enter but it tells me Bad Gateway when entering the credentials: [email protected] /changeme
I think this obviously happens because my nginx is configured incorrectly.
Below I attach data from the .env files used.
main.env.prod
# Configuraciones compartidas
N8N_ENCRYPTION_KEY=6oW9qj3rZx/y+t8tA2sP+a9qWw7N5kXb0cJf8j1xV7o=
N8N_USER_MANAGEMENT_JWT_SECRET=aK7sD/v9h9Q3nZ6xYk2pL8s1bU0f7g4jHn9mX6a3wB/r5yT1uI2oP3l4kZ8nWmJ0iHg6fL/xZ0cR5vB+eL/h=
N8N_HOST=n8n.openboxti.cl
N8N_PROTOCOL=https
N8N_PORT=5678
N8N_SSL_KEY=/etc/nginx/certs/n8n.key
N8N_SSL_CERT=/etc/nginx/certs/n8n.crt
NODE_ENV=production
WEBHOOK_URL=https://n8n.openboxti.cl/
SUB_HOOK=/webhook
SUB_HOOK_TEST=/webhook-test
# Configuraciones específicas del contenedor principal EXECUTIONS_PROCESS deprecated ahora se usa EXECUTIONS_MODE main o own
EXECUTIONS_MODE=queue
N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true
N8N_TRUST_PROXY_ENABLED=true
# Con esto deshabilito los webhooks N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true
nginx-proxy.env.prod
# Configuraciones de Nginx
DISABLE_IPV6=true
N8N_SSL_KEY=/etc/nginx/certs/n8n.key
N8N_SSL_CERT=/etc/nginx/certs/n8n.crt
# Configuraciones de seguridad
CLIENT_MAX_BODY_SIZE=10M
PROXY_READ_TIMEOUT=600s
PROXY_SEND_TIMEOUT=600s
# Configuraciones de rendimiento
WORKER_PROCESSES=auto
WORKER_CONNECTIONS=1024
KEEPALIVE_TIMEOUT=65
# Configuraciones de logging
ACCESS_LOG=/data/logs/access.log
ERROR_LOG=/data/logs/error.log
postgres.env.prod
# Configuracion postgres
DB_TYPE=postgresdb
DB_POSTGRESDB_DATABASE=postgres
DB_POSTGRESDB_HOST=n8n-postgres
DB_POSTGRESDB_PASSWORD=mypasswd
DB_POSTGRESDB_USER=postgres
redis.env.prod
# Configuracion para redis
QUEUE_TYPE=redis
QUEUE_BULL_REDIS_HOST=n8n-redis
QUEUE_BULL_REDIS_PORT=6379
QUEUE_BULL_REDIS_DB=0
webhook1.env.prod
# Configuraciones esenciales para webhooks
N8N_HOST=n8n.openboxti.cl
N8N_PROTOCOL=https
N8N_PORT=5678
WEBHOOK_URL=https://n8n.openboxti.cl/
N8N_SSL_KEY=/etc/nginx/certs/n8n.key
N8N_SSL_CERT=/etc/nginx/certs/n8n.crt
# Configuraciones de seguridad
N8N_ENCRYPTION_KEY=6oW9qj3rZx/y+t8tA2sP+a9qWw7N5kXb0cJf8j1xV7o=
N8N_USER_MANAGEMENT_JWT_SECRET=aK7sD/v9h9Q3nZ6xYk2pL8s1bU0f7g4jHn9mX6a3wB/r5yT1uI2oP3l4kZ8nWmJ0iHg6fL/xZ0cR5vB+eL/h=
# Configuraciones de colas
EXECUTIONS_MODE=queue
QUEUE_TYPE=redis
QUEUE_BULL_REDIS_HOST=n8n-redis
QUEUE_BULL_REDIS_PORT=6379
QUEUE_BULL_REDIS_DB=0
# Configuraciones adicionales
NODE_ENV=production
SUB_HOOK=/webhook
SUB_HOOK_TEST=/webhook-test
N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true
webhook2.env.prod
...
N8N_PORT=5678
...
worker1.env.prod
# Configuraciones compartidas
N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true
N8N_ENCRYPTION_KEY=6oW9qj3rZx/y+t8tA2sP+a9qWw7N5kXb0cJf8j1xV7o=
N8N_USER_MANAGEMENT_JWT_SECRET=aK7sD/v9h9Q3nZ6xYk2pL8s1bU0f7g4jHn9mX6a3wB/r5yT1uI2oP3l4kZ8nWmJ0iHg6fL/xZ0cR5vB+eL/h=
N8N_HOST=n8n.openboxti.cl
N8N_PROTOCOL=https
N8N_PORT=5678
N8N_SSL_KEY=/etc/nginx/certs/n8n.key
N8N_SSL_CERT=/etc/nginx/certs/n8n.crt
NODE_ENV=production
WEBHOOK_URL=https://n8n.openboxti.cl/
SUB_HOOK=/webhook
SUB_HOOK_TEST=/webhook-test
# Configuraciones específicas del worker
N8N_WORKER=true
EXECUTIONS_MODE=queue
QUEUE_TYPE=redis
QUEUE_BULL_REDIS_HOST=n8n-redis
QUEUE_BULL_REDIS_PORT=6379
QUEUE_BULL_REDIS_DB=0
worker2.env.prod
...
N8N_PORT=5678
...
worker3.env.prod
...
N8N_PORT=5678
...
n8n.conf (for proxy mount)
# ------------------------------------------------------------
# n8n.openboxti.cl
# ------------------------------------------------------------
upstream n8n_webhooks {
server n8n-webhook1:5678;
server n8n-webhook2:5678;
}
map $scheme $hsts_header {
https "max-age=63072000; preload";
}
log_format proxy '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
server {
set $forward_scheme https;
set $server "n8n_webhooks";
set $port 80;
listen 80;
#listen [::]:80;
listen 443 ssl http2;
#listen [::]:443;
server_name n8n.openboxti.cl;
# Let's Encrypt SSL
resolver 1.1.1.1 8.8.8.8 valid=300s;
resolver_timeout 5s;
include conf.d/include/letsencrypt-acme-challenge.conf;
include conf.d/include/ssl-ciphers.conf;
ssl_certificate /etc/nginx/certs/fullchain2.pem;
ssl_certificate_key /etc/nginx/certs/n8n.key;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_http_version 1.1;
access_log /data/logs/proxy-host-1_access.log proxy;
error_log /data/logs/proxy-host-1_error.log warn;
# Usamos una location para el caso general, proxy al contenedor de n8n
location / {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_http_version 1.1;
# Proxy!
include conf.d/include/proxy.conf;
proxy_pass http://$server:$port; # <---- Proxy al contenedor de n8n por HTTP
}
# Custom
include /data/nginx/custom/server_proxy[.]conf;
}
# Admin Interface
server {
listen 81 default;
#listen [::]:81 default;
server_name nginxproxymanager;
root /app/frontend;
access_log /dev/null;
location /api {
return 302 /api/;
}
location /api/ {
proxy_pass http://n8n-main:5678/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
}
location / {
proxy_pass http://n8n-main:5678/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
}
}