Queue Mode guide [How to scale up n8n]

n8n Scaling Mode Setup Instructions

Setting Up n8n in Queue Mode
This guide will help you set up a scaled version of n8n using Redis as a mediator and a PostgreSQL database to share workflow information between the main instance and worker instances.

Prerequisites
Install Docker
Ensure Docker is installed on your system. If not, install it from Docker’s official website.

Pre-checks
Before proceeding, stop any existing instances of PostgreSQL, Redis, or n8n running on your local machine using the following command:

sudo systemctl stop

Steps to Set Up n8n in Queue Mode

  1. Create a Docker Network

docker network create n8n

  1. Create the queue.env File
    Create a new file named queue.env to store the required environment variables:

DB_TYPE=postgresdb
DB_POSTGRESDB_DATABASE=postgres
DB_POSTGRESDB_HOST=n8n-postgres
DB_POSTGRESDB_PASSWORD=mypasswd
DB_POSTGRESDB_USER=postgres
EXECUTIONS_MODE=queue
QUEUE_BULL_REDIS_HOST=n8n-redis
N8N_ENCRYPTION_KEY=
N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true

  1. Create Redis Container

docker run --name n8n-redis --network=n8n -d -p 6379:6379 redis

  1. Create PostgreSQL Container

docker run --name n8n-postgres --network=n8n -e POSTGRES_PASSWORD=mypasswd -p 5432:5432 -d postgres

  1. Create n8n Main Instance

docker run --name n8n-main --env-file .n8n/queue.env --network=n8n -p 5678:5678 n8nio/n8n

  1. Create Worker Instances

docker run --name n8n-worker1 --env-file .n8n/queue.env --network n8n n8nio/n8n worker

For additional worker instances, run:

docker run --name n8n-worker2 --env-file .n8n/queue.env --network n8n n8nio/n8n worker

  1. Verify Docker Containers
    Check if all necessary containers are running with:

docker ps

Testing Queue Mode

  1. Open n8n Main Instance
    Access the n8n interface:

http://localhost:5678

  1. Import a Test Workflow
    Copy the workflow JSON from the provided test document, paste it into the n8n workflow editor, activate and save the workflow.

  2. Monitor the Workflow Execution
    Check logs in your terminal windows to ensure tasks are distributed among worker instances.

Second-Level Scaling: Handling Webhooks Separately
To offload webhook handling to a dedicated instance, run:

docker run --name n8n-webhook1 --env-file .n8n/queue.env --network n8n n8nio/n8n webhook

Troubleshooting and Monitoring

  1. Checking Logs

docker logs -f

  1. Checking Redis Connectivity

docker exec -it n8n-redis redis-cli
ping

  1. Checking PostgreSQL Connectivity

docker exec -it n8n-postgres psql -U postgres
SELECT datname FROM pg_database;

  1. Debugging Execution Failures
    Check execution history in the n8n UI under Executions.

  2. Monitoring Resource Usage

docker stats

Conclusion
By following this setup, you successfully configure n8n to run in queue mode with Redis and PostgreSQL, ensuring efficient workload distribution and scalability.

1 Like