Empty "Error from queue" trying to set up worker process

Chase · May 27, 2022, 8:45pm

Describe the issue/error/question

The worker tasks start up successfully. Once running, however, their only further output is to log queue errors every 30 seconds:
Error from queue:

Looking at the code, it appears that the exception is supposed to output after that statement, but it’s blank in the log:
packages/cli/commands/worker/ts:351

logger.error('Error from queue: ', error);

Without error information, the cause has been difficult to diagnose. Does anyone know what could be causing this?

What is the error message (if any)?

Error from queue:

Please share the workflow

N/A

Share the output returned by the last node

N/A

Information on your n8n setup

n8n version: latest in docker
Database you’re using (default: SQLite): postgres
Running n8n with the execution process [own(default), main]:
Running n8n via [Docker, npm, n8n.cloud, desktop app]: Docker

Configuration Details

Main process

The existing production deployment is a task running in AWS ECS, which persists data in an RDS postgres instance. This main process is not yet set to run in queue mode. The intention is to flip it into queue mode once the worker tasks are operational.

Main process ENV

DB_POSTGRESDB_DATABASE n8n
DB_POSTGRESDB_HOST [db url]
DB_POSTGRESDB_PASSWORD [secret]
DB_POSTGRESDB_USER [user name]
DB_TYPE postgresdb
N8N_BASIC_AUTH_ACTIVE true
N8N_BASIC_AUTH_PASSWORD [secret]
N8N_BASIC_AUTH_USER [user name]
N8N_EMAIL_MODE smtp
N8N_ENCRYPTION_KEY [secret]
N8N_HOST [host url]
N8N_LOG_LEVEL info
N8N_PAYLOAD_SIZE_MAX 64
N8N_SMTP_HOST [smtp host url]
N8N_SMTP_PASS [secret]
N8N_SMTP_PORT 2465
N8N_SMTP_SENDER [sender email]
N8N_SMTP_SSL true
N8N_SMTP_USER [user id]
NODE_ENV production
WEBHOOK_URL [webhook url]

Worker process

The new worker processes have been added as tasks in the same ECS cluster. We initially tried hosting the job queue as a MemoryDB instance (AWS’s Redis). The result was the error described above. After attempts to get around the error, we guessed that there was some incompatibility between N8N and MemoryDB. We next tried using a regular Redis task within the ECS cluster instead. Unfortunately, this resulted in the same error.

Worker ENV

DB_POSTGRESDB_DATABASE n8n
DB_POSTGRESDB_HOST [db url]
DB_POSTGRESDB_PASSWORD [secret]
DB_POSTGRESDB_USER [user name]
DB_TYPE postgresdb
EXECUTIONS_MODE queue
N8N_ENCRYPTION_KEY [secret]
N8N_HOST [host url]
N8N_LOG_LEVEL info
N8N_PAYLOAD_SIZE_MAX 64
QUEUE_BULL_REDIS_HOST [host url]
QUEUE_BULL_REDIS_PORT 6379
WEBHOOK_URL [webhook url]

MutedJam · May 30, 2022, 9:38am

Hey @Chase, I tried reproducing the problem you have reported but didn’t run into any trouble here. The exact docker-compose.yml file I have used is this one:

docker-compose.yml

version: '3.8'

services:
  postgresdb:
    image: postgres:13
    restart: unless-stopped
    environment:
      - POSTGRES_USER=n8n
      - POSTGRES_PASSWORD=PsqlPassword1234
    volumes:
      - C:\Users\Tom\Desktop\queue_test\psql_data:/var/lib/postgresql/data

  n8n:
    depends_on:
      - postgresdb
      - redis
    image: n8nio/n8n:0.178.2
    restart: unless-stopped
    environment:
      - EXECUTIONS_MODE=queue
      - NODE_ENV=production
      - N8N_PORT=5678
      - N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_HOST=postgresdb
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=PsqlPassword1234
      - QUEUE_BULL_REDIS_HOST=redis
      - QUEUE_BULL_REDIS_PORT=6379
      - N8N_LOG_LEVEL=debug
      - EXECUTIONS_DATA_PRUNE=true
      - EXECUTIONS_DATA_MAX_AGE=1
    ports:
      - 5678:5678
    volumes:
      - C:\Users\Tom\Desktop\queue_test\n8n_data:/home/node/.n8n

  n8nworker:
    depends_on:
      - postgresdb
      - redis
      - n8n
    image: n8nio/n8n:0.178.2
    restart: unless-stopped
    environment:
      - EXECUTIONS_MODE=queue
      - EXECUTIONS_PROCESS=main
      - NODE_ENV=production
      - N8N_PORT=5679
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_HOST=postgresdb
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=PsqlPassword1234
      - QUEUE_BULL_REDIS_HOST=redis
      - QUEUE_BULL_REDIS_PORT=6379
      - N8N_LOG_LEVEL=debug
      - EXECUTIONS_DATA_PRUNE=true
      - EXECUTIONS_DATA_MAX_AGE=1
    ports:
      - 5679:5679
    volumes:
      - C:\Users\Tom\Desktop\queue_test\n8n_data:/home/node/.n8n
    command: n8n worker

  n8nwebhook:
    depends_on:
      - postgresdb
      - redis
      - n8n
    image: n8nio/n8n:0.178.2
    restart: unless-stopped
    environment:
      - EXECUTIONS_MODE=queue
      - EXECUTIONS_PROCESS=main
      - NODE_ENV=production
      - N8N_PORT=5680
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_HOST=postgresdb
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=PsqlPassword1234
      - QUEUE_BULL_REDIS_HOST=redis
      - QUEUE_BULL_REDIS_PORT=6379
      - N8N_LOG_LEVEL=debug
      - EXECUTIONS_DATA_PRUNE=true
      - EXECUTIONS_DATA_MAX_AGE=1
    ports:
      - 5680:5680
    volumes:
      - C:\Users\Tom\Desktop\queue_test\n8n_data:/home/node/.n8n
    command: n8n webhook

  redis:
    image: redis
    ports:
      - 6379:6379
    restart: unless-stopped

That said, I am not exactly familiar with AWS. Could you confirm whether you’re running into this problem when testing your n8n configuration even outside of AWS?

Chase · June 2, 2022, 1:29pm

Hey @MutedJam , thanks for looking into this. I haven’t tried this configuration locally, and Amazon ECS doesn’t use docker compose yml files for configuration.

However, I went through yours, and compared your settings to what I had. I found that I hadn’t set NODE_ENV, and my N8N_LOG_LEVEL was still set to info. I set the former to production, and the latter to debug. This hasn’t fixed the problem, but I am at seeing the exception populated in the logs instead of coming out blank, which is one of those rare cases getting an error is an improvement:

e[31merrore[39m | e[31mError from queue: e[39m {"errorno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect","file":"worker.js"}

It’s a little messy with the terminal control characters, but it looks like the worker is timing out trying to connect to the queue?

MutedJam · June 3, 2022, 9:58am

Good spot, the timeout here could indeed indicate a communication problem between your n8n instances and Redis. Perhaps @krynble could confirm this?

When facing timeouts myself there is almost always a firewall at play dropping my traffic somewhere along the way. I wonder if this might be the case for your AWS EC2 networking setup as well. Is there a chance your EC2 security group doesn’t allow Redis & n8n to talk to each other?

krynble · June 3, 2022, 11:51am

Hey @Chase and @MutedJam yes, indeed this looks like a network issue.

The expected output from a worker process should look like this

If you start n8n without redis (I stopped the redis container, so it simply doesn’t exist) you’ll get an output like this:

I hope this helps!

Chase · June 8, 2022, 3:38pm

Thanks for the response! Sorry for the lag between replies here, I’m hopping on and off this upgrade.

I’ll look into why those tasks can’t communicate. I am curious as to why I’m getting a different error though. Instead of the “trying to reconnect” and “Unable to connect”, I’m getting the ETIMEOUT exception. There’s also a new thing complaining about “No codex available”, but I think that’s unrelated:

Starting n8n worker...
UserSettings were generated and saved to: /home/node/.n8n/config
2022-06-08T15:28:00.979Z | e[34mdebuge[39m | e[34mNo codex available for: N8nTrainingCustomerDatastore.node.jse[39m {"file":"LoadNodesAndCredentials.js","function":"addCodex"}
2022-06-08T15:28:00.981Z | e[34mdebuge[39m | e[34mNo codex available for: N8nTrainingCustomerMessenger.node.jse[39m {"file":"LoadNodesAndCredentials.js","function":"addCodex"}
n8n worker is now ready
* Version: 0.180.0
* Concurrency: 10
2022-06-08T15:28:14.802Z | e[31merrore[39m | e[31mError from queue: e[39m {"errorno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect","file":"worker.js"}
/usr/local/lib/node_modules/n8n/dist/commands/worker.js:217
throw error;
^
Error: connect ETIMEDOUT
at Socket.<anonymous> (/usr/local/lib/node_modules/n8n/node_modules/ioredis/built/redis/index.js:327:37)
at Object.onceWrapper (node:events:641:28)
at Socket.emit (node:events:527:28)
at Socket.emit (node:domain:475:12)
at Socket._onTimeout (node:net:516:8)
at listOnTimeout (node:internal/timers:559:17)
at processTimers (node:internal/timers:502:7) {
errorno: 'ETIMEDOUT',
code: 'ETIMEDOUT',
syscall: 'connect'
}

krynble · September 2, 2022, 1:08pm

I would still guess this is a network issue.

Can you try downloading and installing redis-cli (it comes with the redis) package and try connecting to your redis from the same host that runs the worker process?

This should help you diagnose network connectivity issues.