### Bug Description
I noticed the warning that I should use `N8N_RUNNERS_ENABLE…D=true` so I turned it on.
At the moment my worker looks like:
```
x-shared: &shared
image: docker.n8n.io/n8nio/n8n:1.81.4
restart: unless-stopped
environment:
- N8N_EDITOR_BASE_URL=https://my-domain.tld
- WEBHOOK_URL=https://my-domain.tld
- DB_TYPE=postgresdb
- DB_POSTGRESDB_HOST=1.2.3.7
- DB_POSTGRESDB_PORT=5432
- DB_POSTGRESDB_DATABASE=n8n-test
- DB_POSTGRESDB_USER=n8n-test
- DB_POSTGRESDB_PASSWORD=EEEEEEEEEEEEEEEEEEEEEE
- EXECUTIONS_MODE=queue
- QUEUE_BULL_PREFIX=n8n-stage
- QUEUE_BULL_REDIS_CLUSTER_NODES=1.2.3.4:6379,1.2.3.5:6379,1.2.3.6:6379
- QUEUE_BULL_REDIS_PASSWORD=DDDDDDDDDDDDDD
- QUEUE_HEALTH_CHECK_ACTIVE=true
- N8N_ENCRYPTION_KEY=CCCCCCCCCCCCCCCCCC
- N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true
# Binary Data
- N8N_AVAILABLE_BINARY_DATA_MODES=s3,filesystem
- N8N_DEFAULT_BINARY_DATA_MODE=s3
# S3
- N8N_EXTERNAL_STORAGE_S3_HOST=storage.googleapis.com
- N8N_EXTERNAL_STORAGE_S3_BUCKET_NAME=n8n-stage-storage
- N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION=europe-west3
- N8N_EXTERNAL_STORAGE_S3_ACCESS_KEY=AAAAAAAAAAAAAAAAAAAAA
- N8N_EXTERNAL_STORAGE_S3_ACCESS_SECRET=BBBBBBBBBBBBBBBBBBBBBBBBBB
# security
- N8N_BLOCK_ENV_ACCESS_IN_NODE=true
# runers
- N8N_RUNNERS_ENABLED=true
volumes:
- n8n_stage_storage:/home/node/.n8n
- n8n_stage_data:/home/node/data:ro
services:
n8n-worker:
<<: *shared
command: worker
```
The issue is that after some executions the container crashes with:
```
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
```
It looks like the container is leaking memory, because even if there are no executions for a long time, the memory does never decrease.
Also there are a lot of `Task (X) rejected by Runner with reason "Offer expired and no open task slots"` entries.
There are no issues when not using the environment variable `N8N_RUNNERS_ENABLED`.
Here is the full log:
```
Task (spu8t-xe) rejected by Runner with reason "Offer expired and no open task slots"
Task (g8vdKEVM) rejected by Runner with reason "Offer expired and no open task slots"
Task (Xmp6YzEb) rejected by Runner with reason "Offer expired and no open task slots"
Task (55hbR0w2) rejected by Runner with reason "Offer expired and no open task slots"
Task (srOzNIJY) rejected by Runner with reason "Offer expired and no open task slots"
Task (MK9GYUtF) rejected by Runner with reason "Offer expired and no open task slots"
Task (EA3sIhw8) rejected by Runner with reason "Offer expired and no open task slots"
Worker started execution 4974 (job 385)
Credentials not found
Credentials not found
Credentials not found
Worker finished execution 4974 (job 385)
Worker started execution 4979 (job 386)
Worker started execution 4982 (job 387)
Worker finished execution 4979 (job 386)
Worker finished execution 4982 (job 387)
Task (DCHekng4) rejected by Runner with reason "Offer expired and no open task slots"
Task (OQZX0IZg) rejected by Runner with reason "Offer expired and no open task slots"
Task (KPLnzrYW) rejected by Runner with reason "Offer expired and no open task slots"
Task (u-EQx-KZ) rejected by Runner with reason "Offer expired and no open task slots"
Task (OFM0ccLr) rejected by Runner with reason "Offer expired and no open task slots"
Worker finished execution 4764 (job 384)
Worker started execution 5046 (job 388)
Task (91aCdsrP) rejected by Runner with reason "Offer expired and no open task slots"
Task (YgP1c1w0) rejected by Runner with reason "Offer expired and no open task slots"
Task (qVIVhgTw) rejected by Runner with reason "Offer expired and no open task slots"
Task (WIYXPfWg) rejected by Runner with reason "Offer expired and no open task slots"
Task (F88b1zbr) rejected by Runner with reason "Offer expired and no open task slots"
Task (J2UJsw8i) rejected by Runner with reason "Offer expired and no open task slots"
Task (Xfdko8RW) rejected by Runner with reason "Offer expired and no open task slots"
Task (rucFHnpj) rejected by Runner with reason "Offer expired and no open task slots"
Task (bFIxCi8U) rejected by Runner with reason "Offer expired and no open task slots"
Task (lx5g2QQN) rejected by Runner with reason "Offer expired and no open task slots"
Task (fsWR5uSM) rejected by Runner with reason "Offer expired and no open task slots"
Task (FgyqOo4M) rejected by Runner with reason "Offer expired and no open task slots"
Task (yZOxE3yQ) rejected by Runner with reason "Offer expired and no open task slots"
Task (JxTX-03f) rejected by Runner with reason "Offer expired and no open task slots"
Task (m-NKnIHe) rejected by Runner with reason "Offer expired and no open task slots"
Task (LNjAHAOl) rejected by Runner with reason "Offer expired and no open task slots"
Task (w5sIU5Tr) rejected by Runner with reason "Offer expired and no open task slots"
Task (KuN8uGlB) rejected by Runner with reason "Offer expired and no open task slots"
Task (ZhXwFcyI) rejected by Runner with reason "Offer expired and no open task slots"
Task (E8aNUyOZ) rejected by Runner with reason "Offer expired and no open task slots"
Task (woagBQ6t) rejected by Runner with reason "Offer expired and no open task slots"
Task (c5covv61) rejected by Runner with reason "Offer expired and no open task slots"
Task (HtCoGG4g) rejected by Runner with reason "Offer expired and no open task slots"
Task (9iMTNY5w) rejected by Runner with reason "Offer expired and no open task slots"
Task (g2x1997T) rejected by Runner with reason "Offer expired and no open task slots"
Task (oWLv9pAu) rejected by Runner with reason "Offer expired and no open task slots"
Task (XzHoKnpa) rejected by Runner with reason "Offer expired and no open task slots"
Task (Ya3SSBrJ) rejected by Runner with reason "Offer expired and no open task slots"
Worker started execution 5269 (job 389)
Worker finished execution 5269 (job 389)
Task (dFgBU2TK) rejected by Runner with reason "Offer expired and no open task slots"
Task (7jmC7LOy) rejected by Runner with reason "Offer expired and no open task slots"
Task (1EuAelfv) rejected by Runner with reason "Offer expired and no open task slots"
Task (LRV_0XDO) rejected by Runner with reason "Offer expired and no open task slots"
Task (J3j7KD8A) rejected by Runner with reason "Offer expired and no open task slots"
Task (yMp2IM_U) rejected by Runner with reason "Offer expired and no open task slots"
Task (3j2GpW19) rejected by Runner with reason "Offer expired and no open task slots"
404 - "404 page not found\n"
404 - "404 page not found\n"
404 - "404 page not found\n"
Worker finished execution 5046 (job 388)
Worker started execution 5324 (job 390)
Task (ksezm1Gc) rejected by Runner with reason "Offer expired and no open task slots"
Task (RwlNxzKv) rejected by Runner with reason "Offer expired and no open task slots"
Task (0pQplVot) rejected by Runner with reason "Offer expired and no open task slots"
Task (IUK54Vlw) rejected by Runner with reason "Offer expired and no open task slots"
Task (GUSaCHoJ) rejected by Runner with reason "Offer expired and no open task slots"
Task (lnVHBepn) rejected by Runner with reason "Offer expired and no open task slots"
Task (RqhtD5o1) rejected by Runner with reason "Offer expired and no open task slots"
Task (aFou2teB) rejected by Runner with reason "Offer expired and no open task slots"
Task (sHjGqTa-) rejected by Runner with reason "Offer expired and no open task slots"
Task (Xo1TfUs3) rejected by Runner with reason "Offer expired and no open task slots"
Task (KO3JhpjO) rejected by Runner with reason "Offer expired and no open task slots"
Task (AjJsNR_X) rejected by Runner with reason "Offer expired and no open task slots"
Task (KmMXyCY0) rejected by Runner with reason "Offer expired and no open task slots"
Task (Qo8s1aKB) rejected by Runner with reason "Offer expired and no open task slots"
Task (uKdvSRYX) rejected by Runner with reason "Offer expired and no open task slots"
Task (9qdcN1F0) rejected by Runner with reason "Offer expired and no open task slots"
Task (It_rlMlf) rejected by Runner with reason "Offer expired and no open task slots"
Task (ZQJQR8Pv) rejected by Runner with reason "Offer expired and no open task slots"
Task (TlHGdjSW) rejected by Runner with reason "Offer expired and no open task slots"
Task (TYehyt-w) rejected by Runner with reason "Offer expired and no open task slots"
Task (YLtivMdp) rejected by Runner with reason "Offer expired and no open task slots"
Task (fdTC0Ykm) rejected by Runner with reason "Offer expired and no open task slots"
Task (CPVv_dzZ) rejected by Runner with reason "Offer expired and no open task slots"
Task (_v624PFs) rejected by Runner with reason "Offer expired and no open task slots"
Task (nLcAAUhp) rejected by Runner with reason "Offer expired and no open task slots"
Task (yy3yc1zc) rejected by Runner with reason "Offer expired and no open task slots"
Task (Y2muwWH3) rejected by Runner with reason "Offer expired and no open task slots"
Task (OBD2tCo3) rejected by Runner with reason "Offer expired and no open task slots"
Task (02ttefj7) rejected by Runner with reason "Offer expired and no open task slots"
Task (bRllFMt2) rejected by Runner with reason "Offer expired and no open task slots"
Task (_J5tdLO7) rejected by Runner with reason "Offer expired and no open task slots"
Task (Bc2HPE16) rejected by Runner with reason "Offer expired and no open task slots"
Task (KpMX1wbF) rejected by Runner with reason "Offer expired and no open task slots"
Task (OZsYEh70) rejected by Runner with reason "Offer expired and no open task slots"
Task (q2h9wZCX) rejected by Runner with reason "Offer expired and no open task slots"
Task (EclrPx2L) rejected by Runner with reason "Offer expired and no open task slots"
<--- Last few GCs --->
[7:0xffffbaa14610] 8293609 ms: Scavenge 2032.7 (2087.9) -> 2024.9 (2087.9) MB, 14.48 / 0.00 ms (average mu = 0.162, current mu = 0.021) allocation failure;
[7:0xffffbaa14610] 8293632 ms: Scavenge 2032.9 (2087.9) -> 2025.1 (2088.2) MB, 4.37 / 0.00 ms (average mu = 0.162, current mu = 0.021) allocation failure;
[7:0xffffbaa14610] 8295314 ms: Mark-Compact 2032.9 (2088.2) -> 2024.3 (2088.2) MB, 1663.45 / 0.00 ms (average mu = 0.171, current mu = 0.180) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
```
After restart:
```
Invalid timestamp value for N8N_RELEASE_DATE: $(date -u +"%Y-%m-%dT%H:%M:%SZ")
User settings loaded from: /home/node/.n8n/config
Found unfinished executions: 4121, 4423, 4425, 4426, 4428, 4434, 4437, 4440, 4443, 4446, 4449, 4453, 4455, 4458, 4460, 4462, 4466, 4467, 4472, 4475, 4478, 4479, 4481, 4487, 4490, 4493, 4496, 4499, 4502, 4505, 4508, 4511, 4513, 4517, 4520, 4523, 4526, 4529, 4532, 4535, 4538, 4541, 4544, 4545, 4550, 4553, 4556, 4559, 4562, 4565, 4568, 4571, 4573, 4577, 4580, 4583, 4586, 4588, 4591, 4593, 4596, 4597, 4600, 4605, 4608, 4611, 4614, 4617, 4620, 4623, 4626, 4629, 4632, 4635, 4638, 4641, 4644, 4645, 4650, 4653, 4656, 4659, 4662, 4665, 4668, 4671, 4673, 4677, 4679, 4683, 4686, 4689, 4692, 4695, 4698, 4701, 4704, 4705, 4706, 4713, 4716, 4719, 4720, 4725, 4728, 4730, 4733, 4736, 4738, 4741, 4744, 4747, 4750, 4755, 4758, 4761, 4764, 4974, 4979, 4982, 5046, 5269, 5324, 5325, 5475
This could be due to a crash of an active workflow or a restart of n8n.
Currently active workflows:
- Import Trigger Schedule (ID: T5pFqhkUiMCcrgM9)
- Jira Webhook (ID: EMzNZ9O3lhWZgUyq)
[Recovery] Logs available, amended execution
[Recovery] Logs available, amended execution
[Recovery] Logs available, amended execution
[Recovery] Logs available, amended execution
n8n Task Broker ready on 127.0.0.1, port 5679
n8n worker is now ready
* Version: 1.81.4
* Concurrency: 10
n8n worker server listening on port 5678
Registered runner "JS Task Runner" (04aT0FvfAo1yRojfpnNlL)
Worker started execution 5476 (job 391)
Task (77eoL3-H) rejected by Runner with reason "Offer expired and no open task slots"
Task (fR_KTLI5) rejected by Runner with reason "Offer expired and no open task slots"
Task (ArY-rW74) rejected by Runner with reason "Offer expired and no open task slots"
Task (iT8FcITY) rejected by Runner with reason "Offer expired and no open task slots"
```
There is also something else I noticed after the restart, there are a lot of `Found unfinished executions` with ids which are listed as `Worker finished execution` in the log before, which is quite confusing, another bug?
# Debug info
## core
- n8nVersion: 1.81.4
- platform: docker (self-hosted)
- nodeJsVersion: 20.18.3
- database: postgres
- executionMode: scaling
- concurrency: -1
- license: enterprise (production)
- consumerId: 03176d77-824f-4ef2-83db-39f7bc022c2f
## storage
- success: all
- error: all
- progress: false
- manual: true
- binaryMode: s3
## pruning
- enabled: true
- maxAge: 336 hours
- maxCount: 10000 executions
## client
- userAgent: mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/133.0.0.0 safari/537.36
- isTouchDevice: false
Generated at: 2025-03-06T15:22:24.288Z
### To Reproduce
1. Setup a worker with `N8N_RUNNERS_ENABLED=true`
2. Run some memory intensive work, like CSV parsing with lot of rows, while still sending some little json webhook executions.
3. Notice how the memory stays high after execution.
### Expected behavior
No memory leak and crashes if `N8N_RUNNERS_ENABLED=true` is set.
### Operating System
docker (self-hosted)
### n8n Version
1.81.4
### Node.js Version
20.18.3
### Database
PostgreSQL
### Execution mode
queue
