As the title states, we are experiencing some difficult to discover OOM issue’s that where not there before.
For some reason all manual started processes get stuck on an OOM. I can start them with the “execute workflow” button, which starts and (for some processes) finishes but as soon as i press the play button of a specific node, it executes the code until that node but the node itself never finishes and reaches an OOM error.
It all started when i tried to run the “orderList multiverse” node which sometime succeeds with an 5mb response after which it immediately crashes with OOM.
In the example below I started the flow pressing the playbutton at the right top of the “updateableVariables1” node on which it got stuck.
All workflows are turned off so there are no other workflows running (except for the webhook for Oauth, which is not triggered in this case until it would reach the “GetOauthMultivers1”, due to the way multivers works the normal oauth does not work )
the “code” node only contains a static response and should never get stuck or reach 8GB memory:
return [{json: {
magentoCount: 0
}}];

using the “execute workflow” leads to the next part but eventually gets killed due to OOM:
previously this all worked without any issue’s but we noticed today that for a couple of days al workflows got stopped due to these OOM’s
we are running this on our own server with the docker-compose setup
our .env contains these as an attempt to lessen the memory used
N8N_LOG_LEVEL=error
N8N_LOG_OUTPUT=file
N8N_LOG_FILE_MAXCOUNT=1
NODE_ENV=production
N8N_DEFAULT_BINARY_DATA_MODE=filesystem
the docker-compose contains:
version: "3"
services:
traefik:
image: "traefik"
restart: always
command:
- "--api=true"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--entrypoints.web.http.redirections.entryPoint.to=websecure"
- "--entrypoints.web.http.redirections.entryPoint.scheme=https"
- "--entrypoints.websecure.address=:443"
- "--certificatesresolvers.mytlschallenge.acme.tlschallenge=true"
- "--certificatesresolvers.mytlschallenge.acme.email=${SSL_EMAIL}"
- "--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json"
ports:
- "80:80"
- "443:443"
volumes:
- ${DATA_FOLDER}/letsencrypt:/letsencrypt
- /var/run/docker.sock:/var/run/docker.sock:ro
n8n:
image: n8nio/n8n
restart: always
ports:
- "127.0.0.1:5678:5678"
labels:
- traefik.enable=true
- traefik.http.routers.n8n.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`)
- traefik.http.routers.n8n.tls=true
- traefik.http.routers.n8n.entrypoints=websecure
- traefik.http.routers.n8n.tls.certresolver=mytlschallenge
- traefik.http.middlewares.n8n.headers.SSLRedirect=true
- traefik.http.middlewares.n8n.headers.STSSeconds=315360000
- traefik.http.middlewares.n8n.headers.browserXSSFilter=true
- traefik.http.middlewares.n8n.headers.contentTypeNosniff=true
- traefik.http.middlewares.n8n.headers.forceSTSHeader=true
- traefik.http.middlewares.n8n.headers.SSLHost=${DOMAIN_NAME}
- traefik.http.middlewares.n8n.headers.STSIncludeSubdomains=true
- traefik.http.middlewares.n8n.headers.STSPreload=true
environment:
- N8N_BASIC_AUTH_ACTIVE=true
- N8N_BASIC_AUTH_USER
- N8N_BASIC_AUTH_PASSWORD
- N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
- N8N_PORT=5678
- N8N_PROTOCOL=https
- N8N_LOG_LEVEL=error
- N8N_LOG_OUTPUT=file
- N8N_LOG_FILE_MAXCOUNT=1
- NODE_ENV=production
- WEBHOOK_TUNNEL_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
- EXECUTIONS_DATA_PRUNE=true
- EXECUTIONS_DATA_MAX_AGE=72
# - EXECUTIONS_PROCESS=main
- N8N_DEFAULT_BINARY_DATA_MODE=filesystem
# - EXECUTIONS_DATA_SAVE_ON_ERROR=all
# - EXECUTIONS_DATA_SAVE_MANUAL_EXECUTIONS=false
# - DB_SQLITE_VACUUM_ON_STARTUP=true
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ${DATA_FOLDER}/.n8n:/home/node/.n8n
- ${DATA_FOLDER}/n8n-files:/files
while the log_level is set to error there are also debug logs? these are the last 50 lines:
{"level":"debug","message":"Start processing node \"Function4\"","metadata":{"file":"WorkflowExecute.js","node":"Function4","timestamp":"2023-01-03T15:42:10.299Z","workflowId":"18"}}
{"level":"debug","message":"Running node \"Function4\" started","metadata":{"file":"WorkflowExecute.js","node":"Function4","timestamp":"2023-01-03T15:42:10.299Z","workflowId":"18"}}
{"level":"debug","message":"Running node \"Function4\" finished successfully","metadata":{"file":"WorkflowExecute.js","node":"Function4","timestamp":"2023-01-03T15:42:10.309Z","workflowId":"18"}}
{"level":"debug","message":"Start processing node \"orderList multiverse\"","metadata":{"file":"WorkflowExecute.js","node":"orderList multiverse","timestamp":"2023-01-03T15:42:10.309Z","workflowId":"18"}}
{"level":"debug","message":"Running node \"orderList multiverse\" started","metadata":{"file":"WorkflowExecute.js","node":"orderList multiverse","timestamp":"2023-01-03T15:42:10.310Z","workflowId":"18"}}
{"level":"debug","message":"Proxying request to axios","metadata":{"file":"NodeExecuteFunctions.js","function":"proxyRequestToAxios","timestamp":"2023-01-03T15:42:10.318Z"}}
{"level":"debug","message":"Received child process message of type processHook for execution ID 252235.","metadata":{"executionId":"252235","file":"WorkflowRunner.js","timestamp":"2023-01-03T15:42:35.241Z"}}
{"level":"debug","message":"Running node \"orderList multiverse\" finished successfully","metadata":{"file":"WorkflowExecute.js","node":"orderList multiverse","timestamp":"2023-01-03T15:42:35.241Z","workflowId":"18"}}
{"level":"debug","message":"Start processing node \"IF10\"","metadata":{"file":"WorkflowExecute.js","node":"IF10","timestamp":"2023-01-03T15:42:35.451Z","workflowId":"18"}}
{"level":"debug","message":"Received child process message of type processHook for execution ID 252235.","metadata":{"executionId":"252235","file":"WorkflowRunner.js","timestamp":"2023-01-03T15:42:35.553Z"}}
{"level":"debug","message":"Running node \"IF10\" started","metadata":{"file":"WorkflowExecute.js","node":"IF10","timestamp":"2023-01-03T15:42:35.452Z","workflowId":"18"}}
{"level":"debug","message":"Running node \"IF10\" finished successfully","metadata":{"file":"WorkflowExecute.js","node":"IF10","timestamp":"2023-01-03T15:42:35.464Z","workflowId":"18"}}
{"level":"debug","message":"Executing hook on node \"orderList multiverse\" (hookFunctionsPush)","metadata":{"executionId":"252235","file":"WorkflowExecuteAdditionalData.js","function":"nodeExecuteAfter","sessionId":"6mas4b5yk7m","timestamp":"2023-01-03T15:42:35.553Z","workflowId":"18"}}
{"level":"debug","message":"Send data of type \"nodeExecuteAfter\" to editor-UI","metadata":{"dataType":"nodeExecuteAfter","file":"Push.js","function":"send","sessionId":"6mas4b5yk7m","timestamp":"2023-01-03T15:42:35.554Z"}}
{"level":"debug","message":"Received child process message of type processHook for execution ID 252235.","metadata":{"executionId":"252235","file":"WorkflowRunner.js","timestamp":"2023-01-03T15:42:35.671Z"}}
{"level":"debug","message":"Executing hook on node \"IF10\" (hookFunctionsPush)","metadata":{"executionId":"252235","file":"WorkflowExecuteAdditionalData.js","function":"nodeExecuteBefore","sessionId":"6mas4b5yk7m","timestamp":"2023-01-03T15:42:35.672Z","workflowId":"18"}}
{"level":"debug","message":"Send data of type \"nodeExecuteBefore\" to editor-UI","metadata":{"dataType":"nodeExecuteBefore","file":"Push.js","function":"send","sessionId":"6mas4b5yk7m","timestamp":"2023-01-03T15:42:35.672Z"}}
{"level":"debug","message":"Start processing node \"GetOauthMultivers2\"","metadata":{"file":"WorkflowExecute.js","node":"GetOauthMultivers2","timestamp":"2023-01-03T15:42:35.716Z","workflowId":"18"}}
{"level":"debug","message":"Running node \"GetOauthMultivers2\" started","metadata":{"file":"WorkflowExecute.js","node":"GetOauthMultivers2","timestamp":"2023-01-03T15:42:35.717Z","workflowId":"18"}}
{"level":"debug","message":"Proxying request to axios","metadata":{"file":"NodeExecuteFunctions.js","function":"proxyRequestToAxios","timestamp":"2023-01-03T15:42:35.719Z"}}
{"level":"debug","message":"Received child process message of type processHook for execution ID 252235.","metadata":{"executionId":"252235","file":"WorkflowRunner.js","timestamp":"2023-01-03T15:42:35.790Z"}}
{"level":"debug","message":"Executing hook on node \"IF10\" (hookFunctionsPush)","metadata":{"executionId":"252235","file":"WorkflowExecuteAdditionalData.js","function":"nodeExecuteAfter","sessionId":"6mas4b5yk7m","timestamp":"2023-01-03T15:42:35.791Z","workflowId":"18"}}
{"level":"debug","message":"Send data of type \"nodeExecuteAfter\" to editor-UI","metadata":{"dataType":"nodeExecuteAfter","file":"Push.js","function":"send","sessionId":"6mas4b5yk7m","timestamp":"2023-01-03T15:42:35.791Z"}}
{"level":"debug","message":"Received child process message of type processHook for execution ID 252235.","metadata":{"executionId":"252235","file":"WorkflowRunner.js","timestamp":"2023-01-03T15:42:35.868Z"}}
{"level":"debug","message":"Executing hook on node \"GetOauthMultivers2\" (hookFunctionsPush)","metadata":{"executionId":"252235","file":"WorkflowExecuteAdditionalData.js","function":"nodeExecuteBefore","sessionId":"6mas4b5yk7m","timestamp":"2023-01-03T15:42:35.868Z","workflowId":"18"}}
{"level":"debug","message":"Send data of type \"nodeExecuteBefore\" to editor-UI","metadata":{"dataType":"nodeExecuteBefore","file":"Push.js","function":"send","sessionId":"6mas4b5yk7m","timestamp":"2023-01-03T15:42:35.869Z"}}
{"level":"debug","message":"Received child process message of type sendMessageToUI for execution ID 252235.","metadata":{"executionId":"252235","file":"WorkflowRunner.js","timestamp":"2023-01-03T15:42:35.869Z"}}
{"level":"debug","message":"Send data of type \"sendConsoleMessage\" to editor-UI","metadata":{"dataType":"sendConsoleMessage","file":"Push.js","function":"send","sessionId":"6mas4b5yk7m","timestamp":"2023-01-03T15:42:35.869Z"}}
{"level":"debug","message":"Received webhook \"GET\" for path \"5f50a821-51b9-4e36-98ec-f5d276f1c3f0\"","metadata":{"file":"ActiveWorkflowRunner.js","function":"executeWebhook","timestamp":"2023-01-03T15:42:35.881Z"}}
{"level":"info","message":"Initializing n8n process","metadata":{"file":"start.js","function":"run","timestamp":"2023-01-03T15:42:44.472Z"}}
{"level":"error","message":"Last session crashed","metadata":{"file":"CrashJournal.js","function":"init","timestamp":"2023-01-03T15:42:44.480Z"}}
{"level":"debug","message":"Lazy Loading credentials and nodes from n8n-nodes-base","metadata":{"credentials":310,"file":"DirectoryLoader.js","function":"loadAll","nodes":389,"timestamp":"2023-01-03T15:42:55.809Z"}}
{"level":"debug","message":"Wait tracker querying database for waiting executions","metadata":{"file":"WaitTracker.js","function":"getWaitingExecutions","timestamp":"2023-01-03T15:42:56.062Z"}}
{"level":"debug","message":"[license] initializing for deviceFingerprint 894cf08c5e93f1e0fe9d141c18deb2a60f7fc8dedf12e32b02eb6c5be667d415","metadata":{"file":"LicenseManager.js","function":"log","timestamp":"2023-01-03T15:42:57.145Z"}}
{"level":"debug","message":"Initializing active workflow \"Multivers OAuth process\" (startup)","metadata":{"file":"ActiveWorkflowRunner.js","function":"init","timestamp":"2023-01-03T15:42:57.674Z","workflowId":15,"workflowName":"Multivers OAuth process"}}
{"level":"debug","message":"Add editor-UI session","metadata":{"file":"Push.js","function":"add","sessionId":"pxssqc326p8","timestamp":"2023-01-03T15:42:57.871Z"}}
{"level":"debug","message":"Add editor-UI session","metadata":{"file":"Push.js","function":"add","sessionId":"6mas4b5yk7m","timestamp":"2023-01-03T15:42:57.877Z"}}
{"level":"verbose","message":"Successfully started workflow \"Multivers OAuth process\"","metadata":{"file":"ActiveWorkflowRunner.js","function":"init","timestamp":"2023-01-03T15:42:57.923Z","workflowId":15,"workflowName":"Multivers OAuth process"}}
{"level":"verbose","message":"Finished initializing active workflows (startup)","metadata":{"file":"ActiveWorkflowRunner.js","function":"init","timestamp":"2023-01-03T15:42:57.925Z"}}
{"level":"debug","message":"Add editor-UI session","metadata":{"file":"Push.js","function":"add","sessionId":"r7eql1zebsc","timestamp":"2023-01-03T15:42:59.282Z"}}
{"level":"debug","message":"Wait tracker querying database for waiting executions","metadata":{"file":"WaitTracker.js","function":"getWaitingExecutions","timestamp":"2023-01-03T15:43:56.080Z"}}
{"level":"debug","message":"Wait tracker querying database for waiting executions","metadata":{"file":"WaitTracker.js","function":"getWaitingExecutions","timestamp":"2023-01-03T15:44:56.096Z"}}
{"level":"debug","message":"Wait tracker querying database for waiting executions","metadata":{"file":"WaitTracker.js","function":"getWaitingExecutions","timestamp":"2023-01-03T15:45:56.117Z"}}
{"level":"debug","message":"Wait tracker querying database for waiting executions","metadata":{"file":"WaitTracker.js","function":"getWaitingExecutions","timestamp":"2023-01-03T15:46:56.136Z"}}
{"level":"debug","message":"Wait tracker querying database for waiting executions","metadata":{"file":"WaitTracker.js","function":"getWaitingExecutions","timestamp":"2023-01-03T15:47:56.156Z"}}
{"level":"debug","message":"Wait tracker querying database for waiting executions","metadata":{"file":"WaitTracker.js","function":"getWaitingExecutions","timestamp":"2023-01-03T15:48:56.176Z"}}
{"level":"debug","message":"Wait tracker querying database for waiting executions","metadata":{"file":"WaitTracker.js","function":"getWaitingExecutions","timestamp":"2023-01-03T15:49:56.194Z"}}
{"level":"debug","message":"Wait tracker querying database for waiting executions","metadata":{"file":"WaitTracker.js","function":"getWaitingExecutions","timestamp":"2023-01-03T15:50:56.214Z"}}
{"level":"debug","message":"Wait tracker querying database for waiting executions","metadata":{"file":"WaitTracker.js","function":"getWaitingExecutions","timestamp":"2023-01-03T15:51:56.234Z"}}
{"level":"debug","message":"Wait tracker querying database for waiting executions","metadata":{"file":"WaitTracker.js","function":"getWaitingExecutions","timestamp":"2023-01-03T15:52:56.254Z"}}
n8n version: 0.209.4
Does someone know how to solve these OOM?