N8n Compute Engine instance randomly crashes

nahueld · April 7, 2023, 2:36pm

Describe the problem/error/question

Our VM running in Compute Engine in GCP using the standard docker images crashes from time to time, when this happens we completely lose access to the VM and we need to reset the VM and the services to continue operating

Our question is, is this a known issue? what type of deployment do you recommend in GCP?

We have tried Cloud Run but it does not work well, this is a known limitation because n8n does not support the main instance scaling
We have tried Compute Engine with the issue described in this post to no success

What else can we do? We don’t have anyone in the team with enough infrastructure experience that can perform the Kubernetes installation

We are open to explore something like

Main instance running in Compute Engine
Workers running in Cloud Run (for scaling) although I haven’t found docs on how to do this exactly

What is the error message (if any)?

The only exception I see in the logs is the following

DEFAULT 2023-04-05T23:16:29.301636334Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] Initializing n8n process
DEFAULT 2023-04-05T23:16:39.831220219Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 editorUiConnected listeners added to [MessageEventBus]. Use emitter.setMaxListeners() to increase limit
DEFAULT 2023-04-05T23:16:39.831373443Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] (Use `node --trace-warnings ...` to show where the warning was created)
DEFAULT 2023-04-05T23:16:39.873698472Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] n8n ready on 0.0.0.0, port 5678

Please share your workflow

N/A

Share the output returned by the last node

N/A

Information on your n8n setup

n8n version: n8nio/n8n:0.219.1
Database (default: SQLite): Postgres 13
n8n EXECUTIONS_PROCESS setting (default: own, main): main
Running n8n via (Docker, npm, n8n cloud, desktop app): docker
Operating system:
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal

nahueld · April 7, 2023, 2:53pm

Also may be related to this one?

Jon · April 7, 2023, 6:18pm

Hey @nahueld,

Welcome to the community

There shouldn’t be any issues with n8n in GCP with Compute, Have you checked the compute logs when it crashes? Docker crashing or locking up shouldn’t cause the VM to stop working so it sounds like there could be a bit to it.

nahueld · April 7, 2023, 7:21pm

Yes, the compute logs are the ones in the ticket, they talk about some memory leak but they don’t add much more information

Jon · April 8, 2023, 2:38am

Hey @nahueld,

They looked like docker logs to me rather than compute OS logs. Can you confirm how you got the logs?

nahueld · April 10, 2023, 12:03am

Correct these are the docker logs I’m sending to Cloud Logging, the VM is running the docker image

Jon · April 10, 2023, 8:58am

It has been a while since I have used compute but does that give you access to the OS itself to install docker and load the image on or does it directly run a docker image?

nahueld · April 10, 2023, 11:59am

It gives you access to the OS, there you can install docker and on top of that run the containers, we are currently using a pre-built image on the VM called “Docker Compose Server on Ubuntu Server 20.04”, and the VM specs are

VM instance: 1 vCPU + 3.75 GB memory
Standard Persistent Disk: 10GB

nahueld · April 19, 2023, 8:21pm

Something I noticed before the instance crashes is the following

DEFAULT 2023-04-19T20:16:34.234218061Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] query is slow: SELECT "SharedWorkflow"."workflowId" AS "SharedWorkflow_workflowId", "SharedWorkflow"."userId" AS "SharedWorkflow_userId", "SharedWorkflow__SharedWorkflow_workflow"."createdAt" AS "SharedWorkflow__SharedWorkflow_workflow_createdAt", "SharedWorkflow__SharedWorkflow_workflow"."updatedAt" AS "SharedWorkflow__SharedWorkflow_workflow_updatedAt", "SharedWorkflow__SharedWorkflow_workflow"."id" AS "SharedWorkflow__SharedWorkflow_workflow_id", "SharedWorkflow__SharedWorkflow_workflow"."name" AS "SharedWorkflow__SharedWorkflow_workflow_name", "SharedWorkflow__SharedWorkflow_workflow"."active" AS "SharedWorkflow__SharedWorkflow_workflow_active", "SharedWorkflow__SharedWorkflow_workflow"."nodes" AS "SharedWorkflow__SharedWorkflow_workflow_nodes", "SharedWorkflow__SharedWorkflow_workflow"."connections" AS "SharedWorkflow__SharedWorkflow_workflow_connections", "SharedWorkflow__SharedWorkflow_workflow"."settings" AS "SharedWorkflow__SharedWorkflow_workflow_settings", "SharedWorkflow__SharedWorkflow_workflow"."staticData" AS "SharedWorkflow__SharedWorkflow_workflow_staticData", "SharedWorkflow__SharedWorkflow_workflow"."pinData" AS "SharedWorkflow__SharedWorkflow_workflow_pinData", "SharedWorkflow__SharedWorkflow_workflow"."versionId" AS "SharedWorkflow__SharedWorkflow_workflow_versionId", "SharedWorkflow__SharedWorkflow_workflow"."triggerCount" AS "SharedWorkflow__SharedWorkflow_workflow_triggerCount", "SharedWorkflow__SharedWorkflow_role"."createdAt" AS "SharedWorkflow__SharedWorkflow_role_createdAt", "SharedWorkflow__SharedWorkflow_role"."updatedAt" AS "SharedWorkflow__SharedWorkflow_role_updatedAt", "SharedWorkflow__SharedWorkflow_role"."id" AS "SharedWorkflow__SharedWorkflow_role_id", "SharedWorkflow__SharedWorkflow_role"."name" AS "SharedWorkflow__SharedWorkflow_role_name", "SharedWorkflow__SharedWorkflow_role"."scope" AS "SharedWorkflow__SharedWorkflow_role_scope" FROM "n8n2"."shared_workflow" "SharedWorkflow" LEFT JOIN "n8n2"."workflow_entity" "SharedWorkflow__SharedWorkflow_workflow" ON "SharedWorkflow__SharedWorkflow_workflow"."id"="SharedWorkflow"."workflowId" LEFT JOIN "n8n2"."role" "SharedWorkflow__SharedWorkflow_role" ON "SharedWorkflow__SharedWorkflow_role"."id"="SharedWorkflow"."roleId"
DEFAULT 2023-04-19T20:16:34.270483125Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] execution time: 1337
DEFAULT 2023-04-19T20:16:34.308839587Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] query is slow: SELECT COUNT(1) AS "cnt" FROM "n8n2"."execution_entity" "ExecutionEntity" WHERE ("ExecutionEntity"."workflowId" = $1 AND NOT("ExecutionEntity"."id" IN ($2, $3, $4, $5, $6, $7, $8, $9, $10))) -- PARAMETERS: [70,"87824","87825","87826","87827","87828","87829","87830","87831","87832"]
DEFAULT 2023-04-19T20:16:34.322493899Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] execution time: 1211
DEFAULT 2023-04-19T20:16:34.335499143Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] query is slow: SELECT COUNT(1) AS "cnt" FROM "n8n2"."execution_entity" "ExecutionEntity" WHERE ("ExecutionEntity"."workflowId" = $1 AND NOT("ExecutionEntity"."id" IN ($2, $3, $4, $5, $6, $7, $8, $9, $10))) -- PARAMETERS: [70,"87824","87825","87826","87827","87828","87829","87830","87831","87832"]
DEFAULT 2023-04-19T20:16:34.338461066Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] execution time: 1017
DEFAULT 2023-04-19T20:16:34.620172585Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] query is slow: SELECT "execution"."id" AS "execution_id", "execution"."finished" AS "execution_finished", "execution"."mode" AS "execution_mode", "execution"."retryOf" AS "execution_retryOf", "execution"."retrySuccessId" AS "execution_retrySuccessId", "execution"."status" AS "execution_status", "execution"."startedAt" AS "execution_startedAt", "execution"."stoppedAt" AS "execution_stoppedAt", "execution"."workflowData" AS "execution_workflowData", "execution"."waitTill" AS "execution_waitTill" FROM "n8n2"."execution_entity" "execution" WHERE ("execution"."workflowId" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46, $47, $48, $49, $50, $51, $52, $53, $54, $55, $56, $57, $58) AND id NOT IN ($59, $60, $61, $62, $63, $64, $65, $66)) AND "execution"."workflowId" = $67 ORDER BY id DESC LIMIT 20 -- PARAMETERS: ["49","50","51","52","53","54","55","56","8","11","14","15","16","19","22","23","24","26","27","31","33","34","37","40","42","44","2","3","5","7","9","10","20","21","29","32","36","38","39","41","43","45","46","48","57","59","60","61","62","63","65","66","67","68","69","70","71","72","87825","87826","87827","87828","87829","87830","87831","87832",70]
DEFAULT 2023-04-19T20:16:34.644474369Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] execution time: 1020
DEFAULT 2023-04-19T20:16:34.648557704Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] query is slow: UPDATE "n8n2"."workflow_statistics" SET "count" = count + 1, "latestEvent" = $1 WHERE ("workflowId" = $2 AND "name" = $3) -- PARAMETERS: ["2023-04-19T20:16:33.566Z",11,"production_success"]
DEFAULT 2023-04-19T20:16:34.678505532Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] execution time: 1045
DEFAULT 2023-04-19T20:16:34.858951238Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] query is slow: SELECT "SharedWorkflow"."workflowId" AS "SharedWorkflow_workflowId", "SharedWorkflow"."userId" AS "SharedWorkflow_userId", "SharedWorkflow__SharedWorkflow_workflow"."createdAt" AS "SharedWorkflow__SharedWorkflow_workflow_createdAt", "SharedWorkflow__SharedWorkflow_workflow"."updatedAt" AS "SharedWorkflow__SharedWorkflow_workflow_updatedAt", "SharedWorkflow__SharedWorkflow_workflow"."id" AS "SharedWorkflow__SharedWorkflow_workflow_id", "SharedWorkflow__SharedWorkflow_workflow"."name" AS "SharedWorkflow__SharedWorkflow_workflow_name", "SharedWorkflow__SharedWorkflow_workflow"."active" AS "SharedWorkflow__SharedWorkflow_workflow_active", "SharedWorkflow__SharedWorkflow_workflow"."nodes" AS "SharedWorkflow__SharedWorkflow_workflow_nodes", "SharedWorkflow__SharedWorkflow_workflow"."connections" AS "SharedWorkflow__SharedWorkflow_workflow_connections", "SharedWorkflow__SharedWorkflow_workflow"."settings" AS "SharedWorkflow__SharedWorkflow_workflow_settings", "SharedWorkflow__SharedWorkflow_workflow"."staticData" AS "SharedWorkflow__SharedWorkflow_workflow_staticData", "SharedWorkflow__SharedWorkflow_workflow"."pinData" AS "SharedWorkflow__SharedWorkflow_workflow_pinData", "SharedWorkflow__SharedWorkflow_workflow"."versionId" AS "SharedWorkflow__SharedWorkflow_workflow_versionId", "SharedWorkflow__SharedWorkflow_workflow"."triggerCount" AS "SharedWorkflow__SharedWorkflow_workflow_triggerCount", "SharedWorkflow__SharedWorkflow_role"."createdAt" AS "SharedWorkflow__SharedWorkflow_role_createdAt", "SharedWorkflow__SharedWorkflow_role"."updatedAt" AS "SharedWorkflow__SharedWorkflow_role_updatedAt", "SharedWorkflow__SharedWorkflow_role"."id" AS "SharedWorkflow__SharedWorkflow_role_id", "SharedWorkflow__SharedWorkflow_role"."name" AS "SharedWorkflow__SharedWorkflow_role_name", "SharedWorkflow__SharedWorkflow_role"."scope" AS "SharedWorkflow__SharedWorkflow_role_scope" FROM "n8n2"."shared_workflow" "SharedWorkflow" LEFT JOIN "n8n2"."workflow_entity" "SharedWorkflow__SharedWorkflow_workflow" ON "SharedWorkflow__SharedWorkflow_workflow"."id"="SharedWorkflow"."workflowId" LEFT JOIN "n8n2"."role" "SharedWorkflow__SharedWorkflow_role" ON "SharedWorkflow__SharedWorkflow_role"."id"="SharedWorkflow"."roleId"
DEFAULT 2023-04-19T20:16:34.859498068Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] execution time: 1180
DEFAULT 2023-04-19T20:16:35.012477455Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] query is slow: SELECT "SharedWorkflow"."workflowId" AS "SharedWorkflow_workflowId", "SharedWorkflow"."userId" AS "SharedWorkflow_userId", "SharedWorkflow__SharedWorkflow_workflow"."createdAt" AS "SharedWorkflow__SharedWorkflow_workflow_createdAt", "SharedWorkflow__SharedWorkflow_workflow"."updatedAt" AS "SharedWorkflow__SharedWorkflow_workflow_updatedAt", "SharedWorkflow__SharedWorkflow_workflow"."id" AS "SharedWorkflow__SharedWorkflow_workflow_id", "SharedWorkflow__SharedWorkflow_workflow"."name" AS "SharedWorkflow__SharedWorkflow_workflow_name", "SharedWorkflow__SharedWorkflow_workflow"."active" AS "SharedWorkflow__SharedWorkflow_workflow_active", "SharedWorkflow__SharedWorkflow_workflow"."nodes" AS "SharedWorkflow__SharedWorkflow_workflow_nodes", "SharedWorkflow__SharedWorkflow_workflow"."connections" AS "SharedWorkflow__SharedWorkflow_workflow_connections", "SharedWorkflow__SharedWorkflow_workflow"."settings" AS "SharedWorkflow__SharedWorkflow_workflow_settings", "SharedWorkflow__SharedWorkflow_workflow"."staticData" AS "SharedWorkflow__SharedWorkflow_workflow_staticData", "SharedWorkflow__SharedWorkflow_workflow"."pinData" AS "SharedWorkflow__SharedWorkflow_workflow_pinData", "SharedWorkflow__SharedWorkflow_workflow"."versionId" AS "SharedWorkflow__SharedWorkflow_workflow_versionId", "SharedWorkflow__SharedWorkflow_workflow"."triggerCount" AS "SharedWorkflow__SharedWorkflow_workflow_triggerCount", "SharedWorkflow__SharedWorkflow_role"."createdAt" AS "SharedWorkflow__SharedWorkflow_role_createdAt", "SharedWorkflow__SharedWorkflow_role"."updatedAt" AS "SharedWorkflow__SharedWorkflow_role_updatedAt", "SharedWorkflow__SharedWorkflow_role"."id" AS "SharedWorkflow__SharedWorkflow_role_id", "SharedWorkflow__SharedWorkflow_role"."name" AS "SharedWorkflow__SharedWorkflow_role_name", "SharedWorkflow__SharedWorkflow_role"."scope" AS "SharedWorkflow__SharedWorkflow_role_scope" FROM "n8n2"."shared_workflow" "SharedWorkflow" LEFT JOIN "n8n2"."workflow_entity" "SharedWorkflow__SharedWorkflow_workflow" ON "SharedWorkflow__SharedWorkflow_workflow"."id"="SharedWorkflow"."workflowId" LEFT JOIN "n8n2"."role" "SharedWorkflow__SharedWorkflow_role" ON "SharedWorkflow__SharedWorkflow_role"."id"="SharedWorkflow"."roleId"
DEFAULT 2023-04-19T20:16:35.030470419Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] execution time: 1327
DEFAULT 2023-04-19T20:16:35.483835412Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] allWorkOrders { json: { records: [] }, pairedItem: { item: 0, input: undefined } }
DEFAULT 2023-04-19T20:16:35.494819237Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] []
DEFAULT 2023-04-19T20:16:44.983636497Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] query is slow: SELECT "User"."createdAt" AS "User_createdAt", "User"."updatedAt" AS "User_updatedAt", "User"."id" AS "User_id", "User"."email" AS "User_email", "User"."firstName" AS "User_firstName", "User"."lastName" AS "User_lastName", "User"."password" AS "User_password", "User"."resetPasswordToken" AS "User_resetPasswordToken", "User"."resetPasswordTokenExpiration" AS "User_resetPasswordTokenExpiration", "User"."personalizationAnswers" AS "User_personalizationAnswers", "User"."settings" AS "User_settings", "User"."globalRoleId" AS "User_globalRoleId", "User"."disabled" AS "User_disabled", "User"."apiKey" AS "User_apiKey", "User__User_globalRole"."createdAt" AS "User__User_globalRole_createdAt", "User__User_globalRole"."updatedAt" AS "User__User_globalRole_updatedAt", "User__User_globalRole"."id" AS "User__User_globalRole_id", "User__User_globalRole"."name" AS "User__User_globalRole_name", "User__User_globalRole"."scope" AS "User__User_globalRole_scope" FROM "n8n2"."user" "User" LEFT JOIN "n8n2"."role" "User__User_globalRole" ON "User__User_globalRole"."id"="User"."globalRoleId" WHERE ( ("User"."id" = $1) ) AND ( "User"."id" IN ($2) ) -- PARAMETERS: ["9c7abeda-4230-4ce3-ba51-c903a32f2fce","9c7abeda-4230-4ce3-ba51-c903a32f2fce"]
DEFAULT 2023-04-19T20:16:44.994493594Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] execution time: 1521
DEFAULT 2023-04-19T20:16:49.277532139Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] query is slow: SELECT "ExecutionEntity"."id" AS "ExecutionEntity_id", "ExecutionEntity"."data" AS "ExecutionEntity_data", "ExecutionEntity"."finished" AS "ExecutionEntity_finished", "ExecutionEntity"."mode" AS "ExecutionEntity_mode", "ExecutionEntity"."retryOf" AS "ExecutionEntity_retryOf", "ExecutionEntity"."retrySuccessId" AS "ExecutionEntity_retrySuccessId", "ExecutionEntity"."status" AS "ExecutionEntity_status", "ExecutionEntity"."startedAt" AS "ExecutionEntity_startedAt", "ExecutionEntity"."stoppedAt" AS "ExecutionEntity_stoppedAt", "ExecutionEntity"."workflowData" AS "ExecutionEntity_workflowData", "ExecutionEntity"."workflowId" AS "ExecutionEntity_workflowId", "ExecutionEntity"."waitTill" AS "ExecutionEntity_waitTill" FROM "n8n2"."execution_entity" "ExecutionEntity" WHERE ("ExecutionEntity"."id" = $1) LIMIT 1 -- PARAMETERS: [55792]
DEFAULT 2023-04-19T20:16:49.282455032Z [resource.labels.instanceId: docker-compose-n8n-vm-vm] execution time: 1413

There is a series of slow queries and my theory is that these build up memory usage, do you know why could this be the case?

BramKn · April 20, 2023, 6:14am

Hi @nahueld

Have a look at this: Workflow on worker taking a lot more memory than on main - #12 by netroy

system · July 19, 2023, 6:14am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.