Extreme memory increase due to infinite manual process

Hey @Joey_Geraeds,

I am fairly sure that line just means the active workflow is enabled and ready to go not that it is running, That log though does bring me back to an earlier question. When you say there is an OOM error where are you seeing that? I would typically expect to see something in the docker log about it.

True, I had thought to see them there as well.
The reason I mentioned the OOM is due to the server logs.
the syslog file contains:

Jan  6 08:07:54 <servername> kernel: [75893.354525] node invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
Jan  6 08:07:54 <servername> kernel: [75893.354806] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=e29e8394b08162c2485873219059233573bd5d5a36c09d492ba89225d6f36a67,mems_allowed=0,global_oom,task_memcg=/docker/e29e8394b08162c2485873219059233573bd5d5a36c09d492ba89225d6f36a67,task=node,pid=71491,uid=1000
Jan  6 08:07:54 <servername> kernel: [75893.354870] Out of memory: Killed process 71491 (node) total-vm:7993180kB, anon-rss:7651052kB, file-rss:4kB, shmem-rss:0kB, UID:1000 pgtables:19024kB oom_score_adj:0
Jan  6 08:13:49 <servername> kernel: [76247.857600] node invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
Jan  6 08:13:49 <servername> kernel: [76247.857834] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=e29e8394b08162c2485873219059233573bd5d5a36c09d492ba89225d6f36a67,mems_allowed=0,global_oom,task_memcg=/docker/e29e8394b08162c2485873219059233573bd5d5a36c09d492ba89225d6f36a67,task=node,pid=82587,uid=1000
Jan  6 08:13:49 <servername> kernel: [76247.857891] Out of memory: Killed process 82587 (node) total-vm:7991804kB, anon-rss:7658632kB, file-rss:4kB, shmem-rss:0kB, UID:1000 pgtables:19076kB oom_score_adj:0
Jan  6 08:16:12 <servername> kernel: [76390.937051] gmain invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jan  6 08:16:12 <servername> kernel: [76390.937371] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/e29e8394b08162c2485873219059233573bd5d5a36c09d492ba89225d6f36a67,task=node,pid=83049,uid=1000
Jan  6 08:16:12 <servername> kernel: [76390.937434] Out of memory: Killed process 83049 (node) total-vm:7997392kB, anon-rss:7649480kB, file-rss:4kB, shmem-rss:0kB, UID:1000 pgtables:19028kB oom_score_adj:0
Jan  6 10:28:33 <servername> kernel: [84332.204414] node invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
Jan  6 10:28:33 <servername> kernel: [84332.204642] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=63bdc117dba2d72ed82a142092e15c8c19101d732da740f69ba644de5bcf3664,mems_allowed=0,global_oom,task_memcg=/docker/63bdc117dba2d72ed82a142092e15c8c19101d732da740f69ba644de5bcf3664,task=node,pid=85094,uid=1000
Jan  6 10:28:33 <servername> kernel: [84332.204699] Out of memory: Killed process 85094 (node) total-vm:8106712kB, anon-rss:7646492kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:19488kB oom_score_adj:0

Are you using the custom image that @netroy linked to yet?

Yes the image: n8nio/n8n:debug-memory-usage

however even when i run in locally, the debug logs for memory start appearing later. around the second “code” node and on the server it is already stuck at the first “code” node as that is the one i tried with testing.

I can try to do a full run and see if that gives some more information/clarification on the memory

While running it completely it passes the first nodes where it is stuck with the “single” node run

this time the log is a little bit different… it now only adds “Killed”

n8n ready on 0.0.0.0, port 5678
Version: 0.209.4
 ================================
   Start Active Workflows:
 ================================
   - Multivers OAuth process (ID: 15)
     => Started

Editor is now accessible via:
https://<url>:5678/
Killed
n8n ready on 0.0.0.0, port 5678
Version: 0.209.4
 ================================
   Start Active Workflows:
 ================================
   - Multivers OAuth process (ID: 15)
     => Started

Editor is now accessible via:
https://<url>:5678/
Killed

I feel like something else is missing here, We don’t have the node outputs in the docker log, n8n isn’t logging the out of memory message we normally see and the only indication is from the host :thinking:

I wonder if the log to file option is tripping up something, Is this machine only used for those 2 docker images and is it a virtual host or a physical?

Trying the full run is not a bad idea just so we can see what is going on, what sort of size is your database file as well?

… looks like another update :smiley:

It looks like n8n isn’t crashing but it is being killed by an outside process or that is how I read that.

Yes very strange, locally it shows me the memory information, yet on the server it does not output.

Right now we are using it only for these 2 docker images:
I am not entirely sure but I think this is a virtual host
8 GB Memory / 160 GB Disk
35GB database.sqlite (mostly due to the logs)

The full run was already done with my previous post

I’am getting the same feeling about the n8n getting shut down by the server itself.
Would it be an option to limit the n8n memory within he docker image?

Hey @Joey_Geraeds,

I believe docker has an option to limit the memory but I am wondering if maybe this issue is all coming from the size of your database.

As you only have a couple of workflows as a test could you export the workflows and credentials then stop the n8n container and rename the sqlite file then start it up again and import the workflows and credentials and see if you hit the same issue.

After adding the docker limits, the image started giving the output which we expected (memory consumption line… rather than lines) but it got killed before showing more.

I’ve followed up with your advise to export workflows + credentials, rename db, import workflow + credentials. This actually seemed to solve our issue, the memory consumption went down from the full 8G to hardly ever going above the 1G. Not sure what’s the cause but it seems to be in the db

MUCH THANKS to you guys @Jon @netroy for helping on this issue!

Can you please check in the database how many rows are there in the execution_entity table ?
I suspect the out of memory issue might be related to that.

Another way to confirm this suspicion would be to set EXECUTIONS_DATA_PRUNE=false and check if that fixes the out-of-memory crashes.

Already tried it with EXECUTIONS_DATA_PRUNE=False but that made no difference.
The db had 8323 executions, but it has reached 18k before without issues

With 8k executions, it most certainly was this issue that got fixed earlier today. You probably did not have an issue with 18k executions before because the issue only affects versions released after December 1st, and you were likely on an unaffected version before.

I just checked out the videos you posted, and the execution always getting stuck at the last node makes me certain that this was the issue. And the since you did not see any logs on the debug image, it’s likely that there was no memory issue in the workflow itself, but in the post-workflow-execution code.