We have 10 workers and 6 webhook nodes running in production. Deployed version is v1.7.1.System is handling upto 100 rps/6000 rpm(webhook) and multiple scheduler workflows with acceptable performance.When we monitor the system metrics, we could find that workers are building up memory gradually and we have to restart N8N set docker every 24 hr. Is it an issue ? We are being just precautionary and we havent faced any crashes as of now. Server config 64 gb ram and 24 core cpu.
Someone like @krynble may be a better shout to ask - but while I’m here, would you happen to have any logs that you could share, as well? 
Hey @Navaneeth_P_K thanks for reporting.
We are not aware of any memory leakage in the app - the node process will frequently increase the heap size to prevent allocating more memory over and over, as long as there’s free memory. This is usually handled by node itself.
Naturally if you have workflows that process a lot of data, the memory usage will grow since n8n keeps all data in memory while a workflow is running, but as soon as an execution ends, n8n saves the data to DB and frees it.
We have a deployment with 4 workers internally and memory does increase up to a point but then the operating system itself handles it. The only crashes we had were with workflows that did handle tons of data, such as massive spreadsheets and crossing data from multiple sources, but this is expected to a certain extent.
worker 3 023-10-31T07:45:34.248Z [Rudder] debug: no existing flush timer, creating new one
2023-10-31T07:45:54.249Z [Rudder] debug: in flush
2023-10-31T07:45:54.249Z [Rudder] debug: cancelling existing flushTimer…
2023-10-31T07:45:54.249Z [Rudder] debug: batch size is 1
2023-10-31T07:46:15.282Z [Rudder] debug: in flush
2023-10-31T07:46:15.282Z [Rudder] debug: cancelling existing timer…
2023-10-31T07:46:15.282Z [Rudder] debug: queue is empty, nothing to flush
2023-10-31T13:45:34.252Z [Rudder] debug: no existing flush timer, creating new one
2023-10-31T13:45:54.253Z [Rudder] debug: in flush
2023-10-31T13:45:54.253Z [Rudder] debug: cancelling existing flushTimer…
2023-10-31T13:45:54.253Z [Rudder] debug: batch size is 1
2023-10-31T13:46:14.632Z [Rudder] debug: in flush
2023-10-31T13:46:14.633Z [Rudder] debug: cancelling existing timer…
2023-10-31T13:46:14.633Z [Rudder] debug: queue is empty, nothing to flush
<— Last few GCs —>
[7:0x7fcc05359010] 220014800 ms: Scavenge 1485.7 (1521.6) → 1482.9 (1522.9) MB, 5.1 / 0.0 ms (average mu = 0.127, current mu = 0.118) allocation failure;
[7:0x7fcc05359010] 220014836 ms: Scavenge 1486.2 (1522.9) → 1483.8 (1523.6) MB, 4.7 / 0.0 ms (average mu = 0.127, current mu = 0.118) task;
[7:0x7fcc05359010] 220016184 ms: Scavenge 1487.0 (1523.6) → 1484.4 (1532.1) MB, 5.1 / 0.0 ms (average mu = 0.127, current mu = 0.118) task;
<— JS stacktrace —>
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2023-10-31T20:52:44.549Z [Rudder] debug: in flush
2023-10-31T20:52:44.550Z [Rudder] debug: batch size is 1
2023-10-31T20:53:04.976Z [Rudder] debug: in flush
2023-10-31T20:53:04.977Z [Rudder] debug: cancelling existing timer…
2023-10-31T20:53:04.977Z [Rudder] debug: queue is empty, nothing to flush
worker 2 2023-10-31T13:45:53.299Z [Rudder] debug: batch size is 1
2023-10-31T13:46:13.669Z [Rudder] debug: in flush
2023-10-31T13:46:13.669Z [Rudder] debug: cancelling existing timer…
2023-10-31T13:46:13.669Z [Rudder] debug: queue is empty, nothing to flush
2023-10-31T19:45:33.293Z [Rudder] debug: no existing flush timer, creating new one
2023-10-31T19:45:53.293Z [Rudder] debug: in flush
2023-10-31T19:45:53.293Z [Rudder] debug: cancelling existing flushTimer…
2023-10-31T19:45:53.293Z [Rudder] debug: batch size is 1
2023-10-31T19:46:14.584Z [Rudder] debug: in flush
2023-10-31T19:46:14.584Z [Rudder] debug: cancelling existing timer…
2023-10-31T19:46:14.584Z [Rudder] debug: queue is empty, nothing to flush
<— Last few GCs —>
[7:0x7f99ccba3010] 243906008 ms: Scavenge 1469.4 (1530.1) → 1464.2 (1531.4) MB, 7.7 / 0.0 ms (average mu = 0.920, current mu = 0.918) task;
[7:0x7f99ccba3010] 243910797 ms: Scavenge 1470.9 (1531.4) → 1465.6 (1532.1) MB, 5.9 / 0.0 ms (average mu = 0.920, current mu = 0.918) task;
[7:0x7f99ccba3010] 243911302 ms: Scavenge 1472.7 (1532.1) → 1467.1 (1549.1) MB, 9.6 / 0.0 ms (average mu = 0.920, current mu = 0.918) task;
<— JS stacktrace —>
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2023-10-31T07:45:53.642Z [Rudder] debug: batch size is 1
2023-10-31T07:46:14.146Z [Rudder] debug: in flush
2023-10-31T07:46:14.146Z [Rudder] debug: cancelling existing timer…
2023-10-31T07:46:14.146Z [Rudder] debug: queue is empty, nothing to flush
2023-10-31T13:45:33.646Z [Rudder] debug: no existing flush timer, creating new one
2023-10-31T13:45:53.648Z [Rudder] debug: in flush
2023-10-31T13:45:53.648Z [Rudder] debug: cancelling existing flushTimer…
2023-10-31T13:45:53.649Z [Rudder] debug: batch size is 1
2023-10-31T13:46:14.748Z [Rudder] debug: in flush
2023-10-31T13:46:14.748Z [Rudder] debug: cancelling existing timer…
2023-10-31T13:46:14.748Z [Rudder] debug: queue is empty, nothing to flush
<— Last few GCs —>
[7:0x7ff7c0dbb010] 220559192 ms: Scavenge 1487.9 (1524.6) → 1485.2 (1525.6) MB, 5.2 / 0.0 ms (average mu = 0.080, current mu = 0.007) allocation failure;
[7:0x7ff7c0dbb010] 220559222 ms: Scavenge 1488.7 (1525.6) → 1485.8 (1525.9) MB, 4.8 / 0.0 ms (average mu = 0.080, current mu = 0.007) allocation failure;
[7:0x7ff7c0dbb010] 220559257 ms: Scavenge 1489.6 (1525.9) → 1486.8 (1534.1) MB, 8.6 / 0.0 ms (average mu = 0.080, current mu = 0.007) allocation failure;
<— JS stacktrace —>
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2023-10-31T21:01:47.013Z [Rudder] debug: in flush
2023-10-31T21:01:47.014Z [Rudder] debug: batch size is 1
2023-10-31T21:02:08.098Z [Rudder] debug: in flush
2023-10-31T21:02:08.098Z [Rudder] debug: cancelling existing timer…
2023-10-31T21:02:08.098Z [Rudder] debug: queue is empty, nothing to flush
We were getting this error FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Hey @Navaneeth_P_K,
Can you share what your workflows are doing? This could just be down to workflow design and they are handling lots of files or objects.
My flows a re of multiple types
- Triggered by a schedule node, Makes a long pool HTTP request typically of around 1 -2 hr (it calls api exposed by rsync)
- Another type of workflow have some code nodes with http nodes.I use http node & have a seperate service just for querying in db since n8n do not support pooling
My request per second in case of webhook is around 60-100 rps.
Is it a lot of data being handled? Having the long http connection is going to keep things busy for a while and depending on how many of those you have it will soon add up.
We are currently working on some new features that will make it easier to monitor a worker though and see what they are doing which could be useful assuming you are on an enterprise license.
I would maybe start with an update as we have made some improvements recently, But generally you will see the memory being used and it will be released but if you are using a lot of code nodes that will cause your memory usage to be higher so it may be worth seeing if you do need them all as each run will create a new sandbox this will be freed up once the workflow is finished but if this happens in a long http connection workflow it will take longer than it needs to for the resources to be released.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.