Hi everyone, I am currently running a self-hosted instance of n8n via Docker and using it to build a highly automated data pipeline. The workflow fetches a dense batch of dynamic system execution telemetry strings from an external sandbox environment, passes the raw data into a Code node for regex filtering, and then attempts to loop through the items to update an external database. The layout works fine with small test payloads, but I am hitting a critical performance wall when processing production-level volume.
The main issue is that whenever the workflow encounters a massive, multi-megabyte nested array structure, the n8n container throws an out-of-memory error and completely crashes. This sets off a cascade of nested issues across the setup: the active execution execution tracking breaks, the queue gets backlogged with retries, and the local SQLite database locks up entirely until I manually force-restart the container via SSH. I am trying to determine if the internal JavaScript Code node is failing to release memory mid-execution because it is processing the entire payload in a single block, or if n8n’s execution data history logging settings are simply inflating the container’s RAM usage beyond its allocated limits.
I have been experimenting with a custom memory-handling template that I obtained as a free download to see if explicitly splitting the incoming JSON array into smaller chunk packets before it hits the parsing engine prevents the system from choking. Unfortunately, even when breaking the data into micro-batches, the continuous background loops still cause a slow memory leak that inevitably kills the workflow worker thread after a few hours of steady automation. I am trying to figure out if it is better practice to offload the heavy data-parsing mechanics completely to an external microservice via a Webhook node, or if there is an environment variable I can configure in my compose file to force aggressive garbage collection between active workflow loops.
If anyone else here has designed high-volume data loops or managed volatile script-generated arrays inside self-hosted nodes without triggering container timeouts or memory spikes, how do you optimize your data buffers? I would be incredibly grateful for any workflow structure advice or performance configurations to keep this pipeline stable.
Hi @Ivonne566 Welcome!
The Code node holding every processed chunk in the same V8 heap across the whole run is what’s leaking, not the execution history logging. Manual chunking inside one Code node execution doesn’t release that memory between iterations, only ending the execution does, and there’s no environment variable that forces GC mid-loop. Use a Loop Over Items node to batch the array, then hand each batch to a sub-workflow through Execute Workflow. Each batch becomes its own execution that ends and frees memory before the next one starts, and the parent workflow only holds the small returned result.
For a stopgap while you redesign, raise the V8 heap instead:
NODE_OPTIONS=--max-old-space-size=<SIZE_IN_MB>
That only raises the ceiling, the same-execution leak still returns once the payload grows past it.
If the SQLite lock keeps happening once the crash stops, move to PostgreSQL, SQLite isn’t built for the concurrent writes a retry backlog produces.
Picking up the second half of your question (whether the “execution data history logging settings are simply inflating the container’s RAM usage”), since it hasn’t been addressed yet:
The execution history mostly pressures your database, not the worker heap. While a run is in progress, the items flowing through the workflow are held in memory regardless of any save settings — that’s the V8-heap ceiling Anshul covered. But when the run finishes, n8n writes the whole execution payload to the DB. With multi-MB nested arrays on SQLite, that’s what bloats the database file and makes the lock contention and queue backlog you described worse.
A few settings worth checking:
EXECUTIONS_DATA_SAVE_ON_SUCCESS=none (and keep EXECUTIONS_DATA_SAVE_ON_ERROR=all so you can still debug failures) — stops writing the huge payload for every successful run.
Pruning is enabled by default, but check EXECUTIONS_DATA_MAX_AGE (default 336 hours = 14 days). With payloads your size, shortening it keeps SQLite far smaller. EXECUTIONS_DATA_PRUNE_MAX_COUNT (default 10,000) is the companion cap.
Leave EXECUTIONS_DATA_SAVE_ON_PROGRESS off unless you really need resume-on-crash — saving progress mid-run multiplies DB writes.
None of this fixes the OOM itself — Anshul’s Loop Over Items + sub-workflow batching is the real fix there. But it removes the compounding DB pressure, and if you migrate to Postgres as suggested, the same settings keep that database healthy too.