Hello there,
I need to work on large dataset, I saw some questions about to add the feature in docker side
But nothing currently borned
Is there an update about an environment variable to pemit to launch a node with the desired stack-size?
Thanks for your help
It looks like your topic is missing some important information. Could you provide the following if applicable.
Thanks @gualter I already read this documentation
The only I could do if to reduce memory consumption by splitting data, but I am not able to that because lines are linked by an id
If I split ont the wrong line, the final result will be be wrong
You can increase the memory to what you need. the info is on the page @gualter shared.
Often there is ways to build your workflow to use batches and still combine the data properly to reduce RAM use though.
@BramKn thanks for your answer
The real problem is not the available RAM in the server, but the stack affected to nodejs, in particular in docker version of n8n
That’s why a new parameter should be fine
If it is possible to transform the workflow to consume less memory, I will say yes!
But I have a file with more than 300 000 lines with 5 columns
Just the loading of the file takes 2mn…
You can already expand the size of memory that is usable. See the docs, on the bottom of the page.
There are ways to split the file by not reading all lines at once. Depends on the file of course. And you can set n8n to use the filesystem for binary data.
@BramKn I am sorry to insist, but it is currently not possible to expand the stack-size in n8n
The parameter stack-size is only available in nodejs command line for security reason, it is no more available in NODE_OPTIONS parameter
I tryed to set max-old-space-size=SIZE but no effect,
Should be working
It is working fine here.
You seem to be doing the wrong thing. what you mention here isnt right. please refer back to the docs 
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.