If I were to propose the way to handle this, i’d say - load the file into a proper database before you do any other processing, filtering, sorting, ordering, grouping etc. Having data loaded in a database will make the rest of the process just so much smoother.
But my usecase is pretty much “read the CSV and forward it to an API after creating md5 hashes for the IDs”, so there is no random access / filtering needed which would justify loading it to a database first.
I think the way I am using, with a combination of sed and head to read only chunks of the CSVs to memory should be efficient enough.
Even if I read the data in chunks from a different source (database), n8n still would keep them in memory.
The interesting part is, that the full file I read and n8n crashes only in the end. So I believe this is some kind of bug?
The use case is pretty simple and we could of course do this outside of n8n. But we need to integrate it into other file watchers for other imports, so this would not be very elegant (and we would have problems to justify the business plan payments if we cannot use n8n for such a seemingly simple usecase)
It sounds like you might be hitting memory limits with such a large file. I’d recommend loading the data into a database first, which will make processing way smoother. You could also try splitting the file into smaller chunks to prevent overloading memory. If you’re using workers, make sure they’re efficiently handling manageable portions of the data.
That’s exactly what I am doing, I split it in small chunks via “see" and load those chunks only. Also according to my logs the full file is processed and then the workflow just crashes right before the end.
I already switched of the save options in the workflow settings so I am wondering how this can be investigated further
The issue you’re dealing with is how n8n handles data. It doesn’t have the ability to read a file line by line with a pointer like a developer normally would have the capability of having.
Recommendations of putting it in a database first are smart, yet are overseeing the issue that you can’t load the file in n8n because of memory limitations.
If you’re limited to only n8n, my recommendation would be to set up a code node (or potentially a separate workflow), that processes the external file using a pointer system (here’s reference to do it in Node JS), and this way you send it to a database (or data table).
From there you can query your data from within n8n.
If you have more freedom from a coding perspective, I would set this all up in a Python install. Just host it on something like Sevalla, and you’re golden.
I think there is some misunderstanding here - maybe because I did not share my workflow. I am posting below now.
According to what I analyzed so far, I assume it’s a problem with finishing the workflow, not a memory leak on the way. I am already reading in chunks.
I am pretty sure that this technique doesn’t use much memory in the n8n node process - but only for the actual chunk of data.
I tried with reading 10000 and 1000 lines chunks and a single chunk seems to fit properly in n8n’s memory. I read such still quite big chunks because I also want to pass data to the Shopware 6 sync API in chunks and not line by line. The lines are pretty short (one line ~ 64 Bytes, but 2,920,269 lines in total, full size 189MB, but a 1000 line chunk would only be 65KB)
As you can see in the workflow I also added a progress logging which prints out the current progress to the docker’s console (when executed via the webhook for testing, instead of a manual test run)
There I can see that the batching runs well until the full file is read (until line 2921001) as printed in the original log (I renamed the Node from earlier “Code1” to “Log current status”)
So I think reading of the file is actually working. But then after everything was read, I get the above error message and I don’t know why.
I think it has to to something with the finalization of the workflow - because it it’s a memory problem during the reading process, I would be a big coincidence that it just happens after everything was read?
The first thing I would highly recommend is to update the nodes to the latest version.
Code node (first, outdated)
Split in Batches (outdated)
Move to file (deprecated)
Spreadsheet file (deprecated)
Set node (last in the wf, outdated)
I’d recommend, you create a new canvas and re-build the workflow from scratch using up to date versions of nodes and nodes which replaced some of the other nodes from over 2 years ago.
Oh wow, I did not know that. Thanks a lot. That was copied from the forum and I rebuilt it now.
It looked quite promissing at first, but then, after the last chunk was read, it too a minute or so and the error appeared again.
I guess I would need to create a subworkflow to read the data as n8n keeps all the previous nodes data in memory, because they still can be accessed … I kind of overlooked that in the initial posting where I got the approach from.
