Tbh I think that file size could be a bit too much to parse at once for most n8n instances. But perhaps we can come up with a workaround. Is the file you are downloading a CSV file or something else that you would need to convert first? And do you have an example file by any chance?
So this is a bit tricky, but I think a possible approach would be to store that huge file on the file system, then read it in chunks. This will only work if you’re not using n8n cloud though and you would need to adjust the paths used in this workflow to your own folder structure. But the basic idea could be to have a parent workflow like this to download a large file, write it to the file system and then prepare batches:
Then call a sub-workflow reading only a limited number of lines at once from that file, do something with it and then return only a very small result set to the parent rather than the full data:
Memory usage was quite low when testing this with a dummy file:
Now this is assuming your CSV file is properly formatted and the only problem is indeed its size. If not It’d be great if you could share an actual example of one of these files (you can of course redact anything confidential, the only thing that matter is that any issues you might have can be reproduced with it).
So this could be a problem on your local filesystem. You might need to swap the file paths used in the workflow to a path that exists on your side and in which you have full permissions. Also, to run the sub-workflow individually you might need to use data pinning first to supply some dummy data.
I’ve tried:
/Users/gabrielebracciali/n8ntemp{{ $binary.data.fileName }}
Here maybe I dont’have full permission, but I have changed into desktop and same permission error.
Hi @Gabriele_Bracciali, is your n8n instance perhaps running as a user other than gabrielebracciali, so it doesn’t have permissions to write into said user directory? What if you create a designated folder on your hard drive and simply allow everyone to write in there (just for testing purposes)?
Hm, I am afraid I don’t know why writing to the file system would fail in your case
I don’t have a Mac though, so perhaps this OS adds some additional isolation layers somewhere along the way. Maybe fellow Mac user @Jon knows more about this?
That’s alright, very large files can be very tricky in n8n. It’s why I thought handling these outside of n8n itself would make things easier, but I was clearly wrong
