Reading CSV in parts - Memory error when loading large file

Questions
1

We’re having an issue trying to read a large csv file from google sheets, it throws the error below:

n8n may have run out of memory while running this execution.

The error doesn’t show up when we use a smaller file, we understand there’s limitation in the resources to be granted to each instance or workflow and we’re trying to work around that.

Particularly we need to know how to read only a section of the file so we can process that, and then go on to the next section instead of loading the whole file into memory in one go. We’re using the loop over N amount of records to process each batch but that only breaks the file in parts after the file is loaded into memory.

How would we go about achieving this sort of fragmented reading of a CSV file? Ideally we would read maybe a thousand or a few thousand records each time instead of the whole thing (several 100k records).

(We’ve seen other similar posts in the forum but none address how to read a CSV file in this manner without loading the entire thing into memory first)

Information on your n8n setup

  • n8n version: 1.106.3
  • Database (default: SQLite): default
  • n8n EXECUTIONS_PROCESS setting (default: own, main): default
  • Running n8n via (Docker, npm, n8n cloud, desktop app): n8n cloud
  • Operating system:
2

Hey @Matias_Odriozola hope all is good. Welcome to the community.

so you want to read without reading?.. Could you tell us more about what is the final goal? Which task you are trying to solve?

3

I’m sorry that sentence wasn’t clear, I don’t want to load the entire file into memory at the same time, but only parts of it at a time. I’ll make an edit on the post to correct this.

Some form of pagination or segmentation is what we’re looking for, to take say, the first 10k records instead of the whole thing, and then go on to the second batch of 10k records, etc.

The task is to read a CSV file and to send requests to an API so the data in the CSV is processed by this API, we’ve gotten the workflow to do this but it hits memory limits if the file we’re using as a source is large.

Thank you for the welcome! The tool seems great so far.

1 Like
4

in self-hosted you could try to do something like

^ this loops over a 100 line csvs and sends it to email 10 lines at a time. Unfortunately “Execute command” node isn’t available on n8n Cloud.

5

Thank you for the response.

I see, If I’m reading this correctly does the execute command node also require the file to be locally stored as well? So this wouldn’t work for something like a file stored in google drive or some FTP correct?

6

Yes, this is for locally stored file. I guess you have two options

  • first load your data into a database and then query the database
  • spin up a custom small service for iterating csv, a api wrapper around sed or implement the reader yourself, which would download a file for a given URL and return content with parameters like offset and limit.
7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.