How to write large number of items to spreadsheet without loading all into memory at once

Questions
1

Hello,

I am retrieving a large amount of items from action network (using pagination), then I separate the pages into individual items, and write them into a spreadsheet (one row per item):

Unfortunately, I am running out of memory. If I understand correctly, this is because N8N tries to retrieve the entire dataset (all pages) at once.

I am sure there is a smarter way to do that, e.g. processing one page at a time, but I can’t figure out how to do it. Could someone give me a starting point?

Information on your n8n setup

  • n8n version: 1.6.1
  • Database (default: SQLite): default
  • n8n EXECUTIONS_PROCESS setting (default: own, main): default
  • Running n8n via (Docker, npm, n8n cloud, desktop app): cloud
  • Operating system: N/A
2

You could control the items you’ve already received by using an ID. You can generate or use some already existing ID. Save it along with the data.

Use filters in your HTTP request to make sure you get less items.

Save the items.

Then you check for the last retrieved item and retrieve the next batch.

You could use timestamps, id or row index to filter what you need.

3

Uh, this is annoying. I can’t use HTTP filters, because I truly want all the data.

I was hoping that since there is already an automation to handle pagination, this would be as simple as telling N8N to process pages one at a time instead of merging everything into one giant batch :-(.

I guess writing the last retrieved page into some datastore and resuming from there would work, but this feels very hacky. For example, how do I tell N8N to re-run the workflow until I’ve received all the data?

4

I am not sure if I completely understand your scenario but I had a case where I needed to retrieved a large amount of items from pimcor herein I kept the whole process in loop till I don’t received an empty return body from the http request node…

5

This sounds promising, actually. Would you be able to share an example? Sorry if it’s obvious what you mean, I’m very new to all things N8N.

7

You can use “Execute workflow” nodes to make the workflow call itself.

  • If you still have data to receive, call the workflow again
  • Else, don’t

Or use a loop, like the @Raj_Agrawal explained.
You can connect the loop to the beggining of the workflow.

image
image1029×615 23.4 KB

1 Like
8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.