How is the best wayy to deal with a lot of data?

[Fernanda_Silva]

Hi, i have a googlesheet with more than 7000 entrys and i have a flow to insert that / update in my Mysql database after a Set and using a compare like my previous question, it worked with fewer data… In this moment my n8n page is down because it used all memory hahah when the page is back i need to find a way to insert that and use my flow! How? Can i use some ETL node? Pentaho maybe? how do i make this performatic?

[shrey-42]

What i usually do, in such cases is to create a ‘repeating loop’. In each iteration, fetch a manageable number of entries, e.g. 100, and then process them and then fetch the next 100 and so on.

[Fernanda_Silva]

How can i do that? with a function? thank you for your answer

[shrey-42]

Adjusting your workflow, it might work out this way:

[shrey-42]

So, it will end up looking like this:

WF1-

WF2-

[Fernanda_Silva]

Thank you very much!