How is the best wayy to deal with a lot of data?

Hi, i have a googlesheet with more than 7000 entrys and i have a flow to insert that / update in my Mysql database after a Set and using a compare like my previous question, it worked with fewer data… In this moment my n8n page is down because it used all memory hahah when the page is back i need to find a way to insert that and use my flow! How? Can i use some ETL node? Pentaho maybe? how do i make this performatic?

What i usually do, in such cases is to create a ‘repeating loop’. In each iteration, fetch a manageable number of entries, e.g. 100, and then process them and then fetch the next 100 and so on.

1 Like

How can i do that? with a function? thank you for your answer

Adjusting your workflow, it might work out this way:


So, it will end up looking like this:




Thank you very much! :partying_face:

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.