Loops and performance issue

[simon_Piquemal]

Hello, I currently have a workflow which aims to update the different elements of a database list. Unfortunately I have several problems. The first and a performance problem. The first time the workflow runs it goes well even if it’s a bit long because of the restrictions of the Notion API. But the second loop will often cause an error due to a non-response from the server. I assume that the train will saturate and accumulate the different elements. So I have 2 questions.
what node to use in addition to desaturate the ram? (I assume nodes to write to disk but I’m not sure)
For the loop system I tried to cut the objects into small elements 50 by 50 and connect the data of this loop to make the second loop but suddenly it does not work and “keeps the first loop in memory”. Which means that it does not execute this second loop on the second pass.

[MutedJam]

Hi @simon_Piquemal, I am sorry you are having trouble.

I’d avoid nesting multiple loops here based on your problem description. Instead, consider moving your inner loop logic into its own workflow and add the Execute Workflow trigger node to it.

Then, at the end of your workflow (on the “Done” output of your Split In Batches node) add a Set node returning only single very small item, something like this:

The small item and the Execute Once option will mean that this workflow will only return a very small dataset when it finishes.

Then in your main workflow use the Execute Workflow node to run the sub-workflow created in the previous step.

Now the sub-workflow would only eat up your RAM for the current batch, but once the sub-workflow finishes processing each batch the RAM would become available again for the next batch.

Hope this helps!

[simon_Piquemal]

Thank you very much, I used a webhook instead of this node but the result is identical. Well my worflow still takes a lot of time but that’s mainly because of the notion API limitations and the large amount of data to process. Thank you for helping me. @MutedJam

[MutedJam]

Ah perfect, using a webhook would of course have a very similar effect provided you only return a small result set. Thanks so much for confirming and for sharing your approach