Aggregate loop

Questions
1

Hi community, I’m doing a http request that fetches 100 items at a time, and keeps looping with an IF-node until there’s no full 100 page results left. In this example it runs 4 times and retrieves a total of 312 items. I need to aggregate all those items of the entire loop into one single array, but that’s where I’m running into issues. The aggregation either only shows the first 100 items, or outputs 3 seperate arrays. I’ve tried working with a Merge-node and “wait for all data to arrive”, but that also doesn’t seem to be the solution.

So straight forward explanation of my workflow: wait for loop to finish → aggregate all items of the seperate http calls into one single array.

Thanks in advance!

PS. The screenshot shows an overview of the workflow. It looks like the aggregate node is inputting 312 items and outputting all of those into 1 item, but for some reason the output is only made up of the first 100 items (first http request).

image
image960×386 40.9 KB

2

Loop Over Items processes each item (e.g., each HTTP call).

After all iterations are complete, the Done output of the loop node emits all results.

Connect the Aggregate node to the Done output to combine all items into a single array.

So the IF you used there is for ?

3

Thanks for the feedback! The IF node is because the loop needs to continue until “next_page_exists = false”, it’s a dynamic loop that has to continue until there are no more 100 items being returned.

1 Like
4

See also the ‘batching’ option for the HTTP request node: