Help looping within pagination

[nucjag]

Hello! I need help with looping

Input: paginated data, each page has some JSON

Algorithm:

1. load next page (if has_more)
2. split
3. loop over all items on the page (Loop2, blue sticker)
4. increment offset
5. get next page (step 1)

I have setup flow, but issue is that:
it loops over Loop2 only once (see console logs)

Question: how to fix and why Loop2 runs only once

n8n 1.85.4 community edition

[barn4k]

hello @nucjag

HTTP Requests node has build-in pagination option, which you can use to receive the whole data

The issue is that the Loop node depends on the $runIndex property of the workflow, which is changed after the last Code node returns payload to the IF node.

The first outer iteration should go as intended, but the second one will break the Loop node, forcing it going directly to the done branch

[nucjag]

@barn4k

pagination option

I have tried pagination, but with it enabled i have a result that ALL data from API comes in (way too much), than it is being splitted and processed…

But because of data size it does not work for me. So, i am trying to ‘paginate’ manually, processing each page after it loads.

@barn4k

$runIndex

is there a way to “reset” this $runIndex ? adding “set” node befor loop did not help…

Algorithm is pretty simpe:

1. if we have more data items, step2, else finish
2. load next 20 data items
3. split loaded item
4. process each item
5. continue to step 1

[barn4k]

You can split the workflow into two parts.

Main WF:

Sub WF

Also, your sub WF will remove the execution data once the execution is finished, so it’s better to use the parent-sub workflows instead of the loop node.

[nucjag]

thank you a lot!