Struggling with pagination and increasing offset

Questions
1

I am trying to get information from an API using offset pagination, but I don’t seem to be able to get the offset number to increase, I have read loads of the post on here but I just cant seem to get it to work, what am I missing here?

It seems to keep just doing the second offset over and over and over.

image
image1882×473 45 KB

SET NODE

image
image1434×722 21.2 KB

HTTP NODE

image
image373×1121 15.8 KB

HTTP OFFSET EXPRESSION
image

HTTP LIMIT EXPRESSION
image

IF NODE
Just says if result is empty then do not continue the loop

INCREASE OFFSET NODE

image
image1445×729 23.5 KB

OFFSET EXPRESSION
image

How do I get the offset number to increase with each loop?

2

Hi @RedPacketSec

The problem you are having is caused by the use of the set node that is outside this loop. So you are in fact getting the same data everytime. so the result is the same everytime.
If the HTTP request returns the offset as well after making the request, you can simply use that one to iterate.

1 Like
3

it does not provide the offset , it only provides the data back.

4

Thats too bad.
I usually use a function node then.
Check if the offset field exists, if it does not set the first offset.
if it does +100 in your case.
In the last set node you can set the offset you get from this function node.
something like:

image
image826×317 7.71 KB

5

you got a sample function code i can try?

6

Yes, somehere in my very nicely documented workflows.

image
image1349×1285 80.8 KB

Oh, I have a few with a nice name. So I found it actually:


const page = "0";
const pageSize = 5;

if(item.page == undefined){
  item.page = page
}
else{
  let page = +item.page;
  let nextPage =  page + pageSize;
  item.page = nextPage;
}


return item;
2 Likes
7

i get error line 4 item not defined

8

@RedPacketSec
not sure what this could be.
does the item exist?

9

this is one area in n8n, that has been a pain for me, pagination using the HTTP node.

i also tried putting the set node in the loop but ultimately it fails as it cant reference itself and ends up with a NaN. which is why I put it out the loop to start with

10

Works fine here

image
image536×571 20.2 KB

I made you the following workflow. It is not a working example at all, but should show you how it would work. This is from a working workflow but cut out all the extra stuffs.

Ps. the page was a string previously because there was some base64 encoding going on here for a specific API.

11

yeah but in your example you are getting something like next page, so you can use that, what i am doing has nothing like that its just results, and i just need it to count of offset upon each loop and increment it.

12

The nextpage is calculated within this function node.

The if checking if the loop needs to continue is indeed using a field returned by the api to check if there is a next page to do. You would need to replace that by a count of te items returned checking if that is the maximum page size.

If you need help with that I will check when I am behind my pc.

13

I do something similar with the NPMJS API I have to keep calling it until I don’t get any data back increasing the offset each time. I then check to see if a property that should exist is returning and if isn’t I assume all is good and I have the lot.

Maybe it will help.

3 Likes
14

This is what i needed, few tweaks but this functionitem i turned into a functionnode and does what i need thanks so much

2 Likes
15

Hi Guys i found another solution by using “$runIndex” that represent the index of iteration

this is the code
{{ $runIndex * start (offset)}}

Thank you for your informations

2 Likes
16

Hi @Idris_Dada

Welcome to the community!

You are responding to an old topic though. So yeah there is deffinitly better solutions out there like the one you mentioned.
You can also have a look at this blogpost: The Good, the Bad, and the Ugly of looping with n8n – n8n Blog

3 Likes