Pagination Offset vs Cursor

Questions
,
1

So i have a URL, lets call it /linky. This only allows me to pull 100 results at a time. The URL allows both/either

Cursor-based Pagination

/linky?event-num-after=199

Offset-based Pagination

Example of retrieving item records 90 to 100
/linky?offset=90&limit=10

What’s the best way to get this to run in a loop, when the initial URL gives no idea, how many results there will be, and dump into one long list?

I think offset might be best for me, but need to make some form of counter for the loop and the first URL need to be plain, without any pagination etc it doesn’t work otherwise

2

I would do the offset based. I did an example in the post below. You have to loop and, when done, merge all executions.

3

will have a look, might have some follow up questions like if your initial page results doesn’t have anything to say there are more results, you just know that there are.

1 Like
4

well i did it a hacky way using cursor but got what I needed , lol

image
image1743×339 29.6 KB

but obviously there is a better way, don’t copy me people, :wink:

3 Likes
5

That might be one of the craziest things I have seen :joy: Nice work.

1 Like
6

my brain was hurting so just hacked it for now.

2 Likes
7

functionality first, elegance later :grinning_face_with_smiling_eyes:

2 Likes
8

Part of the joy of using n8n is the ability to quickly iterate over different worklfow design ideas :slightly_smiling_face:

Get it working then make a second worklfow to play with improving it when you get time.

3 Likes
9

That is exactly my problem right now, There is no cursor/next paging link info coming with my first request, but I know there is 2 more pages

10

Hey @JohnTheAnalyst,

Which API are you dealing with? Most of the ones I have seen offer some kind of indication that there is more data.

11

Weclapp API, I can’t find an indication.

12

If that is the case, you have to iterate until the response is empty.

13

i have many API that give no indication of more data, but there actually is there. Whats your best recommendation for iteration in this situation, one HTTP node for original request, and then a second one to set offset, and make some sort of looop, but need a counter and a way to stop it when no result come back

14

Hey @RedPacketSec,

It would be like @RicardoE105 mentioned, Keep making a call and looping until you get no data, Without having an API to play with that does that it is hard to put a proper example together but the one below would increase a limit and offset counter on each loop. The If node would need to have a condition in it based on the API response to know if there is no data so maybe if the body is empty or doesn’t contain a value then on the false it will merge all the data for you so you can work like normal from there.

15

yeah this is what i was working on, will have a play with your workflow. cheers

1 Like
16

Hey Jon, I am dealing with a similar problem and trying to understand how your workflow works. I can’t understand how the offset value in the “Set” node changes, because the HTTP node takes this value and not the “Increase offset” value after each iteration.

1 Like