Struggling with pagination and increasing offset

I am trying to get information from an API using offset pagination, but I don’t seem to be able to get the offset number to increase, I have read loads of the post on here but I just cant seem to get it to work, what am I missing here?

It seems to keep just doing the second offset over and over and over.

SET NODE

HTTP NODE

HTTP OFFSET EXPRESSION
image

HTTP LIMIT EXPRESSION
image

IF NODE
Just says if result is empty then do not continue the loop

INCREASE OFFSET NODE

OFFSET EXPRESSION
image

How do I get the offset number to increase with each loop?

Hi @RedPacketSec

The problem you are having is caused by the use of the set node that is outside this loop. So you are in fact getting the same data everytime. so the result is the same everytime.
If the HTTP request returns the offset as well after making the request, you can simply use that one to iterate.

1 Like

it does not provide the offset , it only provides the data back.

Thats too bad.
I usually use a function node then.
Check if the offset field exists, if it does not set the first offset.
if it does +100 in your case.
In the last set node you can set the offset you get from this function node.
something like:

you got a sample function code i can try?

Yes, somehere in my very nicely documented workflows.

Oh, I have a few with a nice name. So I found it actually:


const page = "0";
const pageSize = 5;

if(item.page == undefined){
  item.page = page
}
else{
  let page = +item.page;
  let nextPage =  page + pageSize;
  item.page = nextPage;
}


return item;
2 Likes

i get error line 4 item not defined

@RedPacketSec
not sure what this could be.
does the item exist?

this is one area in n8n, that has been a pain for me, pagination using the HTTP node.

i also tried putting the set node in the loop but ultimately it fails as it cant reference itself and ends up with a NaN. which is why I put it out the loop to start with

Works fine here


I made you the following workflow. It is not a working example at all, but should show you how it would work. This is from a working workflow but cut out all the extra stuffs.

Ps. the page was a string previously because there was some base64 encoding going on here for a specific API.

yeah but in your example you are getting something like next page, so you can use that, what i am doing has nothing like that its just results, and i just need it to count of offset upon each loop and increment it.

The nextpage is calculated within this function node.

The if checking if the loop needs to continue is indeed using a field returned by the api to check if there is a next page to do. You would need to replace that by a count of te items returned checking if that is the maximum page size.

If you need help with that I will check when I am behind my pc.

I do something similar with the NPMJS API I have to keep calling it until I don’t get any data back increasing the offset each time. I then check to see if a property that should exist is returning and if isn’t I assume all is good and I have the lot.

Maybe it will help.

2 Likes

This is what i needed, few tweaks but this functionitem i turned into a functionnode and does what i need thanks so much

2 Likes