Hello everyone,
It’s been almost a week I’m trying this thing out, but it refuses to work. Basically, I have an API which I have to paginate in order to get all the records. The API doesn’t return a next URL, or the page number, or any sort of indicator that pages are present after the limit it returns. It is limited to return 25 records as per the documentation. The API is basically OData API, which uses top and skip to paginate through the response. As it is a RESTful API, it also supports the generic offset and limit params when passed. The pagination is working fine in Python until I had this thing to set it up on a workflow. I cannot paste any sort of code here as it is totally confidential, but the basic thing happening is I’m setting up a HTTP node, setting the offset and limit as query parameters, and I’m updating the query parameter offset to $pageCount + 25. And I’m ending the pagination when the length of the response returned is less than 25. Please someone who knows or faced this could give some lead on how to take this forward, as this is building up frustration.
Thanks in Advance!
Hey @Charan_Kalyankar hope all is well.
Could you please expand on what do you need to “take forward” and what exactly is “refusing to work”. From your explanation it sounds like you’ve set this up already, but it is not clear what the problem is.
Heyy @jabbson,
Thanks for taking a look at my problem. Let me break it down for you
Step 1 : I’ve set the URL, required headers and certificates required for the API in HTTP Node.
Step 2 : I’ve added Query Parameters as $offset = 0 and $limit = 25
Step 3 : I’ve enabled pagination in the options, set the Update Query Parameter to update $offset by $pageCount + 25, and Set the expression for the End Pagination when to expression and set it as
{{ $response.$json.body.var.length < 25 }}
After this, I tried hitting the URL, but after fetching around 104 records, the HTTP Node routes to Error Branch saying
“The response received was 5x identical”
and in description it said
“Please check if the Pagination Complete When expression has been configured correctly”.
This is where I am stuck at and I am not able to take it further from this step.
I found one public open data api and tried pagination, this is what ended up working for me (you want to use multiplication to paginate using pageCount and {{ $response.body.isEmpty() }} to check for completeness.
Hey @jabbson,
If this has worked for you, probably should work for me too! Will try this and let you know. If this works out, man, I owe you a burger 
Hey @jabbson,
In the update query params, I can see the offset not being updated, because we are multiplying $pageCount with 50, which is still staying zero for some reason. Here is the screenshot. When i access the pageCount variable it is 0.
You see 0 because at the time of you looking at it, the loop is not iterating, which means the next value for this expression is going to be 0. After the first iteration the count becomes 1, which will make the expression to be 150, then 250 and so on…
I have seen this value after the first iteration. Anyways, I’ve tried replicating what you have suggested and facing this error called “The returned response 5x was identical”.
Sorry, without being able to repro and troubleshoot, it’s hard to guess what the remote server may have not liked (which is what causes the 5xx responses, mentioned in the error).
I know it’s tough doing that. But yeah, I conveyed this to a senior and he said, let’s not worry about pagination now, and pull records defaultly for a set of IDs queried from a postgres table. And BOOM! It automatically paginates when running for those IDs. I don’t really know how. But yeah, this dependency of pagination got resolved.
Thanks for taking time and replying @jabbson! 
