Hey @Kihap69,
Welcome to the community 
At the moment the HTTP Request node doesn’t automatically handle pagination so you would need to build that into your workflow, You can find an example of that here: Handle pagination in HTTP Requests | n8n workflow template
im sure there was a ticket somewhere to vote on related to this too!
Pagination is requested alot in the forums to be implemented in the http nodes etc.
Thanks Jon, however, the API I am making the request to does not return a cursor (or something that points to the next page). How do I overcome this aspect?
TIA
offset and limit.
but i have had so many issues with this in n8n, the loops never seem to work well with updating the offset. I gave up with the pagination for these things and moved over to doing it in a python script externally and grabbing the output via SSH and bringing it back into the flows.
Im waiting for n8n to add pagination into the HTTP node and then i can use it again correctly all in the platform.
But you might get better luck. My next step was to do the entire HTTP request and pagination in a code node, but was easier to do it the way i did it in the end.
Hey @Kihap69,
If it is just paged based you can use this example which will increment the page by one, You will need to tweak this to fit your API but hopefully it helps.
This example loops through the GitHub API to return everything I have starred over the years but the theory should work with most page based APIs.
To increase visibility. Here is the Feature Request in the forum. Please upvote if you are interested in it so that you get informed once it is ready:
You can also already find the current WIP PR here:
It can be tested with the following image docker image:
n8nio/n8n:PR-5993-http-request-pagination
As mentioned, it is still WIP, so incomplete and does not offer the best experience yet.
Thank you Jan, I have voted for this now.
@Jon Thank you so much for this great solution, when I run it on my HTTP node data, on the final IF statement it producing True results instead of False, and due to that can’t combine the output of all data.
I would really appreciate the help!
Hey @82rocker,
What does the data look like when there are no more results from the HTTP request node?
Thanks for the quick response, this is how the data looks on the last item and there is no output data for it.
[
{
"body":{
"breadcrumbs":[
{
"label":"# Employees",
"signal_field_name":"organization_num_employees_ranges",
"value":"1,100",
"display_name":"1-100"
},
{
"label":"Management Level",
"signal_field_name":"person_seniorities",
"value":"owner",
"display_name":"Owner"
},
{
"label":"Management Level",
"signal_field_name":"person_seniorities",
"value":"founder",
"display_name":"Founder"
},
{
"label":"Management Level",
"signal_field_name":"person_seniorities",
"value":"c suite",
"display_name":"C suite"
},
{
"label":"Management Level",
"signal_field_name":"person_seniorities",
"value":"partner",
"display_name":"Partner"
},
{
"label":"Management Level",
"signal_field_name":"person_seniorities",
"value":"head",
"display_name":"Head"
},
{
"label":"Management Level",
"signal_field_name":"person_seniorities",
"value":"vp",
"display_name":"Vp"
},
{
"label":"Management Level",
"signal_field_name":"person_seniorities",
"value":"senior",
"display_name":"Senior"
},
{
"label":"Management Level",
"signal_field_name":"person_seniorities",
"value":"director",
"display_name":"Director"
},
{
"label":"Management Level",
"signal_field_name":"person_seniorities",
"value":"manager",
"display_name":"Manager"
},
{
"label":"Company Locations",
"signal_field_name":"organization_locations",
"value":"United States",
"display_name":"United States"
},
],
"partial_results_only":false,
"disable_eu_prospecting":false,
"partial_results_limit":10000,
"pagination":{
"page":8,
"per_page":10,
"total_entries":69,
"total_pages":7
},
"contacts":[
],
"people":[
],
"model_ids":[
],
"num_fetch_result":null,
"derived_params":{
}
},
"headers":{
"date":"Thu, 27 Jul 2023 18:39:41 GMT",
"content-type":"application/json; charset=utf-8",
"transfer-encoding":"chunked",
"connection":"close",
"vary":"Accept-Encoding, Accept-Encoding, Origin",
"status":"200 OK",
"x-minute-usage":"10",
"cache-control":"max-age=0, private, must-revalidate",
"strict-transport-security":"max-age=3600",
"x-24-hour-requests-left":"1989",
"x-rate-limit-minute":"200",
"x-minute-requests-left":"190",
"x-rate-limit-24-hour":"2000",
"x-hourly-usage":"10",
"x-24-hour-usage":"11",
"x-hourly-requests-left":"390",
"etag":"W/"f424ee377dd1b4a19e19e6f6caf01042"",
"x-frame-options":"ALLOWALL",
"x-rate-limit-hourly":"400",
"x-content-type-options":"nosniff",
"content-security-policy":"frame-ancestors 'self' chrome-extension://alhgpfoeiimagjlnfekdhkjlkiomcapa chrome-extension://ececkagaccnfmkopaiemklekhoimmgpn *.salesforce.com *.lightning.force.com",
"set-cookie":[
"X-CSRF-TOKEN=sVAAek_Gf4ACi2lqnaIxK5SjR436tS5x7JO_8blUqKaoLTc12tWctDwwQJ7w1fsOjufVmj213f86iEu8ZvUHww; path=/; secure; SameSite=Lax",
"_leadgenie_session=pVsA%2FdKHaICm%2Bh7Gz10o2hvsmW1Uemo%2Bgo09MKiRyQI14%2B95mdG7HJCiKcs0azMJaoI%2B7P4kCPok9D%2B3t79EUt4sYSg%2B7C%2FLr7GLLWCSFp0tF%2B2T7M7ucTJB6FiDNZ1nV5b1E5B2Z6%2Bf%2FYxrs%2Ffn6gNTMkRymYEDJ5vIm6FhdyiBpYcx8tMVpR7op2xJGCg4cTgckkDFPxicjr0A8Xq0nBhxgWTrWIKrUbY8enY7I7NYvTK8AibZo4FA2QExyTnJ7CGu2cjaWLogXk2%2FScrgHGX63KxaHTpOboc%3D--rClBj9AahGtUDmym--4uX2bm2VuRa6Ty2tzen66Q%3D%3D; path=/; secure; HttpOnly; SameSite=None",
"GCLB=CIushs_clLW-GA; path=/; HttpOnly; expires=Thu, 27-Jul-2023 18:49:41 GMT"
],
"via":"1.1 google",
"cf-cache-status":"DYNAMIC",
"server":"cloudflare",
"cf-ray":"7ed701a909b16be5-SIN"
},
"statusCode":200,
"statusMessage":"OK"
}
]
Hey @82rocker,
Does your people data contain an id as well? The trick will be finding a key that should exist then updating the if statement to include that, In your case though your API returns a count of records so your check could be something like if page is the same as total_pages
I know this is older, but I just solved this issue with the Mosyle API. It returns total number of items, items per page, and page number. To resolve the question of “Am I done?”, I multiply the number of items per page by the page number. If that results in a value the same or greater than the total number of items, I’m done.
Now my issue is accessing the complete dataset.
I’m thinking that’s going to be using the page numbers, but I don’t know what node I need to access to get to the data.
@Jon When I compare those it again runs the loop but it does not send data on the false condition, here is the complete code of the flow, kindly review thanks.
Hi 82rocker,
When the loop is done, we must combine the successful loops’ data with the one false loop.
I did that with the merge node. Now I have 7 pages in the merge node, each with ~50 records where I need to pull a couple of fields from each to then do another API call on a different service to compare.
No idea how to do this in n8n.
It looks like I’ll need a loop to process each page, then within the page, loop to tear down the JSON into each record, and pull the data out of each record. This seems really inefficient.
I must be making this too hard.
this is why we need pagination built into the HTTP node to help with some of this.
@RedPacketSec it is actually getting very close of being ready to review and then to be released.
Thanks, @russellkg for sharing this, I actually tried that but it was generating duplicate results, @Jon suggestio worked for me!
