I’m trying to use the pagination option in the HTTP Request node, but I can’t figure it out. I reviewed the docs, searched this forum, and watched some videos, but they still won’t work for me.
I’m trying to hit an API, which defaults to 10 records, and then paginate using the page query parameter, adding the information to a Google Sheet.
What is the error message (if any)?
The first 10 records keep being added to the Google Sheet. The page query parameter never increases
It looks like you are mixing up the looping/batching and pagination.
Since your page size is 10, the Loop over items with a batch size 10 will only loop once and add all 10 items to the Google Sheet. However, your path back from the wait should go back into the loop, not back to the http request.
If you route the “done” output from the Loop Over Items back to the HTTP Request node, that might get your next page and get you closer to what you want. You still probably need something in there, like an If - zero pages just after the HTTP Request, to route you around the loop (or just end the workflow) when pagination is done.
Something like this…
EDIT: This is still not right. See later posts. Looping to an HTTP Request node that is set with pagination enabled is NOT the way it works. The retrieval of all pages already happens when the HTTP Request node is reached the one time. WARNING: Looping could duplicate and multiply the calls for each page and require a full restart of n8n.
Thank you for the quick response and sample workflow. I have a few questions:
The HTTP REQUEST just hangs when I run this. If I manually use https://test.engineawesome.app/api/v1/objects/739564f1-cf13-49eb-bdd6-29b1a68eb9da?page=1, it works. Is the pagination setting adding the query parameter to the request? I can’t see it in dev tools.
I’m still confused why the loop goes from “Sheets” to “Loop over items”. The HTTP request pulls in 10 records (?page=1). I get looping over to add to Sheets. But once that’s done, shouldn’t I go back to the HTTP Request node and make another call to get the next 10 records(?page=2).
I think your API must not give you an empty response when it is done, so you might need something different than Pagination Complete When: Response Is Empty Also, I forgot that the pagination feature works on its own without looping the workflow back to it. That is basically telling it to go get all the pages again and again. (Sorry, I didn’t fix that in what I sent previously.)
What you do with the set of items output by the HTTP Request is a completely separate thing.
Here’s a working example on a public test/dummy API that passes the output to a Split Out node instead of a loop. The dummy API returns 2 pages of stuff, and a final page that has no more items in it (3 items total). The 2 pages have 6 items each in the data field of the response, so Split Out forwards 6, then 6, then 0 items (total of 12). The Pagination Complete When setup should also be similar to what you will need.
I really appreciate this, but still not working as expected. I’m not clear on how pagination works. I attached an image, not a working flow. Which option describes how pagination works, or is there a C?
FLOW A: The HTTP Request node paginates over the api, stores everything in memory, and then passes the data in chunks to the next node?
FLOW B: The HTTP Request node is in a loop. It makes a request, passes the response to the next node, things happen, then the HTTP Request node is triggered again, this time increasing {{ $pageCount }} by one, and making another request.
Definitely FLOW A. You should be able to copy/paste the nodes from the last one I sent into a workflow in your own environment, and study how it is working.
That worked! Once you confirmed that the HTTP Request node paginates over the API, stores everything in memory, and then passes data to the next node, it totally made sense.
What confused me was the HTTP node was spinning for minutes and I didn’t know if it was stuck. There were 2100+ records, grabbing 10 at a time, so it took a while.
It isn’t surprising that it appeared to lock up then. The way the workflow looped back, it would have re-entered the http request node 2100 times, and each of those inbound items from the first loop iteration would have fetched the 2100 records again, so loop 1 = 2100 items, loop 2 = 4.41 million items, loop 3 = 9.261 billion items, and so on.
