It always takes extremely weird and development-heavy workaround to simply loop through paginations.
Every time I want to add looping through pagination on different API, this takes me hours for something that should be simple.
Variable set beforehand can’t be iterated easily.
Since I am no developer, I have to resort to vibe coding with AI and hours of correcting for something very simple:
After the current package has been processed, iterate the page and repeat, until no more package.
With an option called “Pagination” in the HTTP request, why are there not some premade template for the most common API types and an integrated iterate/loop function?
This is relevant for practically every workflow where slightly larger amounts of data have to be processed via API.
Every time I go through several posts here, on youtube etc. and every time tutorials are super vague at best ("You have to integrate a loop - yeah, who would’ve guessed)
Keep in mind that not everyone is a deveoper who could code the entire workflow in notepad.
Why no “iterate” function? Why can’t I edit/iterate a variable that has been set in an edit field earlier?
In my case, its rest API, Parameters “limit” and “offset”, and the request also returns “total” number of items. Should be easy enough?
Hi @om-Juri Welcome to the community!
Why don’t you try using the HTTP node’s built in method to handle pagination, which is like update parameters with each request, set Name = offset and Value = {{ $pageCount * <yourLimit> }} so that the n8n loops through all the pages without extra loop node or code node.
Thank you for pointing me to that document. I tried exactly that before.
The API I’m using (as per its documentation) allows for query parameters “limit” and “offset” and it returns a “total”. I know I have 16 packages I want to retrieve. Lets say I want to retrieve 5 with each request.
My problem is, if I set “limit” to 5, it only returns a total of 5.
@om-Juri Understood you can try something like set the limit to 5, and then in the HTTP pagination type would be query name would be offset and now the value should be something like {{ $pageCount * 5 }} so that it always requests the page with like multiplied by 5. something like this:
@om-Juri sounds like the limitation of the pages you are tryna read and that API end point or service. Although pagination with HTTP node works fine in all cases.
Solved it. I thought pagination would continue the workflow for every package instead of accumulating all data together. Which is dangerous as N8N could run out of memory. Using help of AI I made a solution that runs through the workflow once for each package.
Then check with “IF” if total is lower than limit and until true, reiterate. I still think this process is a bit unwieldy as you require a custom code node again, but I learned a lot today. SOLVED