That is not a bad idea although potentially tricky as not all APIs are the same. I guess we could have some different options to look for next URLs in a response body or in a header.
I will add this to our internal tracker as I can see it being useful.
yeah could potentially make a drop down option to use different settings related to this etc
I’m sorry if i’m suddenly join this topic, but I think this one is not possible, even it’s possible it has potential to break the HTTP node it self since every database, platform, etc has their own pagination method, and pagination require more than 1 called every request that you make to get next page
the problem is even this idea can be implemented, this gonna make your HTTP node blocked if the API source having trouble or downtime when the HTTP node automatically request next page data to the API source.
Even with the drop down options, that’s mean HTTP node will call HTTP method again inside it, the other trouble is if when something happen or there is a bug in source API, this gonna loop the HTTP node it self, since you cannot control, the callback with the second call.
but yeah there is nothing impossible but again this is just my opinion…
i am not sure i agree that this will break the http node itself. it may not be simple, but it can be modular and still be implemented
I’m transitioning into n8n from another service (Parabola), just for the sake of having it self-hosted.
But I have faced a blocking point quite early in the switch, which is this, pagination. I feel it’s quite a basic feature for this type of product, as most requests come from an API, and most requests have some form of pagination required to be able to get all results.
Anyway, just wanted to jump on it. And as a reference, Parabola has this perfectly integrated into the node, using the main typical types of pagination. You can see it here.
That looks pretty cool would be good to have something similar in N8N
its something that is a pain in the bum for me even using code nodes
We are actually making some good progress on that. There is a good chance we will get something out soonish.
You can find the PR that Jan has been working on here: feat(HTTP Request Node): Add pagination support by janober · Pull Request #5993 · n8n-io/n8n · GitHub
Nice. Coming along well
Actually more changes locally that I did not commit yet. Will hopefully finish that and commit the next days.
looking forward to seeing it in action will solve many issues for me and clean up many horrible loops and code nodes to consolidated data.
Any news so far?
Hello. Any news about this feature?
Sorry, the whole AI/LangChain push sadly resulted in this not getting finished. Made some changes a few days ago and hope to maybe find some more time the next few weeks.