Pagination included into HTTP Node [GOT CREATED]

Please can you include pagination functionality in the HTTP request node by default, rather than users having to bake in their own javascript function nodes to handle this?

That is not a bad idea although potentially tricky as not all APIs are the same. I guess we could have some different options to look for next URLs in a response body or in a header.

I will add this to our internal tracker as I can see it being useful.

1 Like

yeah could potentially make a drop down option to use different settings related to this etc

1 Like

I’m sorry if i’m suddenly join this topic, but I think this one is not possible, even it’s possible it has potential to break the HTTP node it self since every database, platform, etc has their own pagination method, and pagination require more than 1 called every request that you make to get next page
the problem is even this idea can be implemented, this gonna make your HTTP node blocked if the API source having trouble or downtime when the HTTP node automatically request next page data to the API source.
Even with the drop down options, that’s mean HTTP node will call HTTP method again inside it, the other trouble is if when something happen or there is a bug in source API, this gonna loop the HTTP node it self, since you cannot control, the callback with the second call.

but yeah there is nothing impossible but again this is just my opinion… :joy:

i am not sure i agree that this will break the http node itself. it may not be simple, but it can be modular and still be implemented

1 Like

I’m transitioning into n8n from another service (Parabola), just for the sake of having it self-hosted.
But I have faced a blocking point quite early in the switch, which is this, pagination. I feel it’s quite a basic feature for this type of product, as most requests come from an API, and most requests have some form of pagination required to be able to get all results.

Anyway, just wanted to jump on it. And as a reference, Parabola has this perfectly integrated into the node, using the main typical types of pagination. You can see it here.

3 Likes

That looks pretty cool would be good to have something similar in N8N

1 Like

its something that is a pain in the bum for me even using code nodes

We are actually making some good progress on that. There is a good chance we will get something out soonish.

3 Likes

You can find the PR that Jan has been working on here: feat(HTTP Request Node): Add pagination support by janober · Pull Request #5993 · n8n-io/n8n · GitHub

3 Likes

Nice. Coming along well :slight_smile:

1 Like

Actually more changes locally that I did not commit yet. Will hopefully finish that and commit the next days.

3 Likes

looking forward to seeing it in action :wink: will solve many issues for me and clean up many horrible loops and code nodes to consolidated data.

1 Like

Any news so far?

1 Like

Hello. Any news about this feature?

Sorry, the whole AI/LangChain push sadly resulted in this not getting finished. Made some changes a few days ago and hope to maybe find some more time the next few weeks.

1 Like

New version [email protected] got released which includes the GitHub PR 5993.

2 Likes