API HTTP Pagination

Questions
, ,
1

Describe the problem/error/question

I am trying to solve an pagination issue. The API only returns 100 items at a time, so the calls look like this:

FIRST CALL
offset:0
limit:100
next:/api/v2.20/song/ac32c031-e34f-46fc-acd0-223d505ebebf/playlist/current/spotify?type=all&sortBy=subscriberCount&sortOrder=desc&offset=100&limit=100
previous:[null]
total:189

LAST CALL
offset:100
limit:100
next:[null]
previous:/api/v2.20/song/ac32c031-e34f-46fc-acd0-223d505ebebf/playlist/current/spotify?type=all&sortBy=subscriberCount&sortOrder=desc&offset=0&limit=100
total:189

If you click into the Spotify HTTP node, you will see that I tried to set up the pagination criteria and this worked once when testing and trying to run it for one app get request with one ID.

When I tried to reconnect it to the rest of the flow it kept timing out, which is why I am now trying an alternative way by processing one at a time and handling the pagination manually.

Any help solving that would be amazing, I am stuck for the past 5 hours for this bs and cant figure it out, thanks to everyone!

What is the error message (if any)?

This is ran on a cloud hosted starter plan by the way.

Please share your workflow

Information on your n8n setup

  • n8nVersion: 1.85.4
  • platform: docker (cloud)
  • nodeJsVersion: 20.18.3
  • database: sqlite
  • executionMode: regular
  • concurrency: 5
  • license: community
  • consumerId: 00000000-0000-0000-0000-000000000000

storage

  • success: all
  • error: all
  • progress: false
  • manual: true
  • binaryMode: filesystem

pruning

  • enabled: true
  • maxAge: 168 hours
  • maxCount: 2500 executions

client

  • userAgent: mozilla/5.0 (macintosh; intel mac os x 10_15_7) applewebkit/537.36 (khtml, like gecko) chrome/134.0.0.0 safari/537.36
  • isTouchDevice: false

Generated at: 2025-04-06T21:52:06.857Z

2

Hi @zynera

Have you considered looking into the built-in Pagination feature?

image
image483×510 22.4 KB

I’m pretty sure this could help with your issue.
Also, take a look at Pagination Mode and Pagination Complete When—these might be especially relevant in your case..

3

Hi @mohamed3nan I actually did, if you click on the bottom “Spotify” node you can see that I was able to set up the pagination for a single call, but as mentioned when integreating it into the rest of the workflow it somehow fails which is why I am trying to do it through looping

4

If I understand correctly, you might need to integrate $runIndex into your custom expression for the offset field inside the loop for the manual pagination…