API HTTP Pagination

[zynera]

Describe the problem/error/question

I am trying to solve an pagination issue. The API only returns 100 items at a time, so the calls look like this:

FIRST CALL
offset:0
limit:100
next:/api/v2.20/song/ac32c031-e34f-46fc-acd0-223d505ebebf/playlist/current/spotify?type=all&sortBy=subscriberCount&sortOrder=desc&offset=100&limit=100
previous:[null]
total:189

LAST CALL
offset:100
limit:100
next:[null]
previous:/api/v2.20/song/ac32c031-e34f-46fc-acd0-223d505ebebf/playlist/current/spotify?type=all&sortBy=subscriberCount&sortOrder=desc&offset=0&limit=100
total:189

If you click into the Spotify HTTP node, you will see that I tried to set up the pagination criteria and this worked once when testing and trying to run it for one app get request with one ID.

When I tried to reconnect it to the rest of the flow it kept timing out, which is why I am now trying an alternative way by processing one at a time and handling the pagination manually.

Any help solving that would be amazing, I am stuck for the past 5 hours for this bs and cant figure it out, thanks to everyone!

What is the error message (if any)?

This is ran on a cloud hosted starter plan by the way.

Please share your workflow

Information on your n8n setup

• n8nVersion: 1.85.4
• platform: docker (cloud)
• nodeJsVersion: 20.18.3
• database: sqlite
• executionMode: regular
• concurrency: 5
• license: community
• consumerId: 00000000-0000-0000-0000-000000000000

storage

• success: all
• error: all
• progress: false
• manual: true
• binaryMode: filesystem

pruning

• enabled: true
• maxAge: 168 hours
• maxCount: 2500 executions

client

• userAgent: mozilla/5.0 (macintosh; intel mac os x 10_15_7) applewebkit/537.36 (khtml, like gecko) chrome/134.0.0.0 safari/537.36
• isTouchDevice: false

Generated at: 2025-04-06T21:52:06.857Z

[mohamed3nan]

Hi @zynera

Have you considered looking into the built-in Pagination feature?

I’m pretty sure this could help with your issue.
Also, take a look at Pagination Mode and Pagination Complete When—these might be especially relevant in your case…

[zynera]

Hi @mohamed3nan I actually did, if you click on the bottom “Spotify” node you can see that I was able to set up the pagination for a single call, but as mentioned when integreating it into the rest of the workflow it somehow fails which is why I am trying to do it through looping

[mohamed3nan]

If I understand correctly, you might need to integrate $runIndex into your custom expression for the offset field inside the loop for the manual pagination…