HTTP node pagination

[russellkg]

Pagination is not working for me.

I’m trying to implement pagination within the http node (finally!) In the http node. I have this working for another service and the parameters are very similar. Both give me total records, page number, and records per page.

Simple, right? I wish it were so.

Error message: “The returned response was identical 5x, so requests got stopped”

I’m assuming that means the page counter isn’t being updated.

I’m using the example in the pagination cookbook.

I’ve tried several permutations of the finished algorithm.

I’ve verified that when page 2, 3, 4, etc is passed in the options.page variable, it does pull the right page.

In the http node, I have tried both of the following completed tests:

{{ $response.rows/$response.page_size <= $pageCount }}
{{ $response.body.rows/$response.body.page_size <= $pageCount }}

Here is a json snippet showing the data (minus the devices I’m pulling from the platform.)

{
  "status": "OK",
  "response": [
    {
      "devices": [
          ... Lots of data ...
      ],
      "rows": 115,
      "page_size": 50,
      "page": 1
    }
  ]
}

Here’s my workflow

Share the output returned by the last node

Information on your n8n setup

• n8n version: 1.57.0
• Database (default: SQLite): postgresQL 15
• n8n EXECUTIONS_PROCESS setting (default: own, main): Main
• Running n8n via (Docker, npm, n8n cloud, desktop app): Docker on Digital Ocean
• Operating system: Ubuntu 22.04.4 LTS

[ria]

Hi @russellkg

I think what’s happenening is that you basically have an infinte loop because your complete expression ({{ $response.rows/$response.page_size <= $pageCount }}) never evaluates to true and therefore it keeps fetching the last page over and over and stops eventually after 5 identical responses.

Given your JSON snippet I suspect that the page_size variable will always be 50 and while the rows decrease you will eventually get 0 / 50 which will always be less than the $pageCount because you keep increasing it with each pagination request.

Would it not work for you to set pagination complete when response is empty?

[russellkg]

Hi ria,

Thanks for the reply.

If I request a page that’s out of range, I get the first page. So “response is empty” will never happen.

Rows is the number of contacts in total. It stays the same on each page. However, it looks like my logic is backward. The corrected expression is:

{{ $pageCount * $response.body.page_size >= $response.body.record_count }}

Thanks again,

-Russ