Looping all pages

Questions
, , ,
1

Hello,

I was a few versions behind and now I see there is a way to paginate HTTP requests natively in the same node.
However, I don’t see how to paginate all pages if I have no idea of the total number of pages (changes based on the user request).

I know there are multiple workflow examples in the forum on how to do that but I don’t believe there are any that are up-to-date with regard to the in-node pagination.

The workflow itself knows perfectly when he encountered the same page multiple times, but how can I stop pagination there and output results ?

image
image2394×2248 442 KB

I see I can write an expression that should output true to stop but I don’t know how to do that.

Thank you very much for your help,
Joachim

2

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
3

@Joachim_Brindeau , embedded pagination solution depends on your API response. There is no universal approach as every API has its own way to indicate if there are more pages left. Read the documentation for your API service first. If you still cannot figure it out, can you provide the link to the documentation of the API you query?

As far as I can see that API does not provide any data with regard to pagination. It just gives you all the data if has for your request. Therefore, you might not need any pagination.

4

Thank you for your answer.
The API is not official, I found it inspecting network calls in the console.

Pagination is needed as the official HDB nomenclature (a big list of goods and services one can use to define the scope of a trademark) is around 80k terms.

My main question being: N8N knows when it reaches last page (as shown I when it stops the node with an error message), can I use that same capability to stop the node but still return items?

5

@Joachim_Brindeau , no, n8n cannot know when there are no pages left to query. Your screenshot indicates that the error took place while querying the API, which happen to be due to misconfiguration of the node on your part. The node error-ed to prevent endless loop (I didn’t even know there was such a thing in n8n).

To prevent endless loop the node needs to be reconfigured to utilize the number of pages. This could be done if you know how many total pages are there. The API you use does not tell you that. Therefore, you cannot use the pagination feature in this case. If it worked the old way for you, I would suggest to stick to it while querying this specific API endpoint.

The n8n HTTP Request documentation explains that it can help in two scenarios:

  • when API provides a parameter that would indicate that there are no more pages left or provides an indication of the total items available
  • when API specifically indicates what the next page URL should be

Knowing either of those two could be used in configuring embedded pagination. The API you use does not seem to provide such information.

For clarity, I will give you example of API response that could be leveraged in pagination. Here’s the response from Clockify API

[
  {
    "totals": [
      {
        "totalTime": 4536040,
        "totalBillableTime": 0,
        "entriesCount": 2656
      }
    ],
    "timeentries": [
      . . .
    ]
  }
]

It contains entriesCount (total items). Thus, knowing the total, the number of iterations and the page size, n8n knows how many pages needed to pull all the items (provided you configure it correctly). The $pageCount can be utilized here to automatically update the URL with each iteration until all the items have been pulled in.

Again, the API you use does not seem to provide any information to figure out when to stop. You can use $pageCount to update URL with each iteration in your example. However, n8n will not know when to stop unless your API returns something to indicate “this is the end”. Perhaps it retuns an empty array (I’ve no idea). If that is the case, then yes, you can leverage that and use in the node’s condition when it has to stop.

1 Like
6

Thank you very much for the detailed explanation.
I will definitely try and solve it the clean way.

Didn’t want to sound stubborn, it’s just that most scrapers for example stop when they detect twice the same result page. And I thought if I could catch such an event it would make pagination 10x more flexible to use.

I will check if I manage to parse the total number of results and if not, I’ll go back to an old school loop until pagination can do such a thing.

Thanks again for the explanation and sorry if I sounded stubborn!

7

@Joachim_Brindeau ,

it’s just that most scrapers for example stop when they detect twice the same result page. And I thought if I could catch such an event it would make pagination 10x more flexible to use.

It does make sense and as the error suggested n8n did detect that the same data was returned - “The returned response was identical 5x, so requests got stopped”. Therefore, I would encourage you to create a “Feature Request” (as opposed to “Question”) so that n8n team could consider such an option.

3 Likes