Creating Automated Pagination with n8n's $runIndex

Describe the problem/error/question

I’m trying to create a flow for HTTP requesting a page with pagination. Each time the flow runs, I want it to request the next page in the pagination sequence.

I’ve read the documentation and attempted to use http://neoshop.vn/collections/all?page={{ $runIndex + 1 }}, but the $runIndex variable doesn’t seem to work as expected. The documentation states that it represents “how many times n8n has executed the current node,” following a zero-based index (the first run is 0, the second is 1, and so on).

Currently, the flow keeps repeatedly requesting http://neoshop.vn/collections/all?page=1 without progressing to subsequent pages.

Can you please help me understand why $runIndex isn’t behaving as expected and suggest a solution?

Please share your workflow

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:

@Hien , I’m afraid, neither $runIndex nor other pagination that comes with HTTP Request node can help you. You are just pulling the website pages. Even if you use the page number larger than the website shows, you still pull the whole webpage context with no clear indication that there no pages left.

HTTP Request embedded pagination feature is designed to work with API that provide information when there is no more data left. Your workflow simply downloads the content of the webpage.

To make it work, you would need to do much more. You would actually have to parse the HTML context, which is very unreliable in a sense that the next update to the webpage by the web developer might break your workflow. Unless they provide an API you could use to pull the catalog, I consider this task as mission impossible.

1 Like

@ihortom thank you, but While implementing incremental pagination is technically possible, I wouldn’t consider it a basic task for beginners.

Although I’ve provided a simplified example to illustrate the concept, my primary reason for needing this functionality is to handle websites with unlimited pagination.

@Hien , ok, I can demonstrate how to use pagination in the cases like this. However, the problem remains as you do not have the condition to stop paginating and without it the workflow will be endless till it times out. Therefore, I have to put the condition of max 5 pages just to demonstrate the feature. Also note, that embedded pagination requires n8n version 1.15+ (and you haven’t mentioned what version you use). This allows to engage the variable $pageCount as opposed to $runIndex.

1 Like