Magento 2 pagination with http request node

[jochem]

Describe the problem/error/question

I have a workflow running that retrieves all orders within a certain period and calculates the total number of orders and total revenue. I’m running into an issue where, if there are 2000+ orders in a given period, the Magento server can no longer handle it.

I assume I need to resolve this using pagination. This is new to me, and unfortunately, I haven’t been able to get it to work.

Magento provides pageSize and currentPage: https://developer.adobe.com/commerce/webapi/rest/use-rest/performing-searches/
Additionally, it returns total_count, which likely needs to be used in the Pagination Complete When expression.

Can someone help me get started?

Please share your workflow

Information on your n8n setup

• n8n version: 1.64.2
• Database (default: SQLite):
• n8n EXECUTIONS_PROCESS setting (default: own, main):
• Running n8n via (Docker, npm, n8n cloud, desktop app): Elestio
• Operating system:

[ihortom]

Hey @jochem, according to the API documentation (as I understood it):

• pageSize: number of items on the current (returned) page
• currentPage: the current (returned) page
• total_count: found no description but t appears to indicate total number of items available

If that is so, then you could calculate the number of pages to retrieve with total_count / pageSize formula. I don’t think using n8n variable $pageCount need to be engaged as the API provides it’s own currentPage.

The tricky part is to compose the query parameters that needs updating as it is formed as JSON with nested properties. And I’m not sure if it will work if they are nested.

Moreover, I couldn’t find anything in the documentation specifying how the page-related properties are to be used in the HTTP request. Can you find that? All the three properties you mentioned so far are the one returned by the API, they are not meant to be used with the request to get the data.

[jochem]

Hi @ihortom, thank you for your quick response and for taking the time to address my issue. I hope I fully understand you and can provide the right additional information.

I briefly tested without pagination using the query parameters below.

{
  "searchCriteria": {
    "filterGroups": [
      {
        "filters": [
          {
            "field": "created_at",
            "value": "2024-01-01 00:00:00",
            "condition_type": "gteq"
          }
        ]
      },
      {
        "filters": [
          {
            "field": "created_at",
            "value": "2024-11-31 23:59:59",
            "condition_type": "lteq"
          }
        ]
      },
      {
        "filters": [
          {
            "field": "state",
            "value": "processing,complete",
            "condition_type": "in"
          }
        ]
      }
    ],
    "pageSize": 2,
    "currentPage": 1
  }
}

The output looks as follows:

pageSize: This is always equal to the number specified in the pageSize query parameter. In this case, it remains 2 regardless of currentPage.

currentPage: This is indeed the current (returned) page.

total_count: This is indeed the total number of available items. In this case, it’s 11.

The tricky part is to compose the query parameters that needs updating as it is formed as JSON with nested properties. And I’m not sure if it will work if they are nested.

This does indeed seem to be a challenge. With the settings below, I’m getting an error message.

{
  "errorMessage": "The returned response was identical 5x, so requests got stopped",
  "errorDescription": "Check if \"Pagination Completed When\" has been configured correctly.",
  "errorDetails": {
    "rawErrorMessage": [
      "The returned response was identical 5x, so requests got stopped"
    ]
  },
  "n8nDetails": {
    "nodeName": "HTTP Request1",
    "nodeType": "n8n-nodes-base.httpRequest",
    "nodeVersion": 4.2,
    "itemIndex": 0,
    "time": "3-12-2024, 08:28:47",
    "n8nVersion": "1.64.2 (Self Hosted)",
    "binaryDataMode": "default",
    "stackTrace": [
      "NodeApiError: The returned response was identical 5x, so requests got stopped",
      "    at Object.execute (/usr/local/lib/node_modules/n8n/node_modules/n8n-nodes-base/dist/nodes/HttpRequest/V3/HttpRequestV3.node.js:502:33)",
      "    at processTicksAndRejections (node:internal/process/task_queues:95:5)",
      "    at Workflow.runNode (/usr/local/lib/node_modules/n8n/node_modules/n8n-workflow/dist/Workflow.js:722:19)",
      "    at /usr/local/lib/node_modules/n8n/node_modules/n8n-core/dist/WorkflowExecute.js:711:51",
      "    at /usr/local/lib/node_modules/n8n/node_modules/n8n-core/dist/WorkflowExecute.js:1141:20"
    ]
  }
}

Moreover, I couldn’t find anything in the documentation specifying how the page-related properties are to be used in the HTTP request. Can you find that? All the three properties you mentioned so far are the one returned by the API, they are not meant to be used with the request to get the data.

Hopefully, I understand this question correctly. pageSize and currentPage are included in the query parameters. These, along with the other searchCriteria, are also included in the response (see the example earlier in this message).
total_count is only returned by the API and cannot be part of the query parameters.

I hope this information helps you guide me a step further. If you have any other questions, feel free to let me know.

[ihortom]

The currentPage is actually nested in your filter. It’s rather searchCriteria.currentPage. That is what I meant when mentioned “nested JSON”. Nested query parameters change during pagination might not work.

Also you need to tell HTTP Request node wheh to stop. In your last case you retrieved the very same page many times and the node “realised” the pagination is not working terminating it with the error.

You need to set the condition for pagination completion in the section shown below.

[jochem]

Hi @ihortom ,

Thanks again for your help. With your feedback, I’m getting a better understanding of N8N

I’ve now tested pagination with searchCriteria.currentPage, but it indeed doesn’t seem to work.

This is the first time I’ve needed pagination for an HTTP request. I’ve now built my own pagination. See my workflow below. Is this a logical implementation, or do you have tips for an alternative?