Errors when paginating HTTP Requests

Questions
,
1

To decrease pagination load, I modified these values: “Split in Batch,” “Interval Between Pagination Request”, “Limit Pages” and “Items Per Page.”
When I make a paginated request with a limit of 2 pages, it works perfectly.

#ERROR 1

  • Pagination page limit = none
  • Interval Between Requests = 1000
  • Items Per Batch = 20
ERROR: The service was not ablePreformatted text to process your request
500 - " Index was outside the bounds of the array. body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;} p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px} b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px} H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red } H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon } pre {font-family:"Consolas","Lucida Console",Monospace;font-size:11pt;margin:0;padding:0.5em;line-height:14pt} .marker {font-weight: bold; color: black;text-decoration: none;} .version {color: gray;} .error {margin-bottom: 10px;} .expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:pointer; } @media screen and (max-width: 639px) { pre { width: 440px; overflow: auto; white-space: pre-wrap; word-wrap: break-word; } } @media screen and (max-width: 479px) { pre { width: 280px; } } Server Error in '/' Application. Index was outside the bounds of the array. Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.IndexOutOfRangeException: Index was outside the bounds of the array.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:

[IndexOutOfRangeException: Index was outside the bounds of the array.] System.Collections.Generic.List`1.Add(T item) +60 SS.OpenApi.Code.PartnerGatewayAuthHttpModule.Init(HttpApplication context) in D:\buildAgentFull\work\8e15a453e647e65a\SS.OpenApi\Code\PartnerGatewayAuthHttpModule.cs:41 System.Web.HttpApplication.InitModulesCommon() +94 System.Web.HttpApplication.InitInternal(HttpContext context, HttpApplicationState state, MethodInfo[] handlers) +821 System.Web.HttpApplicationFactory.GetNormalApplicationInstance(HttpContext context) +159 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +187
Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.8.4690.0 "
 Details

ERROR 2

  • Split in Batches
  • Interval Between Pagination Request = 3000
ERROR: The returned response was identical 5x, so requests got stopped
Check if "Pagination Completed When" has been configured correctly.

HTTP Code
500
Stack
NodeApiError: The service was not able to process your request
    at Object.execute (/usr/local/lib/node_modules/n8n/node_modules/n8n-nodes-base/dist/nodes/HttpRequest/V3/HttpRequestV3.node.js:1564:27)
    at Workflow.runNode (/usr/local/lib/node_modules/n8n/node_modules/n8n-workflow/dist/Workflow.js:730:19)
    at /usr/local/lib/node_modules/n8n/node_modules/n8n-core/dist/WorkflowExecute.js:662:53
    at /usr/local/lib/node_modules/n8n/node_modules/n8n-core/dist/WorkflowExecute.js:1064:20

Please share your workflow

Information on your n8n setup

  • n8n version: 1.29.1
  • Database: postgresql
  • Running n8n via: VPS → Docker
  • n8n EXECUTIONS_PROCESS setting (default: own, main): ???
2

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
3

@unicornParty , I believe that your usage of ShipStation API is not accurate which leads you to those errors. I managed to find their documentation corresponding to listing the orders, List Orders API.

Here’s what I think you should change:

  1. Do not use batching, rather control the size of the page. That is add the query parameter pageSize and set it to max value, which is 500.
  2. Replace the condition to complete the pagination “Response is empty” with checking the response payload produced the last page.

With regard to the latter, I cannot be totally sure as I found no explanation for the meaning of properties and am having a guess. Nonetheless, the API returns the following with the list of orders:

  • total
  • page
  • pages

My understanding, that total tells how many orders in the current page, page would confirm the current page number returned, while pages is presumably… no idea actually. Perhaps pages indicate the total number of pages or number of pages remaining.

Therefore, I guess the condition {{ $response.body.pages == 0 }} could work for you as the condition to stop the pagination. I tend to think so because one of the examples shows this

  "total": 2,
  "page": 1,
  "pages": 0

which makes sense to me only when pages represents the number of pages left. Though I would verify that (just pull one page and see the values) - perhaps the example is wrong as the name sounds more to me like number of all pages (not remaining). In the latter case, the condition to complete the pagination would be {{ $response.body.page == $response.body.pages }}.

You also need to take into consideration the rate limit and adjust the delay between pulls of the pages accordingly, which is controlled with “Interval Between Requests (ms)” option of the node.

2 Likes