SuccessFactor Http Request Pagination run endlessly

[weilies_chok]

I am trying to port my postman’s SuccessFactor API into n8n.
Below is the sample response for listing when contain multiple pages

 {
  "d": {
    "results": [
      {
        "personIdExternal": "123",
        "firstName": "Duffin",
        "lastName": "Duffin",
        "gender": "M"
      }
    ],
    "__next": "https://api.sapsf.eu/odata/v2/PerPersonal?$select=personIdExternal,%20firstName,gender,lastName&$filter=lastModifiedDateTime%20ge%20datetimeoffset%272022-01-01T16%3A30%3A00.000Z%27&$format=json&$skiptoken=<my-token>=="
  }
}

And i manually run it from postman, each time using the new url from “__next” to crawl the next page if available. It shows that total is around 4 pages. But when port into n8n (with partial nodes below)

it run endless, keep crawling for next page, forcing me to force-stop at page 10th, which isn’t correct. With same param, it’s expected to stop at page 4th.

Thanks for the support and pointers

Information on your n8n setup

• n8n version: Version 1.81.4
• Database (default: SQLite): default
• n8n EXECUTIONS_PROCESS setting (default: own, main): default
• Running n8n via (Docker, npm, n8n cloud, desktop app): docker
• Operating system: macOS, self-host

also, strange enough, the “__next” URL keep append the redundant query param (as below)

[barn4k]

hello @weilies_chok

You can open the debugging window (F12, Console tab) to see what exactly is passed to the service. Are you sure that it sends an empty response after the 4th page?

[weilies_chok]

let me explain how SAP’s successFactor works in pagination. It returns 1000 items/request and response with “__next” when next page available. Subsequent request must use “skiptoken” in previous response (explain in screenshot below)

here the console, nth helpful for troubleshoot (or i am noob :D)

then i restructure my nodes, can see each node’s loop logged in console

i compare the run in both postman (col F) and n8n (col B), and surprisingly, the skiptoken are identical for first 4 page (response). Then i substitute skip token to retrieve next page’s result.

If you look closely in cell H6, it no longer return “__next” meaning it’s the last available page. But in n8n, i couldn’t explain why it keep spawning with new skiptoken causing it to keep running till i hit stop button

[weilies_chok]

or to be more direct, why the same SkipToken provided in both postman and n8n, postman shows it’s last page but n8n response with “__next”?

[barn4k]

can you share the workflow realted to the pagination part?

I use this kind of behavior quite often and didn’t have any issues with pages.

One suggestion - according to postman, on page 4 the response is not empty (it just don’t have the next link property), so that maybe the reason why n8n keeps trying to paginate. You can try setting the end condition manually with this one:

[weilies_chok]

hi @barn4k , sorry but i think u might missed my point

when postman query pg 4 (with skiptoken), the response will no longer return skiptoken. it tells us, “page 4th is last page”

whereelse using same URL, i query in n8n, the response still contain skiptoken with value. It tells the sys “plz crawl next page”.

Hence regardless wht custom script to put in n8n, it will craw next page forever unless i limit it

below is the code snippet (two diff approaches tried). sorry but i can’t share the complete flow

again i am showing the comparison, with same skiptoken (from pg3 response) value, postman return no __next but n8n has

[barn4k]

Can you check that the fields $format, $fields, $select, $skipToken occurs only once in the __next property? I’m assuming that they may be added to the URL for the subsequent __next property and thus messing with the API

[weilies_chok]

yes, return once in response

{
 ... 
 __next : "https://api.../odata/v2/PerPersonal?$select=personIdExternal,%20firstName,lastName&$filter=lastModifiedDateTime%20ge%20datetimeoffset%272022-01-01T16%3A30%3A00.000Z%27%20&$format=json&$skiptoken=...m93Ijo1MDAwfQ=="
}

below are the network log registered. hope it helps for troubleshoot, as clearly stated querystring have provided skiptoken. Hence “_next” shouldn’t in response (as postman example)

[weilies_chok]

i finally found the answer from other thread… tht’s the missing expression i looking for~ now it stopped

strangle the skiptoken stacked up for each page, getting longer

[barn4k]

Ehm… that was in the post earlier

SuccessFactor Http Request Pagination run endlessly

can you share the workflow realted to the pagination part? [image] I use this kind of behavior quite often and didn’t have any issues with pages. One suggestion - according to postman, on page 4 the response is not empty (it just don’t have the next link property), so that maybe the reason why n8n keeps trying to paginate. You can try setting the end condition manually with this one:

[weilies_chok]

bout the inconsistent of results returned in both postman and n8n, sorry barn4k plz accept my apologize .. . i been working wth two diff SF account now only i noticed. Now i gotto sync internally on which account to be use. But i think the respond of “__next” url with longer and longer query param still a bug

updates:
after using SAME credential in both postman/n8n, the result returned are identical, stopped at page 4 yeah! But jz the minor hiccup, where it keep appending fields repetitively which not a show stopper but nice to resolve