Can't update nested body parameter when trying to paginate request

[TFSho]

Describe the problem/error/question

Can’t get pagination to work properly, keeps sending same request. Trying to update a body parameter that is nested deep in the JSON. Path: query.job.files.$.page. I have tried every possible input for options.pagination.pagination.parameters.parameters[0].name style I can think of but node is still not updating page token on subsequent requests. What am I doing wrong here?

What is the error message (if any)?

The returned response was identical 5x, so requests got stopped
Check if “Pagination Completed When” has been configured correctly.

Please share your workflow

Information on your n8n setup

• n8n version: 1.82.3 (Self Hosted)
• Database (default: SQLite): Default
• n8n EXECUTIONS_PROCESS setting (default: own, main): unknown
• Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
• Operating system: Win11 WSL running Debian

[hubschrauber]

It would help to have an example of the response JSON for a page. In the meantime, here’s a working example of pagination against a simple test service. Maybe you can compare how it is working to what you are doing and make some progress.

[TFSho]

Yeah, I’ve seen that already a few times in other posts and it’s not really applicable to my situation. It’s not modifying a body param, or modifying a nested param, so doesn’t really help. The API is for an app called JobTread, which uses a weird bespoke query language called Pave. Anywho, here is some more details:

Sample Request Body - First Page

{
    "query": {
        "$": {
            "grantKey": "***"
        },
        "job": {
            "$": {
                "id": "22Nj585iGrEU"
            },
            "id": {},
            "files": {
                "$": {
                    "with": {
                        "ft": {
                            "_": "fileTags",
                            "values": {
                                "$": {
                                    "field": "name"
                                }
                            }
                        }
                    },
                    "where": [
                        [
                            "ft",
                            "values"
                        ],
                        "in",
                        [
                            "Photos-After",
                            "Photos-Before",
                            "Photos-Progress"
                        ]
                    ],
                    "page": "",
                    "size": 10,
                    "download": true
                },
                "nodes": {
                    "id": {},
                    "name": {},
                    "url": {
                        "$": {
                            "download": true
                        }
                    },
                    "type": {},
                    "size": {},
                    "fileTags": {
                        "$": {
                            "where": [
                                "name",
                                "like",
                                "Photos%"
                            ]
                        },
                        "nodes": {
                            "name": {},
                            "id": {}
                        }
                    }
                },
                "nextPage": {}
            },
            "location": {
                "account": {
                    "name": {}
                }
            },
            "projectedPriceWithTax": {}
        }
    }
}

Sample Response Body - First Page

{
  "job": {
    "id": "22Nj585iGrEU",
    "files": {
      "nodes": [
        {
          "id": "22P2CtjpdJKX",
          "name": "IMG_3036.jpeg",
          "url": "https://cdn.jobtread.com/GzwA-GVhXurLl6scGgjYgOPEOABqPFEoY5XxQgISNGOyZUCgBQlG4xQVUPJW5tR2Wxie6eohTdAc1kx5CT8.llZpQump-AChfzz9GSwtVOoIVT54hNyP-ZCWXvDWujM?download",
          "type": "image/jpeg",
          "size": 2686297,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-Progress",
                "id": "22P6njvyNqFP"
              }
            ]
          }
        },
        {
          "id": "22P2yREx5bYP",
          "name": "IMG_3322.jpeg",
          "url": "https://cdn.jobtread.com/Cx6AeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQMnlSN2hHWWtLIiwibiI6IklNR18zMzIyLmpwZWcifQM.MRN8TMT2HMCNo1fn_9HL4HPgVOwsYNJjs0wMvg6Dx1s?download",
          "type": "image/jpeg",
          "size": 2708247,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-Progress",
                "id": "22P6njvyNqFP"
              }
            ]
          }
        },
        {
          "id": "22P2yREyKPU8",
          "name": "IMG_3325.jpeg",
          "url": "https://cdn.jobtread.com/GzwA-GVhXakvn9ysbrwAG3CcGAdAjScKZawyvlCBBNNe3TIgMIMEo3GKiqh3olR05srYnu2bOU1TLDYdbQ9_.rZnOKsD_4haG7viGqAgW-2XtnXJ728NnT05TgfC4VXA?download",
          "type": "image/jpeg",
          "size": 2442311,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-Progress",
                "id": "22P6njvyNqFP"
              }
            ]
          }
        },
        {
          "id": "22P3mQuHgsem",
          "name": "IMG_7336.jpeg",
          "url": "https://cdn.jobtread.com/Cx6AeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQM2p5Q1ZDUHRaIiwibiI6IklNR183MzM2LmpwZWcifQM.KFd4Q4c1xQGsX2suti0UCl-NBKWSLrH8RpGCQn42njI?download",
          "type": "image/jpeg",
          "size": 4233234,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-Progress",
                "id": "22P6njvyNqFP"
              }
            ]
          }
        },
        {
          "id": "22P3mQuJLKy7",
          "name": "IMG_3480.jpeg",
          "url": "https://cdn.jobtread.com/Cx6AeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQM2p5Q1ZDdFl4IiwibiI6IklNR18zNDgwLmpwZWcifQM.AqFwYHLl_uOPvjy2EyiRBoSK4NCNdZX6odXvK7naiiY?download",
          "type": "image/jpeg",
          "size": 2901426,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-Progress",
                "id": "22P6njvyNqFP"
              }
            ]
          }
        },
        {
          "id": "22P3mQuJihrL",
          "name": "IMG_3481.jpeg",
          "url": "https://cdn.jobtread.com/GzwA-MWAdaZ--eVQhJAgbrABx4looAHQNwplrFKR8yGFhnZKdwsCHTSETXNcwA_hds1ysMBY09UjXlBeYlHhDw.oox9r2NGAHWyTJl3e1KBsXbE_225JF-e3RQ56gH13JQ?download",
          "type": "image/jpeg",
          "size": 2162868,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-Progress",
                "id": "22P6njvyNqFP"
              }
            ]
          }
        },
        {
          "id": "22P5nHpNEHkb",
          "name": "010- 123 Anywhere St.jpg",
          "url": "https://cdn.jobtread.com/iyWAeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQNW5IcDJ4M3ZCIiwibiI6IjAxMC0gMTU3NSBOZXdwb3J0IExvb3AgTi5qcGcifQM.Mk-9sT37awMVP_xO3DSUmrijWfRbTnLjj9mRLTOFGK0?download",
          "type": "image/jpeg",
          "size": 1286695,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-After",
                "id": "22NnASTxtU35"
              }
            ]
          }
        },
        {
          "id": "22P5nHppuwPV",
          "name": "011- 123 Anywhere St.jpg",
          "url": "https://cdn.jobtread.com/iyWAeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQNW5IcDJ5QWM0IiwibiI6IjAxMS0gMTU3NSBOZXdwb3J0IExvb3AgTi5qcGcifQM.FaIEvzpbNEc6vN0PXrrHYH_Jy2YnyldSRghub0tzp3I?download",
          "type": "image/jpeg",
          "size": 3266668,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-After",
                "id": "22NnASTxtU35"
              }
            ]
          }
        },
        {
          "id": "22P5nHpqSnY4",
          "name": "006- 123 Anywhere St.jpg",
          "url": "https://cdn.jobtread.com/iyWAeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQNW5IcFBDYTdOIiwibiI6IjAwNi0gMTU3NSBOZXdwb3J0IExvb3AgTi5qcGcifQM.d_SfvJmqJQcJAAux8WUlgJGK1OlHShl39Hfr4biHSQE?download",
          "type": "image/jpeg",
          "size": 1633581,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-After",
                "id": "22NnASTxtU35"
              }
            ]
          }
        },
        {
          "id": "22P5nHpvGyMP",
          "name": "009- 123 Anywhere St.jpg",
          "url": "https://cdn.jobtread.com/iyWAeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQNW5IcDJzR1ZLIiwibiI6IjAwOS0gMTU3NSBOZXdwb3J0IExvb3AgTi5qcGcifQM.g2-7RlUw9UPRnom-LnBmQWX_Otu6WpqEBc3Hbv7I_80?download",
          "type": "image/jpeg",
          "size": 3449709,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-After",
                "id": "22NnASTxtU35"
              }
            ]
          }
        }
      ],
      "nextPage": "2RWnPThT6HwS27CS8eaNx77"
    },
    "location": {
      "account": {
        "name": "John Smith"
      }
    },
    "projectedPriceWithTax": 51286.47
  }
}

So at this point in the process, everything seems simple, right? Just pipe $response.body.job.files.nextPage into query.job.files.$.page, set a Completion Expression so that when $response…nextPage returns null, we exit. and I feel I’ve configured that correctly here:

So what we should be seeing is:

Expected Request Body - Second Page

{
    "query": {
        "$": {
            "grantKey": "***"
        },
        "job": {
            "$": {
                "id": "22Nj585iGrEU"
            },
            "id": {},
            "files": {
                "$": {
                    "with": {
                        "ft": {
                            "_": "fileTags",
                            "values": {
                                "$": {
                                    "field": "name"
                                }
                            }
                        }
                    },
                    "where": [
                        [
                            "ft",
                            "values"
                        ],
                        "in",
                        [
                            "Photos-After",
                            "Photos-Before",
                            "Photos-Progress"
                        ]
                    ],
                    "page": "2RWnPThT6HwS27CS8eaNx77",
                    "size": 10,
                    "download": true
                },
                "nodes": {
                    "id": {},
                    "name": {},
                    "url": {
                        "$": {
                            "download": true
                        }
                    },
                    "type": {},
                    "size": {},
                    "fileTags": {
                        "$": {
                            "where": [
                                "name",
                                "like",
                                "Photos%"
                            ]
                        },
                        "nodes": {
                            "name": {},
                            "id": {}
                        }
                    }
                },
                "nextPage": {}
            },
            "location": {
                "account": {
                    "name": {}
                }
            },
            "projectedPriceWithTax": {}
        }
    }
}

Expected Response Body - Second Page

{
  "job": {
    "id": "22Nj585iGrEU",
    "files": {
      "nodes": [
        {
          "id": "22P5nHpvUbWy",
          "name": "013- 123 Anywhere St.jpg",
          "url": "https://cdn.jobtread.com/iyWAeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQNW5IcDJ4NHM4IiwibiI6IjAxMy0gMTU3NSBOZXdwb3J0IExvb3AgTi5qcGcifQM.I7aMeQ3IUYcamDeZIN8i5HjW2XwqtmNMYI7i2GyMXPQ?download",
          "type": "image/jpeg",
          "size": 3489364,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-After",
                "id": "22NnASTxtU35"
              }
            ]
          }
        },
        {
          "id": "22P5nHpvxn4n",
          "name": "004- 123 Anywhere St.jpg",
          "url": "https://cdn.jobtread.com/iyWAeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQNW5IcDMzZFh5IiwibiI6IjAwNC0gMTU3NSBOZXdwb3J0IExvb3AgTi5qcGcifQM.gjUku6Qd6OX07d5zMIyz622tm6PPpqEEP7_hSigAJuQ?download",
          "type": "image/jpeg",
          "size": 3666713,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-After",
                "id": "22NnASTxtU35"
              }
            ]
          }
        },
        {
          "id": "22P5nHpwqZNX",
          "name": "012- 123 Anywhere St.jpg",
          "url": "https://cdn.jobtread.com/iyWAeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQNW5IcDJ6YzlwIiwibiI6IjAxMi0gMTU3NSBOZXdwb3J0IExvb3AgTi5qcGcifQM.3qcUzEdaJqUe8jChP7gc4sNdtcnDw0YK7LJaf0FwrtY?download",
          "type": "image/jpeg",
          "size": 3739627,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-After",
                "id": "22NnASTxtU35"
              }
            ]
          }
        },
        {
          "id": "22P5nHpxJrZQ",
          "name": "002- 123 Anywhere St.jpg",
          "url": "https://cdn.jobtread.com/iyWAeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQNW5IcDMyMzdXIiwibiI6IjAwMi0gMTU3NSBOZXdwb3J0IExvb3AgTi5qcGcifQM.WvYneCbNQ9kXmvKOl5XJS-uBZJYJYbA7cj9IYsQBJ5A?download",
          "type": "image/jpeg",
          "size": 3576067,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-After",
                "id": "22NnASTxtU35"
              }
            ]
          }
        },
        {
          "id": "22P5nHpxSGW3",
          "name": "005- 123 Anywhere St.jpg",
          "url": "https://cdn.jobtread.com/iyWAeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQNW5IcDMycXBoIiwibiI6IjAwNS0gMTU3NSBOZXdwb3J0IExvb3AgTi5qcGcifQM.k6mceOHXdtaCeHxfdaiEm2zh8w2IO_bnU9cIePtF5rw?download",
          "type": "image/jpeg",
          "size": 3819460,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-After",
                "id": "22NnASTxtU35"
              }
            ]
          }
        },
        {
          "id": "22P5nHpy9hNc",
          "name": "003- 123 Anywhere St.jpg",
          "url": "https://cdn.jobtread.com/iyWAeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQNW5IcDMydGpQIiwibiI6IjAwMy0gMTU3NSBOZXdwb3J0IExvb3AgTi5qcGcifQM.EBiVrLSpQXJQEZTcXn9smjWUvxgjO1tyt5H_UN7O1XE?download",
          "type": "image/jpeg",
          "size": 3732452,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-After",
                "id": "22NnASTxtU35"
              }
            ]
          }
        },
        {
          "id": "22P5nHpzu2sh",
          "name": "001- 123 Anywhere St.jpg",
          "url": "https://cdn.jobtread.com/G0sAgMSoa6Q3y5cfBfRUoFEFG3A8qL690EAaOA7Z80WU3uBFAg_NmGyuB0VLsqHQtMgc_Em791jQwFIqRpRpmyTIz6b8TF8wyP4.ADU3Ka_hoW2CrR1Q44Kj1_fDNEEObk59NpnkQjNS_Ic?download",
          "type": "image/jpeg",
          "size": 4259097,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-After",
                "id": "22NnASTxtU35"
              }
            ]
          }
        },
        {
          "id": "22P5nHq39GYz",
          "name": "007- 123 Anywhere St.jpg",
          "url": "https://cdn.jobtread.com/iyWAeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQNW5IcHFXZHkyIiwibiI6IjAwNy0gMTU3NSBOZXdwb3J0IExvb3AgTi5qcGcifQM.g2PMZW4Cb4qZfP0bPlvjWPlmMW-HdRNENCMDEIEFK-I?download",
          "type": "image/jpeg",
          "size": 1637358,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-After",
                "id": "22NnASTxtU35"
              }
            ]
          }
        },
        {
          "id": "22P5nHq6X5Um",
          "name": "008- 123 Anywhere St.jpg",
          "url": "https://cdn.jobtread.com/iyWAeyJiIjoiam9idHJlYWQiLCJrIjoiZmlsZXMvMjJQNW5IcHJCQk11IiwibiI6IjAwOC0gMTU3NSBOZXdwb3J0IExvb3AgTi5qcGcifQM.IQFNFh0ZrerRSkBlWBNftlBlb4eoUkGzl3u4JmxGQgc?download",
          "type": "image/jpeg",
          "size": 3673120,
          "fileTags": {
            "nodes": [
              {
                "name": "Photos-After",
                "id": "22NnASTxtU35"
              }
            ]
          }
        }
      ],
      "nextPage": null
    },
    "location": {
      "account": {
        "name": "John Smith"
      }
    },
    "projectedPriceWithTax": 51286.47
  }
}

Conclusion

Alas, as stated in my OP, I’m not getting the second request or response. It is simply repeating the first request. My working theory at this point is that the pagination parameter in question doesn’t support/isn’t properly reading the dot notation and it’s just not finding the right field to map.

[hubschrauber]

Just a hunch, but trying simple things first… For the pagination -> parameters-> name specify using the alternative JS “map” syntax where the $ appears, like query.job["$"].name

[TFSho]

Yeah I tried query.job.files[“$”].page and a few other variations. None took. Did some other experimenting, if I specify just “query” for the name parameter, it will replace the query parameter (thus giving me an invalid query response for the second request), but as soon as I try to tack on query.job or query[“job”] or any other nested key, it just starts firing off duplicate requests again, not replacing any values anywhere in the subsequent requests… What this tells me is the node is likely looking only in the top level parameters for a key matching our input name value, and when it finds no match because we’ve got some subkey appended to it, it’s simply doing nothing. I’m pretty convinced at this point that this node simply doesn’t support deeply nested body parameters in the pagination feature.

[TFSho]

Current workaround, for anyone who needs it:

[TFSho]

Bumping post