Best-pratice for HTTP-Request that include pagination

I would like to retrieve all records of a certain API-Endpoint. However, that Endpoint only allows me to fetch a maximum of 100 records. Is there a best practice to get all records, i.e. follow the pagination (which may or may not be according to a pagination standard)

My goal is to run them all through a reformatter and then store the endresult in an Excel Spreadsheet.

1 Like

That should be no problem. You can simply create a loop (so create a connection back to an earlier node). Just make sure that you build a node which breakes the loop else it will keep on going forever.

Here a simple example:

1 Like

Although the general idea of a loop is helpful, in the case of working with the Square API, I needed to do an initial fetch and then to filter based on whether the response JSON returned a cursor and use the cursor for the rest of the HTTP requests. This is a working example of that.

{
    "nodes": [
        {
            "parameters": {
                "functionCode": "return items.filter(i => i.json.cursor)"
            },
            "name": "Filter Cursor",
            "type": "n8n-nodes-base.function",
            "typeVersion": 1,
            "position": [
                630,
                370
            ]
        },
        {
            "parameters": {},
            "name": "Merge",
            "type": "n8n-nodes-base.merge",
            "typeVersion": 1,
            "position": [
                930,
                320
            ]
        },
        {
            "parameters": {
                "authentication": "oAuth2",
                "url": "https://connect.squareup.com/v2/catalog/list",
                "options": {},
                "headerParametersUi": {
                    "parameter": [
                        {
                            "name": "Content-type",
                            "value": "application/json"
                        }
                    ]
                },
                "queryParametersUi": {
                    "parameter": [
                        {
                            "name": "types",
                            "value": "ITEM,ITEM_VARIATION,CATEGORY,IMAGE,ITEM_OPTION,ITEM_OPTION_VAL,DISCOUNT,TAX,MODIFIER,MODIFIER_LIST"
                        }
                    ]
                }
            },
            "name": "Initial Fetch",
            "type": "n8n-nodes-base.httpRequest",
            "typeVersion": 1,
            "position": [
                450,
                300
            ],
            "credentials": {
                "oAuth2Api": "Square"
            }
        },
        {
            "parameters": {
                "authentication": "oAuth2",
                "url": "https://connect.squareup.com/v2/catalog/list",
                "options": {},
                "headerParametersUi": {
                    "parameter": [
                        {
                            "name": "Content-type",
                            "value": "application/json"
                        }
                    ]
                },
                "queryParametersUi": {
                    "parameter": [
                        {
                            "name": "types",
                            "value": "ITEM,ITEM_VARIATION,CATEGORY,IMAGE,ITEM_OPTION,ITEM_OPTION_VAL,DISCOUNT,TAX,MODIFIER,MODIFIER_LIST"
                        },
                        {
                            "name": "cursor",
                            "value": "={{$json.cursor}}"
                        }
                    ]
                }
            },
            "name": "Fetch Page",
            "type": "n8n-nodes-base.httpRequest",
            "typeVersion": 1,
            "position": [
                760,
                560
            ],
            "executeOnce": true,
            "credentials": {
                "oAuth2Api": "Square"
            }
        },
        {
            "parameters": {},
            "name": "Output",
            "type": "n8n-nodes-base.noOp",
            "typeVersion": 1,
            "position": [
                1110,
                320
            ]
        }
    ],
    "connections": {
        "Filter Cursor": {
            "main": [
                [
                    {
                        "node": "Fetch Page",
                        "type": "main",
                        "index": 0
                    }
                ]
            ]
        },
        "Merge": {
            "main": [
                [
                    {
                        "node": "Output",
                        "type": "main",
                        "index": 0
                    }
                ]
            ]
        },
        "Initial Fetch": {
            "main": [
                [
                    {
                        "node": "Merge",
                        "type": "main",
                        "index": 0
                    },
                    {
                        "node": "Filter Cursor",
                        "type": "main",
                        "index": 0
                    }
                ]
            ]
        },
        "Fetch Page": {
            "main": [
                [
                    {
                        "node": "Merge",
                        "type": "main",
                        "index": 1
                    },
                    {
                        "node": "Filter Cursor",
                        "type": "main",
                        "index": 0
                    }
                ]
            ]
        }
    }
}
1 Like

I’ve come up with an even better solution as I’m learning the ins and outs of n8n. This one only requires a single HTTP node, and IF node and a function node to combine the results.

{
    "nodes": [
        {
            "parameters": {
                "authentication": "oAuth2",
                "url": "https://connect.squareup.com/v2/catalog/list",
                "options": {},
                "headerParametersUi": {
                    "parameter": [
                        {
                            "name": "Content-type",
                            "value": "application/json"
                        }
                    ]
                },
                "queryParametersUi": {
                    "parameter": [
                        {
                            "name": "types",
                            "value": "ITEM,ITEM_VARIATION,CATEGORY,IMAGE,ITEM_OPTION,ITEM_OPTION_VAL,DISCOUNT,TAX,MODIFIER,MODIFIER_LIST"
                        },
                        {
                            "name": "cursor",
                            "value": "={{$json.cursor || \"\"}}"
                        }
                    ]
                }
            },
            "name": "Fetch Page",
            "type": "n8n-nodes-base.httpRequest",
            "typeVersion": 1,
            "position": [
                150,
                300
            ],
            "executeOnce": true,
            "credentials": {
                "oAuth2Api": "Square"
            }
        },
        {
            "parameters": {
                "functionCode": "const allItems = []\nlet counter = 0\n\nwhile (true) {\n  try {\n    $items(\"Fetch Page\", 0, counter).map(i => i.json.objects.map(o => allItems.push(o)))\n  } catch (err) {\n    break\n  }\n  counter++\n}\n\nreturn allItems.map(o => ({json: o}))\n\n"
            },
            "name": "Combine Results",
            "type": "n8n-nodes-base.function",
            "typeVersion": 1,
            "position": [
                350,
                300
            ]
        },
        {
            "parameters": {
                "conditions": {
                    "string": [
                        {
                            "value1": "={{$json.cursor || \"\"}}"
                        }
                    ]
                }
            },
            "name": "Empty Cursor",
            "type": "n8n-nodes-base.if",
            "typeVersion": 1,
            "position": [
                150,
                120
            ]
        }
    ],
    "connections": {
        "Fetch Page": {
            "main": [
                [
                    {
                        "node": "Empty Cursor",
                        "type": "main",
                        "index": 0
                    }
                ]
            ]
        },
        "Empty Cursor": {
            "main": [
                [
                    {
                        "node": "Combine Results",
                        "type": "main",
                        "index": 0
                    }
                ],
                [
                    {
                        "node": "Fetch Page",
                        "type": "main",
                        "index": 0
                    }
                ]
            ]
        }
    }
}
3 Likes

Thanks a lot for sharing! Is really very appreciated!

Just had a short look at both solutions. The second one looks great and is what I would also suggest!

About the first one. Did that work if there are more than 2 pages? Because I would have assumed that it would not.

Yes, that’s actually the issue that the second one addressed - I probably should have mentioned that! :grinning:

2 Likes