I would like to retrieve all records of a certain API-Endpoint. However, that Endpoint only allows me to fetch a maximum of 100 records. Is there a best practice to get all records, i.e. follow the pagination (which may or may not be according to a pagination standard)
My goal is to run them all through a reformatter and then store the endresult in an Excel Spreadsheet.
That should be no problem. You can simply create a loop (so create a connection back to an earlier node). Just make sure that you build a node which breakes the loop else it will keep on going forever.
Here a simple example:
Although the general idea of a loop is helpful, in the case of working with the Square API, I needed to do an initial fetch and then to filter based on whether the response JSON returned a cursor and use the cursor for the rest of the HTTP requests. This is a working example of that.
{
"nodes": [
{
"parameters": {
"functionCode": "return items.filter(i => i.json.cursor)"
},
"name": "Filter Cursor",
"type": "n8n-nodes-base.function",
"typeVersion": 1,
"position": [
630,
370
]
},
{
"parameters": {},
"name": "Merge",
"type": "n8n-nodes-base.merge",
"typeVersion": 1,
"position": [
930,
320
]
},
{
"parameters": {
"authentication": "oAuth2",
"url": "https://connect.squareup.com/v2/catalog/list",
"options": {},
"headerParametersUi": {
"parameter": [
{
"name": "Content-type",
"value": "application/json"
}
]
},
"queryParametersUi": {
"parameter": [
{
"name": "types",
"value": "ITEM,ITEM_VARIATION,CATEGORY,IMAGE,ITEM_OPTION,ITEM_OPTION_VAL,DISCOUNT,TAX,MODIFIER,MODIFIER_LIST"
}
]
}
},
"name": "Initial Fetch",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 1,
"position": [
450,
300
],
"credentials": {
"oAuth2Api": "Square"
}
},
{
"parameters": {
"authentication": "oAuth2",
"url": "https://connect.squareup.com/v2/catalog/list",
"options": {},
"headerParametersUi": {
"parameter": [
{
"name": "Content-type",
"value": "application/json"
}
]
},
"queryParametersUi": {
"parameter": [
{
"name": "types",
"value": "ITEM,ITEM_VARIATION,CATEGORY,IMAGE,ITEM_OPTION,ITEM_OPTION_VAL,DISCOUNT,TAX,MODIFIER,MODIFIER_LIST"
},
{
"name": "cursor",
"value": "={{$json.cursor}}"
}
]
}
},
"name": "Fetch Page",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 1,
"position": [
760,
560
],
"executeOnce": true,
"credentials": {
"oAuth2Api": "Square"
}
},
{
"parameters": {},
"name": "Output",
"type": "n8n-nodes-base.noOp",
"typeVersion": 1,
"position": [
1110,
320
]
}
],
"connections": {
"Filter Cursor": {
"main": [
[
{
"node": "Fetch Page",
"type": "main",
"index": 0
}
]
]
},
"Merge": {
"main": [
[
{
"node": "Output",
"type": "main",
"index": 0
}
]
]
},
"Initial Fetch": {
"main": [
[
{
"node": "Merge",
"type": "main",
"index": 0
},
{
"node": "Filter Cursor",
"type": "main",
"index": 0
}
]
]
},
"Fetch Page": {
"main": [
[
{
"node": "Merge",
"type": "main",
"index": 1
},
{
"node": "Filter Cursor",
"type": "main",
"index": 0
}
]
]
}
}
}I’ve come up with an even better solution as I’m learning the ins and outs of n8n. This one only requires a single HTTP node, and IF node and a function node to combine the results.
{
"nodes": [
{
"parameters": {
"authentication": "oAuth2",
"url": "https://connect.squareup.com/v2/catalog/list",
"options": {},
"headerParametersUi": {
"parameter": [
{
"name": "Content-type",
"value": "application/json"
}
]
},
"queryParametersUi": {
"parameter": [
{
"name": "types",
"value": "ITEM,ITEM_VARIATION,CATEGORY,IMAGE,ITEM_OPTION,ITEM_OPTION_VAL,DISCOUNT,TAX,MODIFIER,MODIFIER_LIST"
},
{
"name": "cursor",
"value": "={{$json.cursor || \"\"}}"
}
]
}
},
"name": "Fetch Page",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 1,
"position": [
150,
300
],
"executeOnce": true,
"credentials": {
"oAuth2Api": "Square"
}
},
{
"parameters": {
"functionCode": "const allItems = []\nlet counter = 0\n\nwhile (true) {\n try {\n $items(\"Fetch Page\", 0, counter).map(i => i.json.objects.map(o => allItems.push(o)))\n } catch (err) {\n break\n }\n counter++\n}\n\nreturn allItems.map(o => ({json: o}))\n\n"
},
"name": "Combine Results",
"type": "n8n-nodes-base.function",
"typeVersion": 1,
"position": [
350,
300
]
},
{
"parameters": {
"conditions": {
"string": [
{
"value1": "={{$json.cursor || \"\"}}"
}
]
}
},
"name": "Empty Cursor",
"type": "n8n-nodes-base.if",
"typeVersion": 1,
"position": [
150,
120
]
}
],
"connections": {
"Fetch Page": {
"main": [
[
{
"node": "Empty Cursor",
"type": "main",
"index": 0
}
]
]
},
"Empty Cursor": {
"main": [
[
{
"node": "Combine Results",
"type": "main",
"index": 0
}
],
[
{
"node": "Fetch Page",
"type": "main",
"index": 0
}
]
]
}
}
}Thanks a lot for sharing! Is really very appreciated!
Just had a short look at both solutions. The second one looks great and is what I would also suggest!
About the first one. Did that work if there are more than 2 pages? Because I would have assumed that it would not.
Yes, that’s actually the issue that the second one addressed - I probably should have mentioned that! 