How to combine result list from batch api calls

Questions
1

Describe the problem/error/question

I am batching my calls to Yelp API. Currently pulling down batches of 5 results to test. It returns a list of businesses. How do I append each list to a “final” list that is composed of all the businesses from every call made?

Here is what the result looks like:

[
  {
    "businesses": [I want to capture list in here],
    "total": 80,
    "region": {
      "center": {
        "longitude": -122.419418,
        "latitude": 37.774929
      }
    }
  }
]

Please share your workflow

Here is the workflow. I tried using the Aggregate Node, but doesn’t seem like i’m using it correctly.

Share the output returned by the last node

Information on your n8n setup

  • n8n version: 1.59.4
  • Database (default: SQLite): Postgres
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
2

Here is an example, how you can combine the outputs of multiple iterations.
It uses the append function within the last set node if the loop.
For your purpose you should replace the “Retrieve Data” node I put here for generating dummy data with your HTTP request.
You can also define the batch size dynamically.

3

I updated workflow code to use your append example. The problem is I cant see what the array of items contain. I had to set Ignore Type Conversion Errors on, because I get this error: 'final_results' expects a array but we got ' [object Object],[object Object],[object Object],[object Object],[object Object]' [item 0].

But even after turning that on, this is all I see:

image
image982×106 4.39 KB

Updated Workflow Code

4

Could you please pin the data of the HTTP Node? Otherwise it is hard for me to see the issue.
You can also anonymize the data by editing the values of the pinned data. I just need to see the exact data structure it outputs.

5

ok sorry! I have pinned data here

  "meta": {
    "instanceId": "97b1ddf8d2283fc2c25cd0cefc51f69cc0778bf133e8e799939bef35e501ed4f"
  },
  "nodes": [
    {
      "parameters": {},
      "id": "5b1a87e4-7256-47e0-b653-3c5f49d44113",
      "name": "When clicking ‘Test workflow’",
      "type": "n8n-nodes-base.manualTrigger",
      "typeVersion": 1,
      "position": [
        80,
        400
      ]
    },
    {
      "parameters": {},
      "id": "48580445-b58a-4d74-86cb-0486506da9af",
      "name": "Wait",
      "type": "n8n-nodes-base.wait",
      "typeVersion": 1.1,
      "position": [
        1060,
        700
      ],
      "webhookId": "1599c241-e6f4-4613-8115-7971a2ee298a"
    },
    {
      "parameters": {
        "assignments": {
          "assignments": [
            {
              "id": "f692859f-7a1d-4ad5-96b2-8f2cc4c51eb3",
              "name": "offset",
              "value": "={{$json[\"offset\"] || 0}}",
              "type": "number"
            },
            {
              "id": "f922e22a-8473-499d-b968-2ddce86a9a91",
              "name": "limit",
              "value": 5,
              "type": "number"
            },
            {
              "id": "6d9791c0-18b4-4909-b1fd-d4587f771a7b",
              "name": "total_results",
              "value": "={{$json[\"total_results\"] || -1}}",
              "type": "number"
            },
            {
              "id": "6cbf7f2e-920b-4248-afa9-52f83364f723",
              "name": "current_count",
              "value": "={{$json[\"current_count\"] || -1}}",
              "type": "number"
            },
            {
              "id": "b54f6d1b-7018-4354-a85e-11969e1b1738",
              "name": "final_results",
              "value": "={{$json[\"final_results\"] || []}}",
              "type": "array"
            }
          ]
        },
        "options": {}
      },
      "id": "6711d587-442d-4c48-bb8f-35bba5e56b04",
      "name": "Create vars",
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.4,
      "position": [
        400,
        400
      ]
    },
    {
      "parameters": {
        "url": "https://api.yelp.com/v3/businesses/search",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "sendQuery": true,
        "queryParameters": {
          "parameters": [
            {
              "name": "latitude",
              "value": "37.774929"
            },
            {
              "name": "longitude",
              "value": "-122.419418"
            },
            {
              "name": "sort_by",
              "value": "best_match"
            },
            {
              "name": "limit",
              "value": "={{ $json.limit }}"
            },
            {
              "name": "radius",
              "value": "1"
            },
            {
              "name": "offset",
              "value": "={{ $json.offset }}"
            }
          ]
        },
        "options": {}
      },
      "id": "ac66b5e7-7355-4d1b-9bdf-8179b82b1f95",
      "name": "Yelp Get Businesses (Batch)",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [
        820,
        380
      ],
      "credentials": {
        "httpHeaderAuth": {
          "id": "8cvVEt2kfwutfMXE",
          "name": "Yelp Account Key"
        }
      }
    },
    {
      "parameters": {
        "assignments": {
          "assignments": [
            {
              "id": "f692859f-7a1d-4ad5-96b2-8f2cc4c51eb3",
              "name": "offset",
              "value": "={{ $('Create vars').item.json.offset + 50 }}",
              "type": "number"
            },
            {
              "id": "6d9791c0-18b4-4909-b1fd-d4587f771a7b",
              "name": "total_results",
              "value": "={{ $('Yelp Get Businesses (Batch)').item.json.total }}",
              "type": "number"
            },
            {
              "id": "6cbf7f2e-920b-4248-afa9-52f83364f723",
              "name": "current_count",
              "value": "={{ $('Create vars').item.json.current_count +  $('Yelp Get Businesses (Batch)').item.json.businesses.length}}",
              "type": "number"
            },
            {
              "id": "9a20ab41-67f0-4f9d-b881-54cce3262866",
              "name": "final_results",
              "value": "={{ $('Append results').item.json.total_results }}",
              "type": "array"
            }
          ]
        },
        "options": {
          "ignoreConversionErrors": true
        }
      },
      "id": "a3fa0c6b-277a-4d2a-9453-12a329457d2f",
      "name": "Increment",
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.4,
      "position": [
        320,
        680
      ]
    },
    {
      "parameters": {
        "conditions": {
          "options": {
            "caseSensitive": true,
            "leftValue": "",
            "typeValidation": "strict",
            "version": 2
          },
          "conditions": [
            {
              "id": "11789349-ea67-4019-851d-8e52b837e32c",
              "leftValue": "={{ $('Create vars').item.json.total_results }}",
              "rightValue": "={{ $('Create vars').item.json.current_count }}",
              "operator": {
                "type": "number",
                "operation": "gte"
              }
            }
          ],
          "combinator": "and"
        },
        "options": {}
      },
      "id": "6818ef73-8bdf-4715-978f-1132cf942af4",
      "name": "More to download",
      "type": "n8n-nodes-base.if",
      "typeVersion": 2.2,
      "position": [
        580,
        400
      ]
    },
    {
      "parameters": {
        "content": "How do I combine the list of results with each API call?",
        "height": 80,
        "width": 311.1043429501349
      },
      "id": "1841e09d-3f32-4a6e-9f0f-3734a4db5737",
      "name": "Sticky Note3",
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        1000,
        300
      ]
    },
    {
      "parameters": {
        "assignments": {
          "assignments": [
            {
              "id": "0bb0167a-290a-4b6d-bb7a-e38ecbea9099",
              "name": "total_results",
              "value": "=  {{ $('Create vars').item.json.final_results.append($('Get Businesses List').item) }}",
              "type": "array"
            }
          ]
        },
        "options": {
          "ignoreConversionErrors": true
        }
      },
      "id": "a3d45ee5-01ef-4066-8013-10900df1a3d2",
      "name": "Append results",
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.4,
      "position": [
        1220,
        420
      ]
    },
    {
      "parameters": {
        "fieldToSplitOut": "businesses",
        "options": {}
      },
      "id": "c999b73d-cef5-4e8a-a556-ea136d29fdd8",
      "name": "Get Businesses List",
      "type": "n8n-nodes-base.splitOut",
      "typeVersion": 1,
      "position": [
        1000,
        400
      ]
    }
  ],
  "connections": {
    "When clicking ‘Test workflow’": {
      "main": [
        [
          {
            "node": "Create vars",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Wait": {
      "main": [
        [
          {
            "node": "Increment",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Create vars": {
      "main": [
        [
          {
            "node": "More to download",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Yelp Get Businesses (Batch)": {
      "main": [
        [
          {
            "node": "Get Businesses List",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Increment": {
      "main": [
        [
          {
            "node": "Create vars",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "More to download": {
      "main": [
        [
          {
            "node": "Yelp Get Businesses (Batch)",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Append results": {
      "main": [
        [
          {
            "node": "Wait",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Get Businesses List": {
      "main": [
        [
          {
            "node": "Append results",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  },
  "pinData": {
    "Yelp Get Businesses (Batch)": [
      {
        "businesses": [
          {
            "id": "ma-IFK7XaRdVwA9lxNyMXA",
            "alias": "señor-sisig-food-trucks-san-francisco-3",
            "name": "Señor Sisig - Food Trucks",
            "image_url": "https://s3-media3.fl.yelpcdn.com/bphoto/eCuzQUt4gyolCvGRZhyebw/o.jpg",
            "is_closed": false,
            "url": "https://www.yelp.com/biz/se%C3%B1or-sisig-food-trucks-san-francisco-3?adjust_creative=ninSRRD9D-e0eVd-igBL9g&utm_campaign=yelp_api_v3&utm_medium=api_v3_business_search&utm_source=ninSRRD9D-e0eVd-igBL9g",
            "review_count": 1704,
            "categories": [
              {
                "alias": "filipino",
                "title": "Filipino"
              },
              {
                "alias": "foodtrucks",
                "title": "Food Trucks"
              },
              {
                "alias": "asianfusion",
                "title": "Asian Fusion"
              }
            ],
            "rating": 4.2,
            "coordinates": {
              "latitude": 37.7749295,
              "longitude": -122.4194155
            },
            "transactions": [
              "pickup"
            ],
            "price": "$$",
            "location": {
              "address1": "",
              "address2": null,
              "address3": "",
              "city": "San Francisco",
              "zip_code": "94104",
              "country": "US",
              "state": "CA",
              "display_address": [
                "San Francisco, CA 94104"
              ]
            },
            "phone": "+18557474455",
            "display_phone": "(855) 747-4455",
            "distance": 0.22665212004090063
          },
          {
            "id": "pgjbSwK_8DH2o_zQsr3dRQ",
            "alias": "ashyans-lu-ruo-fan-san-francisco",
            "name": "Ashyan's Lu Ruo Fan",
            "image_url": "https://s3-media3.fl.yelpcdn.com/bphoto/mCE1w9N2erQCuDQ76Iam6A/o.jpg",
            "is_closed": false,
            "url": "https://www.yelp.com/biz/ashyans-lu-ruo-fan-san-francisco?adjust_creative=ninSRRD9D-e0eVd-igBL9g&utm_campaign=yelp_api_v3&utm_medium=api_v3_business_search&utm_source=ninSRRD9D-e0eVd-igBL9g",
            "review_count": 32,
            "categories": [
              {
                "alias": "taiwanese",
                "title": "Taiwanese"
              },
              {
                "alias": "fooddeliveryservices",
                "title": "Food Delivery Services"
              }
            ],
            "rating": 4.8,
            "coordinates": {
              "latitude": 37.77713,
              "longitude": -122.41964
            },
            "transactions": [],
            "location": {
              "address1": null,
              "address2": null,
              "address3": null,
              "city": "San Francisco",
              "zip_code": "",
              "country": "US",
              "state": "CA",
              "display_address": [
                "San Francisco, CA"
              ]
            },
            "phone": "+15612127559",
            "display_phone": "(561) 212-7559",
            "distance": 0.22665212004090063
          },
          {
            "id": "AHjpGuYlqpnQSNYKu6prOA",
            "alias": "blue-angels-san-francisco",
            "name": "Blue Angels",
            "image_url": "https://s3-media2.fl.yelpcdn.com/bphoto/7zBtOrW0MYAoRyOPGhii0g/o.jpg",
            "is_closed": false,
            "url": "https://www.yelp.com/biz/blue-angels-san-francisco?adjust_creative=ninSRRD9D-e0eVd-igBL9g&utm_campaign=yelp_api_v3&utm_medium=api_v3_business_search&utm_source=ninSRRD9D-e0eVd-igBL9g",
            "review_count": 171,
            "categories": [
              {
                "alias": "localflavor",
                "title": "Local Flavor"
              }
            ],
            "rating": 3.9,
            "coordinates": {
              "latitude": 37.7771186828613,
              "longitude": -122.419639587402
            },
            "transactions": [],
            "location": {
              "address1": "",
              "address2": "",
              "address3": "",
              "city": "San Francisco",
              "zip_code": "",
              "country": "US",
              "state": "CA",
              "display_address": [
                "San Francisco, CA"
              ]
            },
            "phone": "",
            "display_phone": "",
            "distance": 0.22665212004090063
          },
          {
            "id": "mjJynPiJ8SNpaySRs6zb1w",
            "alias": "dahlia-dell-at-golden-gate-park-san-francisco",
            "name": "Dahlia Dell at Golden Gate Park",
            "image_url": "https://s3-media4.fl.yelpcdn.com/bphoto/_NrMRw5qYJvr7EnR1M0_Yg/o.jpg",
            "is_closed": false,
            "url": "https://www.yelp.com/biz/dahlia-dell-at-golden-gate-park-san-francisco?adjust_creative=ninSRRD9D-e0eVd-igBL9g&utm_campaign=yelp_api_v3&utm_medium=api_v3_business_search&utm_source=ninSRRD9D-e0eVd-igBL9g",
            "review_count": 32,
            "categories": [
              {
                "alias": "gardens",
                "title": "Botanical Gardens"
              }
            ],
            "rating": 4.9,
            "coordinates": {
              "latitude": 37.775,
              "longitude": -122.418333
            },
            "transactions": [],
            "location": {
              "address1": null,
              "address2": null,
              "address3": null,
              "city": "San Francisco",
              "zip_code": "94199",
              "country": "US",
              "state": "CA",
              "display_address": [
                "San Francisco, CA 94199"
              ]
            },
            "phone": "+14156667001",
            "display_phone": "(415) 666-7001",
            "distance": 0.22665212004090063
          },
          {
            "id": "_vV3-W4QDgBzi4pGXThBUg",
            "alias": "soul-skillet-san-francisco",
            "name": "Soul Skillet",
            "image_url": "https://s3-media2.fl.yelpcdn.com/bphoto/1e9G5FXOBhlh5Rx81OChrw/o.jpg",
            "is_closed": false,
            "url": "https://www.yelp.com/biz/soul-skillet-san-francisco?adjust_creative=ninSRRD9D-e0eVd-igBL9g&utm_campaign=yelp_api_v3&utm_medium=api_v3_business_search&utm_source=ninSRRD9D-e0eVd-igBL9g",
            "review_count": 7,
            "categories": [
              {
                "alias": "soulfood",
                "title": "Soul Food"
              },
              {
                "alias": "foodstands",
                "title": "Food Stands"
              },
              {
                "alias": "breakfast_brunch",
                "title": "Breakfast & Brunch"
              }
            ],
            "rating": 4.6,
            "coordinates": {
              "latitude": 37.7771186828613,
              "longitude": -122.419639587402
            },
            "transactions": [],
            "location": {
              "address1": "",
              "address2": "",
              "address3": null,
              "city": "San Francisco",
              "zip_code": "34016",
              "country": "US",
              "state": "CA",
              "display_address": [
                "San Francisco, CA 34016"
              ]
            },
            "phone": "",
            "display_phone": "",
            "distance": 0.22665212004090063
          }
        ],
        "total": 78,
        "region": {
          "center": {
            "longitude": -122.419418,
            "latitude": 37.774929
          }
        }
      }
    ]
  }
}
6

You have to split out the results from the request first, so you can append them one by one. Here is the modified example:

7

Apologies, I was trying to append total_results (number) to final_results (array). But when I append the results from split out, it creates and array from all the fields in the set not. So if you take a look at pinned data in Append results, notice the array of final_results. Seems very counter intuitive.

8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.