Merging items after "split in batches"

Hi

I have a function that outputs a list of locations, which I use to look up the weather. All the data is supposed to be gathered and run through a template for rendering before going to Slack.

I have a setup like this:

How can I gather all the split data items into a single item?

That can be done like this:

{
  "nodes": [
    {
      "parameters": {
        "conditions": {
          "number": [
            {
              "value1": "={{$node[\"Increment\"].data[\"counter\"]}}",
              "value2": 5
            }
          ]
        }
      },
      "name": "IF",
      "type": "n8n-nodes-base.if",
      "typeVersion": 1,
      "position": [
        900,
        550
      ]
    },
    {
      "parameters": {},
      "name": "Start Loop",
      "type": "n8n-nodes-base.noOp",
      "typeVersion": 1,
      "position": [
        500,
        350
      ]
    },
    {
      "parameters": {
        "values": {
          "number": [
            {
              "name": "counter",
              "value": "={{$node[\"Start Loop\"].json[\"counter\"]+1}}"
            }
          ]
        },
        "options": {}
      },
      "name": "Increment",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        700,
        350
      ]
    },
    {
      "parameters": {
        "values": {
          "number": [
            {
              "name": "counter"
            }
          ]
        },
        "options": {}
      },
      "name": "Init Counter",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        300,
        350
      ]
    },
    {
      "parameters": {
        "functionCode": "const allData = []\n\nlet counter = 0;\ndo {\n  try {\n    const items = $items(\"Increment\", 0, counter).map(item => item.json);\n    allData.push.apply(allData, items);\n  } catch (error) {\n    return [{json: {allData}}];  \n  }\n\n  counter++;\n} while(true);\n\n\n"
      },
      "name": "Merge Data",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        1100,
        550
      ]
    }
  ],
  "connections": {
    "IF": {
      "main": [
        [
          {
            "node": "Start Loop",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Merge Data",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Start Loop": {
      "main": [
        [
          {
            "node": "Increment",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Increment": {
      "main": [
        [
          {
            "node": "IF",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Init Counter": {
      "main": [
        [
          {
            "node": "Start Loop",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}
2 Likes

Sweet, thanks!

1 Like

This solution seems kind of arbitrary, as it relies on a known up-front counter, but how do we handle paginated urls, when you don’t know the total number of pages?

Welcome to the community @philipsd6!

That is not correct. The solution does not expect that. It works no matter how many iterations there were.

OK! I posted a new topic for the situation I’m seeing – hope to hear from you there.

As @jan said, it will iterate over the result set until it gets an exception and then just break the iteration and return the result.

If you look at the code it becomes a bit easier to see:

const allData = []

let counter = 0;
do {
  try {
    const items = $items("Increment", 0, counter).map(item => item.json);
    allData.push.apply(allData, items);
  } catch (error) {
    return [{json: {allData}}];  
  }

  counter++;
} while(true);
1 Like

Yes, I see I was off-base in my comment, as it does iterate properly – however, I have noted problems, so I have a new topic opened for my problem – thanks!