Please help me Understand the Different in SplitInBatches vs Direct approach

Questions
,
1

Describe the issue/error/question

I have a workflow where I am passing the input through Split in batches, and later on i have to check condition and filter into different branch where i need to process input. But a a later point i need to access the data from SplitInBatches node as the source however I think because there is if node in between the right index was not passed and i am no longer able to access right data. To Simulate the same I tried to create a dummy workflow.

image
image1920×895 176 KB

What is the error message (if any)?

Please share the workflow

  "meta": {
    "instanceId": "e92fd3d667b8e5f1e1efc1d69eba70b044ec59dc92beae8deaaf76559757d827"
  },
  "nodes": [
    {
      "parameters": {},
      "id": "def7f9a1-52e9-48cb-83a4-c7e281ca662e",
      "name": "Start",
      "type": "n8n-nodes-base.start",
      "typeVersion": 1,
      "position": [
        340,
        200
      ]
    },
    {
      "parameters": {
        "functionCode": "return [\n  {json:{\"syncappid\":1867531079,\"data\":\"a\"}},\n  {json:{\"syncappid\":null,\"data\":\"b\"}},\n  {json:{\"syncappid\":1867531079,\"data\":\"c\"}},\n  {json:{\"syncappid\":null,\"data\":\"d\"}},\n  {json:{\"syncappid\":1867531079,\"data\":\"e\"}}\n];"
      },
      "id": "95c3bbb9-6cf0-4a3a-8fd5-c257c96a9fa8",
      "name": "init",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        560,
        200
      ]
    },
    {
      "parameters": {
        "batchSize": 1,
        "options": {}
      },
      "id": "eb891ef7-bcd0-4082-a6d5-a06d628f044f",
      "name": "SplitInBatches",
      "type": "n8n-nodes-base.splitInBatches",
      "typeVersion": 1,
      "position": [
        760,
        660
      ]
    },
    {
      "parameters": {
        "conditions": {
          "boolean": [
            {
              "value1": "={{typeof $json[\"syncappid\"]!=\"undefined\" && $json[\"syncappid\"]!=\"\" && $json[\"syncappid\"]!=null}}",
              "value2": true
            }
          ]
        }
      },
      "id": "a7527d70-a4a4-4346-a71b-da26ffe3801e",
      "name": "IF",
      "type": "n8n-nodes-base.if",
      "typeVersion": 1,
      "position": [
        820,
        200
      ]
    },
    {
      "parameters": {
        "mode": "wait"
      },
      "id": "dcfa4175-6b8c-452d-bf48-76da8af5daf3",
      "name": "Merge",
      "type": "n8n-nodes-base.merge",
      "typeVersion": 1,
      "position": [
        1280,
        200
      ]
    },
    {
      "parameters": {
        "conditions": {
          "boolean": [
            {
              "value1": "={{$node[\"SplitInBatches\"].context[\"noItemsLeft\"]}}",
              "value2": true
            }
          ]
        }
      },
      "id": "4fe2ef45-ccd7-4d4f-ae43-f9326ca2b069",
      "name": "IF1",
      "type": "n8n-nodes-base.if",
      "typeVersion": 1,
      "position": [
        1620,
        740
      ]
    },
    {
      "parameters": {},
      "id": "925b6fc7-442c-4ad0-b738-8d67b13e2177",
      "name": "NoOp",
      "type": "n8n-nodes-base.noOp",
      "typeVersion": 1,
      "position": [
        1840,
        720
      ]
    },
    {
      "parameters": {
        "options": {}
      },
      "id": "b1f38c81-63be-45ff-b8c5-d65c10c1ad05",
      "name": "Set",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        1060,
        60
      ]
    },
    {
      "parameters": {
        "options": {}
      },
      "id": "68ebb6c7-f470-4359-b3c3-589442f6691c",
      "name": "Set1",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        1060,
        340
      ]
    },
    {
      "parameters": {
        "functionCode": "return [\n  {json:{\"syncappid\":1867531079,\"data\":\"a\"}},\n  {json:{\"syncappid\":null,\"data\":\"b\"}},\n  {json:{\"syncappid\":1867531079,\"data\":\"c\"}},\n  {json:{\"syncappid\":null,\"data\":\"d\"}},\n  {json:{\"syncappid\":1867531079,\"data\":\"e\"}}\n];"
      },
      "id": "ae863339-401b-4ee7-ba64-862163758dfd",
      "name": "init1",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        520,
        660
      ]
    },
    {
      "parameters": {
        "conditions": {
          "boolean": [
            {
              "value1": "={{typeof $json[\"syncappid\"]!=\"undefined\" && $json[\"syncappid\"]!=\"\" && $json[\"syncappid\"]!=null}}",
              "value2": true
            }
          ]
        }
      },
      "id": "2f3cdb81-1e74-45ae-a88b-9c863154bdf0",
      "name": "IF2",
      "type": "n8n-nodes-base.if",
      "typeVersion": 1,
      "position": [
        980,
        660
      ]
    },
    {
      "parameters": {
        "mode": "wait"
      },
      "id": "fefde033-e84d-4d5f-bcc6-f0c8805dac65",
      "name": "Merge1",
      "type": "n8n-nodes-base.merge",
      "typeVersion": 1,
      "position": [
        1440,
        660
      ]
    },
    {
      "parameters": {
        "options": {}
      },
      "id": "5ee15383-7433-49fe-828c-60b8b0222e74",
      "name": "Set2",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        1220,
        520
      ]
    },
    {
      "parameters": {
        "options": {}
      },
      "id": "db9310b9-8415-4b3c-a26e-7e5e77d6aca5",
      "name": "Set3",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        1220,
        800
      ]
    },
    {
      "parameters": {},
      "id": "2fd2cc83-3231-418f-81c9-ed711f219472",
      "name": "NoOp1",
      "type": "n8n-nodes-base.noOp",
      "typeVersion": 1,
      "position": [
        1580,
        200
      ]
    }
  ],
  "connections": {
    "Start": {
      "main": [
        [
          {
            "node": "init",
            "type": "main",
            "index": 0
          },
          {
            "node": "init1",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "init": {
      "main": [
        [
          {
            "node": "IF",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "SplitInBatches": {
      "main": [
        [
          {
            "node": "IF2",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "IF": {
      "main": [
        [
          {
            "node": "Set",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Set1",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Merge": {
      "main": [
        [
          {
            "node": "NoOp1",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "IF1": {
      "main": [
        [
          {
            "node": "NoOp",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "SplitInBatches",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Set": {
      "main": [
        [
          {
            "node": "Merge",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Set1": {
      "main": [
        [
          {
            "node": "Merge",
            "type": "main",
            "index": 1
          }
        ]
      ]
    },
    "init1": {
      "main": [
        [
          {
            "node": "SplitInBatches",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "IF2": {
      "main": [
        [
          {
            "node": "Set2",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Set3",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Merge1": {
      "main": [
        [
          {
            "node": "IF1",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Set2": {
      "main": [
        [
          {
            "node": "Merge1",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Set3": {
      "main": [
        [
          {
            "node": "Merge1",
            "type": "main",
            "index": 1
          }
        ]
      ]
    }
  }
}

Share the output returned by the last node

Information on your n8n setup

  • n8n version: 0.193.3
  • Database you’re using (default: SQLite): Postgres
  • Running n8n with the execution process [own(default), main]: main
  • Running n8n via [Docker, npm, n8n.cloud, desktop app]: npm
2

Hi @mayanktiwari, this seems like quite a complex workflow and I am not entirely sure what’s happening here.

Is there any chance you can merge the loop runs back into a single run following your loop, similar to what’s happening in this example workflow (though possibly not with the allData used there but as plain items instead)? This should make it much easier to work with your loop data in the later stages of your flow.

You could also consider moving each loop into a sub-workflow using the Execute Workflow node. Breaking down complex flows into smaller chunks will make it much easier to understand what your flow does

2 Likes
3

Actually the screenshot is to highlight the main issue,
However at the end I am merging all branches at the last branch where i check the if noItemsLeft in the SplitInBatches.
The example code i provided demonstrate a simple workflow example of the same where you can notice the issue in simple way.

4

Thanks @mayanktiwari, I can now see your example code. However, it’s not clear to me which issue exactly you’re having. Are you having trouble reading data in you expression {{typeof $json["syncappid"]!="undefined" && $json["syncappid"]!="" && $json["syncappid"]!=null}} used in the IF2 node? If so, can you describe which behaviour exactly you are expecting?

5

Thanks for responding,

I mean if you execute the workflow you will notice that the entries are well processed in first set where no Split in Batches used however in the one with Split in Batches all entries are not processed.

6

Hi @mayanktiwari, I am still not 100% sure I understand your expectations :frowning:

Are you perhaps expecting a different behaviour from the Merge node? If you take out the IF/Merge part from your second, the loop will run as described:

Result

image
image1401×340 17.7 KB

Workflow

Is that what you would expect when using the Merge node as well? If so, perhaps you can describe your exact expectation? I know @sirdavidoff is currently working on a Merge node overhaul, so it’d be a good time to share your thoughts here.