Multiple merges in a loop

Describe the problem/error/question

We’re seeing some unexpected behavior trying to deal with multiple paths through a loop. We’re using Merge nodes to make sure we fully finish each loop before starting the next, but when we have multiple Merge nodes, execution hangs on the first.

Please share your workflow

Share the output returned by the last node

The first Merge node never runs.

Information on your n8n setup

  • n8n version: 1.18.2
  • Database (default: SQLite): Postgres
  • n8n EXECUTIONS_PROCESS setting (default: own, main): own
  • Running n8n via (Docker, npm, n8n cloud, desktop app): Docker

@dlerman , it doesn’t look to me that you are using “Loop Over Items” in the way it was intended to. The node is designed to batch the big chunk of data. That is if, say, you have 10 items in the list, you can process it 2 at a time, for example.

I don’t understand what you are trying to achieve here. What is your general goal? Do you even need to utilize “Loop Over Items” node?

Sorry, can give a bit more context: this is a simplification of a workflow that’s failing in production. In the actual workflow, we’re processing ~10,000 items in batches of 100 to avoid pulling too much data from an API at once. For each batch of 100, we have a few conditional checks and based on the outcome of those, take some particular action for each record.

In the first version of this, we just wired all of those paths up to a No-Op and then back to the beginning of the loop, but you get weird behavior since each each conditional node generates a new execution path, so you end up evaluating the loop node too many times. To get each path to complete before starting the next loop, we added some merge nodes, but that’s causing execution to hang without ever going back to the start of the loop.

Thanks @dlerman , it provides more clarity. More questions for you.

  • Why don’t you bring all the 10,000 items together (in batches of 100), filter them out outside of the loop leaving only those you need and only then work on the those that were left standing (could be in another loop if using API again)?

Schematically,

Loop1: accumulate all ➔ Filter (outside of Loop1) ➔ Process remaining (in Loop2 if appropriate)

-deleted because wrong-

Pretty sure this should work.
Are you using the v1 path option? (can be found in the workflow settings)
If so, did you replace the merge nodes if this was set from v0->v1 as well?

Also 2 tips.
For larger data sets I always use a subflow to actually process the batch. As it can help with performance. So that would be: Split → Execute workflow → return back to split
What also might help and allows for parallel processing if the API allows the number of requests, is to push everything to a queue and process them from there. RabbitMQ has nice native nodes in n8n.
(of course it is best to not push complete data sets to a queue, but use an ID or date range or whatever if you can)

Thanks all! It sounds like pulling the logic out of the loop is a great workaround and is what I’ll likely do – but I guess I’m still confused about whether this should work and is therefore a bug or shouldn’t work. A single conditional branch followed by a single merge works fine, even if all the items go to one side of the branch – so apparently a merge can function fine if there are no items in one side of the input – so why wouldn’t two merges work?

Again, thanks for the great workaround!

Thanks @BramKn - it’s a v1 workflow (created and run in v1).

Looks good here

@BramKn is correct.
(I was certain running into exactly the issue I had described.
But after testing it just now, it either had to be related to the specific nodes, or an UI issue when testing the workfow/nodes)

I still stand by my suggestion to take the filtering logic out of the loop. While it might work in some cases it can break the loop in the other. You might encounter a combination of IF conditions where no input will be provided to the Loop node.

1 Like

One thing we want to change: stopping execution in a loop shouldn’t stop the whole workflow; it should just begin the next iteration of the loop instead.

1 Like

Thank @BramKn - I agree that your workflow works, but mine still doesn’t - they’re slightly different in their logic. If you run my exact workflow with pinned input data:

[
  {
    "code": -2
  },
  {
    "code": 11
  }
]

does it complete the loop successfully?

FWIW, I do think the definite takeaway is not to use logic in a loop because the behavior is complex and hard to predict, so that resolves this issue for me, and I thank you all for the help! But in case it’s helpful, I do still think there’s a bug in there…