Hello n8n Community,
I have a workflow where I use two Split Out nodes to process two separate data streams. After both streams are processed, I need to compare their results using the Compare Datasets node.
The problem:
The two Split Out nodes receive and process data at different times. Sometimes, one Split Out finishes much faster than the other. As a result, the Compare Datasets node triggers as soon as the first Split Out finishes, which leads to incorrect or incomplete comparisons because the second dataset is not ready yet.
What I want to achieve:
• I want to ensure that the comparison only happens when both Split Out nodes have finished processing their data.
• If one Split Out finishes first, I need the workflow to “wait” until the second one is done, and only then proceed to Compare Datasets.
Questions:
• What is the best way to synchronize these two branches in n8n, so that Compare Datasets only runs after both Split Out nodes have completed?
• Are there built-in nodes or patterns (like Merge, Wait, or external storage) that can help with this kind of synchronization?
• If not, are there recommended workarounds, such as using Redis, database, or other “assembly point” techniques to collect results and trigger the next step only when both are ready?
Additional context:
• Both data streams are independent and can finish at very different times.
• I am open to using additional nodes or external tools if needed.
• I have seen some discussions about using Merge nodes and external caches (like Redis) for similar async workflows, but I am not sure how to implement this pattern for my specific case.
Thank you for any advice or examples you can provide!
This question clearly describes your workflow structure, the synchronization issue, and what kind of solution you are looking for. It also references the concept of using an external “assembly point” (such as Redis) as described in best practices for async workflows in n8n
If you have AI write your stuff, you should edit out the self-congratulatory statement that AI includes to let you know how awesome a job it did for you.
If you choose to use external storage, and you have little or no control over when, or in what order, you get information, and you need to merge, compare, etc. it will be more reliable to use upsert logic on all sources like:
- If I already have a matching record, then
- lock it → compare it → decide whether to update it → unlock it
- Else
- store a new record
Whether you use external data storage or something that is (usually) more temporary/transient like Redis depends on how long you need to keep the merged data and how much work you want to do to keep things pruned.
Have you tried a Merge with combine → by Matching Fields to collate your sources by a key value before continuing? That should make the workflow wait until the data is finished emitting on both inputs.