Hey guys,
I’m experiencing some very weird behavior from the “Remove Duplicates” node and I created a whole new workflow just to test what was going on. It seems the node is discarding brand new items as duplicates even though they have not been seen before. I’m scratching my head over here thinking what am I doing wrong?
Here’s the test workflow (I can’t get it to show up here for some reason):
What I’m basically doing here is using the “remove items processed from previous executions” setting and clearing deduplication history on first new execution. Then, I’m giving it an input which is outputs correctly and then I’m using the set node to give it the same input but with 2 new/different items.
The node still discards everything even though there are two new items.
My background use case that lead to the discovery of this issue:
My use case here is that I’m using the node within a loop with an API that may return some duplicate items along with new items every now and then, and I want to break the loop as soon as the node no longer receives new items from the API and they’re all duplicates. Obviously because of this issue the loop keeps breaking immaturely because the remove duplicates node discards everything on second input even though there are new items.
What am I doing wrong here?
Thanks!
P.S: Running latest self-hosted n8n via Cloudron.