Hey guys,
Quick question — when the Remove Duplicates node hits the history size limit, does it work like FIFO where the oldest entries get removed automatically to make space for new ones?
Its setup to remove items seen in previous executions, I know there’s a “Clear History” option, but I’m not looking to wipe everything.
Appreciate any insights!
When the Remove Duplicates node reaches its history size limit, it does indeed work like a FIFO (First In, First Out) system. The oldest entries in the history get automatically removed to make space for new entries.
The node keeps track of items it has seen up to the limit you’ve set in the “Max Age/Limit” parameter. Once that limit is reached, the oldest entries will automatically be dropped from memory as new entries come in, without requiring you to manually clear the entire history.
This behavior ensures that your workflow can continue running indefinitely without manual intervention, while still effectively removing duplicates based on your recent execution history. Just make sure you’ve set an appropriate limit value that covers your use case’s time window for duplicate detection.
Hope that helps!
Unfortunately that doesn’t seem to be the case, I was searching the forum for this precise question. I got the following error:
The number of items to be processed exceeds the maximum history size. Please increase the history size or reduce the number of items to be processed.
My history size is 10,000 and the incoming entries are 919. I was hoping there was a way to delete the older entries, but it seems it’s either delete everything or nothing.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.
