FIFO doesn't work for Remove Duplicate Node

Questions
, ,
1

I found a similar question raised months ago and not solved.

Describe the problem/error/question

Hi, there, I am using Remove Duplicate Node to filter those duplicate records by a time trigger. Since the source only returen fix number result and there are old records and new records in one batch. So I am using Remove Duplicate Node and set it to "Remove items Processed in Previous Executions" to get new records only. 
I use keep items where "value is new" and the value to dedupe on is an ID key.

Default history size is untouched and I assume it's 10000.

What is the error message (if any)?

It raised error "The number of items to be processed exceeds the maximum history size. Please increase the history size or reduce the number of items to be processed."

Please share your workflow

My previouse flow is just a http request and get 200 records back.

Share the output returned by the last node

it’s a simple json file with 200 records.

Information on your n8n setup

  • n8n version: 1.110.1
  • Database (default: SQLite):default
  • n8n EXECUTIONS_PROCESS setting (default: own, main): default
  • Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
  • Operating system:
1 Like
2

Same issue here,

image
image937×543 32.8 KB

I can increase the history size, but what is the point? It will just throw the same error again once it reaches the new size limit,

I’m not sure if this node is designed to work as FIFO (First-In, First-Out) but right now it doesn’t seem reliable :confused:

Any suggestions for an auto-clearing approach with this node would be appreciated..