Remove duplicates

[fval90]

Hello,

First, I wanted to thank both the n8n team and the community, they are great

I have a scenario where I make a request to an API that returns a list of items. I want to store them in Supabase, but they can’t be duplicates, each row has an id.

How could I do it? I have tried different ways:

• Merge: combining and and selecting Keep non matches.
• The item list option: this option returns a 413 error. For now I only have about 5k rows, but it will increase considerably.

With the current configuration it keeps returning IDs that are in the supabase database.

Information on your n8n setup

• Running n8n via (Docker):

[giulioandreini]

Hi @fval90
can you share your workflow so we can see how we can help?

The node for that task would be the Item Lists node with its Remove Duplicates operation.
I tried here with 10k demo items and it works:

Let me know if that helps
Best

[fval90]

Sure!

Could you check if the API authentication is hidden?

[giulioandreini]

Hi @fval90
a couple of notes about this workflow:

1. The problem with having the split in batches in that position is that you would remove duplicates only inside a single batch (300 items)
2. The “Fields To Compare” parameter in List Items is empty, double check that one

Could you please send me a demo JSON of your items? In this way we can try to create a demo workflow that handles the same type/amount of data.

Best

[fval90]

Sure, I update the parameter in List Items.

[fval90]

Hi! I edit some element in the workflow and It’s works! Thank for your support

[giulioandreini]

Happy to hear that!