Remove duplicates

Questions
, ,
1

Hello,

First, I wanted to thank both the n8n team and the community, they are great :wink:

I have a scenario where I make a request to an API that returns a list of items. I want to store them in Supabase, but they can’t be duplicates, each row has an id.

How could I do it? I have tried different ways:

  • Merge: combining and and selecting Keep non matches.
  • The item list option: this option returns a 413 error. For now I only have about 5k rows, but it will increase considerably.

image
image1499×434 55.7 KB

image
image504×762 14.9 KB

With the current configuration it keeps returning IDs that are in the supabase database.

Information on your n8n setup

  • Running n8n via (Docker):
4

Hi @fval90
can you share your workflow so we can see how we can help?

The node for that task would be the Item Lists node with its Remove Duplicates operation.
I tried here with 10k demo items and it works:

Let me know if that helps
Best

5

Sure!

Could you check if the API authentication is hidden?

6

Hi @fval90
a couple of notes about this workflow:

  1. The problem with having the split in batches in that position is that you would remove duplicates only inside a single batch (300 items)
  2. The “Fields To Compare” parameter in List Items is empty, double check that one

Could you please send me a demo JSON of your items? In this way we can try to create a demo workflow that handles the same type/amount of data.

Best

7

Sure, I update the parameter in List Items.

9

Hi! I edit some element in the workflow and It’s works! Thank for your support :slight_smile:

10

Happy to hear that!