First, I wanted to thank both the n8n team and the community, they are great
I have a scenario where I make a request to an API that returns a list of items. I want to store them in Supabase, but they can’t be duplicates, each row has an id.
How could I do it? I have tried different ways:
- Merge: combining and and selecting Keep non matches.
- The item list option: this option returns a 413 error. For now I only have about 5k rows, but it will increase considerably.
With the current configuration it keeps returning IDs that are in the supabase database.
Information on your n8n setup
- Running n8n via (Docker):
can you share your workflow so we can see how we can help?
The node for that task would be the Item Lists node with its Remove Duplicates operation.
I tried here with 10k demo items and it works:
Let me know if that helps
Could you check if the API authentication is hidden?
a couple of notes about this workflow:
- The problem with having the split in batches in that position is that you would remove duplicates only inside a single batch (300 items)
- The “Fields To Compare” parameter in List Items is empty, double check that one
Could you please send me a demo JSON of your items? In this way we can try to create a demo workflow that handles the same type/amount of data.
Sure, I update the parameter in List Items.
Hi! I edit some element in the workflow and It’s works! Thank for your support