Deduplication Workflow: Only Add Genuinely New data to Your Database

Built with n8n
, ,
1

What this workflow does

This workflow solves a problem you most likely hit when feeding data (leads, prospects, contacts, anything) into an existing database or Google Sheet.

How it works, step by step

1. Manual Trigger (“Set your input”) Starts the run and fans out into two parallel branches. Easy to swap for a scraper, excel file or any other data output. (you can delete the “New input” to)

2. Two data sources, read in parallel

  • Existing Database — reads the rows already in your destination. Set to executeOnce so it pulls the current state of your database a single time.

  • New input (optional) — reads the incoming batch you want to evaluate.

3. Flag each source with a _type field Each branch passes through a dedicated Set node:

  • Flag Database stamps _type = "Database" on existing records.

  • Flag New Data stamps _type = "New" on incoming records.

Both use Include Other Fields so the original data is preserved and just gets the extra tag. This _type flag is the whole sauce, once both lists are merged into one stream, it’s what lets the Code node tell “already in the database” apart from “incoming/new”.

4. Merge both streams (append mode) A Merge node combines the flagged existing records (Input 1) and flagged new records (Input 2) into one single list.

5. Filter: only new records (0 extra API calls) The heart of the workflow. One Code node does all the deduplication against the merged list:

What it does:

  • Builds a Set of all existing IDs from the items flagged Database. A Set keeps lookups fast even on large lists.

  • Handles both Lead_id and lead_id spellings so a casing difference in your header doesn’t break matching — and you can add more variations.

  • Normalizes every ID with .trim().toLowerCase() so whitespace and capitalization don’t cause false “new” matches.

  • Keeps only the incoming New items whose ID is not already in the existing set.

  • Strips the helper _type field back off before output, so your clean records don’t carry the internal tag.

Because everything is compared in memory against data you already loaded, there are no per-row lookups against the API.

6. Remove Duplicates After filtering out records that already exist, a Remove Duplicates node (comparing on lead_id) catches any duplicates within the incoming “new” batch itself. E.g. the same record appearing twice in the new input. Belt and suspenders.

7. Append the clean records (“Database + new data”) The deduplicated output is appended to your destination sheet. Only genuinely new records get added.

Why I built it this way

The obvious approach to dedup is to loop over each incoming row and query the database to check if it exists. That’s slow and burns one API call per row. By loading both lists once, flagging them, merging, and comparing in a single Code node, the whole job runs in one pass with no extra calls. It scales well, is fast and stays cheap.

Notes / things you can adapt

  • Flag values must match the Code node. The strings in Flag Database (Database) and Flag New Data (New) have to match the values checked in the Code node. Rename both sides together if you want different labels.

  • Matching key. Dedup is done on a single ID field (Lead_id / lead_id). No stable unique ID? Build the key from a combination of fields (e.g. email + company_name) inside the Code node, and point Remove Duplicates at the same logic.

  • Trigger. Swap the Manual Trigger and new input for an scraper, excel sheet or any other way you want to implement it into your workflow.

  • Destination. Any append-capable node works here (Sheets, Postgres, Airtable, etc.) — the dedup logic is storage-agnostic.

For questions you can e-mail me on:

lucas@lca-optimize.com

1 Like
2

The O(1) Set lookup for deduplication is a smart choice - avoids the N+1 API call trap that most people fall into with the “check before insert” pattern. One thing worth adding: if your source data has inconsistent date formats or trailing whitespace in non-ID fields, a small normalization step in the Code node before building the Set will save you from false duplicates slipping through on repeat runs.

1 Like
3

Nguyen,

Thanks for responding!

I have added an trim to lowercase, this works for me, I used linkedin urls as lead_id’s (so it isn’t even nessecary for me). But as you said additional normalizations techniques will greatly improve the reliability of the de-duplication!