How to deduplicate a list of vacancies from an API response within the same payload

Questions
1

Describe the problem/error/question

Each vacancy has a unique external_id, but some appear multiple times in the same API response.

My goal:

Allow exactly one item from each vacancy
Filter out duplicates within the same payload
Without a Supabase check, because the database is still empty

I am using a Code node in “Run Once for All Items” mode to detect duplicates.

What is the error message (if any)?

Please share your workflow

(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)

Share the output returned by the last node

Information on your n8n setup

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
3

@haajj3012 Code node in “Run Once for All Items” mode:

const seen = new Set();
return $input.all().filter(item => {
  const id = item.json.external_id;
  if (seen.has(id)) return false;
  seen.add(id);
  return true;
});

emits one item per unique external_id, drops the rest. if u want to keep a SPECIFIC one (latest by date, highest version, etc) instead of “first seen” lemme know.

4

Hey @haajj3012,
@achamm’s code works perfectly. just adding a
no-code option in case you prefer keeping things visual in
the workflow:

n8n has a built-in Remove Duplicates node that does exactly
this without writing any javascript. setup:

  1. drop a Remove Duplicates node after your api call
  2. operation: “Remove Items With Same Field Value”
  3. compare field: external_id

it’ll filter the array down to unique items in one node, easier
to spot in the canvas later when you’re debugging.

couple of small edge cases worth handling whichever approach
you go with:

  • if external_id is ever missing or null on some items, both
    the code and the node will treat all the nulls as duplicates
    of each other. usually fine, but if you want to keep nulls,
    add a filter before the dedup step to split them out.
  • if the api ever returns the id as a number on some items
    and a string on others, both approaches will treat "123"
    and 123 as different. a quick String(item.json.external_id)
    in the code (or a Set node to normalise the type) fixes that.

since you mentioned supabase is coming later — when you do add
it, the cleanest pattern is to dedup within the payload first
(what you’re doing now), then dedup against supabase as a
second step. keeps the logic clean and easy to test each layer
independently.

5

Great answers from @achamm and @Dharmendra_Kumar – the Set + filter pattern in the Code node is solid, and the visual option with Remove Duplicates is exactly on point for this use case.

Since n8n already supports this scenario very well out of the box, I’d personally avoid a Code node here and lean fully on the built‑in node. It keeps the workflow easier to read and usually more stable in the long run.

For your specific goal (“exactly one item per external_id within the same payload, DB still empty”), you can configure Remove Duplicates like this:

  • Add a Remove Duplicates node right after your API node.
  • Set Operation to Remove Items Repeated Within Current Input.
  • Set Compare → Selected Fields and choose external_id as the field.

This will keep only the first item for each external_id in the current execution and drop all further duplicates – exactly what you described, without any extra code or database checks.

Later, when you start using Supabase and also want to skip vacancies you’ve already processed in previous runs, you can chain a second Remove Duplicates node in Remove Items Processed in Previous Executions mode. That way you get a clean “dedupe within payload → dedupe against history” pattern, all with native nodes.