Adding a ready Code node for the pattern @Deepankar described, since the near-duplicate part is where it usually gets fiddly.
Setup: a Data Table “Get rows” node named Seen (last 7 days, columns canonicalUrl and normTitle, Always Output Data on), and your merged RSS/Reddit items in a node named New Items. Code node mode “Run Once for All Items”:
// n8n Code node, "Run Once for All Items".
// Loops over node "New Items" (feed items with `title` and `link`).
// Previous node "Seen" = Data Table rows with canonicalUrl, normTitle (last ~7 days).
const TRACKING = /^(utm_|fbclid$|gclid$|mc_|ref$|ref_src$|igshid$|si$)/i;
const STOP = new Set(['the','a','an','of','to','in','on','for','and','is','are','with','at','by','from','as','its','new']);
function canonicalUrl(u) {
try {
const x = new URL(u);
x.hash = '';
x.hostname = x.hostname.replace(/^(www|m|amp)\./, '');
for (const k of [...x.searchParams.keys()]) if (TRACKING.test(k)) x.searchParams.delete(k);
x.pathname = x.pathname.replace(/\/amp\/?$/, '/').replace(/\/+$/, '') || '/';
return (x.hostname + x.pathname + (x.search || '')).toLowerCase();
} catch { return (u || '').toLowerCase(); }
}
function normTitle(t) {
return (t || '').toLowerCase().replace(/\s[-|:–—]\s[^-|:–—]{2,40}$/, '') // drop " - Site Name"
.replace(/[^\p{L}\p{N}\s]/gu, ' ').split(/\s+/).filter(w => w && !STOP.has(w)).join(' ');
}
function sameStory(a, b) { // Jaccard on word sets + a floor on shared words
const A = new Set(a.split(' ')), B = new Set(b.split(' '));
let inter = 0; for (const w of A) if (B.has(w)) inter++;
return inter >= MIN_SHARED && inter / (A.size + B.size - inter) >= THRESHOLD;
}
const THRESHOLD = 0.6, MIN_SHARED = 4; // tune on your own feeds
const seen = $('Seen').all().map(i => i.json);
const urls = new Set(seen.map(r => r.canonicalUrl));
const titles = seen.map(r => r.normTitle).filter(Boolean);
const out = [];
for (const item of $('New Items').all()) {
const cu = canonicalUrl(item.json.link), nt = normTitle(item.json.title);
if (urls.has(cu)) continue;
if (nt && titles.some(t => sameStory(t, nt))) continue;
urls.add(cu); titles.push(nt); // also dedupes within this batch
out.push({ json: { ...item.json, canonicalUrl: cu, normTitle: nt } });
}
return out;
I ran it on a small made-up sample: the Verge and TechCrunch versions of the same DevDay story collapse into one, the Shopify story from TechCrunch and Reuters collapses into one, and two different “Show HN” posts both survive. That last case is why MIN_SHARED is there, short titles share “show hn” and plain Jaccard merges them.
Edit: the loop now reads $('New Items'), thanks @Deepankar. With Seen directly upstream, $input would have been the table rows.
For Reddit link posts, map the post’s url field (the linked article) to link, not the permalink, otherwise the Reddit copy never matches the RSS copy.
And write canonicalUrl and normTitle back to the Data Table only after the LLM node succeeds, so a failed run retries the item instead of silently marking it seen.