Still new to n8n so this solution might be simple but any guidance would be appreciated.
Lets say I have this requirement:
I need to collect the latest news from a number of external sources via API and send that to a Google sheet (for example) so I have the latest news from different sources consolidated in 1 place and it updates every day or multiple times a day.
So the steps would be like this I imagine, for a single source:
n8n requests a list of the latest 20 items from an external API. Lets say a “latest news” API.
This will occur on a schedule multiple times a day or whatever feels appropriate for that source to not miss stuff or not get too many dupes.
Process this data into a cleaned/formatted list format or whatever might be required before posting to google sheet.
Post that data to a Google sheet as new entry (insert at the top of sheet) sorted date descending.
Seems simple enough but the problem is that duplicate items often definitely going to appear in each new fetch that were in the previous fetches.
If we fetch items from API every day, 20 per day and only 5 items are new from last fetch, is it possible in n8n to check the incoming data, compare it to the previous fetch(es) and delete any duplicates before posting the new entries to Google sheet?
What is the easiest way to achieve this functionality?
I assume some scripting would be needed for this and perhaps even a database that includes the raw incoming data from all previous fetches to check the new fetch against. Or would you fetch the new news, fetch the existing google sheet entries, compare them somehow and then post only the new unique ones to the google sheet.
Can you point me towards the easiest, simplest, fastest least resource intensive way to achieve this result.
Very much appreciated!