How to store links from RSS Read in datastore and how to remove duplicates to send on Discord server?

Questions
1

Describe the problem/error/question

Hello im working on a project automation on n8n workflow to send cybersecurity news to a discord server.

Here’s how it works:
First, i used schedule trigger node daily or every 24 hour.

Second, i used RSS Read node to take the XML from BleepingComputer website. Inside the output, there is a schema, and inside the schema, there are categories that create a table, such as title, pubDate, author, summary, id, and isoDate. Then, the outputs are sent to the Limit Node.

Third, the Limit Node limits my contents or the items from the RSS Read Node from 25 items down to 1. This one content is forwarded to my Discord Node.

Fourth, the one content from the Limit node is now forwarded to the Discord Node. Inside the disocrd node, I obtained the webhook from my discord channel bot server called CyCOM Bot to allow this bot to send messages with title, author, published date, summary, and link to the official website. It’s like the automatic cyber news daily or weekly.

Here is my problem:

  1. If the contents from BleepingComputer do not change in 24 hours, then the RSS Read will re-fetch the old contents as the previous scheduled Trigger node is activated. Therefore, sending the same content with the same title, pubDate, author, summary, id, and isoDate to my Discord CyCOM bot.

  2. What’s your recommendation on this? I need something to store all data from RSS Read and check if the content is the same. If yes, then skip the content and send nothing.

  3. I wish to have a database WITHIN the workflow and not connecting to external apps.

  4. The problem right now is that i don’t know how the datastore checks if the link (from RSS) is sent to the Discord yet or not. To avoid duplicates

Concerns:

  1. I am concerned that if the system triggers the schedule again, the datastore is refreshed.
  2. Currently, Im testing on datastore. But is it better to use a proper database (e.g., MySQL)?

Please share your workflow

Expected output

I expect the Datastore (Get), located before the IF node, to store the links from the RSS Read.
I expect the IF node to somehow compare this link to the Datastore (Set) behind the Discord server. If the links are the same, it is a duplicate, so skip or don’t send anything.
Otherwise, send the new links, aka new content from the RSS Read.

Information on your n8n setup

  • n8n version: 1.101.4
  • Database (default: SQLite): Using Datastore
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
  • Operating system: Windows 11 Home
2

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.