RSS Feed data remains duplicated

[denverfix]

I’ve created a flow to extract the feed from some sites and forward it to a group on Telegram.

It’s working, however, I’ve noticed that some data is being sent in duplicate.

In the code, I use the following code snippet:

const items = [
  'https://recrutalentos.com.br/feed/',
  'https://empregosnabahia.com.br/feed/',
  'https://alojuca.com.br/feed/',
];

const uniqueUrls = [];
const uniqueItems = items.filter((item) => {
  if (!uniqueUrls.includes(item)) {
    uniqueUrls.push(item);
    return true;
  }
  return false;
}).map((url) => {
  return {
    json: {
      url: url
    }
  };
});

return uniqueItems;

After Split In Batches, use the RSS Feed Read and a new code.

const staticData = this.getWorkflowStaticData('global');

latestRead = staticData.latestRead;

for (let item of items) {
  item.json.latestRead = latestRead || '2021-06-01';
}

return items;

In IF, I use some codes.

Look at my flow:

Even performing the filters, I still receive duplicate feed in Telegram.

Has anyone been there?

What have you done to solve it?

[EmeraldHerald]

Hi @denverfix In order to test this further, can you let us know the following?

• n8n version:
• Database (default: SQLite):
• n8n EXECUTIONS_PROCESS setting (default: own, main):
• Running n8n via (Docker, npm, n8n cloud, desktop app):
• Operating system:

I just tried this on 1.9.3 with your example workflow and at a quick glance, I’m not seeing duplicated titles

[denverfix]

@EmeraldHerald

On the first attempt, it only brought me single data points.

I think the problem is when it runs and finds the same url.

When it doesn’t find new items, it forwards the same feed that has already been forwarded.

I’m using N8N with the latest version 1.9.3

• Database (default: SQLite):

• Running n8n via (Docker, npm, n8n cloud, desktop app):
  Follow all the configurations following these steps using docker
  Docker | n8n Docs

• Operating system:
  Ubuntu - on a vps instance Hetzner.

But I don’t think the problem lies with the installation.

Everything works fine, I use several other workflows.

Do a test using telegram. It forwards without duplication, but if it doesn’t find any new feeds, it will return the same feed that has already been forwarded.

Thanks for trying to help me

[denverfix]

see the print, I receive the feed. But before that he had already forwarded me the same feed.

When it doesn’t find new feeds, it always forwards the same one.

Trying to figure out a solution

[EmeraldHerald]

Hi @denverfix Thanks for confirming your version number, etc. - it’s helpful with testing!

It looks like you’re not updating your latestRead - you’re getting the value of it, but it’s static. If you set staticData.latestRead to whichever date and time you wanted, you should see different behaviour

We have a workflow example here:

Get data from multiple RSS feeds to Telegram | n8n workflow template

In this workflow, it uses the following to ensure that latestRead is updated:

const staticData = getWorkflowStaticData('global');
const newRSSIds = items.map(item => item.json["isoDate"]);
const oldRSSIds = staticData.oldRSSIds; 

if (!oldRSSIds) {
  staticData.oldRSSIds = newRSSIds;
  return items;
}


const actualNewRSSIds = newRSSIds.filter((id) => !oldRSSIds.includes(id));
const actualNewRSS = items.filter((data) => actualNewRSSIds.includes(data.json['isoDate']));
staticData.oldRSSIds = [...actualNewRSSIds, ...oldRSSIds];

return actualNewRSS;

[denverfix]

@EmeraldHerald

I made this change and also included in the IF , {{ $json[“actualNewRSS”].length }} = >0.

This way, if the new feed is longer than 0, it will forward it to true, if it isn’t, it will throw it to false.

Working perfectly now.

Thanks my friend.

Thanks for your help.