What would be the best practice/method to monitor 50+ RSS feeds?

Hi,

So, I want to migrate RSS monitoring from other services to n8n. Right now a third party RSS watcher basically calls n8n webhooks when there’s a new item. However, it’s a paid service and n8n could definitely do it, but I’m not sure what would be considered the best way to implementing it while keeping it easy to maintain.

The big question is: Should there be one process that has a list of RSS feed urls and processes them all at the same time and when there are new items send them to another workflow… Or, would it be better to create individual RSS feed monitor workflows that each call the same second workflow to process the items.

Just curious what others think and pros/cons.

Thanks,
AJ

Hi @ajtatum, welcome to the community :tada:

While n8n can fetch RSS feeds it wouldn’t tell you which items are new. For a really solid system you’d need to implement a polling strategy manually for this, the basic approach is described here:

If you can live with some outages (e.g. like one of the feeds temporarily not available) you could just have your workflow run every hour or so and then simply look at only the items from the last hour.

As for your big questions: I am a fan of easy to read workflows. So where it makes logical sense to split up complex workflows into two or more simpler workflows I’d usually do that.

On the pro side: Your workflows will be much easier to read which also make it much easier to get help if you’re stuck or simply are handing them over to someone else.

On the cons: You have more workflows and your workflow list can eventually become a bit harder to manage. If you use a combination of Webhook/HTTP Request nodes to trigger your sub-workflows you’d also need to have more active workflows and will have more workflow executions which might mean you need a bigger plan when using n8n.cloud.

But what I’m really curious about would be your use case - any chance you can share a few details about what you’re doing with each item?

1 Like

Hey there @MutedJam! Thanks for the reply and sorry for not getting back to you sooner! Basically, I want to build a news aggregator for my WordPress site. I already have WP Webhooks and host n8n myself, so the number of executions, etc doesn’t really matter. I care more about maintainability and management.

Right now I’m paying for a service that essentially pings a n8n webhook whenever 1 out of my ~50 feeds has a new article. Right now I’m still ironing out a few things, such as getting a portion of the content to display on my site (using Mozilla’s Readability API). Anyway, their service is $30/mo and has a nice UI and all… but it’s limited to 50 feeds unless I pay more and I just kind of cringe whenever I think about the fact that I’m paying for that service when I can leverage n8n to read a database to retrieve a list of RSS feed URLS, then scan those RSS feeds for new items, and then do something for new items. I realize in essence I’d end up having a database of links that it would have to check against to determine if there are new items.

Another possibility that I’m playing with is leveraging Inoreader as I already use & pay for that personally. With them, there are two possible roads:

  1. My RSS feeds are grouped into topical folders and you can create a single RSS feed for a folder. So, my WordPress folder that contains ~25 RSS feeds would be one feed. I could have n8n read that single feed.
  2. I could setup a rule so that whenever a new item is added in a particular folder, Inoreader would call a webhook. I’m testing this out to see what data it sends and capturing it.

Using Inoreader has the con of being locked into paying for their Pro plan… which I’ve been on for quite sometime, so it’s not a huge deal, but I just like having something that’s somewhat independent.

Anyway, sorry for going down the rabbit hole a bit. Polling sounds like a good option and I found this workflow and, though it’s somewhat outdated, it could be tweaked and fit my needs: https://n8n.io/workflows/159

I hope that gives you more clarity in what I’m trying to build. Basically, looking to replace a paid service that just does one thing with either JUST n8n or n8n with Inoreader that I already pay for.

Thanks for sharing @ajtatum! I can see where you are coming from.

Personally I use Miniflux as my RSS Reader which has an API but doesn’t support webhooks unfortunately. However, if you’re fine with polling that’d be extremely easy to implement as the Miniflux API supports a lot of parameters allowing filtering. That way you wouldn’t have to manage a separate database for your posts and also won’t have to pay extra for more than 50 feeds.

If you simply want to process new items every 6 hours, a workflow like this would do the job:

You’d get your posts in a really handy format afterwards:

Of course you can (and probably should) refine this for additional resilience (for example fetch all unread posts and mark all processed posts as read afterwards - this will help in case a feed is down temporarily and comes back only after your polling interval is over).

Perhaps Inoreader offers a similar API? If not, Miniflux might be worth considering - it’s free if you self host ($15 per year when hosted by them) and works really well with n8n as I just found out :smiley:

Hope this adds a bit of perspective! If you want to, you could set your polling intervals quite low and could even build your own webhook sending logic in n8n that way.

Thanks @MutedJam! I never heard of Miniflux before and have tried working with FreshRSS and Tiny Tiny RSS, but your idea of using Miniflux’s API sounds like a great idea! It’s definitely better than having a ton of duplicate workflows for each RSS feed in n8n. I’ll have to do some further investigation, but this sounds like it would be a great solution.

Thank you for your input! I really appreciate it!

1 Like

Hi @ajtatum curious to know what you ended up doing. Please share Workflow if you can. Thanks

Hi @myWebmaster, I actually ended up writing a community node for my feed reader: Miniflux node for n8n

This lets me query the Miniflux API for unread posts and marks the posts I have processed as read. A very simple workflow could look like so:

I’ll also post the workflow JSON below for reference, but the forum won’t be able to preview the Miniflux community node of course (as it only knows about standard nodes):

It’s mostly self-explanatory though. The first Miniflux node fetches unread posts from all feeds like so:

The second one marks them as read so they won’t be fetched again on the next run:

That said, it’s been a while since I’ve written this and it might not support all quality of life features newer or official n8n nodes might come with. Still gets the job done for me though :slight_smile:

2 Likes

I do something pretty similar by using Miniflux, which is super easy to setup and self-host. I get all the new entries every 20 minutes, process them in a subflow, and then mark those items as read. Miniflux is setup to retrieve posts every 60 minutes. The biggest hurdle I’ve run into is that it gets blocked every now and then by Cloudflare. For those special RSS feeds, I use a proxy (you can use a socks5 proxy, which is likely provided by a VPN if you’re using one). For the REALLY special sites, I use ScrapeShark and pass in the RSS feed through them.

My “processing” is submitting the posts to WordPress, but only a portion of the post. For this, I use the node modules sanitize-html and truncatize. To submit the data to my site, I rely on WP Webhooks which is SO much easier to work with than then WordPress API.

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.