Compare two databases and keep only the NEW data from the fresher database

Good evening to all,

As I’m slowly getting the hang of n8n (and enjoying it more and more!) I’d like to reactivate a fun sideproject I had set up last year.

Every hour I call up a json file publicly available online from a URL. This file changes every hour, but with some data staying the same, some being added, and some being deleted. I would like to focus only on the ones that are added and set them aside in a database to trigger a series of actions later.

My technical skills unfortunately stop here… I have looked at a lot of things, especially the Merge node with its Remove Key Matches mode…

Do you have any ideas?

Welcome to the community @arthurcorre

How does the data look like? Does it have a timestamp? If it does, you can filter in a function node only the ones added since the last time you checked the URL.

Once I have more information, I can provide better guidance.

Hello, thank you!

Actually the data comes from Twitter: I want to look every hour at what new hashflags (little graphic symbol that appears next to some hashtags) are newly activated on the platform.

Every hour at XX:05 of the day, a new json file is available. In the JSON file, there are indeed timestamps, but they correspond to the precise times of activation and deactivation of these famous hashflags.

If you want to see what this famous file looks like, here it is: (file of December 7, 2021 at 11am)