Inserting documents if they don't already exist in MongoDB (with multiple items)

[Bret_Truchan]

Hello! I have a workflow which works great with one item. I’m essentially creating a system which monitors a few YouTube channels and logs updates into a database (MongoDB). However, I don’t want to log duplicates, so I query the database first to see if a record already exists before inserting it.

For my testing, I have one video already in the database, and another video that isn’t already in the database and should get inserted.

The tricky part is the interplay between MongoDB1 and the If Exists nodes.

The input to MongoDB1 is a list of two items to query. I would have expected these to be queries one after another, and perhaps that’s happening. My query string is {“discovery_uuid”: “{{ $json[“discovery_uuid”] }}”}

MongoDB returns 1 response, which in this specific instance is accurate. It’s the record that already exists. This causes the If Exists node to evaluate to True, preventing anything from being inserted into the database, even the new video.

Do I need to somehow split out these items and processes them one by one?

Please share your workflow

Share the output returned by the last node

[
  {
    "_id": "67351607e425be028401842e",
    "kind": "youtube#searchResult",
    "etag": "hPE7ag09Z3APhbAix_uJ7mipZUc",
    "id": {
      "kind": "youtube#video",
      "videoId": "GcFKJBc8rrI"
    },
    "snippet": {
      "publishedAt": "2024-11-13T12:09:54Z",
      "channelId": "UCP2gnyy_-ToZeIDw6qeI6HA",
      "title": "テレビで活躍する名店シェフの本気の料理がこだわり凄すぎて驚愕しました",
      "description": "後藤シェフのレシピ本『一つ星フレンチ「アムール」が教える ふだん着フレンチ』はこちら https://amzn.to/4frYZvk 『ごと ...",
      "thumbnails": {
        "default": {
          "url": "https://i.ytimg.com/vi/GcFKJBc8rrI/default.jpg",
          "width": 120,
          "height": 90
        },
        "medium": {
          "url": "https://i.ytimg.com/vi/GcFKJBc8rrI/mqdefault.jpg",
          "width": 320,
          "height": 180
        },
        "high": {
          "url": "https://i.ytimg.com/vi/GcFKJBc8rrI/hqdefault.jpg",
          "width": 480,
          "height": 360
        }
      },
      "channelTitle": "George ジョージ",
      "liveBroadcastContent": "none",
      "publishTime": "2024-11-13T12:09:54Z"
    },
    "discovery_datetime": "2024-11-13T13:11:33.939-08:00",
    "discovery_source": "youtube",
    "discovery_type": "video",
    "discovery_uuid": "youtube_GcFKJBc8rrI"
  }
]

Information on your n8n setup

• n8n version: [email protected]
• Database (default: SQLite): N/A (MongoDB in the workflow)
• n8n EXECUTIONS_PROCESS setting (default: own, main): default
• Running n8n via (Docker, npm, n8n cloud, desktop app): n8n cloud
• Operating system: N/A

[Bret_Truchan]

OK, I have a solution. I don’t know if it’s the most elegant solution, but here it is:

[ihortom]

Hey @Bret_Truchan , using Loop in your scenario seems reasonable. You have the list of channels and then you iterated over all the video in that channel. However, I suspect it is still might not work properly due to the logic in “Prep Data” node where you get only the first video ignoring the rest (if there is more than one). The Loop iterates over channels, not videos in the channel for first() to be used harmlessly.

[Bret_Truchan]

@ihortom

However, I suspect it is still might not work

Ah, my apologies, but that’s on purpose. The final version of this workflow is scheduled to run daily, and I only wanted the most recent video posted per channel. If a channel posts more than one video in a day, I’d just use the newest one. In my specific situation, it would be a very unusual for a channel to post more than one video in a day.