Remove duplicates, list of URLs?

Questions
1

Can someone help me remove the duplicates?

[

{

"data": "https://www.bleepingcomputer.com/news/security/actively-exploited-bug-bypasses-authentication-on-millions-of-routers/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/go-rust-net-library-affected-by-critical-ip-address-validation-vulnerability/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/synology-warns-of-malware-infecting-nas-devices-with-ransomware/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/synology-warns-of-malware-infecting-nas-devices-with-ransomware/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/synology-warns-of-malware-infecting-nas-devices-with-ransomware/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/synology-warns-of-malware-infecting-nas-devices-with-ransomware/#comment_form"

},

{

"data": "https://www.bleepingcomputer.com/news/security/australian-govt-warns-of-escalating-lockbit-ransomware-attacks/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/australian-govt-warns-of-escalating-lockbit-ransomware-attacks/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/australian-govt-warns-of-escalating-lockbit-ransomware-attacks/#comments"

},

{

"data": "https://www.bleepingcomputer.com/news/security/actively-exploited-bug-bypasses-authentication-on-millions-of-routers/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/actively-exploited-bug-bypasses-authentication-on-millions-of-routers/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/actively-exploited-bug-bypasses-authentication-on-millions-of-routers/#comment_form"

},

{

"data": "https://www.bleepingcomputer.com/news/security/go-rust-net-library-affected-by-critical-ip-address-validation-vulnerability/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/go-rust-net-library-affected-by-critical-ip-address-validation-vulnerability/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/go-rust-net-library-affected-by-critical-ip-address-validation-vulnerability/#comment_form"

},

{

"data": "https://www.bleepingcomputer.com/news/security/the-week-in-ransomware-august-6th-2021-insider-threat-edition/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/the-week-in-ransomware-august-6th-2021-insider-threat-edition/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/the-week-in-ransomware-august-6th-2021-insider-threat-edition/#comments"

},

{

"data": "https://www.bleepingcomputer.com/news/security/cisco-firewall-manager-rce-bug-is-a-zero-day-patch-incoming/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/cisco-firewall-manager-rce-bug-is-a-zero-day-patch-incoming/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/cisco-firewall-manager-rce-bug-is-a-zero-day-patch-incoming/#comment_form"

},

{

"data": "https://www.bleepingcomputer.com/news/security/computer-hardware-giant-gigabyte-hit-by-ransomexx-ransomware/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/computer-hardware-giant-gigabyte-hit-by-ransomexx-ransomware/"

},

{

"data": "https://www.bleepingcomputer.com/news/security/computer-hardware-giant-gigabyte-hit-by-ransomexx-ransomware/#comment_form"

},

{

"data": "https://www.bleepingcomputer.com/news/security/linux-version-of-blackmatter-ransomware-targets-vmware-esxi-servers/"

}

]

I tried the code snippets but it just doesn’t seem to do what I’m expecting. Is anyone able to leave just a list on non duplicated URLS?

2

The function node should be like this:

const data = [];

for (const item of items) {
  if (data.includes(item.json.data)) continue
  data.push(item.json.data)
}

return data.map((url) => ({ json:{ data: url } }))
Example workflow 
{
  "nodes": [
    {
      "parameters": {},
      "name": "Start",
      "type": "n8n-nodes-base.start",
      "typeVersion": 1,
      "position": [
        -390,
        180
      ]
    },
    {
      "parameters": {
        "functionCode": "return [\n  {\n    json: {\n      data: 'https://www.bleepingcomputer.com/news/security/actively-exploited-bug-bypasses-authentication-on-millions-of-routers/',\n    }\n  },\n    {\n    json: {\n      data: 'https://www.bleepingcomputer.com/news/security/go-rust-net-library-affected-by-critical-ip-address-validation-vulnerability/',\n    }\n  },\n      {\n    json: {\n      data: 'https://www.bleepingcomputer.com/news/security/go-rust-net-library-affected-by-critical-ip-address-validation-vulnerability/',\n    }\n  }\n]\n"
      },
      "name": "Function",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        -130,
        180
      ]
    },
    {
      "parameters": {
        "functionCode": "const data = [];\n\nfor (const item of items) {\n  if (data.includes(item.json.data)) continue\n  data.push(item.json.data)\n}\n\nreturn data.map((url) => ({ json:{ data: url } }))"
      },
      "name": "Function1",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        90,
        180
      ]
    }
  ],
  "connections": {
    "Start": {
      "main": [
        [
          {
            "node": "Function",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Function": {
      "main": [
        [
          {
            "node": "Function1",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}
3 Likes
3

thank you, just what I was looking for

4

Glad that it worked. Have fun.

5

@RicardoE105, hey! How do you change this code to return all the properties of each object?

6

Hey @Roket,

This function might help

const data = [];

for (const item of items) {
  if (data.includes(item.json.data)) continue
  data.push({json:item.json})
}

return data
7

@harshil1712, Hey! After this change, duplicates are returned to me, and the essence of this code is precisely in removing duplicates.

My incoming data looks like this:

[

{

"uuid": "7d5f337c-925d-4202-a0b4-b87ea6d50e9c",

"type": "core",

"process": "b5c2452a-4ef3-46a7-8d9a-0ffa862afa53",

"core": "Диагностика",

"name": "Ядро_1",

"morf": "Да",

"floor": "Да",

"abb": "",

"abb_name": "Абб_1",

"num_part": "1",

"cores": "7d5f337c-925d-4202-a0b4-b87ea6d50e9c , 122beb50-9c7d-4b99-a08f-b6f8273afd9c"

},

{

"uuid": "7d5f337c-925d-4202-a0b4-b87ea6d50e9c",

"type": "core",

"process": "b5c2452a-4ef3-46a7-8d9a-0ffa862afa53",

"core": "Диагностика",

"name": "Ядро_1",

"morf": "Да",

"floor": "Да",

"abb": "",

"abb_name": "Абб_1",

"num_part": "1",

"cores": "7d5f337c-925d-4202-a0b4-b87ea6d50e9c , 122beb50-9c7d-4b99-a08f-b6f8273afd9c"

},

{

"uuid": "7d5f337c-925d-4202-a0b4-b87ea6d50e9c",

"type": "core",

"process": "b5c2452a-4ef3-46a7-8d9a-0ffa862afa53",

"core": "Диагностика",

"name": "Ядро_1",

"morf": "Да",

"floor": "Да",

"abb": "",

"abb_name": "Абб_1",

"num_part": "1",

"cores": "7d5f337c-925d-4202-a0b4-b87ea6d50e9c , 122beb50-9c7d-4b99-a08f-b6f8273afd9c"

},

{

"uuid": "122beb50-9c7d-4b99-a08f-b6f8273afd9c",

"type": "core",

"process": "b5c2452a-4ef3-46a7-8d9a-0ffa862afa53",

"core": "Ремонт",

"name": "Ядро_1",

"morf": "Да",

"floor": "Да",

"abb": "",

"abb_name": "Абб_1",

"num_part": "1",

"cores": "7d5f337c-925d-4202-a0b4-b87ea6d50e9c , 122beb50-9c7d-4b99-a08f-b6f8273afd9c"

},

{

"uuid": "122beb50-9c7d-4b99-a08f-b6f8273afd9c",

"type": "core",

"process": "b5c2452a-4ef3-46a7-8d9a-0ffa862afa53",

"core": "Ремонт",

"name": "Ядро_1",

"morf": "Да",

"floor": "Да",

"abb": "",

"abb_name": "Абб_1",

"num_part": "1",

"cores": "7d5f337c-925d-4202-a0b4-b87ea6d50e9c , 122beb50-9c7d-4b99-a08f-b6f8273afd9c"

},

{

"uuid": "122beb50-9c7d-4b99-a08f-b6f8273afd9c",

"type": "core",

"process": "b5c2452a-4ef3-46a7-8d9a-0ffa862afa53",

"core": "Ремонт",

"name": "Ядро_1",

"morf": "Да",

"floor": "Да",

"abb": "",

"abb_name": "Абб_1",

"num_part": "1",

"cores": "7d5f337c-925d-4202-a0b4-b87ea6d50e9c , 122beb50-9c7d-4b99-a08f-b6f8273afd9c"

}
]

n8n - ▶️321
n8n - ▶️3211522×759 86.4 KB

8

Fingers crossed next week we will release the ‘item list’ node, which will make removing duplicates easy

1 Like
9

@Roket, in that case, the following code snippet can be useful

const jsonObject = items.map(JSON.stringify);
const uniqueSet = new Set(jsonObject);
const uniqueArray = Array.from(uniqueSet).map(JSON.parse);
console.log(uniqueArray);

return uniqueArray;
1 Like
10

@harshil1712, thanks! Works great!