Remove Duplicates Emails from Json

Hi Everyone,

I have some email addresses being sent in by webhook.

I have used:

return items[0].json.data.map(data => { return { json: { data } } });

to output batches of data contained into the webhook, but I would now like to remove duplicates.

I would like to go from:
email1, email2, email2, email1, email3
to
email1, email2, email3
before the workflow continues onto other process’s

I have found this script and put it in a function node:

const items = [1, 2, 3, 2, 3, 1, 5]

const unique = [ ...new Set(items)]  
// [1, 2, 3, 5]

but I do not know how to substitute the [1, 2, 3, 2, 3, 1, 5] with my json of emails.

Please can you help, thank you!!

Kind regards

Paul

The Function-Node would then look like this:

const emails = [ ...new Set(items[0].json.data)];
return emails.map(data => { return { json: { data } } });

Hi Jan, Thanks very much for that, I get it and only unique values are returned as expect, but how do I get the emails into the correct ‘format’ such this function can output correctly.

In other words I have an output from the set node like this:

“data”:
[
{“hit”:
{
“id”:
78722
,
“dateHit”
:
“2020-11-26T16:14:47+00:00”
,
“dateLeft”
:
null
,
“page”
:
null
,
“redirect”
:
[
]
,
“email”
:
{

And need to grab all the email values and output as “email1”, “email2” etc for your function to them remove the duplicates.

Thank you!

Paul

Hi Jan,

To save going in the wrong direction I just wanted to explain the goal again:

I have a webhook that I then use the node set to output under data. This will normally give several results of which have many fields and values as part of each result.

I would like to be able to filter out/remove results that have an email field with a matching value.

If I am not clear please let me know.

Thank you!

Paul

Can you please clean up the example output above to be valid JSON and make sure it gets displayed correctly. Currently I have sadly no idea what is going on there and would have spent quite some time cleaning it up myself. Thanks!

Example:

{
  "key": "value"
}

Hi Jan,

Sorry about that! Here is a cleaned up sample. I have marked the email value as ‘Target Value’ which is what I am trying to filter by duplicates. Thank you!

    {
     "hit":{
      "id":78719,
      "dateHit":"2020-11-26T16:12:57+00:00",
      "dateLeft":null,
      "page":null,
      "redirect":[
         
      ],
      "email":{
         "id":105,
         "name":"xxxxxxxx",
         "subject":"xxxxxx",
         "language":"en",
         "category":[
            
         ],
         "fromAddress":null,
         "fromName":null,
         "replyToAddress":null,
         "bccAddress":null,
         "utmTags":{
            "utmSource":"mautic",
            "utmMedium":"broadcast",
            "utmCampaign":"propertymatch",
            "utmContent":null
         }
      }
   },
   "lead":{
      "id":2441,
      "points":2660,
      "color":null,
      "fields":{
         "core":{
            "email":{
               "id":"6",
               "label":"Email",
               "alias":"email",
               "type":"email",
               "group":"core",
               "object":"lead",
               "is_fixed":"1",
               "properties":"a:0:{}",
               "default_value":null,
               "value":"TARGET VALUE",
               "normalizedValue":"xxxxx"
            }
         }
      }
   }
}

Do I understand correctly that this is the data you receive? In that case, it does sadly not make any sense to me right now. If the data is like you describe above I can not see a way how you could ever receive more than one email address. At least one value above "value":"TARGET VALUE", has to be an array.

Hi Jan, The sample data is one result - the webhook contains multiples of this result. So there will be a number of emails contained in the full webhook output. Thank you!

Hi Jan,

Thinking about it there may be another way. If the bundles can be sorted by email or some other value I could use an IF to say if the last email equals the incoming email return to split in batches node if not match continue to the next node.

Is it possible to not just map but also map and sort by a unique key id contained within the data please?

Kind regards

Paul

Hey @paul2000!

What do you want to do after you filter out (remove duplicate) the emails? The code snippet that Jan shared earlier should give you the expected data.

Can you share sample data with more than 2 emails? The sample data you shared earlier doesn’t help in recreating your workflow. If you can share the workflow that would be much better.

Hi @harshil1712,

Please see the below workflow. Please let me know if you need more information.

{
  "name": "Copy for export",
  "nodes": [
    {
      "parameters": {},
      "name": "Start",
      "type": "n8n-nodes-base.start",
      "typeVersion": 1,
      "position": [
        160,
        -10
      ],
      "disabled": true
    },
    {
      "parameters": {
        "conditions": {
          "string": [
            {
              "value1": "={{$json[\"buyerid\"]}}",
              "operation": "contains",
              "value2": "true"
            }
          ]
        }
      },
      "name": "IF Buyer",
      "type": "n8n-nodes-base.if",
      "typeVersion": 1,
      "position": [
        850,
        190
      ],
      "alwaysOutputData": false
    },
    {
      "parameters": {
        "conditions": {
          "string": [
            {
              "value1": "={{$node[\"Set\"].json[\"rightvalid\"]}}",
              "operation": "contains",
              "value2": "true"
            }
          ]
        }
      },
      "name": "IF Rightval",
      "type": "n8n-nodes-base.if",
      "typeVersion": 1,
      "position": [
        860,
        370
      ]
    },
    {
      "parameters": {
        "httpMethod": "POST",
        "path": "XXX",
        "options": {}
      },
      "name": "Webhook",
      "type": "n8n-nodes-base.webhook",
      "typeVersion": 1,
      "position": [
        160,
        150
      ],
      "webhookId": "XXX"
    },
    {
      "parameters": {
        "requestMethod": "PUT",
        "url": "=https://api.knack.com/v1/objects/object_1/records/{{$json[\"data\"][\"hit\"][\"lead\"][\"fields\"][\"core\"][\"buyerid\"][\"value\"]}}",
        "options": {
          "bodyContentType": "json"
        },
        "bodyParametersUi": {
          "parameter": [
            {
              "name": "field_789",
              "value": "={{$node[\"Set\"].json[\"data\"][\"timestamp\"]}}"
            },
            {
              "name": "field_796",
              "value": "={{$node[\"Set\"].json[\"data\"][\"hit\"][\"lead\"][\"points\"]}}"
            }
          ]
        },
        "headerParametersUi": {
          "parameter": [
            {
              "name": "X-Knack-Application-Id",
              "value": "XXX"
            },
            {
              "name": "X-Knack-REST-API-KEY",
              "value": "XXX"
            }
          ]
        }
      },
      "name": "Update Knack Buyer Page Hit",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 1,
      "position": [
        1090,
        170
      ]
    },
    {
      "parameters": {
        "requestMethod": "PUT",
        "url": "=https://api.knack.com/v1/objects/object_43/records/{{$node[\"Set\"].json[\"data\"][\"hit\"][\"lead\"][\"fields\"][\"core\"][\"rightvalid\"][\"value\"]}}",
        "options": {
          "bodyContentType": "json"
        },
        "bodyParametersUi": {
          "parameter": [
            {
              "name": "field_790",
              "value": "={{$node[\"Set\"].json[\"data\"][\"timestamp\"]}}"
            },
            {
              "name": "field_798",
              "value": "={{$node[\"Set\"].json[\"data\"][\"hit\"][\"lead\"][\"points\"]}}"
            }
          ]
        },
        "headerParametersUi": {
          "parameter": [
            {
              "name": "X-Knack-Application-Id",
              "value": "XXX"
            },
            {
              "name": "X-Knack-REST-API-KEY",
              "value": "XXX"
            }
          ]
        }
      },
      "name": "Update Knack Rightval Page Hit",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 1,
      "position": [
        1090,
        350
      ]
    },
    {
      "parameters": {
        "functionCode": "const waitTimeSeconds = 1;\n\nreturn new Promise((resolve) => {\n  setTimeout(() => {\n    resolve(items);\n  }, waitTimeSeconds * 1000);\n});\n\n"
      },
      "name": "Wait",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        1300,
        350
      ]
    },
    {
      "parameters": {
        "functionCode": "const waitTimeSeconds = 1;\n\nreturn new Promise((resolve) => {\n  setTimeout(() => {\n    resolve(items);\n  }, waitTimeSeconds * 1000);\n});\n\n"
      },
      "name": "Wait2",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        1300,
        170
      ]
    },
    {
      "parameters": {
        "functionCode": "const waitTimeSeconds = 1;\n\nreturn new Promise((resolve) => {\n  setTimeout(() => {\n    resolve(items);\n  }, waitTimeSeconds * 1000);\n});\n\n"
      },
      "name": "Wait1",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        1300,
        520
      ]
    },
    {
      "parameters": {
        "conditions": {
          "string": [
            {
              "value1": "={{$node[\"Set\"].json[\"sellerid\"]}}",
              "operation": "contains",
              "value2": "true"
            }
          ]
        }
      },
      "name": "IF Seller",
      "type": "n8n-nodes-base.if",
      "typeVersion": 1,
      "position": [
        860,
        540
      ]
    },
    {
      "parameters": {
        "requestMethod": "PUT",
        "url": "=https://api.knack.com/v1/objects/object_2/records/{{$node[\"Set\"].json[\"data\"][\"hit\"][\"lead\"][\"fields\"][\"core\"][\"sellerid\"][\"value\"]}}",
        "options": {
          "bodyContentType": "json"
        },
        "bodyParametersUi": {
          "parameter": [
            {
              "name": "field_791",
              "value": "={{$node[\"Set\"].json[\"data\"][\"timestamp\"]}}"
            },
            {
              "name": "field_797",
              "value": "={{$node[\"Set\"].json[\"data\"][\"hit\"][\"lead\"][\"points\"]}}"
            }
          ]
        },
        "headerParametersUi": {
          "parameter": [
            {
              "name": "X-Knack-Application-Id",
              "value": "XXX"
            },
            {
              "name": "X-Knack-REST-API-KEY",
              "value": "XXX"
            }
          ]
        }
      },
      "name": "Update Knack Seller Page Hit",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 1,
      "position": [
        1090,
        520
      ]
    },
    {
      "parameters": {
        "batchSize": 1,
        "options": {
          "reset": false
        }
      },
      "name": "SplitInBatches",
      "type": "n8n-nodes-base.splitInBatches",
      "typeVersion": 1,
      "position": [
        530,
        770
      ],
      "alwaysOutputData": false
    },
    {
      "parameters": {
        "keepOnlySet": true,
        "values": {
          "string": [
            {
              "name": "data",
              "value": "={{$node[\"Webhook\"].json[\"body\"][\"mautic.page_on_hit\"]}}"
            }
          ]
        },
        "options": {}
      },
      "name": "Set1",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        410,
        150
      ]
    },
    {
      "parameters": {
        "functionCode": "return items[0].json.data.map(data => { return { json: { data } } });\nreturn items\n"
      },
      "name": "Function",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        410,
        440
      ]
    },
    {
      "parameters": {
        "values": {
          "string": [],
          "boolean": [
            {
              "name": "rightvalid",
              "value": "={{!!Object.keys($json[\"data\"][\"hit\"][\"lead\"][\"fields\"][\"core\"][\"rightvalid\"][\"value\"]).length}}"
            },
            {
              "name": "buyerid",
              "value": "={{!!Object.keys($json[\"data\"][\"hit\"][\"lead\"][\"fields\"][\"core\"][\"buyerid\"][\"value\"]).length}}"
            },
            {
              "name": "sellerid",
              "value": "={{!!Object.keys($json[\"data\"][\"hit\"][\"lead\"][\"fields\"][\"core\"][\"sellerid\"][\"value\"]).length}}"
            }
          ]
        },
        "options": {}
      },
      "name": "Set",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        680,
        190
      ]
    },
    {
      "parameters": {
        "conditions": {
          "string": [
            {
              "value1": "={{$json[\"data\"][\"hit\"][\"lead\"][\"fields\"][\"core\"][\"email\"][\"value\"]}}",
              "operation": "notEqual",
              "value2": "={{$node[\"Set\"].json[\"data\"][\"hit\"][\"lead\"][\"fields\"][\"core\"][\"email\"][\"value\"]}}"
            }
          ],
          "boolean": []
        }
      },
      "name": "Skip Record",
      "type": "n8n-nodes-base.if",
      "typeVersion": 1,
      "position": [
        660,
        490
      ]
    }
  ],
  "connections": {
    "Webhook": {
      "main": [
        [
          {
            "node": "Set1",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "IF Buyer": {
      "main": [
        [
          {
            "node": "Update Knack Buyer Page Hit",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "IF Rightval",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "IF Rightval": {
      "main": [
        [
          {
            "node": "Update Knack Rightval Page Hit",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "IF Seller",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "IF Seller": {
      "main": [
        [
          {
            "node": "Update Knack Seller Page Hit",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "SplitInBatches",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Update Knack Rightval Page Hit": {
      "main": [
        [
          {
            "node": "Wait",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Update Knack Buyer Page Hit": {
      "main": [
        [
          {
            "node": "Wait2",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Update Knack Seller Page Hit": {
      "main": [
        [
          {
            "node": "Wait1",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Wait2": {
      "main": [
        [
          {
            "node": "IF Rightval",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Set1": {
      "main": [
        [
          {
            "node": "Function",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Function": {
      "main": [
        [
          {
            "node": "SplitInBatches",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "SplitInBatches": {
      "main": [
        [
          {
            "node": "Skip Record",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Wait": {
      "main": [
        [
          {
            "node": "IF Seller",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Wait1": {
      "main": [
        [
          {
            "node": "SplitInBatches",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Set": {
      "main": [
        [
          {
            "node": "IF Buyer",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Skip Record": {
      "main": [
        [
          {
            "node": "Set",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "SplitInBatches",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  },
  "active": false,
  "settings": {
    "timezone": "Europe/London",
    "executionTimeout": 30,
    "saveDataSuccessExecution": "none"
  },
  "id": "117"
}

Thanks for sharing the workflow. Can you also share sample data with more than 2 emails (similar to what you receive in the Webhook node)?

Hi @harshil1712,

Here is a sample from the webhook. It is the most text I can fit in the box.

    {
   "mautic.page_on_hit":[
      {
         "hit":{
            "id":78710,
            "dateHit":"2020-11-26T15:59:41+00:00",
            "dateLeft":null,
            "page":null,
            "redirect":null,
            "email":null,
            "lead":{
               "id":41852,
               "points":0,
               "color":null,
               "fields":{
                  "core":{
                     "email":{
                        "id":"6",
                        "label":"Email",
                        "alias":"email",
                        "type":"email",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "phone":{
                        "id":"8",
                        "label":"Phone",
                        "alias":"phone",
                        "type":"tel",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "points":{
                        "id":"9",
                        "label":"Points",
                        "alias":"points",
                        "type":"number",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":"0",
                        "value":0,
                        "normalizedValue":0
                     },
                     "address1":{
                        "id":"11",
                        "label":"Address Line 1",
                        "alias":"address1",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "address2":{
                        "id":"12",
                        "label":"Address Line 2",
                        "alias":"address2",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "city":{
                        "id":"13",
                        "label":"City",
                        "alias":"city",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "zipcode":{
                        "id":"15",
                        "label":"Post Code",
                        "alias":"zipcode",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "last_active":{
                        "id":"19",
                        "label":"Date Last Active",
                        "alias":"last_active",
                        "type":"datetime",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "attribution_date":{
                        "id":"20",
                        "label":"Attribution Date",
                        "alias":"attribution_date",
                        "type":"datetime",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "attribution":{
                        "id":"21",
                        "label":"Attribution",
                        "alias":"attribution",
                        "type":"number",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:2:{s:9:\"roundmode\";i:4;s:5:\"scale\";i:2;}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "website":{
                        "id":"22",
                        "label":"Website",
                        "alias":"website",
                        "type":"url",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "rightvalid":{
                        "id":"45",
                        "label":"Rightval ID",
                        "alias":"rightvalid",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "maxprice":{
                        "id":"46",
                        "label":"Max Price",
                        "alias":"maxprice",
                        "type":"number",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:2:{s:9:\"roundmode\";s:1:\"3\";s:5:\"scale\";s:0:\"\";}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "buyerid":{
                        "id":"47",
                        "label":"Buyer ID",
                        "alias":"buyerid",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "minimumbeds":{
                        "id":"48",
                        "label":"Minimum Beds",
                        "alias":"minimumbeds",
                        "type":"number",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:2:{s:9:\"roundmode\";s:1:\"3\";s:5:\"scale\";s:0:\"\";}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "sellingposition":{
                        "id":"49",
                        "label":"Selling Position",
                        "alias":"sellingposition",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "buyingposition":{
                        "id":"50",
                        "label":"Buying Position",
                        "alias":"buyingposition",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "lastviewed":{
                        "id":"51",
                        "label":"Last Viewed",
                        "alias":"lastviewed",
                        "type":"date",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "fullname":{
                        "id":"52",
                        "label":"Full Name",
                        "alias":"fullname",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "dearname":{
                        "id":"53",
                        "label":"Dear Name",
                        "alias":"dearname",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "sellerid":{
                        "id":"54",
                        "label":"Seller ID",
                        "alias":"sellerid",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "optinsms":{
                        "id":"55",
                        "label":"Opt-In SMS",
                        "alias":"optinsms",
                        "type":"boolean",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:2:{s:2:\"no\";s:5:\"false\";s:3:\"yes\";s:4:\"true\";}",
                        "default_value":"1",
                        "value":1,
                        "normalizedValue":"true"
                     },
                     "completiondate":{
                        "id":"57",
                        "label":"Completion Date",
                        "alias":"completiondate",
                        "type":"date",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     }
                  },
                  "social":{
                     "facebook":{
                        "id":"23",
                        "label":"Facebook",
                        "alias":"facebook",
                        "type":"text",
                        "group":"social",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     }
                  },
                  "personal":[

                  ],
                  "professional":[

                  ]
               }
            },
            "ipAddress":[

            ],
            "country":"",
            "region":"",
            "city":"",
            "isp":"",
            "organization":"",
            "code":200,
            "referer":"www.google.co.uk",
            "url":"https://www.xxx.co.uk/bungalows-for-sale-xxxx/",
            "urlTitle":"Bungalows For Sale In xxxx | xxx xx",
            "userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15",
            "remoteHost":null,
            "pageLanguage":"en-gb",
            "browserLanguages":[
               "en-gb"
            ],
            "trackingId":"bgh08ltjzovqipgh4vcuvxs",
            "source":null,
            "sourceId":null,
            "query":{
               "page_title":"Bungalows For Sale In xxxx | xxx xx",
               "page_language":"en-gb",
               "page_referrer":"www.google.co.uk",
               "page_url":"https://www.xxx.co.uk/bungalows-for-sale-xxxx/",
               "counter":"0",
               "timezone_offset":"0",
               "resolution":"768x1024",
               "platform":"MacOS",
               "do_not_track":"false"
            }
         },
         "timestamp":"2020-11-26T15:59:41+00:00"
      },
      {
         "hit":{
            "id":78711,
            "dateHit":"2020-11-26T15:59:44+00:00",
            "dateLeft":null,
            "page":null,
            "redirect":null,
            "email":null,
            "lead":{
               "id":41853,
               "points":0,
               "color":null,
               "fields":{
                  "core":{
                     "email":{
                        "id":"6",
                        "label":"Email",
                        "alias":"email",
                        "type":"email",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "phone":{
                        "id":"8",
                        "label":"Phone",
                        "alias":"phone",
                        "type":"tel",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "points":{
                        "id":"9",
                        "label":"Points",
                        "alias":"points",
                        "type":"number",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":"0",
                        "value":0,
                        "normalizedValue":0
                     },
                     "address1":{
                        "id":"11",
                        "label":"Address Line 1",
                        "alias":"address1",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "address2":{
                        "id":"12",
                        "label":"Address Line 2",
                        "alias":"address2",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "city":{
                        "id":"13",
                        "label":"City",
                        "alias":"city",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "zipcode":{
                        "id":"15",
                        "label":"Post Code",
                        "alias":"zipcode",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "last_active":{
                        "id":"19",
                        "label":"Date Last Active",
                        "alias":"last_active",
                        "type":"datetime",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "attribution_date":{
                        "id":"20",
                        "label":"Attribution Date",
                        "alias":"attribution_date",
                        "type":"datetime",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "attribution":{
                        "id":"21",
                        "label":"Attribution",
                        "alias":"attribution",
                        "type":"number",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"1",
                        "properties":"a:2:{s:9:\"roundmode\";i:4;s:5:\"scale\";i:2;}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "website":{
                        "id":"22",
                        "label":"Website",
                        "alias":"website",
                        "type":"url",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "rightvalid":{
                        "id":"45",
                        "label":"Rightval ID",
                        "alias":"rightvalid",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "maxprice":{
                        "id":"46",
                        "label":"Max Price",
                        "alias":"maxprice",
                        "type":"number",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:2:{s:9:\"roundmode\";s:1:\"3\";s:5:\"scale\";s:0:\"\";}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "buyerid":{
                        "id":"47",
                        "label":"Buyer ID",
                        "alias":"buyerid",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "minimumbeds":{
                        "id":"48",
                        "label":"Minimum Beds",
                        "alias":"minimumbeds",
                        "type":"number",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:2:{s:9:\"roundmode\";s:1:\"3\";s:5:\"scale\";s:0:\"\";}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "sellingposition":{
                        "id":"49",
                        "label":"Selling Position",
                        "alias":"sellingposition",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "buyingposition":{
                        "id":"50",
                        "label":"Buying Position",
                        "alias":"buyingposition",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "lastviewed":{
                        "id":"51",
                        "label":"Last Viewed",
                        "alias":"lastviewed",
                        "type":"date",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "fullname":{
                        "id":"52",
                        "label":"Full Name",
                        "alias":"fullname",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "dearname":{
                        "id":"53",
                        "label":"Dear Name",
                        "alias":"dearname",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "sellerid":{
                        "id":"54",
                        "label":"Seller ID",
                        "alias":"sellerid",
                        "type":"text",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     },
                     "optinsms":{
                        "id":"55",
                        "label":"Opt-In SMS",
                        "alias":"optinsms",
                        "type":"boolean",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:2:{s:2:\"no\";s:5:\"false\";s:3:\"yes\";s:4:\"true\";}",
                        "default_value":"1",
                        "value":1,
                        "normalizedValue":"true"
                     },
                     "completiondate":{
                        "id":"57",
                        "label":"Completion Date",
                        "alias":"completiondate",
                        "type":"date",
                        "group":"core",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     }
                  },
                  "social":{
                     "facebook":{
                        "id":"23",
                        "label":"Facebook",
                        "alias":"facebook",
                        "type":"text",
                        "group":"social",
                        "object":"lead",
                        "is_fixed":"0",
                        "properties":"a:0:{}",
                        "default_value":null,
                        "value":null,
                        "normalizedValue":null
                     }
                  },
                  "personal":[

                  ],
                  "professional":[

                  ]
               }
            },
            "ipAddress":[

            ],
            "country":"",
            "region":"",
            "city":"",
            "isp":"",
            "organization":"",
            "code":200,
            "referer":"www.xxx.co.uk",
            "url":"https://www.xxx.co.uk/property/1009-highfields-avenue-xxxx/",
            "urlTitle":"Property For Sale In Highfields Avenue, xxxx £400000 | xxx xx",
            "userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15",
            "remoteHost":null,
            "pageLanguage":"en-gb",
            "browserLanguages":[
               "en-gb"
            ],
            "trackingId":"w6265s2gy6wosq5ct23jeif",
            "source":null,
            "sourceId":null,
            "query":{
               "page_title":"Property For Sale In Highfields Avenue, xxxx £400000 | xxx xx",
               "page_language":"en-gb",
               "page_referrer":"www.xxx.co.uk",
               "page_url":"https://www.xxx.co.uk/property/1009-highfields-avenue-xxxx/",
               "counter":"0",
               "timezone_offset":"0",
               "resolution":"768x1024",
               "platform":"MacOS",
               "do_not_track":"false"
            }
         },
         "timestamp":"2020-11-26T15:59:44+00:00"
      },
      }

Sorry @harshil1712, I do not seem to have mastered paste as code. How do I do that please? Kind regards Paul

No issues. I have fixed it. :slightly_smiling_face:
For future reference, you can wrap the code between ``` and ```.

Hey @paul2000!

I came up with the following code snippet that removes the duplicate entries and gives you unique values. I hope this helps :slightly_smiling_face:

let newArr = [];
let newObj = {};

for(let i in items[0].json.data){
  emailValue=items[0].json.data[i].hit.lead.fields.core.email.value;
  newObj[emailValue] = items[0].json.data[i]
}
for(let i in newObj){
  newArr.push(newObj[i]);
}
return [{json:newArr}];

Thank you so much @harshil1712, I will set it up later today.

Kind regards

Paul

1 Like

Hi @harshil1712,

Please can you tell me how to output the newArr under ‘data’, it is just when the workflow gets to split in batches it thinks there is only one batch, I guess because the each item is under hit and it needs to be data then the list of hits.

Thank You!

Paul

Hi @harshil1712,

Ah ok, I think I have it now. I have to place your code ahead of the mapping function to get it to pass through the split in batches node as desired.

Thank you!

1 Like

I am glad you got it working. Have fun! :slightly_smiling_face: