Import JSON data into a JS function

Hello,

I am collecting data from the Twitter API with an HTTP Get node.

This database (updated every hour) gives indications on hashflags (small graphic symbols appearing behind some hashtags) and notably on their activation time, with a timestamp expressed in milliseconds.

I would like to detect every hour which are the new ones from their respective timestamp. For that I deduce that I must use the Node IF with the date format.

However, to do this, I have to compare two dates (the current one) and the date of each timestamp communicated with dates in ISO8601 format.

My technical skills end here: I don’t know how to import a JSON data into a JS function that would allow me to transform the JSON data into an ISO8601 date.

Can anyone help me? (I’ve already read the documentation and some of the blog topics, don’t consider me lazy in advance, I’m French and may have trouble understanding sometimes)

Here is my current workflow:

{
  "nodes": [
    {
      "parameters": {},
      "name": "Start",
      "type": "n8n-nodes-base.start",
      "typeVersion": 1,
      "position": [
        450,
        550
      ]
    },
    {
      "parameters": {
        "functionCode": "const d = new Date();\nlet year = d.getFullYear();\nlet month = d.getMonth() + 1;\nlet day = d.getDate();\nif(day <= 9)\n    day = '0'+day;\nlet hour = d.getHours();\n\nreturn [\n  {\n    json: {\n      year: year,\n      month: month,\n      day: day,\n      hour: hour,\n    }\n  }\n];\n"
      },
      "name": "Get current date",
      "type": "n8n-nodes-base.function",
      "position": [
        650,
        550
      ],
      "typeVersion": 1
    },
    {
      "parameters": {
        "url": "=https://pbs.twimg.com/hashflag/config-{{$json[\"currentDate\"]}}.json",
        "options": {
          "splitIntoItems": true
        }
      },
      "name": "Get current hashflags",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 1,
      "position": [
        1050,
        550
      ]
    },
    {
      "parameters": {
        "keepOnlySet": true,
        "values": {
          "string": [
            {
              "name": "currentDate",
              "value": "={{$json[\"year\"]}}-{{$json[\"month\"]}}-{{$json[\"day\"]}}-{{$json[\"hour\"]}}"
            }
          ]
        },
        "options": {}
      },
      "name": "Prettier date",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        850,
        550
      ]
    },
    {
      "parameters": {
        "keepOnlySet": true,
        "values": {
          "string": [
            {
              "name": "Nom de la campagne",
              "value": "={{$json[\"campaignName\"]}}"
            },
            {
              "name": "Hashtag",
              "value": "={{$json[\"hashtag\"]}}"
            },
            {
              "name": "URL du hashflag",
              "value": "={{$json[\"assetUrl\"]}}"
            },
            {
              "name": "Timestamp activation",
              "value": "={{$json[\"startingTimestampMs\"]}}"
            },
            {
              "name": "Timestamp désactivation",
              "value": "={{$json[\"endingTimestampMs\"]}}"
            }
          ]
        },
        "options": {}
      },
      "name": "Prettier datas",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        1250,
        550
      ]
    },
    {
      "parameters": {
        "conditions": {
          "dateTime": [
            {}
          ]
        }
      },
      "name": "New hashflag ?",
      "type": "n8n-nodes-base.if",
      "typeVersion": 1,
      "position": [
        1650,
        550
      ]
    },
    {
      "parameters": {},
      "name": "Transform timestamp to ISO8601 date",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        1450,
        550
      ]
    }
  ],
  "connections": {
    "Start": {
      "main": [
        [
          {
            "node": "Get current date",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Get current date": {
      "main": [
        [
          {
            "node": "Prettier date",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Get current hashflags": {
      "main": [
        [
          {
            "node": "Prettier datas",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Prettier date": {
      "main": [
        [
          {
            "node": "Get current hashflags",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Prettier datas": {
      "main": [
        [
          {
            "node": "Transform timestamp to ISO8601 date",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Transform timestamp to ISO8601 date": {
      "main": [
        [
          {
            "node": "New hashflag ?",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}

Not sure if it’s helpful, but I would just simply save some id in static data and after that just remove those which I already get.

Let me know if you need help with that, I think Ill be able to help it this resolve will be acceptable

Following up from what @Shirobachi said, you might want to take a look at this blog post:

It describes the general approach to using static data quite well and might help you.

Good evening, gentlemen,

Thank you for your answers, I have followed the tutorial in the last reply to my post to the letter. Unfortunately, I can’t list the new hashflags released, compared to the workflow of the previous hour. And yet I know that some of them appeared during my tests.

I come back to my previous query: how can I extract the timestamp from each of the data to transform it into a date in ISO8601 format with a formula?

If ever: here is my new workflow just below

{
  "nodes": [
    {
      "parameters": {},
      "name": "Start",
      "type": "n8n-nodes-base.start",
      "typeVersion": 1,
      "position": [
        240,
        300
      ]
    },
    {
      "parameters": {
        "functionCode": "const d = new Date();\nlet year = d.getFullYear();\nlet month = d.getMonth() + 1;\nlet day = d.getDate();\nif(day <= 9)\n    day = '0'+day;\nlet hour = d.getHours();\n\nreturn [\n  {\n    json: {\n      year: year,\n      month: month,\n      day: day,\n      hour: hour,\n    }\n  }\n];\n"
      },
      "name": "Get current date",
      "type": "n8n-nodes-base.function",
      "position": [
        640,
        300
      ],
      "typeVersion": 1
    },
    {
      "parameters": {
        "url": "=https://pbs.twimg.com/hashflag/config-{{$json[\"currentDate\"]}}.json",
        "options": {
          "splitIntoItems": true
        }
      },
      "name": "Get current hashflags",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 1,
      "position": [
        1040,
        300
      ],
      "retryOnFail": true,
      "maxTries": 5,
      "waitBetweenTries": 5000
    },
    {
      "parameters": {
        "keepOnlySet": true,
        "values": {
          "string": [
            {
              "name": "currentDate",
              "value": "={{$json[\"year\"]}}-{{$json[\"month\"]}}-{{$json[\"day\"]}}-{{$json[\"hour\"]}}"
            }
          ]
        },
        "options": {}
      },
      "name": "Prettier date",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        840,
        300
      ]
    },
    {
      "parameters": {
        "triggerTimes": {
          "item": [
            {
              "mode": "everyHour",
              "minute": 45
            }
          ]
        }
      },
      "name": "Cron",
      "type": "n8n-nodes-base.cron",
      "typeVersion": 1,
      "position": [
        440,
        300
      ]
    },
    {
      "parameters": {
        "functionCode": "const new_items = [];\n\n// Get static data stored with the workflow\nconst data = this.getWorkflowStaticData(\"node\");\n\ndata.ids = data.ids || [];\n\nfor (let i = items.length - 1; i >= 0; i--) {\n\n\t// Check if data is already present\n\tif (data.ids.includes(items[i].json.startingTimestampMs)) {\n\t\tbreak;\n\t} else {\n\n\t\t// if new data then add it to an array\n\t\tnew_items.push({\n\t\t\tjson: {\n\t\t\t\tHashtag: items[i].json.hashtag,\n\t\t\t\tLien: items[i].json.assetUrl,\n \t\t\t    Fin: items[i].json.endingTimestampMs,\n\t\t\t},\n\t\t});\n\t}\n}\n\ndata.ids = items.map((item) => item.json.startingTimestampMs);\n\n// return new items\nreturn new_items;\n"
      },
      "name": "Test de fonction",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        1240,
        300
      ]
    }
  ],
  "connections": {
    "Get current date": {
      "main": [
        [
          {
            "node": "Prettier date",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Get current hashflags": {
      "main": [
        [
          {
            "node": "Test de fonction",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Prettier date": {
      "main": [
        [
          {
            "node": "Get current hashflags",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Cron": {
      "main": [
        [
          {
            "node": "Get current date",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}

Hey!

Have a look for this example to learn how to use them, be aware that static data will work only in backgroud. That’s mean that if you click run workflow will not save data to static data, so to test it set webhook and active workflow:

{
  "nodes": [
    {
      "parameters": {
        "functionCode": "// Get static data stored with the workflow\nconst data = this.getWorkflowStaticData(\"node\");\nhashtags = data.knownHashtags\n\n// if first use, then \nif(!hashtags){\n  hashtags = []\n  items.map(item => hashtags.push(item.json.hashtag))\n  data.knownHashtags = hashtags\n  \n  return items\n}\n\n// keep only new\nitems = items.filter(item => hashtags.indexOf(item.json.hashtag) == -1)\n\n// save new, to next time filter them\nitems.map(item => hashtags.push(item.json.hashtag))\ndata.knownHashtags = hashtags\n\nreturn items\nreturn[\n  {\n    \"json\":\n    {\n        \"staticData\": data.knownHashtags,\n      \"hashtags\": hashtags,\n      \"items\":items\n    }\n  }\n\n]\n\n// return new items\n"
      },
      "name": "Test de fonction",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        1180,
        300
      ]
    },
    {
      "parameters": {
        "url": "https://lorem-rss.herokuapp.com/feed?unit=second&interval=5"
      },
      "name": "RSS Feed Read",
      "type": "n8n-nodes-base.rssFeedRead",
      "typeVersion": 1,
      "position": [
        720,
        300
      ]
    },
    {
      "parameters": {
        "keepOnlySet": true,
        "values": {
          "string": [
            {
              "name": "hashtag",
              "value": "={{$json[\"link\"]}}"
            },
            {
              "name": "date",
              "value": "={{$json[\"pubDate\"]}}"
            }
          ]
        },
        "options": {}
      },
      "name": "Set2",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        940,
        300
      ]
    },
    {
      "parameters": {
        "webhookUri": "",
        "text": "={{$json[\"date\"]}} - {{$json[\"hashtag\"]}}"
      },
      "name": "Discord3",
      "type": "n8n-nodes-base.discord",
      "typeVersion": 1,
      "position": [
        1400,
        300
      ]
    },
    {
      "parameters": {
        "interval": 10
      },
      "name": "Interval",
      "type": "n8n-nodes-base.interval",
      "typeVersion": 1,
      "position": [
        460,
        300
      ]
    }
  ],
  "connections": {
    "Test de fonction": {
      "main": [
        [
          {
            "node": "Discord3",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "RSS Feed Read": {
      "main": [
        [
          {
            "node": "Set2",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Set2": {
      "main": [
        [
          {
            "node": "Test de fonction",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Interval": {
      "main": [
        [
          {
            "node": "RSS Feed Read",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}

What code do?
Get dummy data what is 10 last elements (new element every 5 seconds), learn more here.
After that it will push only new data to discord. See function block to see how I filter this data

I used your workflow from question post and changed only date generating bc date 21-12-2021-2 does return 404.

{
  "nodes": [
    {
      "parameters": {
        "functionCode": "// Get static data stored with the workflow\nconst data = this.getWorkflowStaticData(\"node\");\nhashtags = data.knownHashtags\n\n// if first use, then\n// You may want do sth else here to not spam at first use\nif(!hashtags){\n  hashtags = []\n  items.map(item => hashtags.push(item.json.Hashtag))\n  data.knownHashtags = hashtags\n  \n  return items\n}\n\n// keep only new\nitems = items.filter(item => hashtags.indexOf(item.json.Hashtag) == -1)\n\n// save new, to next time filter them\nitems.map(item => hashtags.push(item.json.Hashtag))\ndata.knownHashtags = hashtags\n\nreturn items\nreturn[\n  {\n    \"json\":\n    {\n        \"staticData\": data.knownHashtags,\n      \"hashtags\": hashtags,\n      \"items\":items\n    }\n  }\n\n]\n\n// return new items\n"
      },
      "name": "Test de fonction",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        1000,
        100
      ]
    },
    {
      "parameters": {
        "webhookUri": "https://discord.com/api/webhooks/922278105257766973/zRv_-XE9b_Az67k6WAEJWvUB-t6TSTJJMc-yRH9NSEfMN_NBia6U4W8hyBgRHEQzX7V0",
        "text": "={{$json[\"Hashtag\"]}}"
      },
      "name": "Discord3",
      "type": "n8n-nodes-base.discord",
      "typeVersion": 1,
      "position": [
        1180,
        100
      ]
    },
    {
      "parameters": {
        "functionCode": "const d = new Date();\nlet year = d.getFullYear();\nlet month = d.getMonth() + 1;\nlet day = d.getDate();\nif(day <= 9)\n    day = '0'+day;\nlet hour = d.getHours();\n\nreturn [\n  {\n    json: {\n      year: year,\n      month: month,\n      day: day,\n      hour: hour,\n    }\n  }\n];\n"
      },
      "name": "Get current date3",
      "type": "n8n-nodes-base.function",
      "position": [
        240,
        100
      ],
      "typeVersion": 1
    },
    {
      "parameters": {
        "url": "=https://pbs.twimg.com/hashflag/config-{{$json[\"currentDate\"]}}.json",
        "options": {
          "splitIntoItems": true
        }
      },
      "name": "Get current hashflags3",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 1,
      "position": [
        600,
        100
      ]
    },
    {
      "parameters": {
        "keepOnlySet": true,
        "values": {
          "string": [
            {
              "name": "currentDate",
              "value": "={{$json[\"year\"]}}-{{$json[\"month\"]}}-{{$json[\"day\"]}}"
            }
          ]
        },
        "options": {}
      },
      "name": "Prettier date3",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        400,
        100
      ]
    },
    {
      "parameters": {
        "keepOnlySet": true,
        "values": {
          "string": [
            {
              "name": "Nom de la campagne",
              "value": "={{$json[\"campaignName\"]}}"
            },
            {
              "name": "Hashtag",
              "value": "={{$json[\"hashtag\"]}}"
            },
            {
              "name": "URL du hashflag",
              "value": "={{$json[\"assetUrl\"]}}"
            },
            {
              "name": "Timestamp activation",
              "value": "={{$json[\"startingTimestampMs\"]}}"
            },
            {
              "name": "Timestamp désactivation",
              "value": "={{$json[\"endingTimestampMs\"]}}"
            }
          ]
        },
        "options": {}
      },
      "name": "Prettier datas2",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        800,
        100
      ]
    },
    {
      "parameters": {
        "triggerTimes": {
          "item": [
            {
              "mode": "everyHour"
            }
          ]
        }
      },
      "name": "Cron",
      "type": "n8n-nodes-base.cron",
      "typeVersion": 1,
      "position": [
        60,
        100
      ]
    }
  ],
  "connections": {
    "Test de fonction": {
      "main": [
        [
          {
            "node": "Discord3",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Get current date3": {
      "main": [
        [
          {
            "node": "Prettier date3",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Get current hashflags3": {
      "main": [
        [
          {
            "node": "Prettier datas2",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Prettier date3": {
      "main": [
        [
          {
            "node": "Get current hashflags3",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Prettier datas2": {
      "main": [
        [
          {
            "node": "Test de fonction",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Cron": {
      "main": [
        [
          {
            "node": "Get current date3",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}

Let me know if it helpful or if you have any futher question :slight_smile:

1 Like

Hello,

Thank you for your reply and sorry for the delay.

No wonder you get a 404 when you try, the JSON file is generated every XX:05 or so. So if the execution is done between XXh00 and XXh05 the HTTP call will not work. I didn’t specify that it’s my fault.

On the other hand, to circumvent this problem, you can specify in hard copy the date and the previous time to the current one, in general that works every time.

I will test your code and come back to you.

1 Like