Large JSON result filter

[mvandyk]

Describe the issue/error/question

We have a query that returns a large amount of items. One field is a timestamp “clock” I am trying to filter/reduce the amount of results by only keeping the items that are less than more than 600 seconds different from the preceding item. I am able to do this with Split in Batches but due to the amount of items sometimes returned the workflow crashes.
Is there a way to filter via a function?

What is the error message (if any)?

Please share the workflow

(Select the nodes and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow respectively)

{
“meta”: {
“instanceId”: “3b4f5dc05fe576016b69b65bb579b15d215a729edd26875172032953dda064c6”
},
“nodes”: [
{
“parameters”: {
“authentication”: “apiToken”,
“resource”: “history”,
“parametersUi”: {
“itemids”: [
{
“id”: “=1401621”
}
],
“time_from”: 1661990401,
“time_till”: 1664582399,
“filter”: {
“filter”: [
{
“key”: “value”,
“values”: [
{
“value”: 0
}
]
}
]
},
“limit”: 10,
“sortfield”: [
“clock”
],
“sortorder”: [
{
“sortorder”: “ASC”
}
]
}
},
“id”: “473f088e-e8a0-4e37-a447-4c6259326010”,
“name”: “Zabbix2”,
“type”: “@digital-boss/n8n-nodes-zabbix.zabbix”,
“typeVersion”: 1,
“position”: [
600,
400
],
“credentials”: {
“zabbixTokenApi”: {
“id”: “2”,
“name”: “Zabbix Token account”
}
}
}
],
“connections”: {}
}

Share the output returned by the last node

Currently limited t o10 results… can be thousands.

[
{
“itemid”: “1401621”,
“clock”: “1662702930”,
“value”: “0”,
“ns”: “276997647”
},
{
“itemid”: “1401621”,
“clock”: “1662703110”,
“value”: “0”,
“ns”: “91224002”
},
{
“itemid”: “1401621”,
“clock”: “1662707250”,
“value”: “0”,
“ns”: “448817543”
},
{
“itemid”: “1401621”,
“clock”: “1662709410”,
“value”: “0”,
“ns”: “39214024”
},
{
“itemid”: “1401621”,
“clock”: “1662709470”,
“value”: “0”,
“ns”: “276662476”
},
{
“itemid”: “1401621”,
“clock”: “1662709710”,
“value”: “0”,
“ns”: “259508969”
},
{
“itemid”: “1401621”,
“clock”: “1662720270”,
“value”: “0”,
“ns”: “75857259”
},
{
“itemid”: “1401621”,
“clock”: “1662721170”,
“value”: “0”,
“ns”: “212129837”
},
{
“itemid”: “1401621”,
“clock”: “1662804210”,
“value”: “0”,
“ns”: “187596120”
},
{
“itemid”: “1401621”,
“clock”: “1662804870”,
“value”: “0”,
“ns”: “510817543”
}
]

Information on your n8n setup

• n8n version: 0.203
• Database you’re using (default: SQLite): Postgres
• Running n8n with the execution process [own(default), main]: main
• Running n8n via [Docker, npm, n8n.cloud, desktop app]: Docker

[mvandyk]

OK I figured out how to do this via set node. using $item.last

{{ $item((JSON.stringify($input.last()).split(“:”)[7]).split(“,”)[0]).$node[“Zabbix2”].json[“clock”] - $item(((JSON.stringify($input.last()).split(“:”)[7]).split(“,”)[0] -1)).$node[“Zabbix2”].json[“clock”] }}

I can then add a if node after.

[mvandyk]

Set However does not pass on the value, even though in the editor it works?

[marcus]

Hi @mvandyk,
to filter/reduce your items based on a timediff calculation with previous items you might be better of using the Code node. Using the code node with mode Run once for All Items should have great performance even with thousands of items.

Here is an example workflow using your provided sample data.

[mvandyk]

Thank you very much.