Polling with Function node does not work

Lunoba · July 6, 2022, 9:40pm

helo,

I have tried to customize this to my data:

But I failed. Could you help me out, please? It does not create calender events if there are no new events. But it creates all the events if there is a new event, not just the new one. Strange.

Example polling code

const new_items = [];

// Get static data stored with the workflow
const data = this.getWorkflowStaticData("node");

data.ids = data.ids || [];

for (let i = items.length - 1; i >= 0; i--) {

	// Check if data is already present
	if (data.ids.includes(items[i].json.ID)) {
		break;
	} else {

		// if new data then add it to an array
		new_items.push({
			json: {
				id: items[i].json.ID,
				name: items[i].json.Name,
				email: items[i].json.Email
			},
		});
	}
}

data.ids = items.map((item) => item.json.ID);

// return new items
return new_items;

For this structured data

[
	{
		"id": 0,
		"name": "Sam",
		"email": "[email protected]"
	},
	{
		"id": 1,
		"name": "Ram",
		"email": "[email protected]"
	}
]

How to rewrite the polling code for this structure?

{
"name": "0f9545e151",
"date": "2022-07-06",
"description": "Sync To Do to Googel Calendar with n8n",
},

Thank you for your help

ps: I am on self hosted 0.181.1 version

MutedJam · July 7, 2022, 8:21am

Hi @Lunoba, are you by any chance running your workflow manually? The method to store data described in this blog post will only work for production executions, but not for manual workflow executions (when clicking “Execute Workflow”).

Lunoba · July 7, 2022, 8:49am

Hello @MutedJam

Thank you for trying to help me out.

Yes, I know, I read that part of that blogpost. I just activated the workflow and I got the described situation. I know it wont work if I execute by hand so I just did not that.

MutedJam · July 7, 2022, 3:25pm

Sorry, I must have misread your question this morning. So in your code you are storing data with an ID field:

items.map((item) => item.json.ID);

But your data doesn’t have that ID field, right? So you’d need to rewrite the code to use a suitable field from your own data structure. Is your name field unique by any chance? If so, this might be a suitable candidate.

For a true low-code solution you might, however, want to consider a different approach. You could, for example, write the processed names into a separate Google Sheet and then compare the “processed names” sheet with your incoming data using the Merge Node’s Remove Key Matches mode (similar to this approach, but with a one way sync from your data to the “processed names” sheet instead).

Lunoba · July 7, 2022, 7:47pm

I have tried some version, this is my latest based on your suggestion, but it has 0 data as outpout instead of the new ones. That is correct, name field is unique.

const new_items = [];

// Get static data stored with the workflow
const data = this.getWorkflowStaticData("node");

data.ids = data.ids || [];

for (let i = items.length - 1; i >= 0; i--) {

	// Check if data is already present
	if (data.ids.includes(items[i].json.name)) {
		break;
	} else {

		// if new data then add it to an array
		new_items.push({
			json: {
				name: items[i].json.name,
				date: items[i].json.date,
				description: items[i].json.description
			},
		});
	}
}

data.ids = items.map((item) => item.json.name);

// return new items
return new_items;

Do you have any hints?

ps: I don’t want to use google sheet for this.

MutedJam · July 8, 2022, 7:12am

It doesn’t have to be Google Sheets, it could be pretty much any database. It would mean your flow works manually and in production mode, save you from having to write code and make debugging a lot easier.

Your actual code looks good to me so I am not entirely sure what the problem might be. Can you add a console.log(data) statement to your code and take a look at your server log output to verify the data you have currently stored and compare it with your new items?

Perhaps your break; statement is terminating the loop early, so no additional items are being processed.

Lunoba · July 8, 2022, 8:36pm

I am testing with heroku version:

and I haven’t found that log yet. Do you have any idea?

And I found another similar solution, but I cannot adopt yet. would rewrite to me someone this polling, please?

thank you

MutedJam · July 11, 2022, 6:27am

Hey @Lunoba, the link you have shared uses static data too, so this is essentially the same approach. I’ve tried the below workflow using your existing code:

First time running the workflow returns the example data:

Running it a second time no longer returns the example data as it has been processed already:

Running it a third time after adding a new entry to the end of the mock data array causes only this new entry to be returned:

So it seems your code is working in principle. Can you confirm the full JSON data structure (both before and after adding a new element) you’re working with and also share your workflow?

Lunoba · July 12, 2022, 6:09pm

This is what I have as input

[
{
"name": "e20c1bb9da",
"creation": "2022-07-12 19:58:23.834066",
"modified": "2022-07-12 20:07:47.648285",
"modified_by": "Administrator",
"owner": "[email protected]",
"docstatus": 0,
"parent": null,
"parentfield": null,
"parenttype": null,
"idx": 0,
"status": "Open",
"priority": "Medium",
"color": null,
"date": "2022-07-12",
"description": "<div class="ql-editor read-mode"><p>1958</p></div>",
"reference_type": null,
"reference_name": null,
"role": null,
"assigned_by": null,
"assigned_by_full_name": null,
"sender": null,
"assignment_rule": null,
"_user_tags": null,
"_comments": null,
"_assign": null,
"_liked_by": null,
"_seen": "["Administrator"]"
},
{
"name": "1040f27270",
"creation": "2022-07-12 20:01:23.917217",
"modified": "2022-07-12 20:07:42.631439",
"modified_by": "Administrator",
"owner": "[email protected]",
"docstatus": 0,
"parent": null,
"parentfield": null,
"parenttype": null,
"idx": 0,
"status": "Open",
"priority": "Medium",
"color": null,
"date": "2022-07-12",
"description": "<div class="ql-editor read-mode"><p>2001</p></div>",
"reference_type": null,
"reference_name": null,
"role": null,
"assigned_by": null,
"assigned_by_full_name": null,
"sender": null,
"assignment_rule": null,
"_user_tags": null,
"_comments": null,
"_assign": null,
"_liked_by": null,
"_seen": "["Administrator"]"
}
]

It contains
unique name,
description and
date field as well.

And this is the output of polling node:

[
{
"name": "1040f27270",
"date": "2022-07-12",
"description": "<div class="ql-editor read-mode"><p>2001</p></div>"
},
{
"name": "e20c1bb9da",
"date": "2022-07-12",
"description": "<div class="ql-editor read-mode"><p>1958</p></div>"
}
]

What did I miss? Thank you for keep trying!

Lunoba · July 12, 2022, 6:14pm

Maybe there is one important thing here. I cannot fetch name field from the source, but I get it when I choose * (all fields):

Lunoba · July 23, 2022, 4:08pm

@MutedJam Thank you for trying. Do you have any other idea? Thank you

MutedJam · July 25, 2022, 5:59am

Hey, I must have missed your response, sorry for that.

As said, I don’t think using static workflow data is a good approach because it’s so hard to debug and verify.

I’ll take another look at your example data once I have some time.

MutedJam · July 25, 2022, 3:02pm

Hi @Lunoba, so I tested this once again using the below workflow:

Could you create this exact workflow in your own n8n instance, activate it and then visit the webhook production URL (not the test URL as static data isn’t available for manual executions) in your browser twice?

The first time you open it, both records should come through in the response like so:

The second time you open it, only an empty array should be in the response:

This will verify whether polling in general works for you. If it is, your workflow logic is likely to be the problem here. Could you please share your full workflow if that’s the case?

Lunoba · July 26, 2022, 6:00pm

Hello @MutedJam

I got two lines at the first run, then empty at the second. So your example code does work.

Here is my actual workflow what still push all the first time and does’t new ones later. Check it please when you have some time. Thank you

Lunoba · August 22, 2022, 5:33pm

One of my friend fixed the code, this one works. Thank you for your help. Keep up the good work!

const new_items = [];
 
// Get static data stored with the workflow
const data = this.getWorkflowStaticData("node");
 
data.ids = data.ids || [];
 
for (let i = 0 ; i <  items.length; i++) {
 
	// Check if data is already present
	if (data.ids.includes(items[i].json.name)) {
		continue;
	} else {
 
		// if new data then add it to an array
		new_items.push({
			json: {
				name: items[i].json.name,
				date: items[i].json.date,
				description: items[i].json.description
			},
		});
	}
}
 
data.ids = items.map((item) => item.json.name);
 
// return new items
return new_items;