Help me with data storing

Hi community! :wave:

I’m working on an automation in n8n and have a workflow that follows this logic:

  1. I fetch relevant records (projects) from Airtable.
  2. For each record, I make an HTTP request to get the related comments from an external API.
  3. I merge all comments into a single message and send it to Slack.

The problem I’m facing is: every time the workflow runs, it sends ALL comments again, including those that were already sent previously. What I need is a solution that:

  • Tracks which comments have already been sent to Slack,
  • Sends only new comments the next time the workflow runs (i.e., comments that appeared since the last run).

Ideally, this should work per record/project, so it keeps track of sent comments individually for each record.

I’m looking for the best implementation approach in n8n.

Hi, this is an architectural question tbh. It all depends on what Information is available. If the API call for the comments can be limited by date time that would be a start. If you poll every 5 mins you move the filter window. Obviously this works only when service always runs without stopping. The other option is to check if comments have unique ids or can be marked with custom fields. Like sent and filter on them. Of nothing of the above would work. The only thing I can think of is calculating a hash and storing and comparing it against that.

Reg,
J.

Please share your workflow instead of a screenshot in order to have at least some idea

In Make, I have a Data Store. I stored every pushed comment id into Data Store, in the next run I compare new comment id with those that was already pushed. If there were such id in Data Store it was missed if not it pushed it and wrote this id to Data Store.

Well ok, you have many options to store data. The concepts are the same. Either you host your storage or you get it through the cloud. Or you use an API for storage. Like I said, the concepts are the same. You just need to select.

1 Like

But i want to implement this without any other services. I think there is some built-in features like Data Store in Make

A store between runs. But how persistent and scalable is that solution. Better build something that is independent from the automation solution. That’s my advice

Just use SQLite or something. If will do the job perfectly

1 Like

In any case, thank you for this comment. Because for now i understand that i need some additional API calls to store this id and then compare them with new ones. :pray:

SQLite is like a database but in a file. It works great. But you need to be on self hosted I think. Also make sure you backup the file

1 Like

Thanks. I`m on self hosted version

1 Like

Hi, can you please mark as a solution? Thank you!

How can I do that?)

Under each comment there should be something like a “solution” with check mark. You can tick it


Nothing(

Not sure :frowning: it would give me some points. No worries . Good luck