Retrieve new data by polling - best practise?

Hi all,

current situation is that there should be some action when a new project is created in an project management tool (Redmine). Redmine has a reasonable API, but does not support WebHooks. As I could not be called on project creation, I have to readout the list of projects and trigger some workflow when a new entry is found.
So e.g. call https://www.redmine.org/projects/redmine/issues.json?created_on=>%3D2019-09-29T08:12:32Z every 30sec and increase the timestamp by 30sec, to get every change within last 30sec. Imho this is error-prone and gets complicated beside the happy path (need to modify next request timespan when call fails, etc…).

Does anybody has a verified receipt for such a handling? Actually I have two more sources not supporting WebHooks with similar requirements.

Thank you in advance!

br,
Markus

Hm quite surprised. Would have expected that every reasonable mature system supports webhooks. If they do not, then all solutions are very waisful as polling is really not great.

Anyway the best and most secure way is probably to request all x seconds/minutes the data for x + y seconds/minutes. With y reasonable large to be sure that no new entry will ever be skipped. The processed entries have then to be written to some kind of database like Postgres and whenever a workflow starts it has to check first if entries got processed already and remove the already processed ones. That would then also have the advantage that if, for what reason ever, the polling stops it would be possible to easily process all missed ones. An easier version would be to simply save the id of the last processed entry or the time of it and then request everything after that. Anyway, if a lot of entries have to get processed at once, and some fail, it could become problematic.

If that is a very common problem we could actually think about creating a node which does that. With a mode to add processed entries and another one to filter out already processed ones.

This would be a perfect solution for my problem. I already started doing a workflow manually for this handling, but it’s really getting complicated only to generate the source(!) for the whole pipe.
Actually I could not (should not…) support you in development, as I didn’t have deep experience using JS.

Anything else I could support with?

Ah yes, there many ways to support n8n without writing a line of code. Some of the possibilities are listed on this page:
https://n8n.io/contribute

All help in what form ever is very appreciated!