Check Server Load - wait if greater than XYZ otherwise throttle

Questions
1

My instance

  • K8 with 10 workers and dedicated webhook (docker image)
  • Version 1.18.0
  • Postgresql managed
  • EXECUTIONS_PROCESS - main

My problem 3rd party website/server that has limited resources
(8gb Cloudpanel / WP CLI & REST API)

Running a webhook initiated workflow
I have instances where I may get several webhooks at once which over work the 3rd party server which is a node in the flow, with the server load blowing out and things have a knock on effect.

What is the best way to throttle how many executions can happen on a workflow at once?

Need some sort of queue / throttle, with the queue only proceeding when the 3rd party server load is < XYZ

My only thought would to put the webhooks into a temp database, then have a scheduled workflow to check the database and if records, check the server load via ssh, then loop through the database records - checking load in each loop.

Seems over complicating things

2

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
4

Hi @Dwayne_Taylor, running n8n in queue mode allows you to configure the concurrency. This is documented here: Configuring queue mode | n8n Docs

This setting does not exist on a per-workflow basis, but you could consider setting up a designated n8n instance for this special workflow if you don’t want to risk other workflows being throttled.

Need some sort of queue / throttle, with the queue only proceeding when the 3rd party server load is < XYZ
My only thought would to put the webhooks into a temp database, then have a scheduled workflow to check the database and if records, check the server load via ssh, then loop through the database records - checking load in each loop.

This is more complicated. If a fixed concurrency value as above is not sufficient, such an approach can indeed be a viable alternative.

5

Unloading my learnings for a future reference

@MutedJam - any efficiencies that could be made here? any hidden n8n functionality that would lean it up at all?

Webhook Postgre

  1. dumps the webhook payload into postgre along with ‘x-request-id’, a date, the workflowid and a status
  2. checks if the workflowid has executions running at the moment and how many, if more than X then do nothing
  3. if less than x executions running, then execute 2nd workflow (setup to pull down postgre and run in loop)

Step 2 (2nd workflow)

  • Get postgre records status = new
  • Set the x-request-id
  • loop through records
  • each loop change status of record to processing

  • do something

  • change status of record to complete
  • check server load - if greater than 4.5 wait 3 mins, and check again
  • close loop back and work on next record
1 Like
6

Hi @Dwayne_Taylor, tbh I can’t think of any useful improvements here. But if you have a bit of time it would be great if you can document your use case and the currently missing functionality over in the Feature Requests. This hopefully allows our product team to consider simplifying things going forward :slight_smile:

7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.