Preinstantiate Webhook Process

Feature Requests
1

All workflows get executed in their own separate process. This ensures that all CPU cores get used and that they do not block each other on CPU intensive tasks.

I was very refreshed and impressed when @jan admitted that:

and that got me thinking… why wait for a webhook request to spin up that new thread?

Is it reasonable to spawn these separate processes in advance to speed up routing (without the disadvantages of enabling EXECUTIONS_PROCESS=main) ?

I am trying to dig into the code enough to understand how and when these “separate processes” are spawned.
Found this getExecuteHookFunctions which must be somehow involved and i asked a question in-line on this vintage github commit.

Any pointers will be much appreciated

2

found this

set up docker compose

i use traefik, so my config that i tested looks different, but one needs to understand how to set up a reverse proxy to run in this mode anyway.

my initial question about pre-instantiation still stands, as it seems there is still a bit of overhead in the system.

3

I continued to test and made a flow that mocks the 2 api calls that i do to discord and notion:

Screenshot 2021-06-29 at 20.38.51
Screenshot 2021-06-29 at 20.38.511021×441 62.3 KB

response time is consistently under 1000ms… so its actually quite ok.
I would need some more detailed internal benchmarking to get any deeper into it.

here are debug level logs for this workflow:

n8n-wh_1     | 2021-06-29T19:59:11.075591964Z 2021-06-29T19:59:11.074Z | debug    | Received webhoook "POST" for path "order-test" {"file":"ActiveWorkflowRunner.js","function":"executeWebhook"}
n8n-wh_1     | 2021-06-29T19:59:11.183544516Z Started with ID: 23
n8n-wh_1     | 2021-06-29T19:59:11.185237187Z 2021-06-29T19:59:11.183Z | debug    | Executing hook (hookFunctionsPush) {"executionId":"32","workflowId":2,"file":"WorkflowExecuteAdditionalData.js","function":"workflowExecuteBefore"}
n8n-wh_1     | 2021-06-29T19:59:11.188374224Z 2021-06-29T19:59:11.187Z | verbose  | Started execution of workflow "Vino no api test" from webhook with execution ID 32 {"executionId":"32","file":"WebhookHelpers.js","function":"executeWebhook"}
n8n-queue_1  | 2021-06-29T19:59:11.222318551Z 2021-06-29T19:59:11.221Z | info     | Start job: 23 (Workflow ID: 2 | Execution: 32) {"file":"worker.js","function":"runJob"}
n8n-queue_1  | 2021-06-29T19:59:11.236200639Z 2021-06-29T19:59:11.235Z | verbose  | Workflow execution started {"workflowId":2,"file":"WorkflowExecute.js","function":"processRunExecutionData"}
n8n-queue_1  | 2021-06-29T19:59:11.238098976Z 2021-06-29T19:59:11.237Z | debug    | Start processing node "Webhook" {"node":"Webhook","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.249893491Z 2021-06-29T19:59:11.237Z | debug    | Running node "Webhook" started {"node":"Webhook","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.249995784Z 2021-06-29T19:59:11.238Z | debug    | Running node "Webhook" finished successfully {"node":"Webhook","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.250054866Z 2021-06-29T19:59:11.238Z | debug    | Start processing node "IF address OK" {"node":"IF address OK","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.250068616Z 2021-06-29T19:59:11.238Z | debug    | Running node "IF address OK" started {"node":"IF address OK","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.250095521Z 2021-06-29T19:59:11.239Z | debug    | Running node "IF address OK" finished successfully {"node":"IF address OK","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.262546078Z 2021-06-29T19:59:11.239Z | debug    | Start processing node "IF1" {"node":"IF1","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271402818Z 2021-06-29T19:59:11.240Z | debug    | Running node "IF1" started {"node":"IF1","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271452308Z 2021-06-29T19:59:11.240Z | debug    | Running node "IF1" finished successfully {"node":"IF1","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271464550Z 2021-06-29T19:59:11.240Z | debug    | Start processing node "IF2" {"node":"IF2","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271474420Z 2021-06-29T19:59:11.241Z | debug    | Running node "IF2" started {"node":"IF2","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271484852Z 2021-06-29T19:59:11.241Z | debug    | Running node "IF2" finished successfully {"node":"IF2","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271495341Z 2021-06-29T19:59:11.242Z | debug    | Start processing node "Mock Discord and preprocess" {"node":"Mock Discord and preprocess","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271512356Z 2021-06-29T19:59:11.242Z | debug    | Running node "Mock Discord and preprocess" started {"node":"Mock Discord and preprocess","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271522850Z 2021-06-29T19:59:11.257Z | debug    | Running node "Mock Discord and preprocess" finished successfully {"node":"Mock Discord and preprocess","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271532912Z 2021-06-29T19:59:11.258Z | debug    | Start processing node "Mock Notion Res" {"node":"Mock Notion Res","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271542250Z 2021-06-29T19:59:11.258Z | debug    | Running node "Mock Notion Res" started {"node":"Mock Notion Res","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271551910Z 2021-06-29T19:59:11.264Z | debug    | Running node "Mock Notion Res" finished successfully {"node":"Mock Notion Res","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271608934Z 2021-06-29T19:59:11.265Z | debug    | Start processing node "Set Response" {"node":"Set Response","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271620827Z 2021-06-29T19:59:11.265Z | debug    | Running node "Set Response" started {"node":"Set Response","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271630564Z 2021-06-29T19:59:11.266Z | debug    | Running node "Set Response" finished successfully {"node":"Set Response","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271638840Z 2021-06-29T19:59:11.267Z | debug    | Start processing node "Set Response1" {"node":"Set Response1","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271649367Z 2021-06-29T19:59:11.267Z | debug    | Running node "Set Response1" started {"node":"Set Response1","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271658897Z 2021-06-29T19:59:11.268Z | debug    | Running node "Set Response1" finished successfully {"node":"Set Response1","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271668868Z 2021-06-29T19:59:11.269Z | debug    | Start processing node "Merge" {"node":"Merge","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271680910Z 2021-06-29T19:59:11.269Z | debug    | Running node "Merge" started {"node":"Merge","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271690917Z 2021-06-29T19:59:11.269Z | debug    | Running node "Merge" finished successfully {"node":"Merge","workflowId":2,"file":"WorkflowExecute.js"}
n8n-queue_1  | 2021-06-29T19:59:11.271700729Z 2021-06-29T19:59:11.270Z | verbose  | Workflow execution finished successfully {"workflowId":2,"file":"WorkflowExecute.js","function":"processSuccessExecution"}
n8n-wh_1     | 2021-06-29T19:59:11.302571795Z 2021-06-29T19:59:11.302Z | debug    | Executing hook (hookFunctionsPush) {"executionId":"32","workflowId":2,"file":"WorkflowExecuteAdditionalData.js","function":"workflowExecuteAfter"}

Screenshot 2021-06-29 at 20.59.37
Screenshot 2021-06-29 at 20.59.37797×273 22.8 KB

4

What is quite clear is that the routing logic seems to be taking much more time than the internal execution of the workflow, as the workflow is reporting very fast run time (34ms):
Screenshot 2021-06-29 at 21.02.30

1 Like