Worker processes question

Hello!

I have a workflow that receives a sheet with a lot of lines (each line correspond to a lead, with their informations). Each line get processed one at a time, but it take a long time (1,5s per line). You have a solution to receive the sheet, and create some workers to run multiple batches at a time?

PS: The sheet is received from upload in Amazon S3, using the Amazon SNS Trigger to listen for a upload.

Edit: If I create 2 instances of n8n, and use the first to collect the sheet and split them, so send the batches via API, without wait for a response, and receive the data in the second instance, controlled by the queue mode (to limit the number of simultaneous executions), its may work?

Thanks!

Information on your n8n setup

  • n8n version: 0.210.2
  • Database you’re using: PostgreSQL
  • Running n8n via Docker

Welcome to the community @maycon ! Great to have you with us!

Yes, sounds about right. You can configure the workers (via the concurrency flag) to run multiple workflows in parallel. By default it is set to 10. How many it is able to run without issues depends on the workflows you are running and the specs of the machine the worker is running on.

But there should be no need to have two totally separate n8n instances for that. One in queue mode should be enough. After all can you also start multiple workflows on the same instance by calling a Webhook-Node via the HTTP Request one.

But if you are not in a rush (so if it does not matter if it processes them within 5 minutes or 5h) there are also simpler ways. For example, sending them all to RabbitMQ and configuring the trigger node to only process X requests in parallel.

2 Likes

Whats the difference between using workers and not use them to process 2 or more workflows?

Actually I have a workflow that triggers another, 2 times at the same time, without set the queue mode. My main workflow activates another 2, so I wait until them finishes to activate 2 more.

I have something like that to start 2 workflows at the same time:

The difference is that with workers the workload gets distributed across different workers. Without them, a single n8n instance processes all. Meaning there is a higher chance of it being overloaded and so potentially crashing.
Also if a workflow in n8n queue mode crashes because of memory issues then it just takes down that worker, in own/main mode mode there is a larger chance that it takes down whole n8n (in main mode actually 100%).

1 Like

Hmmm ok, thanks!

Now I have another question: In the final of the execution (I think this is it), n8n will save it, and this is causing this problem:

2023-01-31T17:53:34.040Z | debug    | Running node "NoOp" started "{ node: 'NoOp', workflowId: '2', file: 'WorkflowExecute.js' }"
2023-01-31T17:53:34.040Z | debug    | Running node "NoOp" finished successfully "{ node: 'NoOp', workflowId: '2', file: 'WorkflowExecute.js' }"
2023-01-31T17:53:34.041Z | debug    | Executing hook (hookFunctionsSave) "{\n  executionId: '175',\n  workflowId: '2',\n  file: 'WorkflowExecuteAdditionalData.js',\n  function: 'workflowExecuteAfter'\n}"
2023-01-31T17:53:34.041Z | debug    | Save execution data to database for execution ID 175 "{\n  executionId: '175',\n  workflowId: '2',\n  finished: true,\n  stoppedAt: 2023-01-31T17:53:34.041Z,\n  file: 'WorkflowExecuteAdditionalData.js',\n  function: 'workflowExecuteAfter'\n}"

<--- Last few GCs --->

[7:0x7f968f17b400]   925516 ms: Scavenge 1989.9 (1998.4) -> 1986.8 (2000.5) MB, 2.8 / 0.1 ms  (average mu = 0.942, current mu = 0.950) external memory pressure 
[7:0x7f968f17b400]   925991 ms: Mark-sweep (reduce) 2070.4 (2083.5) -> 2067.1 (2074.2) MB, 31.6 / 0.3 ms  (+ 94.5 ms in 11 steps since start of marking, biggest step 92.2 ms, walltime since start of marking 462 ms) (average mu = 0.893, current mu = 0.809)

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

<--- Last few GCs --->

[7:0x7f93cffc8400]    98623 ms: Scavenge 2032.7 (2041.6) -> 2032.0 (2045.6) MB, 8.6 / 0.1 ms  (average mu = 0.839, current mu = 0.745) allocation failure 
[7:0x7f93cffc8400]    98900 ms: Mark-sweep (reduce) 2115.0 (2128.3) -> 2114.6 (2120.8) MB, 64.3 / 0.1 ms  (+ 7.1 ms in 5 steps since start of marking, biggest step 6.4 ms, walltime since start of marking 277 ms) (average mu = 0.845, current mu = 0.852) fi

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

In my docker logs, i see all the execution data passing on my screen (it contains the processed data). I’ve disabled the “save execution in sucess” option and that solved it, but i need it enabled.
How can I increase the heap memory, using Docker? My droplet have 8GB…

Normally it should use all the memory available by default.

But if you work with a lot of binary data, then it is best if you use filesystem mode for that:

If you activate it, should your memory issue be solved.

1 Like

I’ve installed RabbitMQ to use a simple queue, and it worked, thank you!
Now I will try to configure workers, but for now, my problem is solved :slight_smile:

Glad to hear. And thanks for reporting back.

Have fun!

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.