Does redis trigger uses the queue mode?

Questions
1

Hi!

Right now we are having scalability issues related to the kafka trigger that does not uses the queue mode, so all messages are received and processed on the main node. We are looking how could we change that, one idea is to just put the kafka message to a redis channel, that other workflow listing that a redis channel could pick up and run the workflow. So the question is, if we do that are we going to be able to process those Redis triggers in the worker nodes?

Share the output returned by the last node

Information on your n8n setup

  • n8n version: 0.236
  • Database (default: SQLite): Postgresql 13.10
  • Running n8n via (Docker, npm, n8n cloud, desktop app): AWS Elasticbeanstalk
  • Operating system: Amazon Linux
3

Hi @sebasortiz.dev, I just had a look into this on my end and am wondering if there might be a misunderstanding here.

In a queue mode setup, the main instance would handle all triggers as per the docs:

one main instance receiving workflow information (e.g. triggers) and the worker instances performing the executions.

This should be no different for triggers other than the Kafka Trigger. The workflow itself would then run on the respective workers. You can see this best when enabling debug logging:

n8n-with-kafka-n8n_main-1    | Started with job ID: 2 (Execution ID: 7)
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:02.157Z | info     | Start job: 2 (Workflow ID: LnouSq8ZMljEm3Pg | Execution: 7) "{ file: 'worker.js', function: 'runJob' }"
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:02.161Z | verbose  | Workflow execution started "{\n  workflowId: 'LnouSq8ZMljEm3Pg',\n  file: 'WorkflowExecute.js',\n  function: 'processRunExecutionData'\n}"
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:02.162Z | debug    | Start processing node "Kafka Trigger" "{\n  node: 'Kafka Trigger',\n  workflowId: 'LnouSq8ZMljEm3Pg',\n  file: 'WorkflowExecute.js'\n}"
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:02.163Z | debug    | Running node "Kafka Trigger" started "{\n  node: 'Kafka Trigger',\n  workflowId: 'LnouSq8ZMljEm3Pg',\n  file: 'WorkflowExecute.js'\n}"
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:02.163Z | debug    | Running node "Kafka Trigger" finished successfully "{\n  node: 'Kafka Trigger',\n  workflowId: 'LnouSq8ZMljEm3Pg',\n  file: 'WorkflowExecute.js'\n}"
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:02.163Z | debug    | Start processing node "Wait" "{\n  node: 'Wait',\n  workflowId: 'LnouSq8ZMljEm3Pg',\n  file: 'WorkflowExecute.js'\n}"
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:02.164Z | debug    | Running node "Wait" started "{\n  node: 'Wait',\n  workflowId: 'LnouSq8ZMljEm3Pg',\n  file: 'WorkflowExecute.js'\n}"
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:04.165Z | debug    | Running node "Wait" finished successfully "{\n  node: 'Wait',\n  workflowId: 'LnouSq8ZMljEm3Pg',\n  file: 'WorkflowExecute.js'\n}"
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:04.166Z | debug    | Start processing node "No Operation, do nothing" "{\n  node: 'No Operation, do nothing',\n  workflowId: 'LnouSq8ZMljEm3Pg',\n  file: 'WorkflowExecute.js'\n}"
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:04.166Z | debug    | Running node "No Operation, do nothing" started "{\n  node: 'No Operation, do nothing',\n  workflowId: 'LnouSq8ZMljEm3Pg',\n  file: 'WorkflowExecute.js'\n}"
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:04.166Z | debug    | Running node "No Operation, do nothing" finished successfully "{\n  node: 'No Operation, do nothing',\n  workflowId: 'LnouSq8ZMljEm3Pg',\n  file: 'WorkflowExecute.js'\n}"
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:04.167Z | verbose  | Workflow execution finished successfully "{\n  workflowId: 'LnouSq8ZMljEm3Pg',\n  file: 'WorkflowExecute.js',\n  function: 'processSuccessExecution'\n}"
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:04.167Z | debug    | Executing hook (hookFunctionsSaveWorker) "{\n  executionId: '7',\n  workflowId: 'LnouSq8ZMljEm3Pg',\n  file: 'WorkflowExecuteAdditionalData.js',\n  function: 'workflowExecuteAfter'\n}"
n8n-with-kafka-n8n_worker-1  | 2023-10-10T13:22:04.167Z | debug    | Save execution data to database for execution ID 7 "{\n  executionId: '7',\n  workflowId: 'LnouSq8ZMljEm3Pg',\n  finished: true,\n  stoppedAt: 2023-10-10T13:22:04.167Z,\n  file: 'sharedHookFunctions.js',\n  function: 'updateExistingExecution'\n}"
n8n-with-kafka-n8n_main-1    | 2023-10-10T13:22:04.178Z | debug    | Reading execution data for execution 7 from db for PostExecutionPromise. "{ file: 'WorkflowRunner.js' }"

Here you can see the main instance n8n-with-kafka-n8n_main-1 kicking off the execution and then passing it on the worker n8n-with-kafka-n8n_worker-1.

Is this what you are seeing on your end? Or is your entire workflow execution taking place on the main instance?

2 Likes
4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.