Webhook triggers stuck in Queue mode

Describe the problem/error/question

I’m trying to set up a local dev environment using docker-compose (see attached file). When I create a workflow with a webhook trigger, it works in test mode, but when I try to invoke it (on the main or worker process) it gets queued, but not executed (see screenshots).

version: "3.9"
services:
  n8n-main:
    image: "docker.n8n.io/n8nio/n8n:0.225.0"
    depends_on:
      "n8n-redis":
        condition: service_started
      "n8n-mysql":
        condition: service_healthy
    healthcheck:
      test: wget --no-verbose --tries=1 --spider http://localhost:5678/healthz || exit 1
      start_period: 15s
      timeout: 10s
      retries: 5
      interval: 5s
    environment:
      - N8N_ENCRYPTION_KEY=asdqwe123
      - N8N_LOG_LEVEL=debug
      # ----- mysql ----- #
      - DB_TYPE=mysqldb
      - DB_MYSQLDB_DATABASE=n8n
      - DB_MYSQLDB_HOST=n8n-mysql
      - DB_MYSQLDB_USER=n8n
      - DB_MYSQLDB_PASSWORD=n8nPW
      # ----- disable telemetry ----- #
      - N8N_PERSONALIZATION_ENABLED=false
      - N8N_HIRING_BANNER_ENABLED=false
      - N8N_DIAGNOSTICS_ENABLED=false
      # ----- execution setting ----- #
      - EXECUTIONS_PROCESS=main
      - EXECUTIONS_TIMEOUT_MAX=30
      - EXECUTIONS_MODE=queue
      # ----- redis ----- #
      - QUEUE_BULL_REDIS_HOST=n8n-redis
      - QUEUE_BULL_REDIS_PORT=6379
      - QUEUE_BULL_REDIS_TIMEOUT_THRESHOLD=3000
      - QUEUE_HEALTH_CHECK_ACTIVE=true
      #- N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true
    links:
      - "n8n-redis:n8n-redis"
      - "n8n-mysql:n8n-mysql"
    ports:
      - "127.0.0.1:5678:5678"
    restart: on-failure

  n8n-worker1:
    image: "docker.n8n.io/n8nio/n8n:0.225.0"
    entrypoint: ["n8n", "webhook"]
    depends_on:
      n8n-main:
       condition: service_healthy
    environment:
        - N8N_ENCRYPTION_KEY=asdqwe123
        - DB_TYPE=mysqldb
        - DB_MYSQLDB_DATABASE=n8n
        - DB_MYSQLDB_HOST=n8n-mysql
        - DB_MYSQLDB_USER=n8n
        - DB_MYSQLDB_PASSWORD=n8nPW
        - EXECUTIONS_MODE=queue
        - QUEUE_BULL_REDIS_HOST=n8n-redis
        - QUEUE_BULL_REDIS_PORT=6379
    ports:
      - "127.0.0.1:5679:5678"

  n8n-redis:
    image: "redis:6-alpine"

  n8n-mysql:
    image: "mysql:5.7-oracle"
    environment:
    - MYSQL_ROOT_PASSWORD=my-secret-pw
    - MYSQL_DATABASE=n8n
    - MYSQL_USER=n8n
    - MYSQL_PASSWORD=n8nPW 
    healthcheck:
      interval: 30s
      test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost"]
      timeout: 5s
      retries: 10
    ports:
      - "127.0.0.1:3306:3306"
  
  # ----- test endpoint ----- #
  web1:
    image: strm/helloworld-http

So when I manually trigger this workflow or invoke the test endpoint it works, but not by invoking it via the production endpoints (neither on the worker nor on the main process).

  • n8n version: 0.225.0
  • Database (default: SQLite): MySQL 5.7
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
  • Operating system: macOS 13.3.0

Welcome to the community @sl_lp!

First, I see that you are using MySQL. I advise you to switch to a different database as we will probably drop support soon.

To your actual question. Test executions get executed on the main instance, production ones on the workers. So that makes me believe that the workers are either not up or not connecting. If you look at the output of the worker instances, what does it say?

Hey @jan, thank you for the kind welcome!

I’ll post a full debug log (starting with a new instance, importing the workflow example and triggering the webhook on both the main and the worker process, see also the console output below).

MySQL is currently the only option for the deployment, would that be problem going forward?

n8n-tests-n8n-main-1     | UserSettings were generated and saved to: /home/node/.n8n/config
n8n-tests-n8n-main-1     | 2023-04-20T07:54:23.558Z | debug    | Lazy Loading credentials and nodes from n8n-nodes-base "{\n  credentials: 312,\n  nodes: 402,\n  file: 'DirectoryLoader.js',\n  function: 'loadAll'\n}"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.304Z | warn     | Migrations in progress, please do NOT stop the process. "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.304Z | debug    | Starting migration IntroducePinData1654090101303 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.316Z | debug    | Finished migration IntroducePinData1654090101303 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.321Z | debug    | Starting migration AddJsonKeyPinData1659895550980 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.323Z | debug    | Finished migration AddJsonKeyPinData1659895550980 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.327Z | debug    | Starting migration CreateWorkflowsEditorRole1663755770894 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.328Z | debug    | Finished migration CreateWorkflowsEditorRole1663755770894 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.330Z | debug    | Starting migration WorkflowStatistics1664196174002 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.350Z | debug    | Finished migration WorkflowStatistics1664196174002 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.353Z | debug    | Starting migration CreateCredentialUsageTable1665484192213 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.401Z | debug    | Finished migration CreateCredentialUsageTable1665484192213 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.404Z | debug    | Starting migration RemoveCredentialUsageTable1665754637026 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.409Z | debug    | Finished migration RemoveCredentialUsageTable1665754637026 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.415Z | debug    | Starting migration AddWorkflowVersionIdColumn1669739707125 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.430Z | debug    | Finished migration AddWorkflowVersionIdColumn1669739707125 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.433Z | debug    | Starting migration AddTriggerCountColumn1669823906994 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.444Z | debug    | Finished migration AddTriggerCountColumn1669823906994 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.447Z | debug    | Starting migration MessageEventBusDestinations1671535397530 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.454Z | debug    | Finished migration MessageEventBusDestinations1671535397530 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.456Z | debug    | Starting migration RemoveWorkflowDataLoadedFlag1671726148420 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.468Z | debug    | Finished migration RemoveWorkflowDataLoadedFlag1671726148420 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.478Z | debug    | Starting migration DeleteExecutionsWithWorkflows1673268682475 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.528Z | debug    | Finished migration DeleteExecutionsWithWorkflows1673268682475 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.530Z | debug    | Starting migration AddStatusToExecutions1674138566000 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.550Z | debug    | Finished migration AddStatusToExecutions1674138566000 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.553Z | debug    | Starting migration CreateLdapEntities1674509946020 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.639Z | debug    | Finished migration CreateLdapEntities1674509946020 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.643Z | debug    | Starting migration PurgeInvalidWorkflowConnections1675940580449 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.644Z | debug    | Finished migration PurgeInvalidWorkflowConnections1675940580449 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.647Z | debug    | Starting migration MigrateExecutionStatus1676996103000 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.649Z | debug    | Finished migration MigrateExecutionStatus1676996103000 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.651Z | debug    | Starting migration UpdateRunningExecutionStatus1677236788851 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.652Z | debug    | Finished migration UpdateRunningExecutionStatus1677236788851 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.655Z | debug    | Starting migration CreateVariables1677501636753 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.663Z | debug    | Finished migration CreateVariables1677501636753 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.665Z | debug    | Starting migration CreateExecutionMetadataTable1679416281779 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.695Z | debug    | Finished migration CreateExecutionMetadataTable1679416281779 "{ file: 'migrationHelpers.js', function: 'logMigrationEnd' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.700Z | info     | Initializing n8n process "{ file: 'start.js', function: 'init' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.701Z | debug    | [license] initializing for deviceFingerprint e263dfa0eec821411a63489df913871e978421a72809440c307b1696ff37981c "{ file: 'LicenseManager.js', function: 'log' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:24.887Z | warn     | Migrations finished. "{ file: 'migrationHelpers.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:26.080Z | debug    | Wait tracker querying database for waiting executions "{ file: 'WaitTracker.js', function: 'getWaitingExecutions' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:26.155Z | debug    | Health check started! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:26.158Z | debug    | Health check completed successfully! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:27.136Z | debug    | Initializing event bus... "{ file: 'MessageEventBus.js', function: 'initialize' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:27.139Z | debug    | Initializing event writer "{ file: 'MessageEventBus.js', function: 'initialize' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:27.141Z | debug    | Checking for unsent event messages "{ file: 'MessageEventBus.js', function: 'initialize' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:27.142Z | debug    | Start logging into /home/node/.n8n/n8nEventLog.log  "{ file: 'MessageEventBus.js', function: 'initialize' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:27.143Z | debug    | MessageEventBus initialized "{ file: 'MessageEventBus.js', function: 'initialize' }"
n8n-tests-n8n-main-1     | n8n ready on 0.0.0.0, port 5678
n8n-tests-n8n-main-1     | Version: 0.225.0
n8n-tests-n8n-mysql-1    | 2023-04-20T07:54:27.471011Z 3 [Note] InnoDB: Stopping purge
n8n-tests-n8n-mysql-1    | 2023-04-20T07:54:27.474404Z 3 [Note] InnoDB: Resuming purge
n8n-tests-n8n-main-1     | 
n8n-tests-n8n-main-1     | Editor is now accessible via:
n8n-tests-n8n-main-1     | http://localhost:5678/
n8n-tests-n8n-worker1-1  | UserSettings were generated and saved to: /root/.n8n/config
n8n-tests-n8n-worker1-1  | n8n ready on 0.0.0.0, port 5678
n8n-tests-n8n-worker1-1  | Version: 0.225.0
n8n-tests-n8n-worker1-1  | Webhook listener waiting for requests.
n8n-tests-n8n-main-1     | 2023-04-20T07:54:31.226Z | debug    | Health check started! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:31.227Z | debug    | Health check completed successfully! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:35.605Z | debug    | Add editor-UI session "{ sessionId: 'z0ap39cue99', file: 'abstract.push.js', function: 'add' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:36.387Z | debug    | Health check started! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:36.391Z | debug    | Health check completed successfully! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:41.469Z | debug    | Health check started! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:41.470Z | debug    | Health check completed successfully! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:46.511Z | debug    | Health check started! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:46.512Z | debug    | Health check completed successfully! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:50.006Z | verbose  | Updating versionId for workflow 1 for user da5e6d84-9f33-459d-beef-49c978596916 after saving "{\n  previousVersionId: '11ab87f8-9830-4e8b-8b75-c3572680003e',\n  newVersionId: '4208e1fa-3fe7-4dd4-980d-293f851291f9',\n  file: 'workflows.services.js',\n  function: 'update'\n}"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:51.588Z | debug    | Health check started! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:51.589Z | debug    | Health check completed successfully! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:56.656Z | debug    | Health check started! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:54:56.658Z | debug    | Health check completed successfully! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:55:00.081Z | debug    | Received webhook "GET" for path "23d93889-c164-43bd-aca4-206a09a26975" "{ file: 'ActiveWorkflowRunner.js', function: 'executeWebhook' }"
n8n-tests-n8n-main-1     | Started with job ID: 1 (Execution ID: 1)
n8n-tests-n8n-main-1     | 2023-04-20T07:55:00.136Z | debug    | Executing hook (hookFunctionsPush) "{\n  executionId: '1',\n  sessionId: undefined,\n  workflowId: '1',\n  file: 'WorkflowExecuteAdditionalData.js',\n  function: 'workflowExecuteBefore'\n}"
n8n-tests-n8n-main-1     | 2023-04-20T07:55:00.141Z | verbose  | Started execution of workflow "My workflow" from webhook with execution ID 1 "{\n  executionId: '1',\n  file: 'WebhookHelpers.js',\n  function: 'executeWebhook'\n}"
n8n-tests-n8n-main-1     | 2023-04-20T07:55:01.739Z | debug    | Health check started! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:55:01.740Z | debug    | Health check completed successfully! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:55:06.810Z | debug    | Health check started! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:55:06.811Z | debug    | Health check completed successfully! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-worker1-1  | Started with job ID: 2 (Execution ID: 2)
n8n-tests-n8n-main-1     | 2023-04-20T07:55:11.876Z | debug    | Health check started! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:55:11.877Z | debug    | Health check completed successfully! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:55:16.925Z | debug    | Health check started! "{ file: 'AbstractServer.js' }"
n8n-tests-n8n-main-1     | 2023-04-20T07:55:16.926Z | debug    | Health check completed successfully! "{ file: 'AbstractServer.js' }"

[localhost ~/git/n8n-tests] curl http://localhost:5678/webhook/23d93889-c164-43bd-aca4-206a09a26975                                                                                                                                                    
{"message":"Workflow was started"}% 

[localhost ~/git/n8n-tests] curl http://localhost:5679/webhook/23d93889-c164-43bd-aca4-206a09a26975                                                                                                                                                                                                                                                                                                                                                                                            
{"message":"Workflow was started"}% 

Yes, MySQL will probably go away. So best to switch to Postgres.

It looks like you do not actually start a worker at all. I just looked again through your configuration. The worker is configured to start in “webhook” mode instead of the “worker”.

Can you try to change the line:

entrypoint: ["n8n", "webhook"]

to

entrypoint: ["n8n", "worker"]

Good catch - that was the culprit, thanks!

Glad to hear! Have fun!

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.