Test Webhook and Rest Callback - Cannot GET Error - on queue setup

prononext · December 24, 2022, 4:02pm

Dear Community,

on my docker compose setup I cannot receive any test webhooks and rest callbacks for my credentials.

I always get the following errors when trying to get the test webhook:
(after clicking “Listen for Test Event”)

Cannot GET /test/f02tdc527-f85c-41fa-a603-aa5atdsd0cdf44

and like this when trying to update or add my oauth credentials

Cannot GET /rest/f02tdc527-f85c-41fa-a603-aa5atdsd0cdf44

This is my Setup

Docker Compose File

version: '3.9'

volumes:
  db_storage:
  n8n_storage:
  redis_storage:
  letsencrypt_storage:

## n8n Environment Variables for Main, Worker and Webhook
x-shared: &shared
    image: n8nio/n8n:${N8N_VERSION_TAG}
    restart: always
    environment:
      - GENERIC_TIMEZONE
      ### - n8n Security
      - N8N_PORT
      - N8N_USER_MANAGEMENT_DISABLED
      - N8N_BASIC_AUTH_ACTIVE
      - N8N_BASIC_AUTH_USER
      - N8N_BASIC_AUTH_PASSWORD
      - N8N_ENCRYPTION_KEY
      ### - n8n Customization
      - N8N_DIAGNOSTICS_ENABLED
      - N8N_PERSONALIZATION_ENABLED
      - N8N_HIRING_BANNER_ENABLED
      ### - n8n - Email
      - N8N_EMAIL_MODE
      - N8N_SMTP_HOST
      - N8N_SMTP_PORT
      - N8N_SMTP_USER
      - N8N_SMTP_PASS
      - N8N_SMTP_SENDER
      - N8N_SMTP_SSL
      ### - n8n Execution
      - N8N_LOG_LEVEL
      - N8N_DISABLE_PRODUCTION_MAIN_PROCESS
      - EXECUTIONS_MODE
      - EXECUTIONS_PROCESS
      - EXECUTIONS_DATA_SAVE_ON_SUCCESS
      - EXECUTIONS_DATA_SAVE_ON_ERROR
      - EXECUTIONS_DATA_PRUNE
      - EXECUTIONS_DATA_MAX_AGE
      - N8N_DEFAULT_BINARY_DATA_MODE
      - N8N_AVAILABLE_BINARY_DATA_MODES
      - NODE_OPTIONS
      ### - n8n Database
      - DB_TYPE
      - DB_POSTGRESDB_PORT
      - DB_POSTGRESDB_DATABASE=${POSTGRES_DB}
      - DB_POSTGRESDB_USER=${POSTGRES_NON_ROOT_USER}
      - DB_POSTGRESDB_PASSWORD=${POSTGRES_NON_ROOT_PASSWORD}
      - DB_POSTGRESDB_HOST
      - DB_LOGGING_MAX_EXECUTION_TIME
      ### - n8n Queue
      - QUEUE_BULL_REDIS_PORT
      - QUEUE_BULL_REDIS_HOST
      - QUEUE_HEALTH_CHECK_ACTIVE
      ### - Webhook
      - WEBHOOK_URL
      - N8N_ENDPOINT_WEBHOOK
      - N8N_ENDPOINT_WEBHOOK_TEST
      - N8N_ENDPOINT_WEBHOOK_WAIT
      - N8N_SKIP_WEBHOOK_DEREGISTRATION_SHUTDOWN
    depends_on:
      - postgres
      - redis
    volumes:
      - n8n_storage:/home/node/.n8n
      - ${N8N_LOCAL_STORAGE}:/files

services:

####### - Traefik Reverse Proxy
  traefik:
    image: traefik:${TRAEFIK_VERSION_TAG}
    restart: on-failure
    command:
      - "--log.level=INFO"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.myresolver.acme.dnschallenge=true"
      - "--certificatesresolvers.myresolver.acme.dnschallenge.provider=route53"
      - "--certificatesresolvers.myresolver.acme.email=${LETSENCRYPT_EMAIL}"
      - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
    environment:
      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY
      - AWS_HOSTED_ZONE_ID
    volumes:
      - letsencrypt_storage:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock

####### - ProstgreSQL Databse
  postgres:
    image: postgres:${POSTGRESQL_VERSION_TAG}
    restart: always
    environment:
      - POSTGRES_USER
      - POSTGRES_PASSWORD
      - POSTGRES_DB
      - POSTGRES_NON_ROOT_USER
      - POSTGRES_NON_ROOT_PASSWORD
    volumes:
      - db_storage:/var/lib/postgresql/data
      - ${DB_INIT_FILE}:/docker-entrypoint-initdb.d/init-data.sh
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -h localhost -U ${POSTGRES_USER} -d ${POSTGRES_DB}']
      interval: 5s
      timeout: 5s
      retries: 10

####### - Redis  Queue Databse
  redis:
    image: redis:${REDIS_VERSION_TAG}
    restart: always
    volumes:
      - redis_storage:/data
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
      interval: 5s
      timeout: 5s
      retries: 10

####### - n8n Main Process
  n8n-main:
    <<: *shared
    command: ${N8N_MAIN_COMMAND}
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.n8n-main.rule=Host(`${N8N_UI_SUBDOMAIN}.${DOMAIN_NAME}`)"
      - "traefik.http.routers.n8n-main.entrypoints=websecure"
      - "traefik.http.routers.n8n-main.tls.certresolver=myresolver"
      - "traefik.http.middlewares.n8n-main.headers.SSLRedirect=true"
      - traefik.http.middlewares.n8n-main.headers.STSSeconds=315360000
      - traefik.http.middlewares.n8n-main.headers.browserXSSFilter=true
      - traefik.http.middlewares.n8n-main.headers.contentTypeNosniff=true
      - traefik.http.middlewares.n8n-main.headers.forceSTSHeader=true
      - traefik.http.middlewares.n8n-main.headers.SSLHost=${DOMAIN_NAME}
      - traefik.http.middlewares.n8n-main.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.n8n-main.headers.STSPreload=true

####### - n8n Workers
  n8n-worker-1:
    <<: *shared
    command: ${N8N_WORKER_COMMAND}

  n8n-worker-2:
    <<: *shared
    command: ${N8N_WORKER_COMMAND}

  n8n-worker-3:
    <<: *shared
    command: ${N8N_WORKER_COMMAND}

####### - n8n Webhook
  n8n-webhook:
    <<: *shared
    command: ${N8N_WEBHOOK_COMMAND}
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.n8n-webhook.rule=Host(`${N8N_WEBHOOK_SUBUDOMAIN}.${DOMAIN_NAME}`)"
      - "traefik.http.routers.n8n-webhook.entrypoints=websecure"
      - "traefik.http.routers.n8n-webhook.tls.certresolver=myresolver"
      - "traefik.http.middlewares.n8n-webhook.headers.SSLRedirect=true"
      - traefik.http.middlewares.n8n-webhook.headers.STSSeconds=315360000
      - traefik.http.middlewares.n8n-webhook.headers.browserXSSFilter=true
      - traefik.http.middlewares.n8n-webhook.headers.contentTypeNosniff=true
      - traefik.http.middlewares.n8n-webhook.headers.forceSTSHeader=true
      - traefik.http.middlewares.n8n-webhook.headers.SSLHost=${DOMAIN_NAME}
      - traefik.http.middlewares.n8n-webhook.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.n8n-webhook.headers.STSPreload=true

Environment Variables

######## Public Domains
DOMAIN_NAME=example.com
N8N_UI_SUBDOMAIN=n8n
N8N_WEBHOOK_SUBUDOMAIN=webhook.n8n
WEBHOOK_URL=https://webhook.n8n.example.com/

######## Services volumes and storage locoations
DB_INIT_FILE=/opt/n8n/init-data.sh
N8N_LOCAL_STORAGE=/local-files

######## Services Versions
TRAEFIK_VERSION_TAG=v2.9.6
POSTGRESQL_VERSION_TAG=12.13
REDIS_VERSION_TAG=alpine
N8N_VERSION_TAG=0.209.2

####### - Traefik Reverse Proxy
[email protected]
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_HOSTED_ZONE_ID=

####### - ProstgreSQL Databse
POSTGRES_USER=
POSTGRES_PASSWORD=
POSTGRES_DB=
POSTGRES_NON_ROOT_USER=
POSTGRES_NON_ROOT_PASSWORD=

####### - n8n Processes
GENERIC_TIMEZONE=CET
N8N_MAIN_COMMAND=/bin/sh -c "sleep 5; n8n start"
N8N_WORKER_COMMAND=/bin/sh -c "sleep 5; n8n worker --concurrency=50"
N8N_WEBHOOK_COMMAND=/bin/sh -c "sleep 5; n8n webhook"
## Security
N8N_PORT=5678
N8N_USER_MANAGEMENT_DISABLED=false
N8N_BASIC_AUTH_ACTIVE=true
N8N_BASIC_AUTH_USER=root
N8N_BASIC_AUTH_PASSWORD=
N8N_ENCRYPTION_KEY=
## Customization
N8N_DIAGNOSTICS_ENABLED=false
N8N_PERSONALIZATION_ENABLED=false
N8N_HIRING_BANNER_ENABLED=false
## Email
N8N_EMAIL_MODE=smtp
N8N_SMTP_HOST=smtp.example.com
N8N_SMTP_PORT=587
N8N_SMTP_USER=
N8N_SMTP_PASS=
N8N_SMTP_SENDER=n8n <[email protected]>
N8N_SMTP_SSL=false
## Execution
N8N_LOG_LEVEL=error
N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true
EXECUTIONS_MODE=queue
EXECUTIONS_PROCESS=main
EXECUTIONS_DATA_SAVE_ON_SUCCESS=none
EXECUTIONS_DATA_SAVE_ON_ERROR=all
EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_MAX_AGE=14
N8N_DEFAULT_BINARY_DATA_MODE=filesystem
N8N_AVAILABLE_BINARY_DATA_MODES=filesystem
NODE_OPTIONS="--max-old-space-size=4096"

## Database
DB_TYPE=postgresdb
DB_POSTGRESDB_PORT=5432
DB_POSTGRESDB_HOST=postgres
DB_LOGGING_MAX_EXECUTION_TIME=0
## Queue
QUEUE_BULL_REDIS_PORT=6379
QUEUE_BULL_REDIS_HOST=redis
QUEUE_HEALTH_CHECK_ACTIVE=true
## Webhook
N8N_SKIP_WEBHOOK_DEREGISTRATION_SHUTDOWN=true
N8N_ENDPOINT_WEBHOOK=prod
N8N_ENDPOINT_WEBHOOK_TEST=test
N8N_ENDPOINT_WEBHOOK_WAIT=wait

On traefik, on n8n, webhook, main or worker there are not logs given at the time of error.

The production webhooks are working for me. Also when running without the webhook service, test webhooks and rest callbacks are working on the main n8n service.

I read many post of people trying to make webhooks work on another path of the main n8n domain, but they always used a workaround like non dockerized nginx to get it routed. So I just tried to make the webhook on a seperated sub-domain and it was working for production calls, just test webhooks and rest-callbacks have some communication problem.

Does anyone have a idea what it could be and how to solve it?

jan · December 24, 2022, 6:13pm

Did you try without the environment variable N8N_ENDPOINT_WEBHOOK_TEST=test?

Even though it exists (since the very early days) does nobody really use it and it is quite possible that it causes problems. If it does we should then probably simply remove it as I do not think it is worth the work to support it.

prononext · December 24, 2022, 7:54pm

I removed the N8N_ENDPOINT_WEBHOOK_TEST environment variable.

And got the same:

Cannot GET /webhook-test/f02tt527-f85c-41fa-a603-ca5add0cdx44

of course after rebuilding shutting down and rebuilding the complete stack to be sure.

jan · December 24, 2022, 8:11pm

Very strange. Are you sure you make those requests only to the main instance?

prononext · December 24, 2022, 8:43pm

Yes I use 1:1 the above metioned docker-compose setup.

In this setup there is the webhook service, which is working on production webhooks without any problem, only test and oauth callbacks are not working.

  n8n-webhook:
    <<: *shared
    command: bin/sh -c "sleep 5; n8n webhook"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.n8n-webhook.rule=Host(`${N8N_WEBHOOK_SUBUDOMAIN}.${DOMAIN_NAME}`)"
      - "traefik.http.routers.n8n-webhook.entrypoints=websecure"
      - "traefik.http.routers.n8n-webhook.tls.certresolver=myresolver"
      - "traefik.http.middlewares.n8n-webhook.headers.SSLRedirect=true"
      - traefik.http.middlewares.n8n-webhook.headers.STSSeconds=315360000
      - traefik.http.middlewares.n8n-webhook.headers.browserXSSFilter=true
      - traefik.http.middlewares.n8n-webhook.headers.contentTypeNosniff=true
      - traefik.http.middlewares.n8n-webhook.headers.forceSTSHeader=true
      - traefik.http.middlewares.n8n-webhook.headers.SSLHost=${DOMAIN_NAME}
      - traefik.http.middlewares.n8n-webhook.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.n8n-webhook.headers.STSPreload=true

I tried to make the setup as structured as possible. Guess I will make a pull request to n8n Github with it as soon as this last problem is fixed, cause there is no real production ready docker-compose available right now with all components clearly included.

prononext · January 13, 2023, 6:40pm

The problem still persists with v0.210.2

prononext · January 16, 2023, 12:45pm

I found where the problem is.

If you are working in queue mode and have a different webhook url for your webhook service container the production webhooks are working, but test webhooks and any OAuth Redirect will not (not the ones that the UI shows you).

I guess the logic with n8n is that it takes the WEBHOOK_URL environment variable and applies it to all webhooks, test and production, but when using a seperate webhook container that will not work as n8n is not executing test webhooks on that container, it only uses it for production webhooks.

I tested back and forth and finally edited the given test webhook url, and replaced it with the UI URL instead of the URL from the env WEBHOOK_URL - and it worked.

That workaround is only good for webhooks and not for OAuth Redirect URLs as it will result in “400: redirect_uri_mismatch”.

To solve this issue I would suggest one of the following solutions:

Show the main N8N URL on test webhooks and OAuth Redirect etc. and not the webhook url, or
Just accept test webhooks and oauth urls on the webhook service container instead of the main instance.

I hope that will be solved.

Jon · January 16, 2023, 1:20pm

Hey @prononext,

I am not sure I follow, So n8n will use the WEBHOOK_URL option for test and live endpoints but we would never expect the webhook-test calls to be sent to a webhook worker or worker instance they should always be directed to the main instance.

It sounds like maybe the oauth and webhook-test urls are being passed to the webhook workers instead of the main instance which would cause issues.

prononext · January 16, 2023, 1:39pm

Yes thats what I also thought, test webhooks and oauth calls should be handled in the main instance and not the webhook production worker.

The problem is that, when a WEBHOOK_URL is specified it uses that for all production webhooks, test-webhooks and oauth redirects.

That makes no sense for me, as the WEBHOOK_URL variable is configured on the webhook worker and should only be used for production webhooks.

So like mentioned the easy way would be to just NOT use and NOT display the WEBHOOK_URL for any test-webhooks and oauth redirects.

Jon · January 16, 2023, 1:56pm

Hey @prononext,

I would have expected the webhook_url to be defined on both the production webhook worker and the main instance although we document that it should be on the main instance.

If we didn’t display the webhook_url for test webhooks and oauth I am not entirely sure how someone would then work out what url they need to use.

It sounds like what you might actually want is N8N_EDITOR_BASE_URL to be used for oauth and webhook test calls and then webhook_url to be used for everything else so that you can use 2 urls in your routing along with uri routing in the load balancer.

prononext · January 16, 2023, 3:28pm

Yes adding N8N_EDITOR_BASE_URL solved the problem.
Thats what I missed in the ENV Forest
Thank you very much.

system · January 23, 2023, 3:29pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.