Using WEBHOOK_URL in Kubernetes

Questions
,
#1

Hi team,

I’ve been implementing n8n in Kubernetes using a custom helm chart inspired by https://github.com/a5r0n/n8n-chart

It’s all working perfectly well and users are really happy with it.
We have one main process, one webhook, one redis and some auto scaling workers (from 3 to 10 pods). The main process and the webhook have their own ingress & DNS entries.

Description

When using webhooks, end users are often confused with URLs which are working fine for Production but not for Testing.
I’ve setup WEBHOOK_URL = https://n8n-webhook.mydomain
But it seems that only the “Production URL” is working properly with it.
When using the “Test URL”, it also displays the same url (except webhook-test instead of webhook) but users need to be pointing to the main DNS https://n8n.mydomain

So, is there a way of having two different variables for WEBHOOK_URL? That will avoid confusion for users testing their webhooks. Or maybe do you recommend to have custom web server rules depending on context?

Information on your n8n setup

  • n8n version: 0.195.5
  • Database you’re using (default: SQLite): Postgresql
  • Running n8n with the execution process [own(default), main]: own
  • Running n8n via [Docker, npm, n8n.cloud, desktop app]: docker/kubernetes
#2

Hey @Romuald_BARON,

It sounds like you have managed to do a lot of work there with the setup, I think when it comes to scaling the bit most people miss is all of the workflow editing is done on the main instance and that also applies to testing workflows that way you are able to see the execution data in the canvas and play with it.

Our recommended approach is on our scaling docs which I have pasted below for you which isn’t exactly what you are after but hopefully it helps answer the question.

Note: Manual workflow executions still occur on the main process and the default URL for these is /webhook-test/*. Make sure that these URLs route to your main process.
#3

Thanks @Jon for your answer.

So, I’ve been setting up the routing using pathType in ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#ingress-v1-networking-k8s-io
metadata:
  name: n8n-https
  annotations:
    traefik.ingress.kubernetes.io/router.priority: "1"
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.entrypoints: "https"
spec:
  tls:
    - hosts: [ n8n.mydomain ]
  rules:
      - host: n8n.mydomain
        http:
          paths:
            - path: /webhook
              pathType: Prefix
              backend:
                service:
                  name: n8n-webhook
                  port:
                    number: 5678
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: n8n-main
                  port:
                    number: 5678

And it’s working, but not all the time…
I’m getting the following error randomly (it sometimes works…) with nothing displayed in the n8n main/webhook logs:

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot POST /webhook-test/mywebhook</pre>
</body>
</html>

And same error for webhook, except Cannot POST /webhook/mywebhook of course…
Any idea what could be causing such behavior?

Thanks a lot!

#4

Hey @Romuald_BARON,

I have 2 thoughts…

  1. The webhook is not set up to allow POST requests
  2. The connection is being routed to the wrong location so maybe webhook for -test and main for just webhook.
#5

I’ve finally fixed my issue.
By creating 2 ingress with different priority, I was able to check the /webhook first and point to the webhook service before pointing all other requests to the main:

# /webhook pointing to webhook service
apiVersion: networking.k8s.io/v1
kind: Ingress # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#ingress-v1-networking-k8s-io
metadata:
  name: n8n-https
  annotations:
    traefik.ingress.kubernetes.io/router.priority: "100"
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.entrypoints: "https"
spec:
  tls:
    - hosts: [ n8n.mydomain ]
  rules:
      - host: n8n.mydomain
        http:
          paths:
            - path: /webhook
              pathType: Prefix
              backend:
                service:
                  name: n8n-webhook
                  port:
                    number: 5678

# All traffic to Main
apiVersion: networking.k8s.io/v1
kind: Ingress # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#ingress-v1-networking-k8s-io
metadata:
  name: n8n-https
  annotations:
    traefik.ingress.kubernetes.io/router.priority: "1"
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.entrypoints: "https"
spec:
  tls:
    - hosts: [ n8n.mydomain ]
  rules:
      - host: n8n.mydomain
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: n8n-main
                  port:
                    number: 5678

Everything is now working perfectly.

Thanks for your help!

3 Likes