Error with k8s scaling

Hello,

I want to use n8n with workers.
So I followed the following procedure:

I was also helped by the contribution of @a5r0n Scaling N8N on kubernetes with Helm Chart

So I created my own k8s descriptors adapted to my cluster.
Currently I only have one worker pod and one webhook pod.

When I open the webhook pod log I get this. OK cool.

UserSettings were generated and saved to: /root/.n8n/config
Webhook listener waiting for requests.
n8n ready on 0.0.0.0, port 5678
Version: 0.157.0

I have a problem when I want to connect to the webhook pod. I still have a 404 error.

Here is my descriptor for the deployment of the webhook

apiVersion: apps/v1
kind: Deployment
metadata:
  name: n8n-webhook
  labels:
    app: n8n-webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: n8n-webhook
  template:
    metadata:
      labels:
        app: n8n-webhook
    spec:
      containers:
        - name: n8n-webhook
          image: n8nio/n8n:0.157.0
          command:
            - n8n
          args:
            - webhook
          envFrom:
            - secretRef:
                name: n8n-credentials
            - configMapRef:
                name: n8n-config-default
            - configMapRef:
                name: n8n-config
          ports:
            - name: http
              containerPort: 5678
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /healthz
              port: 5678
          readinessProbe:
            httpGet:
              path: /healthz
              port: 5678

When I go to this pod and run the following command, I get a 404 error while it’s local.

#> curl -I http://0.0.0.0:5678
HTTP/1.1 404 Not Found
X-Powered-By: Express
Access-Control-Allow-Origin: http://localhost:8080
Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, PATCH, DELETE
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, sessionid
Content-Security-Policy: default-src 'none'
X-Content-Type-Options: nosniff
Content-Type: text/html; charset=utf-8
Content-Length: 140
Vary: Accept-Encoding
Date: Wed, 05 Jan 2022 13:01:43 GMT
Connection: keep-alive
Keep-Alive: timeout=5

I do not understand. While doing a test, I removed the “webhook” argument. This turns this pod into the main n8n with the following log.

Initializing n8n process
UserSettings were generated and saved to: /home/node/.n8n/config
n8n ready on 0.0.0.0, port 5678
Version: 0.157.0
 ================================
   Start Active Workflows:
 ================================
   - Test
     => Started

Editor is now accessible via:
https://my.domain.com:5678/

By running the exact same curl -I http://0.0.0.0:5678 command, everything works fine.
This makes me say that this is a question with n8n.
Could this be related to a particular configuration of n8n that does not work with the webhook instance?

Here is the configuration sent to all instances of n8n (main, webhook, worker).

GENERIC_TIMEZONE=Europe/Paris
TZ=Europe/Paris
N8N_BASIC_AUTH_ACTIVE=true
DB_TYPE=postgresdb
N8N_PORT=5678
EXECUTIONS_DATA_SAVE_ON_ERROR=all
EXECUTIONS_DATA_SAVE_ON_SUCCESS=none
EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_MAX_AGE=672
EXECUTIONS_MODE=queue
N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true
N8N_SKIP_WEBHOOK_DEREGISTRATION_SHUTDOWN=true
QUEUE_BULL_REDIS_HOST=redis
QUEUE_BULL_REDIS_PORT=6379
QUEUE_BULL_REDIS_DB=0
N8N_PROTOCOL=https
N8N_HOST=my.domain.com
VUE_APP_URL_BASE_API=https://my.domain.com/
WEBHOOK_TUNNEL_URL=https://my.domain.com/
DB_POSTGRESDB_HOST=#########
DB_POSTGRESDB_PORT=########
DB_POSTGRESDB_USER=########
DB_POSTGRESDB_PASSWORD=########
DB_POSTGRESDB_DATABASE=########
POSTGRES_NON_ROOT_USER=########
POSTGRES_NON_ROOT_PASSWORD=########
N8N_BASIC_AUTH_USER=########
N8N_BASIC_AUTH_PASSWORD=########
N8N_ENCRYPTION_KEY=########

I admit that I do not understand why I have a 404 error with the webhook and not with the main.
An idea ?

Thank you

— EDIT 05/01/2021 14:20 —

To provide an additional answer, when I run a curl http: //0.0.0.0:5678/healthz, I get the correct return.
The error would come from n8n.

{"status": "ok"} / data # 

Hi @vanitom, it seems you are querying the root path / of your webhook worker using curl.

The path used by n8n for incoming webhooks would, however, be something like /webhook/*.

So could you try setting up a workflow with a webhook node, activate the workflow using the toggle in the upper right and then send your curl request to a valid production webhook path? The path would be visible in the UI in the webhook node’s details:
image

Note that n8n would also respond with a 404 if a webhook isn’t registered on the path you are using (or you are using a HTTP verb not configured in your webhook node) or the workflow isn’t active. So a 404 response can also occur if n8n is working as expected.

Thanks @MutedJam

I started testing with test urls (ex: https: // my-domain.com / webhook-test / d7de8d9f-1555-47f5-848e-781888a51943)

I got 404 errors when in fact the WEBHOOK instance only accepts “NOT TEST” urls. That’s right ?

As per the respective docs, the test webhooks will still be executed by the main process:

Manual workflow executions still occur on the main process and the default URL for these is /webhook-test/*. Make sure that these URLs route to your main process.

The webhook workers will only handle /webhook/* (without the -test) traffic:

  • Redirect any request that matches /webhook/* to the webhook servers pool
  • All other paths (the n8n internal API, the static files for the editor, etc.) should get routed to the main process

Effectively using a real URL in PROD, it works.
Merci

What disturbed me is the fact that the errors are not even between the following urls:

http://my.n8n-domain.com/webhooks/dont-exists

{"code": 404, "message": "The requested webhook \" GET sdfsdf \ "is not registered.", "hint": "The workflow must be active for a production URL to run successfully. You can activate the workflow using the toggle in the top-right of the editor. Note that unlike test URL calls, production URL calls aren't shown on the canvas (only in the executions list) "," stack ":" ResponseError: The requested webhook \ " GET sdfsdf \ "is not registered. \ N at ActiveWorkflowRunner.executeWebhook (/usr/local/lib/node_modules/n8n/dist/src/ActiveWorkflowRunner.js:94:23)\n at processTicksAndRejections (internal / process / tasksqui : 93: 5) \ n at async /usr/local/lib/node_modules/n8n/dist/src/WebhookServer.js:47:24 "}

http://my.n8n-domain.com/

Cannot GET /

http://my.n8n-domain.com/dont-exists

Cannot GET /dont-exists
1 Like

Excellent, glad to hear this works! I can totally see how the other 404 can be irritating (every non-2xx code makes me nervous), though in this case it would be expected.

Thanks a lot for confirming :slight_smile:

Yes it is irritating not to have enough element to debug.

Actually what was complicated for me was that the 404 error message generated by n8n (via Express) is the same as the one returned by k8s. We therefore do not know where exactly the k8s error and all these levels (ingress, services, pods, …) or even n8n comes from.

Thanks to you @MutedJam