Describe the problem/error/question
I have deployed n8n on GKE using helm charts. Everything is working fine, except the webhook processor.
I have three deployments.
1: main n8n
2: n8n worker
3: n8n webhook processor
All three deployments are working fine on an individual level. I can access the UI via the browser. If I create a webhook and send a request to it, the main node accepts it, and sends the workflow to the worker node. This works fine.
However I want that the webhook processing is handled by the webhook processor. I have explicitly set N8N_DISABLE_PRODUCTION_MAIN_PROCESS to FALSE in the main n8n instance.
The webhook processor node is working fine by itself. I can see from the logs that it is waiting for requests at 0.0.0.0 at port 5678. This webhook processor node is exposed via a ClusterIP service.
The service is listening on port 80, and targeting everything to port 5678.
logs from webhook processor node
Ingress
I have an ingress that is configured to route everything to the main n8n node. However, I want that everything sent to /webhook is sent to the webhook processor node.
For this, I added a new path, and configured it to point to the exposed service.
This is where the problem starts. For some reason, the health checks fail for this.
This can be visualized by the following load balancing monitoring chart.
I have tried to troubleshoot this extensively, but to no avail. There are no logs that can help.
These are ingress logs
Code
webhook-processor.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
service: {{ .Values.webhook.serviceName }}
name: n8n-webhook-processors
namespace: {{ .Release.Namespace | quote }}
spec:
replicas: {{ .Values.webhook.replicaCount }}
selector:
matchLabels:
service: {{ .Values.webhook.serviceName }}
template:
metadata:
labels:
service: {{ .Values.webhook.serviceName }}
spec:
containers:
- name: n8n-webhook-processor
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
command: ["n8n", "webhook"]
env:
- name: N8N_ENCRYPTION_KEY
valueFrom:
secretKeyRef:
name: n8n-secret
key: EncryptionKey
- name: DB_TYPE
value: postgresdb
- name: DB_POSTGRESDB_HOST
value: {{ .Values.postgresql.host | quote }}
- name: DB_POSTGRESDB_PORT
value: {{ .Values.postgresql.port | quote }}
- name: DB_POSTGRESDB_DATABASE
value: {{ .Values.postgresql.database | quote }}
- name: DB_POSTGRESDB_USER
valueFrom:
secretKeyRef:
name: {{ .Values.postgresql.secretName }}
key: POSTGRES_NON_ROOT_USER
- name: DB_POSTGRESDB_PASSWORD
valueFrom:
secretKeyRef:
name: {{ .Values.postgresql.secretName }}
key: POSTGRES_NON_ROOT_PASSWORD
- name: EXECUTIONS_MODE
value: {{ .Values.n8n.basicAuth.executionMode | quote }}
- name: WEBHOOK_URL
value: {{ .Values.webhook.webhookUrl | quote }}
- name: N8N_PORT
value: "5678"
- name: N8N_BASIC_AUTH_ACTIVE
value: {{ .Values.n8n.basicAuth.active | quote }}
- name: N8N_BASIC_AUTH_USER
value: {{ .Values.n8n.basicAuth.user | quote }}
- name: N8N_BASIC_AUTH_PASSWORD
valueFrom:
secretKeyRef:
name: n8n-secret
key: N8N_BASIC_AUTH_PASSWORD
- name: QUEUE_BULL_REDIS_HOST
value: {{ .Values.n8n.basicAuth.queue.host | quote }}
- name: QUEUE_BULL_REDIS_PORT
value: {{ .Values.n8n.basicAuth.queue.port | quote }}
- name: QUEUE_HEALTH_CHECK_ACTIVE
value: "true"
resources:
requests:
cpu: {{ .Values.webhook.resources.requests.cpu }}
memory: {{ .Values.webhook.resources.requests.memory }}
volumeMounts:
- mountPath: /home/node/.n8n
name: {{ .Values.persistence.name }}
volumes:
- name: {{ .Values.persistence.name }}
persistentVolumeClaim:
claimName: {{ .Values.persistence.name }}
webhook-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
service: {{ .Values.webhook.serviceName }}
name: {{ .Values.webhook.serviceName }}
namespace: {{ .Release.Namespace | quote }}
spec:
type: {{ .Values.webhook.serviceType }}
ports:
- name: {{ .Values.webhook.portName | quote }}
port: 80
targetPort: 5678
protocol: TCP
selector:
service: {{ .Values.webhook.serviceName }}
ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: n8n-ingress
namespace: n8n
annotations:
kubernetes.io/ingress.global-static-ip-name: n8n-ingress-controller
spec:
defaultBackend:
service:
name: {{ .Values.service.label }}
port:
number: 80
rules:
- host: {{ .Values.n8n.basicAuth.ingressUrl | quote }}
http:
paths:
- path: /webhook
pathType: Prefix
backend:
service:
name: {{ .Values.webhook.serviceName }}
port:
number: 80
- path: /
pathType: Prefix
backend:
service:
name: {{ .Values.service.label }}
port:
number: 80
What is the error message (if any)?
Initially I was getting this error when I was testing the webhook
However, now I am getting this.
Information on your n8n setup
- n8n version: 1.62.6
- Database (default: SQLite): postgresql
- n8n EXECUTIONS_PROCESS setting (default: own, main): queue mode
- Running n8n via (Docker, npm, n8n cloud, desktop app): Google Kubernetes Engine
- Operating system: Container Optimized OS