Test Webhooks for my Self-hosted railway Deployment with Webhook processor + workers do not work

Giulio_Pezzulli · July 10, 2024, 2:36pm

Describe the problem/error/question

I am deploying this template on railway:

It has 5 services:

PostgreSQL
Redis
Primary n8n
Worker n8n
Webhook processor N8n

I am advised to have these set in my primary n8n
EXECUTIONS_MODE=queue
N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true
WEBHOOK_URL=https://webhook-processor-production-[redacted].up.railway.app/

(https://webhook-processor-production-[redacted].up.railway.app/ is my public domain for my webhook processor instance)
Which allows for a good ui and editor experience.

However the problem i have is that test webhooks respond with:

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot POST /webhook-test/9210ae3a-b25a-4273-8127-2712f03577e9</pre>
</body>
</html>

However production webhooks work when the workflow is active.

This result is reversed when i change:
WEBHOOK_URL=https://primary-production-[redacted].up.railway.app/webhook-test/9210ae3a-b25a-4273-8127-2712f03577e9

i.e test webhooks work, production ones do not.

Basically i do not know how to also open up the test webhooks endpoint while also having production endpoints go through the load balancing that the webhook processor does for me.

Webhook processor instance env vars:

DB_POSTGRESDB_DATABASE=${{Postgres.POSTGRES_DB}}
DB_POSTGRESDB_HOST=${{Postgres.PGHOST_PRIVATE}}
DB_POSTGRESDB_PASSWORD=${{Postgres.POSTGRES_PASSWORD}}
DB_POSTGRESDB_PORT=${{Postgres.PGPORT_PRIVATE}}
DB_POSTGRESDB_USER=${{Postgres.POSTGRES_USER}}
DB_TYPE=postgresdb
ENABLE_ALPINE_PRIVATE_NETWORKING=true
EXECUTIONS_MODE=queue
N8N_ENCRYPTION_KEY=${{Primary.N8N_ENCRYPTION_KEY}}
N8N_LISTEN_ADDRESS=::
PORT=5678
QUEUE_BULL_REDIS_HOST=${{Redis.REDISHOST}}
QUEUE_BULL_REDIS_PASSWORD=${{Redis.REDIS_PASSWORD}}
QUEUE_BULL_REDIS_PORT=${{Redis.REDISPORT}}
QUEUE_BULL_REDIS_USERNAME=${{Redis.REDISUSER}}

Primary worker env vars

DB_POSTGRESDB_DATABASE=${{Postgres.POSTGRES_DB}}
DB_POSTGRESDB_HOST=${{Postgres.PGHOST_PRIVATE}}
DB_POSTGRESDB_PASSWORD=${{Postgres.POSTGRES_PASSWORD}}
DB_POSTGRESDB_PORT=${{Postgres.PGPORT_PRIVATE}}
DB_POSTGRESDB_USER=${{Postgres.POSTGRES_USER}}
DB_TYPE=postgresdb
ENABLE_ALPINE_PRIVATE_NETWORKING=true
EXECUTIONS_MODE=queue
N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true
N8N_ENCRYPTION_KEY=[redacted]
N8N_LISTEN_ADDRESS=::
PORT=5678
QUEUE_BULL_REDIS_HOST=${{Redis.REDISHOST}}
QUEUE_BULL_REDIS_PASSWORD=${{Redis.REDIS_PASSWORD}}
QUEUE_BULL_REDIS_PORT=${{Redis.REDISPORT}}
QUEUE_BULL_REDIS_USERNAME=${{Redis.REDISUSER}}
WEBHOOK_URL=https://webhook-processor-production-[redacted].up.railway.app/

NOTE: I have redacted sensitive information about my instances with [redacted]

Information on your n8n setup

**n8n version: Version: 1.48.3
**Database: postgres
**n8n EXECUTIONS_PROCESS setting (default: own, main): default
**Running n8n via (Docker, npm, n8n cloud, desktop app): docker railway template
**Operating system: whatever docker uses probably linux

n8n · July 10, 2024, 2:36pm

It looks like your topic is missing some important information. Could you provide the following if applicable.

n8n version:
Database (default: SQLite):
n8n EXECUTIONS_PROCESS setting (default: own, main):
Running n8n via (Docker, npm, n8n cloud, desktop app):
Operating system:

Jon · July 10, 2024, 3:01pm

Hey @Giulio_Pezzulli,

We do not support Railway so some of this is going to come down to your knowledge of that platfrom.

Typically when using webhook workers a loadbalancer would be used to make sure that /webhook-test/* goes to the main instance of n8n and /webhook/* routes to the pool of webhook workers. Have you configured your loadbalancer for this or are you sending everything to the webhook worker?

Giulio_Pezzulli · July 10, 2024, 3:06pm

So all railway is doing is getting this docker image: https://hub.docker.com/r/n8nio/n8n

and deploying it, and uses this start command:
n8n webhook

So from what i understand. It must be using whatever default configuration is set from the n8n docker container, and it’s logic is all determined by the defaults and the env vars i put in.

Giulio_Pezzulli · July 10, 2024, 3:09pm

Here are the start commands for each of the 3 n8n instances i have deployed

Webhook Processor
n8n webhook

Worker
n8n worker

Primary
n8n start

Each of them use the exact same docker container i referenced above

Jon · July 10, 2024, 3:09pm

Hey @Giulio_Pezzulli,

We don’t include a load balancer (which is a requirement for webhook workers in our docs) so I suspect at some point you must be telling n8n which url to use. Does Railway generate different URLs for the main instance and the webhook worker?

Giulio_Pezzulli · July 10, 2024, 3:16pm

Ah, I thought the webhook processor was the load balancer.

So railway generates a public domain for the Webhook Processor and the Primary instances. The Webhook worker however is not public facing and has a private internal domain - which I am not exactly sure how the other instances interact with it as that private domain is not passed as an env var to the other two.

Giulio_Pezzulli · July 10, 2024, 3:19pm

So apparently railway does have inbuilt load balancing as I can increase the replica’s of the Webhook processor instance and the Worker instance. And they mention it here:

juanmart1981 · July 11, 2024, 1:21am

Have you solved this? i’m encountering a very similar issue if not the

It works on /webhook-test/ but doesn’t work on /webhook/

Jon · July 11, 2024, 6:43am

Hey @Giulio_Pezzulli,

So in your case the webhook worker needs to be public, You would set the WEBHOOK_URL env option to your webhook worker (if you only have one if you have multiple it would point to your load balancer).

Then you would need to set N8N_EDITOR_BASE_URL to be the URL for the main instance. That should solve the issue.

Giulio_Pezzulli · July 11, 2024, 9:48am

Okay I think I am a bit confused, i just want to clarify.

Current Setup

Primary

start command: n8n start
public domain: true
WebhookURL: set to public webhook processor

Webhook Processor

start command: n8n webhook
publicDomain: true

Worker

start command: n8n worker
public domain: false

When you say “Webhook Worker” - do you mean the Worker instance with start command “n8n worker”?

Changes you suggest

Primary

start command: n8n start
public domain: true
WebhookURL: change this to Worker public domain
N8N_EDITOR_BASE_URL: Primary instance public domain (itself)

Webhook Processor

start command: n8n webhook
publicDomain: true

Worker

start command: n8n worker
public domain: change to true

Here are my questions:

Is this correct?
What does the webhook processor actually do then?

What doesn’t make sense here is that the Webhook Processor instance doesn’t seem to be doing anything now. So i will also try to see if this combination works with just the added n8n_editor_base_url set to the primary instance.

Thanks for all your support so far

Giulio_Pezzulli · July 11, 2024, 10:50am

Okay so as it stands I have actually done the following and it worked:

Primary

start command: n8n start
public domain: true
WebhookURL: Kept this pointing to public domain of Webhook Processor
N8N_EDITOR_BASE_URL: Primary instance public domain (itself) - this is what was missing!

Webhook Processor

this stayed the same

start command: n8n webhook
publicDomain: true

Worker

this stayed the same, i did not make it public

start command: n8n worker
public domain: kept it to false

Conclusion

So in the end, what I was missing was the N8N_EDITOR_BASE_URL was not set on my primary instance.

Follow up Questions

What i still would like to understand is:

What does the Webhook Processor instance do (the “n8n webhook” start command)
which instances should i have replica’s for and have a load balancer on top of? from what i understand, the Worker processes the jobs, and the webhook processor processes the incoming request, so i would need a load balancer for both?

Jon · July 11, 2024, 12:10pm

Hey @Giulio_Pezzulli.

The webhook processor handles webhook requests, this would be what outside users or systems interact with for inbound data items.
This is up to you, You can’t have multiple main instances without an enterprise license but you can have as many workers and webhook processors as you want. The only thing you would need to load balance is the webhook processors, This is covered in the documentation here: Configuring queue mode | n8n Docs

bruno_po · July 29, 2024, 11:39am

I solved the problem using Self-hosted Railway Deployment. I’ll share how it worked for me, which might help you. Directly from the Railway panel, during the N8N installation, choose the worker, select the three dots, and apply a redeploy.

system · October 27, 2024, 11:39am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.