Webhook works, but after a while it starts returning 404s

[Jorijn]

Hi. I have a couple of webhooks running. At some point I duplicated a workflow and a conflict happened with the webhook identifier. The DB contents indicated no webhooks were registered but they were showing in the UI. I then recreated all webhook triggers for these workflows and the DB looked fine again.

After I restart n8n, the workflows work just fine. But, after a while they start returning 404s:

➜  ~ curl -X POST xxx
{"code":404,"message":"The requested webhook \"POST xxx\" is not registered.","hint":"The workflow must be active for a production URL to run successfully. You can activate the workflow using the toggle in the top-right of the editor. Note that unlike test URL calls, production URL calls aren't shown on the canvas (only in the executions list)"}%

Information on your n8n setup

• n8n version: 1.2.1
• Database (default: SQLite): postgres (saas digitalocean)
• Running n8n via (Docker, npm, n8n cloud, desktop app): kubernetes
• Operating system: linux

Any help would be greatly appreciated.

[Jon]

Hey @Jorijn,

Welcome to the community

Have you checked the n8n container output to see if it shows any errors? Sometimes if a webhook workflow hits an error it can disable itself but to also check it isn’t a webhook-test url you are using right?

Do you know how long or how manu requests it takes to get the 404s as well?

[Jorijn]

Hey Jon

Thanks for the quick reply.

I confirmed, I am using the production URL.

When looking back, it appears to be showing this error:

2023-08-04 02:13:34.388	
query failed: INSERT INTO "public"."workflow_statistics"("count", "latestEvent", "name", "workflowId") VALUES ($1, $2, $3, $4) -- PARAMETERS: [1,"2023-08-04T00:13:34.372Z","data_loaded","2"]
2023-08-04 02:13:33.105	
query failed: INSERT INTO "public"."workflow_statistics"("count", "latestEvent", "name", "workflowId") VALUES ($1, $2, $3, $4) -- PARAMETERS: [1,"2023-08-04T00:13:33.096Z","data_loaded","2"]

2023-08-04 02:13:33.105	
error: error: duplicate key value violates unique constraint "pk_workflow_statistics"
2023-08-04 01:12:35.920	
error: error: duplicate key value violates unique constraint "pk_workflow_statistics"
2023-08-04 01:12:35.920	
query failed: INSERT INTO "public"."workflow_statistics"("count", "latestEvent", "name", "workflowId") VALUES ($1, $2, $3, $4) -- PARAMETERS: [1,"2023-08-03T23:12:35.915Z","data_loaded","1"]
2023-08-04 01:12:35.573	
error: error: duplicate key value violates unique constraint "pk_workflow_statistics"

also:

		2023-08-09 13:59:30.201	
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-08-09 13:59:30.201	
    at Object.executeWebhook (/usr/local/lib/node_modules/n8n/dist/WebhookHelpers.js:437:15)
2023-08-09 13:59:30.201	
ResponseError: getaddrinfo EAI_AGAIN <url of database server>
2023-08-09 13:59:30.201	
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:

I haven’t been able to pinpoint when the webhook disables.

Could these trigger the disabling of the webhook?

[jan]

I assume k8s is the problem. It is probably configured to start a new container and only then turn off the old one. That will cause problem as the old one will delete the database entries when it turns off.
So you either would have to:

• configure it so that it first shuts down the old container before the new one gets started or
• set the environment variable N8N_SKIP_WEBHOOK_DEREGISTRATION_SHUTDOWN=true

[Jorijn]

Hey @jan - thanks for pointing me in the right direction. I’ve added the environment flag and restarted the deployment twice to clear out the olds settings.

Now, I’m setting messages like this:

│  ================================                                                                                                                                                                                │
│    Start Active Workflows:                                                                                                                                                                                       │
│  ================================                                                                                                                                                                                │
│    - <name here> (ID: 1)                                                                                                                                              │
│ query failed: INSERT INTO "public"."webhook_entity"("workflowId", "webhookPath", "method", "node", "webhookId", "pathLength") VALUES ($1, $2, $3, $4, DEFAULT, DEFAULT) -- PARAMETERS: ["1","snip- │
│ error: error: duplicate key value violates unique constraint "PK_snip"

The record is not removed, so it’s already there. That makes sense. Can I conclude that this message is harmless?