Self-hosted n8n does not recover automatically after postgres restart

[vkarbovnichy]

I found a great thread about healthchecks for docker containers

Docker image suggestion: Add healthcheck

The idea is: Add a healthcheck to the n8n docker images by default utilizing n8n’s /healthz status endpoint (which must be enabled for this to work - since it is not enabled by default, that may be a point of contention here? Or maybe it’s not a big deal.) My use case: To allow for out of the box Docker health monitoring of n8n containers. With it enabled, docker can perform it’s own management steps if a container becomes unhealthy, the user can see that it’s health in docker ps output, etc. …

I tried to implement that strategy, but the /healthz endpoint still return HTTP 200 OK when my postgres container is stopped.

This is the proof:

root@user:~# docker ps -a
CONTAINER ID   IMAGE                            COMMAND                  CREATED       STATUS                      PORTS
                                              NAMES
04b49fd7a88d   docker.n8n.io/n8nio/n8n:1.47.3   "tini -- /docker-ent…"   12 days ago   Up 48 minutes               0.0.0.0:5678->5678/tcp, :::5678->5678/tcp                                  n8n-infra-n8n-1
c7a5d6677707   postgres:16                      "docker-entrypoint.s…"   12 days ago   Exited (0) 31 minutes ago
                                              n8n-infra-postgres-1
7140108c7903   traefik:v3.1                     "/entrypoint.sh --ap…"   12 days ago   Up 7 days                   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   n8n-infra-traefik-1
root@user:~# docker exec n8n-infra-n8n-1 wget -S http://127.0.0.1:5678/healthz -O /dev/null
Connecting to 127.0.0.1:5678 (127.0.0.1:5678)
  HTTP/1.1 200 OK
  Content-Type: application/json; charset=utf-8
  Content-Length: 15
  ETag: W/"f-VaSQ4oDUiZblZNAEkkN+sX+q3Sg"
  Date: Sat, 31 Aug 2024 20:08:13 GMT
  Connection: close

saving to '/dev/null'
null                 100% |********************************|    15  0:00:00 ETA
'/dev/null' saved

These are last log lines of the n8n container:

Editor is now accessible via:
https://n8n.octoflow.ru:5678/
DatabaseError: terminating connection due to administrator command
Error: Connection terminated unexpectedly
Error: Connection terminated unexpectedly

I tried to check what /rest/login gives in this situation, but it works normally somehow

root@user:~# docker exec n8n-infra-n8n-1 wget -S http://127.0.0.1:5678/rest/login -O /dev/null
Connecting to 127.0.0.1:5678 (127.0.0.1:5678)
  HTTP/1.1 401 Unauthorized
wget: server returned error: HTTP/1.1 401 Unauthorized

In my browser it does not finish because browser sends correct cookies and n8n parses them normally and then tries to go to the database, not able to continue, so doesn’t work.

Now I have no idea what to test to know if n8n is working normally.