Getting 403 trying to access self-hosted installation on subdomain hosted on DigitalOcean droplet

Questions
1

Setup

I have a self-hosted installation of n8n. For this I’ve used Digital Ocean droplet with Docker . I have a subdomain “n8n” and an A record. Basically I went along Digital Ocean | n8n Docs so the installation folder is that of a user with admin privilages (/home/user_name/)

My .env file lloks like this (of course user_name and example.com are my user name and domain):

# Replace <directory-path> with the path where you created folders earlier
DATA_FOLDER=/home/user_name/n8n-docker-caddy

# The top level domain to serve from, this should be the same as the subdomain you created above
DOMAIN_NAME=example.com

# The subdomain to serve from
SUBDOMAIN=n8n

# DOMAIN_NAME and SUBDOMAIN combined decide where n8n will be reachable from
# above example would result in: https://n8n.example.com

# Optional timezone to set which gets used by Cron-Node by default
# If not set New York time will be used
GENERIC_TIMEZONE=Europe/Berlin

# The email address to use for the SSL certificate creation
[email protected]

and the docker-compose.yml is the default

version: "3.7"

services:
  caddy:
    image: caddy:latest
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ${DATA_FOLDER}/caddy_data:/data
      - ${DATA_FOLDER}/caddy_config:/config
      - ${DATA_FOLDER}/caddy_config/Caddyfile:/etc/caddy/Caddyfile

  n8n:
    image: n8nio/n8n
    restart: always
    ports:
      - 5678:5678
    environment:
      - N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - NODE_ENV=production
      - WEBHOOK_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
      - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
    volumes:
      - ${DATA_FOLDER}/local_files:/files
      - ${DATA_FOLDER}/.n8n:/home/node/.n8n
volumes:
  caddy_data:
    external: true
  caddy_config:

The n8n version is 1.1.1.

Problem

The problem is that I get a random (?) 403 when I try to access the n8n via subdomain. What I mean by random:

  1. I’ve already access the install via subdomain, created a an admin user account and set a workflow there via using my default browser (Firefox dev).
  2. The workflow works. I can access the installation without problem via http://[droplet IP address]:5678 on my default browser.
  3. I sometimes get 403 forbidden access when going to n8n subdomain on my default browser. By sometimes I mean that two days ago it worked. Then it didn’t, so after finding some post I’ve run sudo docker run --rm -it --user root -v ~/.n8n:/home/node/.n8n --entrypoint chown n8nio/base:16 -R node:node /home/node/.n8n which solved the problem but then again I got 403 today, all aon my default browser.
  4. When accessing on my computer but from a different browser (standard Firefox) I could reach the n8n subdomain without problem. Did that today so not sure if it will work tommorow (will let you know).
  5. When accessing from a different computer (using standard Firefox) I could reach the n8n subdomain. Again, did that today so not sure if it will work tommorow (will let you know).

I feel like this is something similar to Init Problem: 403 Error - N8N, Nginx Proxy Manager, and Docker but my setup is different and I’m not a materfull sysadmin so I am not sure if this is valid in my setup and how to approach that.

If you need me to show some logs or test running a specific command line then please be a bit more precise with what and where to run :slight_smile:

4

Hi @Matiasko :wave: Welcome to the community! :tada:

I’m not too sure on this one, but would you also happen to have any server logs from DigitalOcean that might have some more details on this or when it stops working? it’s very strange it worked at one point, and then stopped :thinking:

1 Like
5

Yesterday it worked again on my default browser. Today it gives 403 on both dev and standard Firefox. Still available via http://[droplet IP address]:5678

If you meant container logs then here’s output of docker compose logs -t --since 2023-08-21 --tail all

n8n-docker-caddy-n8n-1  | 2023-08-21T20:04:13.793201437Z n8n ready on 0.0.0.0, port 5678
n8n-docker-caddy-n8n-1  | 2023-08-21T20:04:13.818241768Z Migrations in progress, please do NOT stop the process.
n8n-docker-caddy-n8n-1  | 2023-08-21T20:04:14.096453637Z Initializing n8n process
n8n-docker-caddy-n8n-1  | 2023-08-21T20:04:16.911897642Z Version: 1.3.1
n8n-docker-caddy-n8n-1  | 2023-08-21T20:04:17.035783577Z  ================================
n8n-docker-caddy-n8n-1  | 2023-08-21T20:04:17.036192192Z    Start Active Workflows:
n8n-docker-caddy-n8n-1  | 2023-08-21T20:04:17.036632628Z  ================================
n8n-docker-caddy-n8n-1  | 2023-08-21T20:04:17.036936844Z    - My workflow (ID: loGsc17yiJvhJIWk)
n8n-docker-caddy-n8n-1  | 2023-08-21T20:04:17.581701992Z      => Started
n8n-docker-caddy-n8n-1  | 2023-08-21T20:04:17.584401912Z 
n8n-docker-caddy-n8n-1  | 2023-08-21T20:04:17.584450905Z Editor is now accessible via:
n8n-docker-caddy-n8n-1  | 2023-08-21T20:04:17.584457498Z https://n8n.mateuszkalamarz.pl:5678/
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.933759555Z {"level":"info","ts":1692648248.9301057,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.935564078Z {"level":"warn","ts":1692648248.9354033,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.951691253Z {"level":"info","ts":1692648248.9514787,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.953166632Z {"level":"info","ts":1692648248.9526803,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.953213130Z {"level":"info","ts":1692648248.9527133,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.953773182Z {"level":"info","ts":1692648248.9536574,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.953988440Z {"level":"info","ts":1692648248.9539208,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.954363672Z {"level":"info","ts":1692648248.9542823,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.954553753Z {"level":"info","ts":1692648248.9544797,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.954699098Z {"level":"info","ts":1692648248.9545808,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["n8n.mateuszkalamarz.pl"]}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.959542925Z {"level":"info","ts":1692648248.9593923,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.959690194Z {"level":"info","ts":1692648248.959598,"msg":"serving initial configuration"}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.960047471Z {"level":"info","ts":1692648248.9599795,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003ca380"}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.960786754Z {"level":"info","ts":1692648248.9606555,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:08.963849141Z {"level":"info","ts":1692648248.963657,"logger":"tls","msg":"finished cleaning storage units"}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:29.586991090Z {"level":"error","ts":1692648269.586737,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"n8n:5678","duration":0.025854363,"request":{"remote_ip":"213.156.100.36","remote_port":"62735","client_ip":"213.156.100.36","proto":"HTTP/2.0","method":"GET","host":"n8n.mateuszkalamarz.pl","uri":"/assets/open-sans-latin-400-normal-b34551ae.woff2","headers":{"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["n8n.mateuszkalamarz.pl"],"Sec-Fetch-Dest":["font"],"Sec-Fetch-Mode":["cors"],"Accept":["application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"X-Forwarded-For":["213.156.100.36"],"Accept-Encoding":["identity"],"Sec-Fetch-Site":["same-origin"],"Cookie":[],"Referer":["https://n8n.mateuszkalamarz.pl/assets/index-6122886e.css"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/117.0"],"Te":["trailers"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"n8n.mateuszkalamarz.pl"}},"error":"writing: http2: stream closed"}
n8n-docker-caddy-caddy-1  | 2023-08-21T20:04:29.588407285Z {"level":"error","ts":1692648269.5882406,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"n8n:5678","duration":0.026502924,"request":{"remote_ip":"213.156.100.36","remote_port":"62735","client_ip":"213.156.100.36","proto":"HTTP/2.0","method":"GET","host":"n8n.mateuszkalamarz.pl","uri":"/assets/open-sans-latin-700-normal-d1a17abb.woff2","headers":{"X-Forwarded-Host":["n8n.mateuszkalamarz.pl"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["identity"],"Sec-Fetch-Mode":["cors"],"Cookie":[],"Referer":["https://n8n.mateuszkalamarz.pl/assets/index-6122886e.css"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/117.0"],"Sec-Fetch-Dest":["font"],"Te":["trailers"],"X-Forwarded-For":["213.156.100.36"],"X-Forwarded-Proto":["https"],"Accept":["application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"n8n.mateuszkalamarz.pl"}},"error":"writing: http2: stream closed"}
n8n-docker-caddy-caddy-1  | 2023-08-21T22:39:47.310981379Z {"level":"error","ts":1692657587.3105536,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"n8n:5678","duration":0.003872795,"request":{"remote_ip":"213.156.100.36","remote_port":"52096","client_ip":"213.156.100.36","proto":"HTTP/2.0","method":"GET","host":"n8n.mateuszkalamarz.pl","uri":"/n8n-logo-collapsed.svg","headers":{"Sec-Fetch-Dest":["image"],"X-Forwarded-For":["213.156.100.36"],"X-Forwarded-Proto":["https"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/117.0"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate, br"],"Referer":["https://n8n.mateuszkalamarz.pl/workflows"],"Sec-Fetch-Mode":["no-cors"],"Accept":["image/avif,image/webp,*/*"],"X-Forwarded-Host":["n8n.mateuszkalamarz.pl"],"Sec-Fetch-Site":["same-origin"],"Cookie":[]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"n8n.mateuszkalamarz.pl"}},"error":"writing: http2: stream closed"}
n8n-docker-caddy-caddy-1  | 2023-08-22T07:04:09.118853831Z {"level":"info","ts":1692687849.1184688,"logger":"tls.cache.maintenance","msg":"advancing OCSP staple","identifiers":["n8n.mateuszkalamarz.pl"],"from":1692989998,"to":1693288798}
6

Hey @Matiasko,

It sounds like it could be an issue with a cache or n8n is crashing, Are you using Cloudflare at all for your domain?

7

Not at all. And I’m not sure about the crashing as I can always access the install via IP url.

8

Now I’m just guessing but could this be due to having wildcard SSL from Let’s Encrypt set for the domain and subdomains. I might be wrong but isn’t there a separate certificate for the n8n?

9

Hey @Matiasko,

A wildcard certificate will be fine I use one for my n8n instance, I guess if using the IP is always working then the issue is going to be outside of n8n. Where do you manage your DNS and is there any special caching happening at all?