"The DNS server returned an error, perhaps the server is offline."

Describe the problem/error/question

Upon fixing my nginx reverse proxy issue preventing the socket from connecting, I receive the following error for all requests made. (Extra odd since the exact setup worked 2 months ago perfectly fine)

What is the error message (if any)?

I have:

  • completely removed and reinstalled the latest docker & docker compose.
  • deleted and recreated the images
  • tried to add the custom DNS directive to the docker compose file (ignores entirely, checking the n8n image’s resolv.conf shows ‘127.0.0.1’)
  • I also set my server’s resolv.conf to 1.1.1.1, 1.0.0.1. Nothing
  • I could not find in docs if there were any config settings or ENV vars I could set
  • forcing the dns via a debug dockerd dockerd --debug --dns 1.1.1.1 also does nothing

docker-compose.yaml:

version: '3'

services:
  traefik:
    image: 'traefik'
    command:
      - '--api=true'
      - '--api.insecure=true'
      - '--api.dashboard=true'
      - '--providers.docker=true'
      - '--providers.docker.exposedbydefault=false'
      - '--entrypoints.websecure.address=:443'
      - '--certificatesresolvers.mytlschallenge.acme.tlschallenge=true'
      - '--certificatesresolvers.mytlschallenge.acme.email=${SSL_EMAIL}'
      - '--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json'
    ports:
      - '8443:443'
      - '8080:80'
    dns:
      - 1.1.1.1
    volumes:
      - ${DATA_FOLDER}/letsencrypt:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock:ro

  initContainer:
    image: busybox
    command: ['sh', '-c', 'chown -R 1000:1000 /home/node/.n8n']
    volumes:
      - ${DATA_FOLDER}/.n8n:/home/node/.n8n

  n8n:
    image: docker.n8n.io/n8nio/n8n
    ports:
      - '0.0.0.0:5678:5678'
    dns:
      - 1.1.1.1
    labels:
      - traefik.enable=true
      - traefik.http.routers.n8n.rule=Host(`${DOMAIN_NAME}`)
      - traefik.http.routers.n8n.tls=true
      - traefik.http.routers.n8n.entrypoints=websecure
      - 'traefik.http.routers.n8n.rule=PathPrefix(`/${SUBFOLDER}{regex:$$|/.*}`)'
      - 'traefik.http.middlewares.n8n-stripprefix.stripprefix.prefixes=/${SUBFOLDER}'
      - 'traefik.http.routers.n8n.middlewares=n8n-stripprefix'
      - traefik.http.routers.n8n.tls.certresolver=mytlschallenge
      - traefik.http.middlewares.n8n.headers.SSLRedirect=true
      - traefik.http.middlewares.n8n.headers.STSSeconds=315360000
      - traefik.http.middlewares.n8n.headers.browserXSSFilter=true
      - traefik.http.middlewares.n8n.headers.contentTypeNosniff=true
      - traefik.http.middlewares.n8n.headers.forceSTSHeader=true
      - traefik.http.middlewares.n8n.headers.SSLHost=${DOMAIN_NAME}
      - traefik.http.middlewares.n8n.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.n8n.headers.STSPreload=true
    environment:
      - N8N_HOST=${DOMAIN_NAME}
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - NODE_ENV=production
      - N8N_PATH
      - WEBHOOK_URL=https://${DOMAIN_NAME}${N8N_PATH}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ${DATA_FOLDER}/.n8n:/home/node/.n8n
    depends_on:
      initContainer:
        condition: service_completed_successfully

n8n container output:

| User settings loaded from: /home/node/.n8n/config
n8n-1            | Last session crashed
n8n-1            | Initializing n8n process
n8n-1            | n8n ready on 0.0.0.0, port 5678
n8n-1            | Version: 1.57.0
n8n-1            |
n8n-1            | Editor is now accessible via:
n8n-1            | https://0.0.0.0:5678/
n8n-1            | 2024-09-05T23:04:59.569Z [Rudder] error: Response error code: EAI_AGAIN
n8n-1            | 2024-09-05T23:05:04.786Z [Rudder] error: Response error code: EAI_AGAIN
n8n-1            | 2024-09-05T23:05:10.235Z [Rudder] error: Response error code: EAI_AGAIN
n8n-1            | 2024-09-05T23:05:16.177Z [Rudder] error: Error: EAI_AGAIN
n8n-1            | AxiosError: getaddrinfo EAI_AGAIN telemetry.n8n.io
n8n-1            | Error: getaddrinfo EAI_AGAIN telemetry.n8n.io
n8n-1            | AxiosError: getaddrinfo EAI_AGAIN telemetry.n8n.io
n8n-1            | Error: getaddrinfo EAI_AGAIN telemetry.n8n.io
AxiosError: getaddrinfo EAI_AGAIN redacted.teamwork.com
n8n-1            | Error: getaddrinfo EAI_AGAIN redacted.teamwork.com
n8n-1            | 2024-09-05T23:05:43.719Z [Rudder] error: Response error code: EAI_AGAIN
n8n-1            | 2024-09-05T23:05:48.961Z [Rudder] error: Response error code: EAI_AGAIN
n8n-1            | 2024-09-05T23:05:54.446Z [Rudder] error: Response error code: EAI_AGAIN
n8n-1            | Error fetching feature flags Error [PostHogFetchNetworkError]: Network error while fetching PostHog
n8n-1            |     at new PostHogFetchNetworkError (/usr/local/lib/node_modules/n8n/node_modules/posthog-node/lib/index.cjs.js:740:16)
n8n-1            |     at PostHog.<anonymous> (/usr/local/lib/node_modules/n8n/node_modules/posthog-node/lib/index.cjs.js:1129:51)
n8n-1            |     at step (/usr/local/lib/node_modules/n8n/node_modules/posthog-node/lib/index.cjs.js:135:27)
n8n-1            |     at Object.throw (/usr/local/lib/node_modules/n8n/node_modules/posthog-node/lib/index.cjs.js:84:53)
n8n-1            |     at rejected (/usr/local/lib/node_modules/n8n/node_modules/posthog-node/lib/index.cjs.js:71:36)
n8n-1            |     at processTicksAndRejections (node:internal/process/task_queues:95:5) {
n8n-1            |   error: TypeError: fetch failed
n8n-1            |       at node:internal/deps/undici/undici:12502:13
n8n-1            |       at processTicksAndRejections (node:internal/process/task_queues:95:5) {
n8n-1            |     [cause]: Error: getaddrinfo EAI_AGAIN ph.n8n.io
n8n-1            |         at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:109:26) {
n8n-1            |       errno: -3001,
n8n-1            |       code: 'EAI_AGAIN',
n8n-1            |       syscall: 'getaddrinfo',
n8n-1            |       hostname: 'ph.n8n.io'
n8n-1            |     }
n8n-1            |   },
n8n-1            |   [cause]: TypeError: fetch failed
n8n-1            |       at node:internal/deps/undici/undici:12502:13
n8n-1            |       at processTicksAndRejections (node:internal/process/task_queues:95:5) {
n8n-1            |     [cause]: Error: getaddrinfo EAI_AGAIN ph.n8n.io
n8n-1            |         at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:109:26) {
n8n-1            |       errno: -3001,
n8n-1            |       code: 'EAI_AGAIN',
n8n-1            |       syscall: 'getaddrinfo',
n8n-1            |       hostname: 'ph.n8n.io'
n8n-1            |     }
n8n-1            |   }
n8n-1            | }
n8n-1            | 2024-09-05T23:06:00.415Z [Rudder] error: Error: EAI_AGAIN

Information on your n8n setup

  • n8n version: latest
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app): docker
  • Operating system: Ubuntu 20.04

More details directly from dockerd debug stream. It seems to be respecting the 1.1.1.1 DNS from somewhere…

I may just misunderstand how docker works.

ERRO[2024-09-05T18:49:55.798904109-05:00] [resolver] failed to query external DNS server  client-addr="udp:172.18.0.3:32784" dns-server="udp:1.1.1.1:53" error="read udp 172.18.0.3:32784->1.1.1.1:53: i/o timeout" question=";telemetry.n8n.io.\tIN\t AAAA"
ERRO[2024-09-05T18:49:55.798938969-05:00] [resolver] failed to query external DNS server  client-addr="udp:172.18.0.3:52818" dns-server="udp:1.1.1.1:53" error="read udp 172.18.0.3:52818->1.1.1.1:53: i/o timeout" question=";telemetry.n8n.io.\tIN\t A"
ERRO[2024-09-05T18:49:57.297642177-05:00] [resolver] failed to query external DNS server  client-addr="udp:172.18.0.3:39507" dns-server="udp:1.1.1.1:53" error="read udp 172.18.0.3:39507->1.1.1.1:53: i/o timeout" question=";telemetry.n8n.io.\tIN\t A"
ERRO[2024-09-05T18:49:57.297678186-05:00] [resolver] failed to query external DNS server  client-addr="udp:172.18.0.3:57586" dns-server="udp:1.1.1.1:53" error="read udp 172.18.0.3:57586->1.1.1.1:53: i/o timeout" question=";telemetry.n8n.io.\tIN\t AAAA"

Bumping for visibility. Appreciate the help.

Hey @ThinkBot,

The error is pretty much saying that there is a DNS issue, This is unlikely to be caused by your reverse proxy as a reverse proxy is only used for inbound traffic not outbound traffic.

For some of the issues you could set the diagnostics option to false which will stop the anonymous telemetry calls like the ph.n8n.io errors you are seeing but it may not solve the issue with the nodes for that you will need to make sure your docker container is using a DNS server which responds correctly which 1.1.1.1 should be doing.

Do you get the error all the time or is it intermittent and does it apply to all external hosts?

@Jon
Thanks for the response. It is 100% consistent with n8n only and all requests thru it. All other running services on my dedi server encounter no dns based issues (although most are not run via docker).

I even enforced port 53 open via UFW. No dice.

I think its just extra odd that dockerd is attempting to use 1.1.1.1 yet still fails.

Hey @ThinkBot,

Yeah that is a bit strange, It could also be container network related maybe. Do you have n8n running on a different docker network that maybe doesn’t have outside world access?

I am not quite sure why this changed things…

But thru many hours of testing and investigation. I finally fixed this. Simply adding:

network_mode: bridge to the n8n container in your docker compose fixes this issue.
I don’t fully understand why, but the bug seems to be related to a bug that exists all the way back in 2016: dns configuration option does not work on docker-compose==1.6.0 · Issue #2847 · docker/compose · GitHub.

Despite network mode bridge being default when no mode is attached.

All my DNS settings were correct on host, and within docker compose otherwise.

For refernce:
Docker compose: v2.29.2
Dockerd/docker: 27.2.0, build 3ab4256

2 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.