Self-hosted N8N on Coolify in a Hetzner VPS can't connect to Google services

lamurian · March 1, 2025, 6:28pm

Describe the problem/error/question

I have a problem when authorizing access to BigQuery and Google Gemini nodes in my workflow. I use the same workflow in the cloud version, and everything works well. However, when I tried the exact same workflow in a self-hosted N8N instance within Coolify, it refuses to authorize my access. Google BigQuery node was authenticated using OAuth2, as suggested by the documentation. Google Gemini node was authenticated using API Key from Google AI Studio, also as suggested by the documentation.

What is the error message (if any)?

This is the associated error logs obtained from N8N canvas for BigQuery node:

Issues:

    There was a problem loading the parameter options from server: "403 - "\n\n \n \n Error 403 (Forbidden)!!1\n \n \n

    403. That’s an error.\n

    Your client does not have permission to get URL /bigquery/v2/projects/tonal-run-451811-t9/datasets/news_aggregator/tables/curated_news from this server. That’s all we know.\n""

So, I use the Coolify instance to access the N8N container terminal with:

docker exec -it $n8n_container sh

Then, I tried to manually send an HTTP request to Gemini API:

export GEMINI_API="my_gemini_api"
wget --header="Content-Type: application/json" \
     --post-data='{
       "contents": [{
         "parts":[{"text": "Write a story about a magic backpack."}]
       }]
     }' \
     "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=${GEMINI_API_KEY}" \
     -O -

Which gives me the following error:

Connecting to generativelanguage.googleapis.com (216.58.210.170:443)
wget: server returned error: HTTP/1.1 403 Forbidden

I also tried to use the IPv6 address:

wget --header="Content-Type: application/json" \
     --post-data='{
       "contents": [{
         "parts":[{"text": "Write a story about a magic backpack."}]
       }]
     }' \
     "https://2a00:1450:4026:802::200a/v1beta/models/gemini-1.5-flash:generateContent?key=${GEMINI_API_KEY}" \
     -O -

Which gives me the following error:

Connecting to 2a00:1450:4026:802::200a ([2a00:1450:4026:802::200a]:443)
wget: can't connect to remote host: Network unreachable

On a side note, N8N can access other instances. I tried with DeepSeek and Groq credentials, and there’s no issue. Google services are the only ones refused to work.

Please share your workflow

Share the output returned by the last node

Only an error because I can’t use the Google BigQuery and Google Gemini nodes.

Information on your n8n setup

n8n version: The latest docker version
Database (default: SQLite): SQLite
n8n EXECUTIONS_PROCESS setting (default: own, main):
Running n8n via (Docker, npm, n8n cloud, desktop app): Docker, deployed via Coolify
Operating system: Ubuntu

lamurian · March 1, 2025, 6:33pm

I tested the connection using Coolify terminal, and it gets connected to Google Gemini using the API Key. So, I suppose the issue isn’t the Hetzner VPS, Coolify instance, nor the API Key.

lamurian · March 1, 2025, 6:35pm

This is the Docker compose file deployed in Coolify:

services:
  n8n:
    image: docker.n8n.io/n8nio/n8n
    environment:
      - SERVICE_FQDN_N8N_5678
      - 'N8N_EDITOR_BASE_URL=${SERVICE_FQDN_N8N}'
      - 'WEBHOOK_URL=${SERVICE_FQDN_N8N}'
      - 'N8N_HOST=${SERVICE_URL_N8N}'
      - 'GENERIC_TIMEZONE=${GENERIC_TIMEZONE:-Europe/Berlin}'
      - 'TZ=${TZ:-Europe/Berlin}'
    volumes:
      - 'n8n-data:/home/node/.n8n'
    healthcheck:
      test:
        - CMD-SHELL
        - 'wget -qO- http://127.0.0.1:5678/'
      interval: 5s
      timeout: 20s
      retries: 10

And this is the .env:

GENERIC_TIMEZONE=Asia/Jakarta
NODE_ENV=production
SERVICE_FQDN_N8N=https://flow.konsulin.care
SERVICE_URL_N8N=flow.konsulin.care
NODE_VERSION=22.12.0
DOMAIN_NAME=konsulin.care
N8N_HOST=flow.konsulin.care
N8N_PORT=5678
N8N_PROTOCOL=https
N8N_RUNNERS_ENABLED=true
SUBDOMAIN=flow
TZ=Asia/Jakarta
WEBHOOK_URL=https://flow.konsulin.care/

Yo_its_prakash · March 1, 2025, 8:02pm

The hertzner Ip might be flagged by Google (sometimes the hertzner users have faced that)

check if it true, by sending a Bigquerry CURL to google (to debug from your hertzner server):

curl -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
    "https://bigquery.googleapis.com/bigquery/v2/projects/YOUR_PROJECT_ID/datasets"

If that cant be accessed either, let me know

lamurian · March 1, 2025, 8:19pm

Hi @Yo_its_prakash, thank you for your suggestion. I sent two HTTP requests directly from the server, one request to Gemini and the other to BigQuery. These requests can access the API endpoints, no issue at all. So, in short:

Accessing directly from the VPS → Success
Accessing from Coolify terminal → Success
Accessing from N8N docker instance → Failure, 403 (Forbidden) error
Accessing from N8N web interface → Failure, 403 (Forbidden) error

Command 1:

wget --header="Content-Type: application/json" \
     --post-data='{
       "contents": [{
         "parts":[{"text": "Write a story about a magic backpack."}]
       }]
     }' \
     "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=${GEMINI_API_KEY}" \
     -O -

Ouput:

  "candidates": [
    {
      "content": {
        "parts": [
          {
            "text": "... (Omitted for brevity)"
          }
        ],
        "role": "model"
      },
      "finishReason": "STOP",
      "avgLogprobs": -0.632054860674703
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 8,
    "candidatesTokenCount": 547,
    "totalTokenCount": 555,
    "promptTokensDetails": [
      {
        "modality": "TEXT",
        "tokenCount": 8
      }
    ],
    "candidatesTokensDetails": [
      {
        "modality": "TEXT",
        "tokenCount": 547
      }
    ]
  },
  "modelVersion": "gemini-1.5-flash"
}

Command 2:

curl -H "Authorization: Bearer $TOKEN" \
    "https://bigquery.googleapis.com/bigquery/v2/projects/tonal-run-451811-t9/datasets"

Output:

{
  "kind": "bigquery#datasetList",
  "etag": "DRditU/h99+ZpjD3FhBxXQ==",
  "datasets": [
    {
      "kind": "bigquery#dataset",
      "id": "tonal-run-451811-t9:news_aggregator",
      "datasetReference": {
        "datasetId": "news_aggregator",
        "projectId": "tonal-run-451811-t9"
      },
      "location": "US"
    }
  ]
}

Michel_Morelli · March 2, 2025, 12:45am

Hi, have you tried to don’t use n8n via Docker but via nginx (its job is to act as a proxy) ?

lamurian · March 2, 2025, 4:03am

Hi @Michel_Morelli, thank you for your advice. I’m not sure how to achieve that in Coolify; might you have further suggestions on how to achieve that?

On a side note, I found a similar problem reported as an issue in the N8N repo. The underlying cause of this issue seems to be unclear. I already tried to reconfigure the credentials, nodes, etc., but I don’t know how to reconfigure the server side.

lamurian · March 2, 2025, 4:06am

In Coolify, I found these proxy settings. Could this be somewhat relevant to your suggestions?

Update: I tested the workflow without resolving the issues with Gemini and/or BQ. There appears to be a problem when sending an HTTP request from N8N. The workflow can fetch data from the internet, as we see in the successful Read RSS node. But it can’t send an HTTP request.

Forbidden - perhaps check your credentials? [item 2]
ERROR: The request could not be satisfied 403 ERROR The request could not be satisfied. Request blocked. We can’t connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
Generated by cloudfront (CloudFront) Request ID: pXdNthx9YaocG3VDyBxI2I4MYi1_6nSnmHScjmGYGkQKss7i8P0ilg==

The errors occur when accessing specific URLs. I reconfigured the node to continue on errors, and as expected, it’s able to hit Groq API but unable to hit Gemini nor BQ APIs.

lamurian · March 2, 2025, 8:16pm

After digging around, I discovered the problem was that Google blocked Hetzner’s IP addresses. Other people have faced similar issues, and their solution was to disable IPv6 and then reserve the floating IPv4. Another solution is to get a new IPv4. In my case, I host N8N to another server. After migrating to AWS, everything works as intended. I’ll put it here as a note if someone also faces the same problem when deploying N8N in Hetzner with/without Coolify.

brahimh · March 2, 2025, 11:07pm

I’m thinking about migrating from DigitalOcean to HETZNER, is this a problems with HETZNER in general?
Is not having the IPv4 what’s causing the problem?

lamurian · March 2, 2025, 11:39pm

There’s no direct confirmation from Google, but it seems related to Google’s blacklist. Hetzner reuses IP addresses (or so some people say), and some of them are blacklisted by Google. There are many other possibilities, so I think sharing the sources with you is better.

Sources:

https://www.reddit.com/r/hetzner/comments/12kg2vt/deleted_by_user/
Why am I seeing 403 Forbidden when trying to send events? – Sentry Help Center
https://managingwp.io/live-blog/google-cloud-and-hetzner-connectivity-issues-gridpane-specifics/
429 google drive errors back for anyone using Hetzner, even with cloudflare or google dns used? - #93 by codsane - Help and Support - rclone forum
GSuite-Uploads durch Google für bestimmte Hetzner IP-Adressen blockiertBorns IT- und Windows-Blog

Personally, I’d still use Hetzner, considering its cost-effectiveness. I’m hosting N8N in AWS just because I need to use Google Services. Other than that, I deployed most of my apps in Hetzner.

brahimh · March 3, 2025, 2:42am

Many thanks for sharing
I use google services too.

Did you try using Cloudflare DNS? (As mentioned in one of the sources you shared)

solomon · March 3, 2025, 2:54am

Thanks for sharing the solution.

I read following this topic because I was curious about what could be happening.

Minijus_Savickas · March 6, 2025, 8:19pm

I also recommend trying Hostinger’s VPS for n8n. It comes with a 1-click n8n template that requires no extra configuration, and I’ve found it works flawlessly without any issues from Google services.

bigfather · March 6, 2025, 8:25pm

@Minijus_Savickas bro, can your hostinger vps run puppeteer community node? i cant get it to run in hetznerxcoolify. n8n-nodes-puppeteer

Minijus_Savickas · March 7, 2025, 12:56pm

@bigfather What are the errors while trying to run this? As far as I know, puppeteer worked for me previously in Hostinger’s VPS. There’s also extended version of this node n8n-nodes-puppeteer-extended, maybe this would work for you? You can also share a specific workflow that you have, so that I could test it out for you

system · March 14, 2025, 12:57pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.