Http proxy problem

a1ekseev · October 31, 2021, 5:42pm

I added in n8n.env

HTTP_PROXY="http://10.20.0.1:1080"
HTTPS_PROXY="http://10.20.0.1:1080"

And everything worked successfully, until the latest version was updated.
When I start Workflow, I get a 400 Bad Request error.

StackTrace

ERROR RESPONSE
NodeApiError: Bad request - please check your parameters
    at Object.apiRequest (/usr/local/lib/node_modules/n8n/node_modules/n8n-nodes-base/dist/nodes/Telegram/GenericFunctions.js:84:15)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Object.checkExists (/usr/local/lib/node_modules/n8n/node_modules/n8n-nodes-base/dist/nodes/Telegram/TelegramTrigger.node.js:148:47)
    at async ActiveWorkflowRunner.addWorkflowWebhooks (/usr/local/lib/node_modules/n8n/dist/src/ActiveWorkflowRunner.js:205:39)
    at async ActiveWorkflowRunner.add (/usr/local/lib/node_modules/n8n/dist/src/ActiveWorkflowRunner.js:349:13)
    at async /usr/local/lib/node_modules/n8n/dist/src/Server.js:471:21
    at async /usr/local/lib/node_modules/n8n/dist/src/ResponseHelper.js:86:26 {
  cause: Error: Request failed with status code 400
      at createError (/usr/local/lib/node_modules/n8n/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/local/lib/node_modules/n8n/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/local/lib/node_modules/n8n/node_modules/axios/lib/adapters/http.js:269:11)
      at IncomingMessage.emit (events.js:327:22)
      at IncomingMessage.EventEmitter.emit (domain.js:467:12)
      at endReadableNT (internal/streams/readable.js:1327:12)
      at processTicksAndRejections (internal/process/task_queues.js:80:21) {
    config: undefined,
    request: undefined,
    response: {
      status: 400,
      statusText: 'Bad Request',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html><head><title>400 Bad Request</title></head>\r\n' +
        '<body><h2>400 Bad Request</h2></body></html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON],
    cause: {},
    error: '<html><head><title>400 Bad Request</title></head>\r\n' +
      '<body><h2>400 Bad Request</h2></body></html>\r\n',
    statusCode: 400,
    options: {
      url: 'https://api.telegram.org/botTOKEN/getWebhookInfo',
      method: 'post',
      headers: [Object],
      transformRequest: undefined,
      transformResponse: undefined,
      paramsSerializer: undefined,
      timeout: 300000,
      adapter: undefined,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      validateStatus: undefined,
      transitional: [Object],
      'axios-retry': [Object],
      data: undefined,
      httpsAgent: undefined
    }
  },
  node: {
    parameters: { updates: [Array], additionalFields: [Object] },
    name: 'TG WebHook',
    type: 'n8n-nodes-base.telegramTrigger',
    typeVersion: 1,
    position: [ 170, 500 ],
    webhookId: 'ID',
    credentials: { telegramApi: [Object] }
  },
  timestamp: 1635701784120,
  httpCode: '400',
  description: '<html><head><title>400 Bad Request</title></head>\r\n' +
    '<body><h2>400 Bad Request</h2></body></html>\r\n'

The same request on the same server through curl + proxy succeeds.
Also, when there is an error in the service, I can see the request in the proxy log.

Version: 0.146.0

RicardoE105 · October 31, 2021, 9:27pm

We change the library to make http requests from request to axios. That env variable HTTP_PROXY and HTTPS_PROXY are only valid for the request library and not axios. To keep using the request library set the env variable N8N_USE_DEPRECATED_REQUEST_LIB=true

RedPacketSec · November 2, 2021, 4:58pm

yeah the new library is breaking lots of things for me, using HTTP request nodes, I’m going to have to go back to the old library.

a1ekseev · November 2, 2021, 6:05pm

Why haven’t you updated this document? n8n/BREAKING-CHANGES.md at master · n8n-io/n8n · GitHub

The new library will not support HTTP RPOXY?

When will support for N8N_USE_DEPRECATED_REQUEST_LIB be removed ?

RicardoE105 · November 2, 2021, 6:15pm

Because we do not support the env variables that an underline library uses, we only offer support for the env variable in our docs.

I guess there is an equivalent in Axios to do so using the env variables. However, why are you using env variables in the first place and not the poxy parameter in the HTTP node? What Am I missing?

I am not sure, but I will say that until the migration to Axios is stable enough, and we have fixed all the issues that arose.

RicardoE105 · November 2, 2021, 6:16pm

It will help if you share the issues you are having so that we can fix them.

a1ekseev · November 2, 2021, 6:32pm

All requests must go through a proxy, because there is no internet on the server.

It is expensive to configure in each HTTP node, besides, how can I configure proxy for Telegram Trigger, Trello Triger?

RedPacketSec · November 2, 2021, 8:35pm

these have been logged in other tickets.

RedPacketSec · November 2, 2021, 8:35pm

having a universal proxy toggle would be handy

RicardoE105 · November 2, 2021, 8:42pm

Ahh, I see how setting a global proxy is useful for your use-case. According to the docs, with Axios you should be able to do the same setting the env variable https_proxy and https_proxy

a1ekseev · November 6, 2021, 4:52pm

I set http_proxy and https_proxy in the .env file, the error remains the same.
What am I doing wrong?

RicardoE105 · November 7, 2021, 12:38pm

I assigned this to one of my coworkers that have a better insight.

krynble · November 8, 2021, 1:19pm

Hello everyone.

I just tested using the environment variables mentioned http_proxy and https_proxy and it works fine.

I even hacked Axios’ source code to output a message whenever it was checking for proxy operation and it worked. Axios follows these environment variables exactly like the request library used to do.

So, my setup:

squid running locally on port 3128
n8n workflow using http request node, making a basic GET request to any web page
no proxy set manually for http request node
I am “watching” for squid’s access log file changes, and I can see the requests being proxied via squid correctly.

I attached some screenshots of the setup:

Screenshot from 2021-11-08 14-17-32

This is Squid’s access log:

Which leads me to believe the problem is not with the proxy itself.

Could you please try running this locally on a setup that does not require proxy (could be your local machine) so we can try to pinpoint what’s the problem? Maybe it’s something else related to the request, like Content-type, body or something else.

a1ekseev · January 7, 2022, 4:05pm

Sorry for the long reply.

Looks like a problem within the docker container, but rather an issue somewhere on the network.
The problem is clearly in the library, but in the implementation of the proxy application, because when I use N8N_USE_DEPRECATED_REQUEST_LIB=true, everything starts working, installing Telegram Webhook, Trello Webhook, but with the new Axios library - does not work.

There is a docker-compose installation in the archive, just run it and see the errors in the n8n container logs.

a1ekseev · January 12, 2022, 8:22pm

Can you tell me if you’ve been able to check?

krynble · January 17, 2022, 12:57pm

Hey @a1ekseev

There is something strange and I believe it could be related to the proxy configuration you’re using.

I tried your example and called a service that detects whenever an HTTP request is received, and I did not get any call.

This basically means that the request did not get past the proxy. Could be something related to user-agent perhaps? I’m not exactly sure.

I can tell that when using squid I could get the requests to work fine, therefore I can only assume it’s something with the 3proxy.

Unfortunately I cannot help further as I have no knowledge about this proxy tool nor how to diagnose this. n8n is proxying the requests via squid correctly therefore proxying seems to be fine.

a1ekseev · January 17, 2022, 6:41pm

Have you tested squid + n8n in docker?

a1ekseev · January 19, 2022, 3:40pm

Could you share your docker configuration? Where you have Squid + n8n running.

krynble · February 8, 2022, 4:20pm

Hey sorry for the delay @a1ekseev I lost your notification.

I did set up squid using a local installation on my own PC so I don’t have a docker configuration to share with you, sorry.

I also used the default settings just to test if proxying was working and it was ok, so nothing fancy on my setup.

a1ekseev · April 9, 2022, 6:34pm

Sorry for the long reply.
Finally got around to checking with Squid.
I took the public Squid container from Datadog and checked, overall the problem is the same.

My configuration:

docker-compose.yml

version: "3"

networks:
 test_network:
  name: test_network
  driver: bridge
  ipam:
   driver: default
   config:
    - subnet: 172.18.0.0/24

services:
  postgresql:
    restart: unless-stopped
    image: postgres:12-alpine
    container_name: postgresql
    volumes:
      - ./init-db.sql:/docker-entrypoint-initdb.d/init.sql
      - ./postgresql/data:/var/lib/postgresql/data
    networks:
      - test_network
    ports:
      - 127.0.0.1:5432:5432
    env_file:
      - postgres.env

  n8n:
    image: n8nio/n8n
    container_name: n8n
    restart: unless-stopped
    volumes:
      - ./n8n:/root/.n8n
    networks:
      - test_network
    ports:
      - 127.0.0.1:5678:5678
    env_file:
      - n8n.env

  squid:
    image: datadog/squid:latest
    container_name: squid
    restart: unless-stopped
    ports:
      - 127.0.0.1:3128:3128
    volumes:
      - ./squid/logs:/var/log/squid
    networks:
      - test_network

n8n.env

HTTP_PROXY=http://squid:3128
HTTPS_PROXY=http://squid:3128

GENERIC_TIMEZONE=Europe/Moscow

DB_TYPE=postgresdb
DB_POSTGRESDB_HOST=postgresql
DB_POSTGRESDB_PORT=5432
DB_POSTGRESDB_DATABASE=n8n
DB_POSTGRESDB_USER=postgres
DB_POSTGRESDB_PASSWORD=postgres

N8N_BASIC_AUTH_ACTIVE=true
N8N_BASIC_AUTH_USER=test
N8N_BASIC_AUTH_PASSWORD=test

N8N_HOST=localhost
N8N_PROTOCOL="https"

postgres.env

POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres

ini-db.sql

CREATE DATABASE "n8n";

I made a curl call from the host machine and the request works.
I also installed curl inside the container and the request also works.

1649528451.564      0 172.18.0.4 TAG_NONE/501 4113 POST https://telemetry.n8n.io/v1/batch - HIER_NONE/- text/html
1649528452.044      0 172.18.0.4 TAG_NONE/501 4113 POST https://telemetry.n8n.io/v1/batch - HIER_NONE/- text/html
1649528452.888      0 172.18.0.4 TAG_NONE/501 4113 POST https://telemetry.n8n.io/v1/batch - HIER_NONE/- text/html
1649528471.320      0 172.18.0.4 TAG_NONE/501 4113 POST https://telemetry.n8n.io/v1/batch - HIER_NONE/- text/html
1649528471.549      0 172.18.0.4 TAG_NONE/501 4113 POST https://telemetry.n8n.io/v1/batch - HIER_NONE/- text/html
1649528471.969      0 172.18.0.4 TAG_NONE/501 4113 POST https://telemetry.n8n.io/v1/batch - HIER_NONE/- text/html
1649528472.888      0 172.18.0.4 TAG_NONE/501 4113 POST https://telemetry.n8n.io/v1/batch - HIER_NONE/- text/html
1649528539.323    354 172.18.0.1 TCP_MISS/200 389 GET http://httpbin.org/ip - HIER_DIRECT/34.231.5.222 application/json
1649528677.743    425 172.18.0.4 TCP_MISS/200 389 GET http://httpbin.org/ip - HIER_DIRECT/3.231.0.150 application/json

The network stack is different when you run in a container or on a host machine.