N8N - Self-hosted performance check

foxq · July 25, 2024, 3:13pm

Hi guys,

We are experiencing significant issues with our webhook processing setup in n8n. Despite configuring multiple webhook processors, we’re encountering bottlenecks and performance issues.

Our goal is to pull around 300,000 records from a HubSpot workflow directly into an n8n webhook. However, we’re facing frequent timeouts, with the subdomains becoming unreachable. Although we can access the server via IP, the webhook URLs remain inaccessible.

Thanks!

Spec

version: '3.8'

services:
  rabbitmq:
    image: rabbitmq:3-management
    environment:
      - RABBITMQ_DEFAULT_USER=username
      - RABBITMQ_DEFAULT_PASS=*****
    ports:
      - "5672:5672"
      - "15672:15672"
    healthcheck:
      test: ["CMD", "rabbitmq-diagnostics", "status"]
      interval: 30s
      retries: 3
      start_period: 30s
      timeout: 10s
    networks:
      - n8n_network

  traefik:
    restart: always
    image: traefik:v2.5
    command:
      - "--api.dashboard=true"
      - "--api.insecure=true"  
      - "--providers.docker=true" 
      - "--entrypoints.web.address=:8081" 
      - "--entrypoints.dashboard.address=:8082"
    ports:
      - "8081:8081"
      - "8082:8082"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
      - n8n_network

  redis:
    image: redis:latest
    command: ["redis-server", "--appendonly", "yes"]
    deploy:
      replicas: 1
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 30s
      retries: 3
      start_period: 30s
      timeout: 10s
    networks:
      - n8n_network

  n8n_editor:
    image: n8nio/n8n:1.21.1
    depends_on:
      redis:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy
    environment:
      - N8N_ENCRYPTION_KEY=C13fnPiJbk9v1atS
      - EXECUTIONS_MODE=queue
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=<hidden>
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=<hidden>
      - DB_POSTGRESDB_USER=<hidden>
      - DB_POSTGRESDB_PASSWORD=<hidden>
      - WEBHOOK_URL=https://n8n-webhook.domain.com
      - N8N_HOST=n8n-editor.domain.com
      - N8N_PORT=5680
      - N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true
    ports:
      - "5680:5680"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.n8n_editor.rule=Host(`n8n-editor.domain.com`)"
      - "traefik.http.services.n8n_editor.loadbalancer.server.port=5680"
      - "traefik.http.routers.n8n_editor.entrypoints=web"
    networks:
      - n8n_network

  n8n_webhook:
    image: n8nio/n8n:1.21.1
    restart: always
    depends_on:
      - redis
      - rabbitmq
    command: webhook
    environment:
      - N8N_ENCRYPTION_KEY=C13fnPiJbk9v1atS
      - EXECUTIONS_MODE=queue
      - DB_TYPE=postgresdb
      - N8N_HOST=n8n-editor.domain.com
      - DB_POSTGRESDB_HOST=<hidden>
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=<hidden>
      - DB_POSTGRESDB_USER=<hidden>
      - DB_POSTGRESDB_PASSWORD=<hidden>
      - WEBHOOK_URL=https://n8n-webhook.domain.com
      - N8N_PORT=5681
      - QUEUE_MODE=redis
      - QUEUE_BULL_REDIS_HOST=redis
      - QUEUE_BULL_REDIS_PORT=6379
      - QUEUE_BULL_REDIS_DB=1
      - RABBITMQ_DEFAULT_USER=username
      - RABBITMQ_DEFAULT_PASS=*******
      - RABBITMQ_HOST=rabbitmq
      - N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.n8n_webhook.rule=Host(`n8n-webhook.domain.com`)"
      - "traefik.http.services.n8n_webhook.loadbalancer.server.port=5681"
      - "traefik.http.routers.n8n_webhook.entrypoints=web"
      - "traefik.http.middlewares.rateLimit.rateLimit.average=1000"
      - "traefik.http.middlewares.rateLimit.rateLimit.burst=2000"
      - "traefik.http.routers.n8n_webhook.middlewares=rateLimit@docker"
    deploy:
      replicas: 3
    networks:
      - n8n_network

  n8n_worker:
    image: n8nio/n8n:1.21.1
    depends_on:
      - redis
      - rabbitmq
    command: worker --concurrency=3
    environment:
      - N8N_ENCRYPTION_KEY=C13fnPiJbk9v1atS
      - EXECUTIONS_MODE=queue
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=<hidden>
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=<hidden>
      - DB_POSTGRESDB_USER=<hidden>
      - DB_POSTGRESDB_PASSWORD=<hidden>
      - QUEUE_MODE=redis
      - QUEUE_BULL_REDIS_HOST=redis
      - QUEUE_BULL_REDIS_PORT=6379
      - QUEUE_BULL_REDIS_DB=1
      - RABBITMQ_DEFAULT_USER=username
      - RABBITMQ_DEFAULT_PASS=*******
      - RABBITMQ_HOST=rabbitmq
    deploy:
      replicas: 4
    networks:
      - n8n_network

networks:
  n8n_network:
    external: true

n8n · July 25, 2024, 3:13pm

It looks like your topic is missing some important information. Could you provide the following if applicable.

n8n version:
Database (default: SQLite):
n8n EXECUTIONS_PROCESS setting (default: own, main):
Running n8n via (Docker, npm, n8n cloud, desktop app):
Operating system:

barn4k · July 25, 2024, 9:33pm

hello @foxq

Are you talking about 300k requests per second? Or what is the timeframe?
Plus the performance will totally depend on your workflow.

BramKn · July 26, 2024, 9:07am

I have had the same issue with hubspot.
For this and other webhooks you use that don’t need an actual reply I always use RabbitMQ to offload the incoming data directly. This way the server does not get overloaded and you can process things after in a controlled way.

After setting this up we never had an issue anymore.

Also, welcome to the community!

foxq · July 26, 2024, 9:29am

Hi @BramKn and @barn4k,

Thank you very much for your prompt feedback!

@barn4k, I can’t specify the exact timeframe for the entire amount of data being processed (they seem to keep coming one by one, but then the subdomain gets a 520 error)

To give you more context:

After triggering the HubSpot workflow (which involves selecting 300k records and making a POST request to n8n), my n8n instance handles the first 5K records fine (n8n <> RabbitMQ). However, suddenly, the traffic from the subdomain (both editor and webhook) results in a 520 error. I can still access the n8n editor via direct IP, and I can see that the requests continue to come in.

Considering @BramKn’s suggestion, I suspect I might have misconfigured something because that’s precisely the approach I followed.

The workflow structure is very basic (for testing purposes for now):
1º Webhook Node:
- Method: POST
- Response: Immediate
2º Send Message to RabbitMQ Node:
- Mode: Queue
- Arguments: x-queue-type: quorum

Allow me to share with you my vhost (apache webserver):
<VirtualHost *:80>
ServerName n8n-webhook.domain.com
ProxyRequests Off
ProxyPreserveHost On
ProxyVia Full
<Proxy *>
Require all granted

ProxyPass http://localhost:8081/
ProxyPassReverse http://localhost:8081/

Note: 8081 (traefik port)

Do you guys have any suggestions?

Thank you once again!

BramKn · July 26, 2024, 9:36am

Hi @foxq

Not sure about the config on RabbitMQ node, I always use the basic config with just the “send input data” enabled.
But that should not be the issue.

More investigation on the proxy side needs to be done I guess. Is it properly loadbalancing for example?

To answer the question/remark on n8n front-end, that makes sense as it is a seperate container so that this stays up and running is to be expected.

To be clear I have had this setup without load balancer with hubspot webhooks and rabbitMQ and never had an issue with it. But I do think you can slow it down a bit on the hubspot side as well, my client did that so no idea what exactly can be done there.

foxq · July 26, 2024, 9:51am

Hey @BramKn,

The RabbitMQ node doesn’t seem to be the issue. I’ve tried many different configurations. Let me share more details about the setup, including Traefik and the RabbitMQ dashboard.

I’ve checked the logs for each container, and everything appears to be functioning correctly. It seems that port 80 is receiving too much traffic, causing it to become unreachable.

Any thoughts?

Cheers!

barn4k · July 26, 2024, 10:50am

I see that traefik is listening for ports 8080 and 8081, but you have configured the hubspot to send data via webhook on port 443 (HTTPS). I’m not familiar with traefik, but if you have 5k of requests going into n8n, traefik should see them

BramKn · July 26, 2024, 12:57pm

Not sure. Like @barn4k I am not familiar with Traefik.
The requests don’t seem to continue to n8n.
When the webhooks come in, have you checked the resources used by the n8n webhook workers? are they getting overloaded/crashing?

foxq · July 26, 2024, 1:13pm

That’s probably where the problem is @barn4k, some port conflict. I’ve some ideas in mind, but I need some time to test them out

I will share my feedback with you, very soon.

Really appreciate your help @barn4k and @BramKn

Cheers!

system · October 24, 2024, 1:14pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.