Execution data is written to the database even when it is disabled

Questions
1

Describe the problem/error/question

I run n8n self-hosted in queue-mode with a docker-compose-file. When queue-mode is enabled the execution data is written fully into the database. The settings to disable this in the server environment or the workflow itself seems to have no impact. This is a problem for me because in some workflows I need to process multiple MB of data from an SQL database. Did I miss something or is there an other way to avoid the data load in public.execution_data?

Please share your workflow

I have created a minimal docker-compose which can reproduce the problem:

docker-compose-n8n.yml 
volumes:
  postgres_data:
  redis_data:

x-healthcheck: &healthcheck_base
  interval: 5s
  timeout: 5s
  retries: 10
  start_period: 15s

x-n8n-environment-base: &n8n_environment_base
  QUEUE_HEALTH_CHECK_ACTIVE: "true"
  QUEUE_HEALTH_CHECK_PORT: "5678"
  N8N_ENCRYPTION_KEY: "${N8N_ENCRYPTION_KEY}"
  N8N_PROTOCOL: "${PROTOCOL}"
  N8N_HOST: "${HOST}"
  N8N_PORT: "${PORT}"
  EXECUTIONS_MODE: "queue"
  N8N_RUNNERS_MODE: "external"
  N8N_RUNNERS_BROKER_LISTEN_ADDRESS: "0.0.0.0"
  N8N_RUNNERS_BROKER_PORT: "5679"
  OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS: "true"
  QUEUE_BULL_REDIS_HOST: "redis"
  QUEUE_BULL_REDIS_PORT: "6379"
  EXECUTIONS_DATA_SAVE_ON_SUCCESS: "none"
  EXECUTIONS_DATA_SAVE_ON_ERROR: "none"
  EXECUTIONS_DATA_SAVE_ON_PROGRESS: "false"
  EXECUTIONS_DATA_SAVE_MANUAL_EXECUTIONS: "false"
  DB_TYPE: "postgresdb"
  DB_POSTGRESDB_HOST: "${N8N_DB_HOST}"
  DB_POSTGRESDB_PORT: "${N8N_DB_PORT}"
  DB_POSTGRESDB_DATABASE: "${N8N_DB_NAME}"
  DB_POSTGRESDB_USER: "${N8N_DB_USER}"
  DB_POSTGRESDB_PASSWORD: "${N8N_DB_PW}"
  N8N_CONCURRENCY_PRODUCTION_LIMIT: "10"

x-n8n-worker: &n8n_worker
  image: docker.n8n.io/n8nio/n8n:${N8N_VERSION}
  command: worker
  depends_on:
    n8n-main:
      condition: service_healthy
  environment:
    <<: [*n8n_environment_base]
    N8N_RUNNERS_AUTH_TOKEN: "${N8N_RUNNERS_AUTH_TOKEN}"
  volumes:
    - ./local-files/.n8n:/home/node/.n8n
    - ./local-files:/files
  healthcheck:
    <<: *healthcheck_base
    test: ["CMD-SHELL", "wget -q --spider http://localhost:5678/healthz || exit 1"]

x-n8n-taskrunner: &n8n_taskrunner
  image: n8nio/runners:${N8N_VERSION}
  volumes:
    - ./n8n-task-runners.json:/etc/n8n-task-runners.json:ro
  healthcheck:
    <<: *healthcheck_base
    test: ["CMD-SHELL", "pgrep -f 'node.*runner' || exit 1"]

services:
  redis:
    image: redis:7-alpine
    container_name: redis
    volumes:
      - redis_data:/data
    healthcheck:
      <<: *healthcheck_base
      test: ["CMD", "redis-cli", "ping"]
      interval: 1s
      timeout: 3s

  postgres:
    image: postgres:16-alpine
    container_name: postgres
    environment:
      POSTGRES_USER: "${N8N_DB_USER}"
      POSTGRES_PASSWORD: "${N8N_DB_PW}"
      POSTGRES_DB: "${N8N_DB_NAME}"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      <<: *healthcheck_base
      test: ["CMD-SHELL", "pg_isready -U ${N8N_DB_USER} -d ${N8N_DB_NAME}"]

  n8n-main:
    image: docker.n8n.io/n8nio/n8n:${N8N_VERSION}
    container_name: n8n-main
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      <<: *n8n_environment_base
    ports:
      - "5678:5678"
    volumes:
      - ./local-files/.n8n:/home/node/.n8n
      - ./local-files:/files
    healthcheck:
      <<: *healthcheck_base
      test: ["CMD-SHELL", "wget -q --spider http://localhost:5678/healthz || exit 1"]
      start_period: 30s

  n8n-worker-1:
    <<: *n8n_worker
    container_name: n8n-worker-1

  n8n-taskrunner-1:
    <<: *n8n_taskrunner
    container_name: n8n-taskrunner-1
    depends_on:
      n8n-worker-1:
        condition: service_healthy
    environment:
      N8N_RUNNERS_AUTH_TOKEN: "${N8N_RUNNERS_AUTH_TOKEN}"
      N8N_RUNNERS_TASK_BROKER_URI: "http://n8n-worker-1:5679"
      N8N_RUNNERS_LAUNCHER_HEALTH_CHECK_PORT: "5680"
.env (demo) 
N8N_VERSION=2.18.5
N8N_RUNNERS_AUTH_TOKEN=JuI1qCnVy0BSiTrYfXQ-SzI_lShC3xVHGP2u0pP4S1s
GENERIC_TIMEZONE=Europe/Berlin
SUBDOMAIN=localhost
PROTOCOL=http
HOST=localhost
PORT=5678
WEBHOOK_URL=http://localhost/
N8N_ENCRYPTION_KEY=aN8P_TIp7_nFdOs6BVXOPOF0ayk_t4iQa2JsrvZ31qs
N8N_DB_HOST=postgres
N8N_DB_PORT=5432
N8N_DB_NAME=n8n
N8N_DB_USER=n8n
N8N_DB_PW=B7nZQ6OxW67DXEObBwWSXnCJnkU1nZGpSdn-lBxjtvA
n8n-task-runners.json 
{
  "task-runners":
  [
    {
      "runner-type": "javascript",
      "workdir": "/home/runner",
      "command": "/usr/local/bin/node",
      "args":
      [
        "--disallow-code-generation-from-strings",
        "--disable-proto=delete",
        "/opt/runners/task-runner-javascript/dist/start.js"
      ],
      "health-check-server-port": "5681",
      "allowed-env":
      [
        "PATH",
        "GENERIC_TIMEZONE",
        "NODE_OPTIONS",
        "NODE_PATH",
        "N8N_RUNNERS_AUTO_SHUTDOWN_TIMEOUT",
        "N8N_RUNNERS_TASK_TIMEOUT",
        "N8N_RUNNERS_MAX_CONCURRENCY",
        "N8N_SENTRY_DSN",
        "N8N_VERSION",
        "ENVIRONMENT",
        "DEPLOYMENT_NAME",
        "HOME"
      ],
      "env-overrides":
      {
        "NODE_FUNCTION_ALLOW_BUILTIN": "*",
        "NODE_FUNCTION_ALLOW_EXTERNAL": "*",
        "N8N_RUNNERS_HEALTH_CHECK_SERVER_HOST": "0.0.0.0"
      }
    },
    {
      "runner-type": "python",
      "workdir": "/home/runner",
      "command": "/opt/runners/task-runner-python/.venv/bin/python",
      "args": ["-I", "-B", "-X", "disable_remote_debug", "-m", "src.main"],
      "health-check-server-port": "5682",
      "allowed-env":
      [
        "PATH",
        "N8N_RUNNERS_LAUNCHER_LOG_LEVEL",
        "N8N_RUNNERS_AUTO_SHUTDOWN_TIMEOUT",
        "N8N_RUNNERS_TASK_TIMEOUT",
        "N8N_RUNNERS_MAX_CONCURRENCY",
        "N8N_SENTRY_DSN",
        "N8N_VERSION",
        "ENVIRONMENT",
        "DEPLOYMENT_NAME"
      ],
      "env-overrides":
      {
        "N8N_RUNNERS_STDLIB_ALLOW": "*",
        "N8N_RUNNERS_EXTERNAL_ALLOW": "*"
      }
    }
  ]
}
workflow

Steps to reprodurce

  • Start server
  • Execute workflow
  • Connect to database, example: docker exec -it postgres psql -U n8n -d n8n
  • Check size of last execution, example: SELECT “executionId”, pg_column_size(“data”) FROM public.execution_data;

Information on your n8n setup

  • n8n version: 2.18.5
  • Database: Postgres
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
  • Operating system: Rocky Linux 9.7 (Blue Onyx)
4

Unfortunately the entries aren’t trimmed after the execution.

If the data would be only this much while the workflow is running, than this wouldn’t be optimal but manageable.

The data from the sql request is directly processed and trimmed down, but the request itself has multible MB of data.

Even if the request is turned down to one result per workflow it would be very much data and also the total data on the database is still the same amount.

I can’t see how this would work with an sql-request.

In the production-system this is allready implemented, but we can’t just delete all data very fast. It’s only with some workflows with heavy data load where we need to find a solution.

Is there any other way to avoid the execution_data of specific workflows (after they finished their execution)?

6

Hi @TimoK , Welcome !

This is an architectural constraint of queue mode, the worker writes the results to the database because workers and the main process are separate containers. The execution data has to go through Postgres during execution, regardless of your save settings. The EXECUTIONS_DATA_SAVE_ON_SUCCESS: “none” setting should clean up those rows after the execution completes, but since you’re confirming they persist, that cleanup isn’t happening.

Your case is worth filing as a github issue since you have a clean minimal reproduction (your docker-compose is excellent for that).

As a workaround until this is fixed, you could add a scheduled query that cleans up execution_data rows where the corresponding execution_entity status is success or error.

Let me know !

7

Thanks for the information and the advise.

I created a bug report there and will close this question to avoid confusion:

I have also created a temporary workaround:

cleanup execution_data.json (4.0 KB)

*edit: I will let this question close itself because I can’t find the option to close it myself.

**edit2: The workaround-workflow must be adjusted for the own needs. For testing the time-interval and the datasize are set up very small!