Buffered Logging for Performance Improvement

Feature Requests
,
1

The problem is:

Direct database writes for execution logs in n8n create a significant performance bottleneck, limiting workflow throughput despite the number of workers and webhook instances.
Users must choose between having logs (with limited performance) or disabling logs entirely (losing crucial operational visibility).

The solution is:

Implement optional buffered logging to batch database writes, similar to how Nginx and other high-performance systems handle logs. This allows maintaining comprehensive logging while significantly improving throughput by reducing database I/O operations.

The idea is:

Implement an optional buffered logging system for execution logs (as a funtionality that can be enabled or disabled), to improve throughput while maintaining logging capabilities. The system would:

  1. Buffer execution logs before writing to database:

    • In-memory buffer with configurable size
    • Configurable flush interval
    • Optional Redis-based buffer for distributed setups

  2. Configuration via environment variables:

   DB_LOGGING_BUFFER_ENABLED=true
   DB_LOGGING_BUFFER_SIZE=512k      # Buffer size before flush
   DB_LOGGING_FLUSH_INTERVAL=1m     # Time-based flush interval
  1. Workflow-level configuration override:
    image
    image1952×1270 143 KB

My use case:

Running n8n in production on Kubernetes with high-throughput workflows where:

  1. Current throughput is limited due to database logging overhead

  2. Need to maintain execution logs for:

    • Compliance requirements
    • Debugging capabilities
    • Performance monitoring
    • Security auditing

  3. Currently forced to choose between performance (disabling logs) or having logs (with limited throughput)

I think it would be beneficial to add this because:

  1. Solves Critical Performance Issues:

    • Reduces database write operations through batching
    • Improves workflow execution throughput
    • Reduces database connection pool pressure
    • Maintains logging capabilities without performance penalty

  2. Provides Flexibility:

    • Configurable per workflow
    • Different settings for webhook/worker/main processes
    • Optional Redis support for distributed setups

  3. Overall Features:

  • Maintains compliance capabilities (logging, monitoring, etc.)
  • Supports high-throughput scenarios
  • No compromise between performance and logging
  • Follows industry standard practices

Any resources to support this?

  1. Current Performance Issue:

  2. Similar Implementations:

  3. Architecture Support:

    • Works with n8n’s existing queue mode (Configuring queue mode | n8n Docs)
    • Compatible with both standalone and distributed setups
    • Optional Redis integration for distributed scenarios

The implementation could start with in-memory buffering and later add Redis support for distributed setups, providing a scalable solution that grows with users’ needs.

2

I think that is a good idea!