Backgrounds
The current Kafka Trigger node appears to limit consumption to a single partition at a time, processing messages sequentially. While this ensures message ordering within a single partition, it creates severe limitations:
-
Throughput Bottleneck: The inability to consume from multiple partitions concurrently prevents the system from scaling, significantly limiting throughput under heavy message loads.
-
Uncontrolled Parallelism: The existing general workflow
Paralleloption is unusable as it triggers excessive, uncontrolled message consumption, leading to system overload.
We need a dedicated mechanism to leverage the parallel nature of Kafka partitions.
The Proposed Solution is…
Add a Concurrency option to the Kafka Trigger node settings.
This option will allow users to explicitly define the maximum number of partitions the consumer should process simultaneously, providing controlled resource utilization and scaling.
Reference: https://kafka.js.org/docs/consuming#a-name-concurrent-processing-a-partition-aware-concurrency