Code Node Execution Hangs When Offloaded to Workers (GKE/Queue Mode)

I’m seeking assistance with a persistent hang issue when attempting to run workflows that contain a Code Node using the Manual Execute button.

When the environment variable OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS is set to true on the Main Node, all manual executions of workflows containing a Code Node (specifically Python) immediately hang in the UI.

Successful Configuration Checks

I’ve confirmed the following critical items are correctly configured and working:

  1. Network Connectivity (Task Broker): Workers and Runners can successfully reach the Primary Node’s Task Broker URI (Confirmed via nc -vz).

  2. Queue Connectivity: Worker nodes successfully connect to and communicate via the Redis queue

  3. Runner Registration: The Runner Pods successfully connect to the Main Node and register their services. The Main Node logs show: Registered runner "launcher-python"

  4. Hanging Log State: The Runner logs show the process is stalled waiting for the task offer:

    INFO [launcher:py] Waiting for launcher's task offer to be accepted...
    
    
  5. Environment Variables: All runner-related variables are consistent across Main, Workers, and Runners, including:

    • N8N_RUNNERS_MODE: external

    • N8N_RUNNERS_AUTH_TOKEN (Confirmed identical via clean secret reference).

    • N8N_RUNNERS_TASK_BROKER_URI: http://n8n-primary-service:5679

Has anyone encountered this specific failure with OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS in a K8s queue setup, and is there a known variable that needs to be explicitly set on the Worker or Main node to resolve this possible internal task routing issue?

Information on n8n setup

  • n8n version: 1.115.3
  • Database: postgres
  • n8n setup: Main, Worker, Runner
  • Running n8n via: GKE (Google’s Kubernetes)
  • Operating system: Standard N8N containers

Has anyone seen or does anyone know of any full examples that shows a containerized setup with a main ui, workers, runners and one that also uses queues. I’ve seen a ton of examples, but no full setup that does it all. I’m sure its just one environment variable we have wrong or something silly. The online docs are very high level but don’t show main → workers w/ queue → runners.