[Architecture] Webhook "Respond when last node finishes" hangs indefinitely on Serverless (Cloud Run) due to Redis TCP Drop

Questions
,
1

Describe the problem/error/question

We are running n8n in Queue Mode on Google Cloud Run (Serverless). We discovered a massive hidden cost trap regarding Webhook nodes configured to “Respond when last node finishes”.
If our n8n instance receives a webhook after being idle for >10 minutes, the worker executes the job perfectly and instantly (in ~3 seconds). However, the Main instance hangs indefinitely, waiting for the worker’s response. Because Main is stuck waiting, the HTTP request stays open, and Google Cloud Run bills us for “Active CPU Time” for up to 15+ minutes until the internal Cloud Run timeout kills it.

What is the error message (if any)?

There is no explicit error in n8n. The webhook simply never returns a response to the client. Cloud Run metrics show the request latency spiking to ~1000 seconds (16.7 minutes).

Information on your n8n setup

• n8n version: 2.30.5
• Database: Postgres (Cloud SQL)
• Queue: Redis (Upstash Serverless / Memorystore)
• Running n8n via: Google Cloud Run (Queue Mode, Main & Worker separated)
• Operating system: Node.js 24 (Custom Docker Image)

Root Cause Analysis:

The issue is a “Silent Drop” of the Redis Pub/Sub TCP connection by the Google Cloud VPC / NAT. Google silently drops idle TCP connections after exactly 10 minutes without sending a TCP RST packet.
When Main wakes up/receives a webhook at minute 15, it pushes the job to Redis (which works, because write commands create a new connection or auto-reconnect), but Main waits for the worker’s completion signal on the old, dead Pub/Sub ( SUBSCRIBE ) channel. Main has no idea the channel is dead, so it waits forever.

Our Current Workarounds:

1. High-Frequency Ping: We increased our synthetic health checks to run every 9 minutes so the Redis connection is never idle for >10 minutes.
2. Dummy Keep-Alive: Triggering a dummy workflow (Webhook → NoOp) every 5 minutes via Cloud Scheduler just to keep the SUBSCRIBE channel warm.

My Question to the Community & n8n Team:

Is there a way to pass ioredis keep-alive options (like enableKeepAlive: true ) via n8n environment variables to prevent silent NAT drops? For Postgres, we fixed this using Cloud SQL TCP keepalive flags, but we can’t do this for Redis since n8n hides the Redis connection settings. How are other enterprise users solving this on serverless infrastructures?

2

Hi @yoel77
n8n does expose ioredis TCP keep-alive on the queue Redis client through env vars, added in 2.7.0 (PR #23902), so your 2.30.5 already has it. This is the equivalent of enableKeepAlive for the Bull and Pub/Sub (SUBSCRIBE) connections. Set it on both main and workers:

QUEUE_BULL_REDIS_KEEP_ALIVE=true
QUEUE_BULL_REDIS_KEEP_ALIVE_DELAY=5000
QUEUE_BULL_REDIS_KEEP_ALIVE_INTERVAL=5000

Only QUEUE_BULL_REDIS_KEEP_ALIVE=true is required; the delay and interval already default to 5000ms, so probes go out every 5s and the SUBSCRIBE connection never sits idle long enough for Google’s NAT to silently drop it at 10 minutes. With that in place you can retire the 9-minute synthetic ping and the dummy keep-alive workflow.

3

Hi mate,

Thanks for sharing your issue with us , I have not run my projects personally in google cloud yet but base of what you have described it sounds the problem less about n8n and more about long-lived sub connections on serverless infrastructure.

4

Reply an Anshul_Namdev & ShahbazFalahian:

Hi @Anshul_Namdev and @ShahbazFalahian,

Thank you both so much for taking the time to read into our architecture issue!

@Anshul_Namdev - This is an absolute game-changer! I just searched through the official n8n documentation (and even asked the n8n AI bot), and interestingly, these QUEUE_BULL_REDIS_KEEP_ALIVE variables are nowhere to be found. The docs currently only list keep-alive options for PostgreSQL, which led us to believe it wasn’t natively supported for Redis.

Since you mentioned it was added in PR #23902, it looks like an undocumented lifesaver for everyone running n8n on serverless infrastructures like Google Cloud Run or AWS ECS!

We will immediately add QUEUE_BULL_REDIS_KEEP_ALIVE=true to both our Main and Worker containers and run some long-term idle tests. If this forces ioredis to keep the connection warm and prevents the GCP NAT from silently dropping the SUBSCRIBE channel, this elegantly solves the entire architecture trap.

Thank you again for pointing out this undocumented gem! I highly recommend the n8n team to add this to the official Queue Mode docs, as serverless deployments are becoming the standard.

5

Spot on about the root cause: the subscriber channel dying silently due to NAT timeouts. If the platform won’t let you pass keep-alive flags directly, one reliable trick is to wrap the client connection: in your worker startup, before subscribing, set socket keep-alive via OS-level settings or through a proxy that does. For serverless containers, adding --sysctl net.ipv4.tcp_keepalive_time=600 (or similar) to the container runtime often does the job. I have a ready-to-go proof-of-concept that demonstrates this pattern inside a containerized environment like yours—no charges, just want to see if it fits. If you’d like, shoot me a DM and I’ll spin it up for you.

6

Good that the keepalive vars solved it, and yoel77 is right that they are undocumented, which is why almost nobody running on serverless finds them. One caveat on the sysctl route above: net.ipv4.tcp_keepalive_time only does anything if the socket actually has SO_KEEPALIVE enabled, which is precisely what QUEUE_BULL_REDIS_KEEP_ALIVE=true turns on. So the env var is the load-bearing part and the sysctl just tunes the interval. Worth confirming the probes really do survive your NAT (ss -o on a worker will show the keepalive timer on the Redis socket), because some managed NATs only reset their idle timer on data, not on bare keepalive segments.

The reason I would not call this fully closed: keepalive fixes exactly one of the ways this hangs, the idle drop, and the architecture has others that produce the identical symptom and the identical bill. “Respond when last node finishes” holds the client’s HTTP connection open while Main waits on a cross-instance Pub/Sub completion signal. That signal is also lost if a worker OOMs or restarts mid-job, if Main scales to zero and loses the subscriber, or, specific to your setup, if you are using Upstash serverless for the Bull and Pub/Sub backend: its serverless tier is not built for long-lived SUBSCRIBE the way a persistent Memorystore instance is, and a dropped pub/sub message there is silent. In every one of those, Main hangs again and Cloud Run bills wall-clock CPU up to the 15 minute timeout. Same trap, different trigger, and keepalive cannot see any of them.

So the durable fix on scale-to-zero infrastructure is to stop making the HTTP response depend on that synchronous cross-instance signal for anything that is not guaranteed sub-second. Respond immediately from the webhook (200 plus a job id), let the worker run async, and deliver the result via a callback to the caller or a status endpoint they poll. That retires the entire class (idle drop, worker crash, pub/sub loss) instead of patching one cause, and it kills the cost trap outright, because Main is not holding an open request while it waits. Where you genuinely need the answer in the same request, keep “Respond when last node finishes” for fast, bounded workflows only.

Last thing, because this one is invisible by design: put a Cloud Run alert on request latency above roughly 60 seconds. Whatever the next cause turns out to be, a hang shows up first as a latency spike, not as an n8n error, so an alert on that metric turns “we found it on the bill” into “we got paged in minutes.” On a queue-mode instance that scales to zero, that alert is what catches the failure mode keepalive does not.

7

Thanks for letting us know about this, We have created CV-6 as the internal dev ticket to look into it.

8

Hi @Adam13y,

Thank you for the incredibly detailed and insightful response! You hit the nail on the head regarding the architectural risks of “Respond to Webhook: When last node finishes” in a serverless scale-to-zero environmen.

Just to give some context on our architecture: We are running a highly modular “Automation-as-a-Service” platform on Cloud Run (Scale-to-Zero) with a dedicated Redis instance to ensure strict bulkheading and failure isolation.

We actually follow your advice to the letter for our production workflows! All heavy customer-facing workflows are strictly set to “Respond Immediately”. The Main container pushes the job to the queue, fires back a 200 OK instantly, and Cloud Run immediately scales the CPU to zero, giving
us a true zero-cost idle state while the workers handle the heavy lifting.

The only reason we use “Wait for last node to finish” is specifically for a dedicated /deep-healthz webhook. This is our synthetic monitoring probe. We deliberately want it to travel the entire chain synchronously (Main → Redis → Worker → Redis → Main). Furthermore, this specific workflow also pings our custom n8n MCP Server and the Gemini API. By doing this, we effectively test 90% of our core n8n functionality in one go. If a worker OOMs, or if Pub/Sub loses a message, we want this specific request to hang and fail, so our external monitoring correctly registers a cluster failure.

Regarding the Cloud Run timeout: You also solved another mystery for us! We previously had the Cloud Run timeout set to 3600s (1 hour). That perfectly explains why our Main container used to hang for exactly one hour during a silent drop, before finally giving up, restarting, and processing health checks cleanly again. We’ve now tightened this to 600s (10 minutes), which gives us a sharper cutoff but still leaves room if a rare complex workflow absolutely needs to wait for the last node.

That being said, your suggestion to put a GCP alert on request latency > 60s is absolutely brilliant. Since we use “Respond immediately” for almost everything, and our deep health check only takes ~7 seconds, any HTTP latency over 60s on the Main container is a guaranteed anomaly. Catching a silent hang at the 60s mark saves us from finding out via the billing dashboard. We are implementing that alert across our units.

For the keepalives, setting QUEUE_BULL_REDIS_KEEP_ALIVE=true and QUEUE_BULL_REDIS_KEEP_ALIVE_DELAY=10000 (since n8n 2.7.0) perfectly solved the idle drop on our GCP NAT.

Really appreciate the deep dive. The fact that the n8n core team opened CV-6 based on this discussion shows how incredibly valuable this community is! :rocket:

9

Thanks for validating the NAT timeout root cause! You are 100% spot on regarding the OS-level concept.

However, as @Anshul_Namdev pointed out above, since n8n 2.7.0, the core team actually exposed the underlying ioredis socket keep-alive settings directly via environment variables!

Setting QUEUE_BULL_REDIS_KEEP_ALIVE=true along with QUEUE_BULL_REDIS_KEEP_ALIVE_DELAY=10000 tells the Node.js socket to natively fire
empty ACKs. I just implemented it and it beautifully defeats the 10-minute Google Cloud NAT drop without requiring any --sysctl kernelhacks or proxy sidecars.

Really appreciate the offer for the PoC though, it’s great to see the community jumping in on complex serverless networking issues!:smiley:

10

We have taken a look at this issue and have confirmed that this does look likely to be a bug, It has been passed to one of our engineering teams to be resolved.