Safely Stop n8n Workers Without Losing In Progress Jobs

Questions
1

I’m running n8n in queue mode with multiple workers, and I’m trying to handle deployments and worker restarts safely.
The problem is that a worker can be processing a job when it receives a shutdown signal. If it gets killed before the workflow finishes, the job may remain marked as processing even though no worker is actually handling it.
I’m considering a heartbeat/lease approach:
UPDATE jobs
SET lease_until = NOW() + INTERVAL ‘2 minutes’
WHERE id = $1
AND worker_id = $2
AND status = ‘processing’;
Then another worker could reclaim the job once the lease expires.
For production, would you combine graceful shutdown with leases, or rely entirely on n8n’s queue behavior?
How would you make sure a job isn’t processed twice during the shutdown/recovery window?

Describe the problem/error/question

What is the error message (if any)?

Please share your workflow

(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)

Share the output returned by the last node

Information on your n8n setup

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
2

Hey @Greg_John, while you wait for a response, here are some things that might help:

Suggested resources

Automatically matched to your question.

Docs:

Forum:

@Niffzy, @Emmas - you’ve helped with similar issues before, can you take a look?

Automatically suggested by n8n’s community bot. It’s a pilot - please share feedback here.

3

Hi @Greg_John I would use graceful shutdown + leases, rather than relying on either one alone.

When a worker receives a shutdown signal, it should stop accepting new work and give active executions time to finish. But you still need a lease because the worker could crash before that happens.

For example:

UPDATE jobs
SET lease_until = NOW() + INTERVAL '2 minutes'
WHERE id = $1
AND worker_id = $2
AND status = 'processing'
RETURNING id;

The worker renews the lease while processing. If it disappears, another worker can reclaim the job after lease_until expires.

I’d also make the actual job processing idempotent. A lease prevents most duplicate processing, but there’s always a small race where the original worker finishes after its lease expires.

The flow becomes:

Claim Job
↓
Start Lease
↓
Process + Heartbeat
↓
Complete Job
↓
Graceful Shutdown

And recovery:

Find Expired Leases
↓
Reclaim Job
↓
Retry / Continue
4

Hey Greg!
One thing before building a lease table: n8n queue mode already does this for its own executions. Workers take a lock on the Bull job in Redis and keep renewing it. Two env vars control it:

QUEUE_WORKER_LOCK_DURATION=60000   # lease length in ms
QUEUE_WORKER_LOCK_RENEW_TIME=10000 # how often the worker renews it

So if your jobs table is just mirroring n8n executions, I’d skip it. You’d be reimplementing what Bull does underneath.

For the shutdown side, send SIGTERM (not SIGKILL). The worker stops pulling new jobs and waits up to N8N_GRACEFUL_SHUTDOWN_TIMEOUT seconds for running executions. Default is 30. Set it longer than your slowest workflow.

The part that bites people: your orchestrator has its own timeout, and it wins. Docker Compose’s stop_grace_period defaults to 10s, so a worker with a 30s graceful window still gets killed at 10s. Kubernetes terminationGracePeriodSeconds defaults to 30. Make the orchestrator number bigger than the n8n one, with some slack.

n8n-worker:
  environment:
    - N8N_GRACEFUL_SHUTDOWN_TIMEOUT=300
  stop_grace_period: 330s

On “processed twice”: since 2.0, n8n no longer auto-retries stalled jobs (QUEUE_WORKER_MAX_STALLED_COUNT was removed, see the 2.0 breaking changes). So on a hard kill the risk is a lost execution, not a duplicate. Your recovery has to be something you trigger yourself: a re-send from the source, or a scheduled workflow that re-queues whatever didn’t finish.

That’s where idempotency matters, and I agree with @Niffzy there. Put an idempotency key on the side effect itself (unique constraint + INSERT ... ON CONFLICT DO NOTHING, or a “processed” row keyed by the source record ID). Then a retry is harmless whether it came from you or anywhere else.

If jobs is actually your own business queue (a workflow polls it for work), then yes, leases make sense. Claim with FOR UPDATE SKIP LOCKED so two workers can’t grab the same row, and reclaim on lease_until < NOW().

5

Hi @Greg_John, one caveat with the “processed” row approach: if an external API succeeds and the worker dies before saving that row, recovery can repeat the action. Saving the row first creates the opposite problem: recovery skips an action that might never have happened.

For external calls, use the destination API’s supported idempotency mechanism with a key identifying the original event and action. Reuse that key on recovery, not a new n8n execution ID.

For PostgreSQL-only changes, commit the deduplication record and business update together. If the external API doesn’t support idempotency, check whether the action succeeded before retrying an uncertain result.

6

What I’d do in production

I’d lean on the combination of graceful shutdown + the built-in locks, and treat any custom lease as optional extra insurance rather than the main plan.

1. Graceful shutdown first
Set N8N_GRACEFUL_SHUTDOWN_TIMEOUT high enough for your longest realistic jobs (e.g. 5–15 minutes if you have slow ones). In Kubernetes match terminationGracePeriodSeconds to that value (or a bit higher). On SIGTERM the worker should stop accepting new work and try to finish whatever it’s already running. Most of the time that alone prevents the “stuck processing” situation.

2. Tune the lock settings
Make QUEUE_WORKER_LOCK_DURATION comfortably longer than your longest expected job + some buffer. If a job regularly takes 3–4 minutes, don’t leave the lock at 60s. Raise the renew interval a bit too if needed. This keeps legitimate long jobs from looking stalled while still recovering reasonably fast when a worker really dies.

3. Don’t fight the built-in system with a full custom DB lease unless you have a strong reason. Adding your own lease_until / worker_id logic on top of n8n’s Redis locks can create race conditions and double-ownership situations. If you already have your own jobs table outside of n8n’s execution system, a light lease there can be useful as a secondary safety net, but for pure n8n workflows I’d stick with what Bull already does.