N8n Queue Mode: What happens to a job when a worker crashes or restarts?

Questions
1

We are running n8n in Queue Mode on Kubernetes and would like to clarify how n8n handles worker crashes and stalled executions.

Our main question is about job recovery when a worker crashes or is restarted.

Suppose a worker picks up an execution from the Redis queue and starts processing it, but the worker crashes before the execution is completed.

  1. Will another available worker detect the stalled job and take it over?

  2. If so, does the new worker continue the execution from where the previous worker stopped, or does it restart the execution from the beginning?

  3. Is this worker-failover/stalled-job recovery behavior officially supported in the current n8n Queue Mode architecture?

  4. The documentation mentions that queue_worker_max_stalled_count is deprecated. Does that mean stalled-job detection/recovery itself is still supported, with this particular configuration option simply no longer being used?

  5. What is the current recommended configuration/behavior for handling worker crashes in Queue Mode?

Observation

We have also observed an interesting behavior when multiple workflows are triggered concurrently.

Initially, some executions appear as Error in the n8n UI shortly after they start. However, after some time, those same executions continue processing and eventually finish with Success.

Because this happens during periods of higher concurrent workload, we suspect that a worker may be becoming overloaded, crashing, or restarting, and that another worker may subsequently be picking up a stalled job.

However, we are not sure whether this is actually what is happening.

Specifically, we would like to understand:

  • If a worker crashes while processing an execution, what status does n8n initially record for that execution?

  • Can an execution temporarily appear as Error/Failed and subsequently continue and become Success?

  • If another worker takes over a stalled job, how does n8n determine what state to resume from?

  • Does the second worker resume the execution from the last persisted state, or does it execute the workflow again from the beginning?

  • If the execution was already marked as crashed or error, how can it subsequently continue and eventually become successful?

  • Could what we are observing instead be caused by a UI/status update arriving before the actual execution state is finalized?

Environment

  • n8n version: n8nio/n8n:latest

  • Deployment: Kubernetes

  • Execution mode: Queue Mode

  • Database: PostgreSQL

  • Queue: Redis

  • Main pods: 1

  • Worker pods: 1 min , 10 max

  • Approximate concurrent executions: 10

We would appreciate clarification on the current n8n Queue Mode worker-failure/stalled-job recovery mechanism, especially how execution state is persisted and recovered when a worker crashes or restarts.

2

Based on the current architecture, when a job is reclaimed by the reaper and picked up by another worker, the execution restarts from the beginning.

n8n does not currently support “checkpointing” or “resuming” from the exact mid-workflow state of a crashed Node.js process. While there is a setting EXECUTIONS_DATA_SAVE_ON_PROGRESS that saves progress for visibility, it is not designed to allow a different worker to pick up the exact memory state of a previous execution. To handle this in your workflows, it is recommended to make your nodes idempotent (e.g., using unique IDs or checking if a record already exists before creating it) so that a restart does not cause duplicate side effects.

Your observation of executions transiently showing “Error” before “Success” is a known phenomenon in distributed Queue Mode environments, likely caused by one of the following:

  • Transient “Crashed” State: When a worker crashes or becomes unresponsive (e.g., due to heavy load or a Kubernetes pod restart), the main instance or the reaper may detect the “stalled” execution and mark it as Error or Crashed in the database/UI. However, if the execution was actually still processing (just very slowly) or if the worker recovered and finished the task, the final state recorded in the database will be Success. The UI reflects these state changes as they happen.
  • Race Conditions/UI Lag: In high-concurrency scenarios, the status updates from different workers and the main instance can arrive at the database/UI with slight delays. An execution might be flagged as an error due to a timeout/heartbeat failure, but the actual completion signal arrives shortly after, updating the status to success.
  • Worker Overload: If your workers are hitting resource limits (CPU/Memory) in Kubernetes, they might experience “stop-the-world” GC pauses or slow heartbeats. This makes the main instance think the worker has failed, triggering the error status, even if the worker eventually recovers and completes the job.

This still applies

I believe this should be posted under the Question category, not feature request

3

Does a retry happen on the same worker, or can another worker pick up the job?

According to the documentation, QUEUE_WORKER_MAX_STALLED_COUNT is deprecated and retries should not happen. We also haven’t observed any scenario where another worker picks up a job after the original worker crashes.

However, we have seen cases where a job is initially marked as Error in the UI, but after some time it is rerun and eventually succeeds. Similarly, in some cases where a job is Canceled from the UI, it later runs again and succeeds.

This is causing some confusion around the actual rerun/retry mechanism. Could you please clarify what triggers these reruns and whether the rerun can be picked up by the same worker or a different worker?

4

Hi @Mahavir_Kathed — the short version is: on current Queue Mode, a crashed worker’s job is generally not automatically handed to another worker the way older Bull stalled-job retries worked. That matches what you’re seeing with QUEUE_WORKER_MAX_STALLED_COUNT being deprecated and with no clear “another worker took over mid-run” behavior.

What usually happens instead:

  1. Crash / restart — the in-flight execution is marked crashed/error (or left looking broken in the UI for a bit). There is no mid-workflow checkpoint handoff to a different worker. If anything runs again, it is typically a new execution from the start, not a resume.

  2. Same worker vs another worker — automatic reclaim of the same Redis job by a different worker is not the reliable path anymore. A later Success you see is more often either (a) the original worker was only slow / under GC / heartbeat pressure and eventually finished and wrote Success, or (b) something upstream started a new run (webhook client retry, parent workflow retry, manual Retry, overlapping cron, Error Workflow, etc.). Check whether the Success row shares the same execution ID as the earlier Error/Canceled row. Same ID → status flapping on one run. Different IDs → a separate rerun.

  3. Canceled then Success — cancel is best-effort. If the worker never got the cancel (or finished before cancel landed), it can still complete and land on Success. A later Success with a new execution ID means something retried outside that cancel.

  4. What to do operationally — pin an exact n8n image (avoid :latest drift across scaled workers), watch worker pod restarts / OOMKilled / memory pressure, and design side effects to be idempotent so a full restart from the trigger is safe. For business truth, log completion in your own store rather than trusting a transient UI status alone.

Happy to dig into a specific execution ID pair (Error then Success) if you can share whether the IDs matched.

5

I tested exactly this on 10 October on n8n 2.42.6 in queue mode (1 main, 1 worker, Postgres, Redis, Docker Compose), with a workflow that takes 40 seconds:

1-3. Does another worker take over? In my test: no. When the worker was killed in the middle (docker kill), the execution ended as crashed and was never run again - also not after the worker came back. That matches the source: n8n sets the queue’s maxStalledCount to 0 (packages/cli/src/scaling/scaling.service.ts), so a stalled job is not handed out a second time. I only had one worker, so I did not test a second worker picking it up - but with that setting I would not expect it.

What surprised me: a normal restart loses the run too. docker compose restart worker 5 seconds into the run: the worker logged Waiting for 1 active executions to finish..., Docker stopped it after its default 10 seconds, and the execution ended as crashed. After the restart the worker logs Found unfinished executions: ... and This could be due to a crash of an active workflow or a restart of n8n.

5. What helped: give the worker time to finish before it is stopped. Two timers matter - the container stop timeout (Docker: 10 s by default) and n8n’s own N8N_GRACEFUL_SHUTDOWN_TIMEOUT (30 s by default for workers, per the docs). With a 90 s stop timeout and N8N_GRACEFUL_SHUTDOWN_TIMEOUT=80, the same 40-second run survived a restart and a recreate (the restart simply took ~36 s). Keep n8n’s value a little below the container’s.

On Kubernetes the matching knob should be the pod’s terminationGracePeriodSeconds (default 30 s) - I have not tested that, and with autoscaling 1-10 workers it matters on every scale-down, so it is worth a test in your cluster.

What stays lost either way: runs longer than the grace period, and workers that are killed (OOM, node loss). So the advice above about idempotent steps still applies, plus an alert for executions that never finished.

I wrote the test up with the full table and log lines here: n8n queue mode: a running workflow ends as "crashed" when the worker restarts - why, and the fix (my own page - I also sell an n8n hosting kit, the guide itself is free).