Reliable orchestration of an external local agent: claims, heartbeats and late callbacks

[Guido_Renneberg]

We use n8n Cloud with Data Tables to coordinate jobs processed by an external agent on a Windows machine. This is a separate application worker, not an n8n queue-mode worker.

We want to distinguish job claim from actual processing start and safely handle crashes, retries and delayed callbacks.

What architecture would you recommend for:

1. Atomically claiming a job and assigning a unique attempt ID?
2. Renewing a lease through heartbeats and recovering expired attempts?
3. Rejecting late callbacks from an earlier attempt after reassignment?
4. Preventing duplicate effects when execution succeeds but its acknowledgement is lost?

Can Data Tables reliably support the necessary conditional updates, or would you recommend an external transactional database? Would a Wait node with an authenticated resume webhook simplify completion handling, alongside separate heartbeat tracking?

We would appreciate a reference workflow or documented pattern.

[Zeke_2007]

This is a lease and fencing problem, and it’s solvable with one table and a few rules. In order:

1. Claiming and the attempt ID. Claim with a single conditional update, not a read followed by a write: set status='claimed', a fresh attempt_id and lease_expires_at, but only WHERE status='queued' (or the lease has expired). Treat the claim as won only if exactly one row changed. Then use a separate start call, carrying the attempt_id, to move the job to running. That cleanly separates “claimed” from “actually started.”

2. Heartbeats and recovery. The agent renews the lease with its attempt_id, and the update applies only if that attempt is still the current one. A schedule workflow in n8n finds attempts whose lease has expired, requeues them with attempt_no + 1, and after a maximum number of attempts moves them to dead-letter and alerts you.

3. Late callbacks. Use the attempt_id as a fencing token. Every callback carries it, and the final update applies only if it matches the current attempt. Otherwise return a 409 and log it. A slow attempt that wakes up after reassignment can’t overwrite anything.

4. Lost acknowledgements. Make completion idempotent: if the same attempt_id reports the same result again, return the stored result and change nothing. On the agent side, keep results in a local outbox and retry until acknowledged. Pass a stable idempotency key (job_id + step) to any downstream effect.

Data Tables or a database? The whole design rests on one atomic conditional update that tells you how many rows changed. I wouldn’t trust any table for that until you’ve tested it with many parallel claims. A transactional database (Postgres) gives you the guarantee directly, with n8n orchestrating around it.

Wait node? I’d keep the state in the table, not in a paused execution. A resume URL belongs to one execution, which makes duplicate and late callbacks harder to reason about. A plain authenticated webhook workflow that does the fenced update is simpler.

If you’d rather have this designed and kept running for you, Pond AI builds, runs and maintains custom agent systems like this, and sketches the whole flow on one page before any build starts. There’s a 7-day refund .

One question that decides the lease length and how strict step 4 must be: what’s your longest job runtime, and what does running a job twice cost you?

Disclosure: I’m affiliated with Pond AI.

[Akanazubaba]

Welcome @Guido_Renneberg . One subtlety nobody has spelled out: Data Tables will very likely let two workers claim the same row, and that is not a bug, it is the design boundary you have to plan for.

Two things to do. First, run the whole pattern in two lanes inside the same structure, a claims lane and a separate state lane, so the atomic conditional update only ever touches the row it fences and you can rebuild current state from history if the live row gets messy. Second, make the callback endpoint deeply idempotent and replayable: store every attempt and every result as its own record keyed by attempt id, return the recorded result if the same attempt reports twice, and reconcile from those records rather than from a single column that gets overwritten. On the Data Tables question specifically, test many parallel claims before you trust it, and define up front the point where you move to Postgres, which is when you need real transactions across multiple rows or when claim contention grows.

That is the whole model laid out as a contract, three append only tables on the right and five workflows on the left, with the ownership arrows showing which workflow writes which table:

Read it as a contract. The attempts table is the append only truth and jobs holds only current state, so you can always rebuild one from the other. Workflow A is the claim cycle, conditioned so it only ever touches the jobs row it fences. Workflow B is heartbeat renewal, which extends the lease only while the current attempt id matches and returns zero rows otherwise, and that zero row result is your signal that the worker has been superseded. Workflow C is the callback endpoint, idempotent on the idempotency key and returning the stored result on a repeat. Workflow D is the reaper, moving expired attempts to attempt number plus one and dead lettering after the cap. Workflow E is reconciliation, comparing attempts against the outbox delivered flag and replaying anything undelivered. The coloured tags on each workflow are the tables it is allowed to touch, which is how you keep two workers from ever writing the same row.

I am a Pond AI affiliate. Pond AI builds, runs and maintains custom AI agents, and for orchestration like this it drafts the full architecture free before any build begins, with the draft yours to keep, and a 7 day refund cover on custom built by Pond AI

What does a lost or duplicated job cost you in money or trust?

[Odoemelamair]

@Guido_Renneberg a warm welcome to the community. The mechanics of claiming and fencing has been attended to, so let me add the part that usually breaks production leases: the claim update has to be legal only from a state where claiming makes sense, or a crashed worker’s row gets picked up while it is still running.

Concretely, the conditional update should match status queued OR running with lease_expires_at in the past, never a bare “not claimed” check. That single guard is what lets a slow worker be safely replaced. Pair it with a monotonic attempt_no on the row and a lock token that increments every time the row changes hands, so out of order acknowledgements cannot be applied. When an acknowledgement arrives, apply it only if the token it carries is the current one and the row is still in the attempt that produced it, otherwise drop it as stale and log it. And set the lease from the longest job runtime plus a margin, because too short a lease is what manufactures false crash recoveries.

This is the four step cycle, drawn as a ring so you can see that a job only ever moves forward, and that the single way back is through an expired lease:

Read the ring left to right and around: queued to claimed by one atomic update that reports exactly one changed row, claimed to running by a separate start call carrying the attempt id so claim and start stay distinct, and running to expired when the lease runs out, at which point the reaper requeues at attempt no plus one or dead letters it. The guard in the centre is the point of the whole thing, it runs on every hand off and matches queued OR running with an expired lease, which is what lets a slow worker be replaced cleanly and a crashed attempt come back to the start instead of hanging forever.

I am affiliated with Pond AI. Pond AI builds, runs and maintains custom AI agents, and for a system like this it would map the whole orchestration on one page before any build starts, a map that is yours to keep. Custom builds through Pond AI come with a 7 day refund.

Do your jobs have a bounded runtime, or can one legitimately run for hours?

[Chineydu]

Hi @Guido_Renneberg welcome to the forum. The fencing token protects the database row, but it does nothing for the side effect on your Windows machine, and that is the part that will actually bite you.

Here is the ordering that matters. The agent should write its result to a durable local outbox and flush that before it calls back, so a crash between doing the work and telling n8n cannot lose the result. On completion the workflow should update the row and insert the idempotency record in one transaction, keyed on job id plus a step identifier, so a replayed acknowledgement is a no op instead of a duplicate effect. For anything the agent does outward, a payment, a file write, a message, pass that same key to the downstream system and let it deduplicate. The rule to hold in your head is that exactly once is fiction across process boundaries, so you build effectively once with idempotency plus a durable outbox plus atomic commit.

Here is that ordering as the callback sequence, four numbered gates, and note that the effect in gate four never runs until the first three have cleared:

The agent sends one completion envelope carrying the job id, attempt id, attempt number, an idempotency key made of job id plus step, a status, a result and a checksum, and it only sends after the result is durable in its own outbox. Gate one looks the attempt up by attempt id. Gate two is the branch that matters: if the attempt is already terminal, return the stored result and stop, and if the attempt id is not the current one, return 409 and log the stale callback. Gate three is the atomic move, marking the attempt terminal and writing the result row keyed on the idempotency key together. Only then does gate four call downstream, passing that same key. The sweep on the side is reconciliation, which requeues expired claims and replays acknowledgements for succeeded rows with no delivered flag, and that is what recovers the lost acknowledgement case safely.

I am affiliated with Pond AI, Pond AI builds, runs and maintains custom AI agents, and this is exactly the reliability work it does, with a free architecture map drawn before any commitment and there is a 7 day refund cover on builds done by Pond AI.

If a job runs twice, is that merely wasteful for you or genuinely harmful?

[sdflemin]

was curious about the gap between committing the result and calling downstream. if the downstream action succeeds but the ack gets lost, how do you reconcile before retrying without duplicating the effect?