Hi, I have a question about a better way to handle data stream, currently the setup is like the image I upload below. If you look on the Postgres node you could see that it receive 137 items and on the column inside my table there’s status column with bunch of tags and it used to filter if process already done for each data.
In correct scenario this setup will work perfectly but when there’s interrupted in the process that some data is process and the rest isn’t the problem arise. You could see the problem at Event Relevancy Screening sub-workflow that this node run twice, one with 123 data (137-14) and the second is 137.
Personally this data flow is good because each time the certain process finish the data gets tagged and if there’s some data that hasn’t through certain process it can do it in the next execution but the downside is the problem that I’ve got right now, so is there a solution for this problem or anyone has better data flow structure that I can learn?
The double run happens because the parent passes all 137 items into the sub-workflow in one go, and on a retry the “not yet processed” filter and the full list both end up feeding it. A pattern that has worked better for me:
Fetch only unfinished rows at the start (e.g. WHERE status <> 'done'), and before handing a row on, set it to something like processing with a timestamp in the same query (UPDATE ... SET status='processing', started_at=now() ... RETURNING *). That way a second execution can’t pick up the same rows.
Use Loop Over Items with a small batch size (10–20) and call the sub-workflow once per batch, so one failure only affects that batch.
Have the sub-workflow write its own result back per row (done, or failed with the error text) as its last step, instead of tagging everything at the end of the parent.
Add a small cleanup step that resets rows stuck in processing for longer than, say, 30 minutes back to pending, so an interrupted run gets picked up next time.
With that, each row is processed once per stage, and an interrupted run simply continues where it stopped. What triggers the parent workflow at the moment — a schedule, or something else?
The double run comes from the loop-back wires, not from your status tags.
What your screenshot shows:
Select rows from a table gets 137 items in. A Postgres node runs its query once per incoming item, so it ran 4 times (✓4, 548 items total).
The outputs of Duplicate Finder and Event Relevancy Screening are wired back into Select rows. Each time a sub-workflow finishes, its items re-enter Select → Switch and get routed again. That matches Event Relevancy Screening firing once with the first batch (123) and again with the full set (137).
A structure that keeps your “resume where it stopped” idea but runs each stage once:
Remove the loop-back connections.
Make it linear, one stage per status: Trigger → Select (status = new) → Duplicate Finder → Select (status = deduped) → Event Relevancy Screening → Select (status = screened) → Vectorize
Each Select reads the table by status, so after an interrupted run every stage simply picks up what is left.
On every Select node, open Settings and turn on Execute Once, so the query runs one time instead of once per incoming item.
Inside each sub-workflow, write the new status as the last step, so a row only moves on when its stage really finished.
To verify: run it once; each Execute Workflow node should show ✓1, and the item count out of each Select should match a SELECT count(*) … WHERE status = … on the table.
Use your actual status values. Each child writes the next status after its work succeeds, and all three Select nodes receive the trigger directly. Remove the feedback wires. Newly advanced rows can be picked up on the next scheduled run.
The useful restart case to check is zero new rows, with some deduped rows still waiting. With the default empty-result behavior, a linear Select(new) → … → Select(deduped) chain stops at the first empty Select. Independent entry branches let screening resume even when there is nothing left to deduplicate.
Start with one active polling execution. For an external operation that supports idempotency keys, use a stable key such as row-id:stage, so retrying after “operation succeeded, status update failed” reuses the same operation.
A small restart check: seed 0 new, 2 deduped, and 1 screened row; trigger the workflow and check that the two screening jobs and the already-waiting vectorization job run. Run it again and verify that completed stages stay completed.
I’d first check whether those 123 and 137 items are coming from separate executions or two paths in the same execution.
The status tags help, but only after they’re saved. If another run picks up the records before that happens, you can still process the same items twice
I’d use Postgres to claim each item before processing, then mark that step complete only after it succeeds. That also gives interrupted runs a cleaner way to resume
And Duplicate Finder node will generate ‘duplicate’ or ‘unique’ tag. This is what data that exist in the status object. You said
each status queue its own entry branch from the trigger
It feels like you propose dynamic tagging system but I don’t really sure I grasp your solution and I also still hard to understanding how it solve the data stream problem.
candidate → Duplicate Finder unique → Event Relevancy Screening relevant → Vectorize duplicate → terminal state, no further processing vectorize → either waiting for vectorization or completed, depending on how you currently use that value
The important change is that these stages would not feed back into one another through the same execution path. Each stage would start by selecting only the rows waiting for that specific step.
For example, Event Relevancy Screening should select only rows currently marked unique. After each row is screened successfully, it changes that row to either relevant or another terminal status. Vectorize then selects only relevant rows.
This helps with interruption because a later scheduled run can resume directly from whichever statuses remain. If there are zero candidate rows but 123 unique rows, the screening branch can still run without depending on the Duplicate Finder branch producing output first.
It does not by itself prevent two overlapping executions from selecting the same unique rows. For that, I’d add an atomic claim before processing, for example changing unique to screening only if the row is still unique. Only the execution that successfully claims the row should continue.
So the main distinction is:
status says which stage should process the row next;
claim/in-progress state prevents two executions from processing that row at the same time;
final status update happens only after the stage succeeds.
I’d first confirm what vectorize means in your current table: waiting to be vectorized, or already vectorized. Those should ideally be two different states
Your status field holding several tags at once is probably the confusing part: one Select by status can match the same row at more than one stage. I’d keep a single column for what runs next (candidate, unique, screening) and move the old tags into a separate history array. Each branch then selects only its own value, and claims the row with an UPDATE that returns it while that value is unchanged, so two overlapping runs cannot pick the same row.
The 137 on the second run is likely because the sub-workflow call gets the full Postgres output again after a partial failure. Filter on the status column in the query itself (WHERE status not tagged for that step) so each stage only pulls unprocessed rows. Then an interrupted run just picks up the leftovers next time, and the sub-workflow only ever sees what it needs.
Keep your existing status tags and Switch conditions. The change is how the nodes are connected. My new/deduped/screened labels were placeholders for your existing stage filters.
In your screenshot, both child workflows return to Select rows from a table. Each return starts another read of the table and another pass through Switch. Select shows four runs and 548 total items (4 × 137); Screening receives 123 items in one run and 137 in another.
Start with this smaller edit to your current graph:
Trigger → Select rows → Switch
├─ Vectorize News Record → save status → end
├─ Duplicate Finder → save status → end
└─ Event Relevancy → save status → end
Remove the two child-to-Select feedback wires. Keep the status updates inside the children, after each row’s work succeeds. Each poll reads the current database rows and routes that batch once. Newly updated rows move to their next stage on the next poll.
For example, if your Switch sends unique records to Screening: a row already tagged unique enters Screening in this poll. A different row that Duplicate Finder tags unique becomes eligible for Screening in the next poll. Their existing tags determine the route.
For a restart test, connect a Manual Trigger directly to Select and seed one row ready for Duplicate Finder and two rows ready for Screening. First run: Select runs once; Screening receives the two already-ready rows once. Second run: the newly unique row becomes ready for Screening. Successfully completed rows follow your existing next-stage filters. In production, use a Schedule Trigger for these processing polls so leftover rows resume after an interruption; keep one active polling execution during this check.
Could you paste one real row’s complete status value and the three Switch conditions? Your example lists the keys; their values and matching rules will let me map the routes exactly to your current setup.
I understand that linear process is the straight forward system but I want to reduce the node pathways. When the process is linear, if the process is halted in the middle, I need to start again from process A-B-C but in my case since I already have the tag I skip the process that I already done and in the picture you could see that the execution doesn’t go into Vectorize News Record sub-workflow because it is already been processed.
This is the design that will reduce the friction in the linear process but the downside is that I get this new problem. I’m wondering if there’s new system or model that can skip the process that I already done while handling the problem that I found.
@ezraluandre you can keep the tags and the skipping. The double run isn’t caused by them; it comes from re-reading the table every time a child finishes. So read once and let each row carry its own state through the stages:
Trigger → Select rows (Execute Once) → IF needs Duplicate Finder → Duplicate Finder → IF needs Relevancy Screening → Relevancy Screening → IF needs Vectorize → Vectorize
Each IF checks that row’s tags. True goes into the child; false goes straight to the next IF, skipping the stage.
Each child writes its tag to the table, then returns the row with the updated tags, so the next IF sees the new state. This is the part that replaces the re-read.
After an interruption, the next run reads the tags again and each row skips what it has already finished. An already vectorized row passes through all three IFs untouched.
The next IF gets items from two wires (the child and the false branch), so it can show 2 runs. That’s fine here: the two sets of rows are different, so no row gets processed twice.
It’s still a few nodes more than your current graph, but each stage appears once and there are no wires going backwards. If you’d rather have fewer nodes and don’t mind rows moving one stage per execution, the smaller edit @blucca posted above also works.
To verify: interrupt a run halfway, run it again, and compare the item counts going into each child with the rows still missing that tag. No row should enter a child it already finished.
@ezraluandre I rebuilt your graph on n8n 2.47.7 with a Postgres table of 137 rows and stand-in stages that count what they receive:
Your numbers came back. I interrupted Duplicate Finder at row 123 and ran again: Duplicate Finder got 14, and Screening ran twice, with 137 and 123, so those 123 rows were screened twice. The tags are fine. Both sub-workflows are wired back to Select, and after an interruption rows leave the Switch on two outputs at once.
Two structures kept the skipping and ran each stage once (14, then 137), also after I killed the n8n container mid-stage:
Stage queues: Trigger → Vectorize → Duplicate Finder → Screening. Each sub-workflow starts with its own query (has the previous tag, lacks mine) and always returns one item.
Skip and merge: read once, then per stage IF → sub-workflow → Merge. That’s @minimolo’s IF chain plus a Merge. Without the Merge the next stage still ran as two calls (14 + 123), with different rows, so nothing was done twice.
I’d pick the first one: three nodes, no wires back. A chain of Selects in the parent stopped at the first empty Select, like @blucca warned, and left rows behind.
Didn’t test: your real stages, two runs at once, n8n Cloud. At most the one row in flight at the kill was redone. I assumed text, and jsonb is included too.