Master Coordinator for N number of subworkflows

Questions
1

Hi, I have been trying to figure out how to leverage a master workflow(100K records which need to be enriched N times) as a main coordinator. So for each batch,

  1. It has to be enriched (A)
    N. It has to be enriched (N)
    Upon completion, it will be passed to Master or Post Processing workflow ( from the same batch). Any ideas or best practice to share?

Thanks,
Brandon

2

I share the following ideas or suggestions with you.

1- Divide processing into reusable sub-workflows:
Create modular sub-workflows for each enrichment stage (Enrichment A, Enrichment N, etc.).
Use the Execute Workflow node to call these sub-workflows from the master workflow.

2- Use queueing/batch control:
Processing 100,000 records directly can create bottlenecks.
Use the SplitInBatches node to split records into small batches.
You can set limits (e.g., 1,000 per batch) and control the speed.

3- Dynamic thread chaining:
If the number of enrichments changes (is dynamic), you can use iterative logic:
Use a Loop/While node (a conditional structure with “IF + Set + Execute Workflow”) to apply sequential enrichments.

4- Master Flow Control:
The master flow can control:

  • Data ingestion.
  • teration through enrichments.
  • Final validation or consolidation.
  • Sending to a post-processing or storage flow.

5- Persistence and Recovery:
Uses temporary databases or files if the flow can fail (checkpointing).
Optional: Use HTTP Request nodes if the enrichments are external APIs.

1 Like
3

Thanks, Erick. For example (new to n8n),
Master -batch(100) → sends to SF(A), SF(B) via Execute SubWorkflow(SF)

  • I assume we need to pass the same batchid to SF(A) and SF(B)
  • As it’s async, what is the logic to use at Master or a Post-processing WF to collect signal that (SF(A), SF(B) is done completely and then a full post processing can be done on the same batch.

I tried a WAIT(Respond to WH) but it was flaky. Need a better coordinator logic.

Thanks again!

1 Like