Parallel processing and missing outputs

Tips & Tricks
1

Hi there,

I managed to emulate parallel processing natively on N8N with the below workflow:

The tasks it performs are below as well:

When I performing below command run, I got the output {"finishedSet":[{"a":1,"b":4,"c":9}]}, even though I expected {"finishedSet":[{"a":1,"b":4,"c":9}, {"a":1,"b":4,"c":9}]}. Can you see any issue on my approach? Thanks!!

curl -X POST "https://your_n8n_url_here/webhook-test/parallel" -H "Content-Type: application/json" -d '{ "requests": [ { "request_id": 1, "method": "POST", "url": "https://your_n8n_webhook_url_here/webhook/post-subworkflow", "data": { "a": 1, "b": 2, "c": 3 } }, { "request_id": 2, "method": "POST", "url": "https://your_n8n_webhook_url_here/webhook/post-subworkflow", "data": { "a": 1, "b": 2, "c": 3 } } ] }'

You can try with GET method as well.

curl -X POST "https://your_n8n_url_here/webhook-test/parallel" -H "Content-Type: application/json" -d '{ "requests": [ { "request_id": 1, "method": "POST", "url": "https://your_n8n_webhook_url_here/webhook/post-subworkflow", "data": { "a": 1, "b": 2, "c": 3 } }, { "request_id": 2, "method": "GET", "url": "https://your_n8n_webhook_url_here/webhook/post-subworkflow" ] }'

2

Hey, you could have used the “Execute workflow” node instead.

It’s actually the method recommend for this.

3

I may be doing some bad practices at this point, but as you can see, it seems a HTTP request node seems more flexible than using the Sub-workflow node, don’t you think?

Additionally, this workflow seems a good prototype for a parallel processing node, don’t you think?

4

Yes, this is a prevalent workaround for ‘parallel processing’ in n8n.

But, because the ‘Execute Workflow’ node has also been improved significantly since then (especially because you can now choose to wait, or not, for sub-workflow completion), i’d say that’s currently the optimal method.

5

Hi Shrey, thanks for joining the conversation with your expertise. There is a potential scale on analysis of such problem. I will give a glimpse with my educated perspective and insights. Feel free to improve or criticize at will. In this context, ‘object’ means ‘JSON object’.

Given a list of requests, we may endow it with such properties:

  • id: Unique identification of the request;
  • created_at: Timestamp when the entry was created;
  • url: URL without protocol (notation $subdomain.$domain);
  • method: HTTP request method (GET, POST, PUT, DELETE, PATCH);
  • data: Object with necessary key-value entries for the request. If it does not apply, it is an empty object {};
  • depends_on: Array of objects with properties id and status_codes;
  • timeout: Timeout in seconds;
  • retries: An object with properties backoff_strategy and max_retries;
  • priority: An string with values low, medium and high;
  • batch_group: An identifier to address batch segregation, necessary if there are too many requests with different priority and/or dependencies;

Due dependencies, our request list is a set of dependency graphs, whose execution order is from leaf to root, not to mention their priorities. For the sake of simplicity, assume each graph will succesfully execute (a miracle): in this case, the data from a dependendy will be placed on object data for their parents. In the meanwhile some node may time out or be retried max_retries times. We may define a request as succesful, which output is an object with properties id and result, or an error , with properties id and error. Once one of the nodes, either after max_retries times or actual request error, the respective graph failed.

Does it make sense?

6

If you are still with me, I improved OP workflows, but I am failing to retrieve response to the caller.

You can try test pasting on N8N the workflows separately and then perform the POST request, as below, for example.

export WEBHOOK_URL="n8n.webhook.example.com"

curl -X POST "https://$WEBHOOL_URL/webhook/request"      -H "Content-Type: application/json"      -d '{
    "nodes": [
        { "id": 1, "method": "GET", "url": "https://$WEBHOOL_URL/webhook/get-subworkflow", "depends_on": [] },
        { "id": 2, "method": "POST", "url": "https://$WEBHOOL_URL/webhook/post-subworkflow", "data": { "a": 1, "b": 2, "c": 3 }, "depends_on": [1]}
    ]
}'

Workflows

Parallel tasks manager

Share the output returned by the last node

image
image1207×320 34.6 KB

image
image703×562 14.7 KB

1 Like