How to aggregate data from HTTP Request in a loop

[akimrasulov]

I want to aggregate the results from the HTTP Request, while a value of query parameter is not empty. It’s paginated API request.

Describe the problem/error/question

I’m unable to update the key parameter in the URL when doing the HTTP Request, to fetch data from another request.

What is the error message (if any)?

No error. Just there is always the same results and Aggregate node is not populating

Please share your workflow

Share the output returned by the last node

I expect to receive the results in the following form at the end:

{
  "rows": [
    {
      "resourceName": "people/c3770678780834392853",
      "etag": "%EgcBAgkuNz0+GgQBAgUHIgxUaTRDNkVHcnNyMD0=",
      "names": [{}]
    },
    {
      "resourceName": "people/c8741785810902161775",
      "etag": "%EgcBAgkuNz0+GgQBAgUHIgx0TC8xQ3JvaW5GWT0=",
      "names": [{}]
    },
    {
      "resourceName": "people/c5943575406754642423",
      "etag": "%EgcBAgkuNz0+GgQBAgUHIgx521hZQXdFQ1ByUT0=",
      "names": [{}]
    },
    {
      "resourceName": "people/c3243575406754642423",
      "etag": "%EgcBAgkuNz0+GgQBAgUHIgxTK1hZQXdFQ1ByUT0=",
      "names": [{}]
    },
    {
      "resourceName": "people/c2405952239657711278",
      "etag": "%EgcBAgkuNz0+GgQBAgUHIgxTL3FIbjNNV25Obz0=",
      "names": [{}]
    },
    {
      "resourceName": "people/c1813535610825405760",
      "etag": "%EgcBAgkuNz0+GgQBAgUHIgwxbnpXVFp5anFKbz0=",
      "names": [{}]
    }
  ]
}

Information on your n8n setup

• n8n version: 1.84.1
• Database (default: SQLite):
• n8n EXECUTIONS_PROCESS setting (default: own, main):
• Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
• Operating system: Win10

Updated workflow with pinned data without Aggregate node

[Olek]

Nodes’ outputs are immutable, meaning that expressions like $('NodeName').all()|first() return a copy of data, so any changes thereto only available within the current run of a node that tries to mutate the data.

You also do not actually need Aggregate node as you are doing aggregation within the Code node. The Code node output doesn’t account for its previous run output.

Could you share the output of Fetch Contacts node (any run of it). Preferably as pinned data in the node.

[akimrasulov]

Oh ok. So basically it doesn’t do anything outside the node itselft. What is the solution then, what
approach should I go with ?

I’ve update my OP with the new workflow with pinned data and without aggregate node

[Olek]

Upd: previous version most likely has put the workflow into infinite loop. The order of inputs to Merge matters. My apologies.

A bit of a blind shot but try this:

We need:

• a NoOp node just as a loop re-entry point and a bypass
• Merge to have both previous iteration data and new data from Fetch Contacts paired together

Code node just appends new data to rows (well, along with copying previous content of rows whatever it is) and bypasses nextPageToken. Note the Code node mode is set to Once for each item. There will be only item in the output from Merge but this makes the code cleaner with no need to decompose and traverse $input.all.

Please test and lmk if it worked. And if it does, please mark this post as a Solution.

P.S. Do not forget to unpin data.

[Gy_N]

Thank you for this.

[darrell_tw]

You can try this so you don’t need to loop

Since the $nextPageToken already in response.

You can use {{ $response.nextPageToken }}

[Gy_N]

I am using the Monday API, which uses graphql queries and returns a cursor token, which then have to provided in the next query.
The loop solution works for 7-8k values, but after then my smaller self hostes setup crashes because of the data duplications.
I have not figured out how to implement http module pagination with cursor tokens…