Help to Accumulate Json data

[Meghna_jose]

##I have a workflow that fecthes data from a url . But it’s paginated, so I need to pass the page parameter and loop back until the value of certain parameter(called “after”) is null. After the run, final I’m able to get only the data of last page. How can I accumulate the data of previous page and store it as one JSON array.

What is the error message (if any)?

Please share your workflow

The code node has script as below:

let queryExpression = `item.json.list[?source.type=='ClaimExcelReport'].{file:filename,id:source.primary_id,url:download_uri}`;
console.log($input);
return $jmespath($input,queryExpression);

Also Here in the code, I’m unable to fetch the data of “GetReportList”, how can I access it and transorm?

Share the output returned by the last node

Information on your n8n setup

• n8n version: 0.205.0
• Database (default: SQLite): PostgreSQL
• **n8n EXECUTIONS_PROCESS setting (default: own, main):**main
• **Running n8n via (Docker, npm, n8n cloud, desktop app):**Azure
• **Operating system:**Windows

[EmeraldHerald]

Hi @Meghna_jose We have an open feature request for pagination in the HTTP Request node, you might want to vote on it and leave a comment here: Pagination included into HTTP Node - #9 by jan

In the meantime, you might be able to use a solution like what this user has as a fix:

HTTP API Pagination

Happy to say that i was able to solve it using ChatGPT-4 not sure if this is the best solution but here is what i did and it worked for me The solution relies on saving the next_page_id to the global variables of the execution and then retrieve it again. This is the code that is in the Config URL node: // Get the global workflow static data const workflowStaticData = $getWorkflowStaticData('global'); // Access its data let next_page_id = workflowStaticData.next_page_id || null; let url; i…

[Meghna_jose]

Thank you @EmeraldHerald . I was able to implement the logic for pagination. But I’m facing difficulty in accumulating data.
This is the code below I used:

const allData = [];

let counter = 0;
let hasNextPage = true;

(async () => {
  while (hasNextPage) {
    try {
      
      const response = await $items("GetReportList", 0, counter); 
      const jsonData = response[0].json.results;
      console.log(jsonData)

      
      allData.push(...jsonData);
      console.log(allData)

     
      const nextPageCursor = response[0].json.cursor;
      if (nextPageCursor && nextPageCursor.after) {
        counter++;
      } else {
        hasNextPage = false;
      }
    } catch (error) {
      console.error(`Error fetching data: ${error}`);
      hasNextPage = false;
    }
  }

  
  
  return [{"list":allData}];
})();

I’m getting an error as below:

I’m using function code mode as “Return Once for all items”

[EmeraldHerald]

Hi @Meghna_jose It’s a little difficult without having access to what your workflow returns at each step as an example, but could you try this?

const allData = [];

let counter = 0;
let hasNextPage = true;

(async () => {
  while (hasNextPage) {
    try {
      const response = await $items("GetReportList", 0, counter); 
      const jsonData = response[0].json.results;
      
      allData.push(...jsonData);
      
      const nextPageCursor = response[0].json.cursor;
      if (nextPageCursor && nextPageCursor.after) {
        counter++;
      } else {
        hasNextPage = false;
      }
    } catch (error) {
      console.error(`Error fetching data: ${error}`);
      hasNextPage = false;
    }
  }

  const outputData = allData.map(item => ({ item }));
  
  return outputData;
})();

[Meghna_jose]

Thanks for the help @EmeraldHerald .