Handle pagination HTTP Requests with result combination

[_jd]

Hi everyone, freshly arrived in the n8n community, I’m strugling with an automation based on an external API.

What I want to achieve

Call an external PublicationEvents API everyday to retrieve new publications from the day before and share a report on our chat platform (= a daily diggest).

The workflow

• The API is based on an offset + limit mechanism for pagination (no date filter)
• The API returns a “pagination” object with the next offset URL + a “data” object with the return documents (see JSON below)

To stop the HTTP Request loop after retrieving all D-1 items, I want to go over the pagination until I find an item with “createdAt = day - 2”.

Issue

Based on this example: Handle pagination in HTTP Requests | n8n workflow template

I’m struggling with the 2 parts API response (pagination / data):

• I need to store the pagination next URL for the next run
• I need to store each “data” object in order to combine them all at the end
• I need to manipulate the “data” object to check the createdAt of the last item to decide wether I run another call or run the combination steps

Thanks for your help!

[
    {
        "pagination": {
            "limit": 100,
            "offset": 0,
            "nextPath": "/api/events.list?limit=100&offset=100"
        },
        "data": [
            {
                "id": "f6437205-e32f-4f48-84f4-4bf8e75ca198",
                "name": "published.docuement",
                "collectionId": "2f15453e-3013-4946-9922-426f6393d44c",
                "documentId": "8cd458a5-02e8-459c-824c-be77baac3881",
                "createdAt": "2023-02-16T14:03:20.533Z",
                "data": {
                    "title": "Test Document 1"
                },
                "actor": {
                    "id": "4cd362b0-39ed-48ee-a269-beb34818b8a7",
                    "name": "Joe",
                    "avatarUrl": "https://foo.com/avatars/"
                }
            },
            {
                "id": "f6437205-e32f-4f48-84f4-4bf8e75ca198",
                "name": "published.docuement",
                "collectionId": "2f15453e-3013-4946-9922-426f6393d44c",
                "documentId": "8cd458a5-02e8-459c-824c-be77baac3881",
                "createdAt": "2023-02-16T14:03:20.533Z",
                "data": {
                    "title": "Test Document 2"
                },
                "actor": {
                    "id": "4cd362b0-39ed-48ee-a269-beb34818b8a7",
                    "name": "Joe",
                    "avatarUrl": "https://foo.com/avatars/"
                }
            }       ],
        "status": 200,
        "ok": true
    }
]

[MutedJam]

Hi @_jd, welcome to the community

It’s a bit hard to be certain about this without actually being able to test the API in question and without knowing which version of n8n you are running.

But based on your example data I think something like below could do the job:

This is based on the assumption that your API supports a limit and an offset query parameter. The limit remains static, while the offset parameter is the current $runIndex times 100. So on the first run, this flow will use an offset of 0, then 100, then 200 etc.

The workflow uses an IF node to check whether a pagination.nextPath field is present and non-empty in your response to determine whether to continue looping. This might need some tuning if your API would always return this field, even when done with looping.

I hope this helps you get started!

[_jd]

Hi @MutedJam, thanks for your feedback.

I spent some time trying to make it work with what seems to be the “standard” solution using “map” + “push” (see Combine all data node - here), but for some reason, I keep getting errors with this code node.

Yet the following alternative did the trick for me (found in another similar n8n topic):

let results = [],
  i = 0;

do {
  try {
    results = results.concat($("HTTP Request").all(0, i));
  } catch (error) {
    return results;
  }
  i++;
} while (true);

Hopes it will help someone in the same situation