SendGrid - Retrieve all Global Suppressions (Pagination)

Questions
,
1

I’m trying to set up pagination for the SendGrid API’s unsubscribe endpoint in n8n desktop version, but I’m unable to get the pagination working correctly using the HTTP Request node’s built-in pagination options.

API Details

Endpoint: https://api.sendgrid.com/v3/suppression/unsubscribes
Pagination type: offset-based

Parameters:
limit: 500
offset: increments by 500

Current Response
The API returns pagination information in the Link header. Example response header:
Copy"link": “https://api.sendgrid.com/v3/suppression/unsubscribes?%3Doffset=0&limit=500&offset=500; rel="next"; title="2", https://api.sendgrid.com/v3/suppression/unsubscribes?%3Doffset=0&limit=500&offset=0; rel="prev"; title="1", https://api.sendgrid.com/v3/suppression/unsubscribes?%3Doffset=0&limit=500&offset=30000; rel="last"; title="61", https://api.sendgrid.com/v3/suppression/unsubscribes?%3Doffset=0&limit=500&offset=0; rel="first"; title="1"”

What I’ve Tried

Offset Pagination:

Type: Offset
Tried using $pageCount * 500 for offset
Result: Returns same data multiple times

URL Pagination:

Type: URL
Tried extracting next URL from Link header
Result: Unable to get it working correctly

Complete Expression:

Tried using {{ !$response.headers.link?.includes(‘rel=“next”’) }}
Result: Pagination doesn’t stop correctly

How can I properly configure the HTTP Request node’s pagination settings to work with SendGrid’s API? The API provides next/prev/first/last URLs in the Link header - is there a way to utilize this with n8n’s built-in pagination?

2

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
3

Welcome to the community @dan1 !

Tip for sharing information

Pasting your n8n workflow

Ensure to copy your n8n workflow and paste it in the code block, that is in between the pairs of triple backticks, which also could be achieved by clicking </> (preformatted text) in the editor and pasting in your workflow.

```
<your workflow>
```

That implies to any JSON output you would like to share with us.

Make sure that you have removed any sensitive information from your workflow and include dummy or pinned data with it!

I’m trying to picture how the response payload looks like. Could you paste an example of the whole payload returned in response to a single request (excluding the actual data)? I haven’t come across linked headers before.

4

The output without pagination settings returns a simple JSON containing user email and creation date:

[
  {
    "email": "[email protected]",
    "created": 1736764944
  },
}

If I enable “Include Response Headers and Status” it will also add headers to the output:

{
  "headers": {
    "server": "nginx",
    "date": "Mon, 13 Jan 2025 10:50:56 GMT",
    "content-type": "application/json",
    "transfer-encoding": "chunked",
    "connection": "close",
    "vary": "Accept-Encoding,Origin",
    "strict-transport-security": "max-age=15724800; includeSubDomains",
    "link": "<https://api.sendgrid.com/v3/suppression/unsubscribes?limit=500&offset=500>; rel="next"; title="2", <https://api.sendgrid.com/v3/suppression/unsubscribes?limit=500&offset=0>; rel="prev"; title="1", <https://api.sendgrid.com/v3/suppression/unsubscribes?limit=500&offset=33000>; rel="last"; title="67", <https://api.sendgrid.com/v3/suppression/unsubscribes?limit=500&offset=0>; rel="first"; title="1"",
    "twilio-request-id": "------",
    "x-envoy-upstream-service-time": "87",
    "content-security-policy": "frame-ancestors 'none'",
    "cache-control": "no-store",
    "referrer-policy": "strict-origin-when-cross-origin",
    "x-content-type-options": "nosniff",
    "x-ratelimit-remaining": "599",
    "x-ratelimit-reset": "4",
    "x-ratelimit-limit": "600",
    "powered-by": "SGGateway"
  },
  "statusCode": 200,
  "statusMessage": "OK"
}
5

Thanks for the output. I had to format it for you to have a better visibility into actual data.

It looks like the built-in pagination cannot work in this case. You would have to utilise an old-fashion solution with the Loop node. It is described in The Good, the Bad, and the Ugly of looping with n8n – n8n Blog.

For it to work, you would need to parse the value in the link. For example,

1 Like
6

image
image1415×540 69 KB

Thank you for the suggestion! I created a proper loop and didn’t even need to need the links from the response.

However, I’ve encountered another issue: Once I completed all the loops, I was certain that the merge node could combine all the items into one from the loop. I’m pretty sure there’s a fix for this, but I’m really stuck at the moment. Do you have any suggestions?

I am using the “Choose Branch” mode in the Merge Node. I tried with others, too but to no avail.

I know this is a different kind of issue, but it is still relevant to paginating and actually making sure the data is aggregated in the output.

7

Hey @dan1 , no, Merge node cannot combine that data as it’s produced by different iterations (72 in your case, one item in each iteration). For Merge to be able to merge/combine the data you need to have them all as part of the same iteration, which is hardly possible to achieve with this approach. You could engage Code node with some Javascript logic to process all the iterations but how do you signal that all the iterations have completed?

I would suggest to come back to the solution I laid out.

2 Likes
8

YES! This was the missing piece of the puzzle! Thank you very much! I just added it after the “if body length” node and got it working)

9

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.