Prevent data from aggregating between loops

Questions
1

Describe the problem

I’m aggregating data from an http query to hubspot in a postgresql table.

Read data → store in table → check if more pages → read data starting where it left off → etc.

It slows down significantly after each subsequent run, and slows basically to a halt after the 4th run. It’s only getting 200 records from HS each time. Here’s a video of what that looks like.

I am fairly sure this is happening because n8n is running the PostgreSQL node on all data each time, rather than just the data from that loop. so by the 3rd run its running on 600 records instead of just the 200.

How can i prevent it from accumulating data each loop?

Troubleshooting thus far

  • switching to 10 records at a time → slows down after the 6th loop
  • different starting points (different start createdate) → no change whatsoever
  • i’m already using an “edit fields” node to try to ditch the old data

What is the error message (if any)?

No error message, just extreme slowness.

Please share your workflow

Share the output returned by the last node

“HS get all objects with filters” output:

[
  {
    "body": {
      "total": 54985,
      "results": [
        {
          "id": "46002974602",
          "properties": {
            "billtoid": "22482",
            "createdate": "2025-10-15T14:10:20.298Z",
            "dealname": "CONTROL SPECIALTIES  INC TD52 -  S1537169",
            "e_sorder_id": "S1537169.0001",
            "hs_lastmodifieddate": "2025-12-26T20:46:23.299Z",
            "hs_object_id": "46002974602",
            "shiptoid": "22482"
          },
          "createdAt": "2025-10-15T14:10:20.298Z",
          "updatedAt": "2025-12-26T20:46:23.299Z",
          "archived": false,
          "url": "https://app.hubspot.com/contacts/47554685/record/0-3/46002974602"
        },
        {
          "id": "46002974603",
          "properties": {
            "billtoid": "348",
            "createdate": "2025-10-15T14:10:20.298Z",
            "dealname": "MR. QUOTATION  -  S1512379",
            "e_sorder_id": "S1512379.0001",
            "hs_lastmodifieddate": "2025-12-20T18:36:22.782Z",
            "hs_object_id": "46002974603",
            "shiptoid": "348"
          },
          "createdAt": "2025-10-15T14:10:20.298Z",
          "updatedAt": "2025-12-20T18:36:22.782Z",
          "archived": false,
          "url": "https://app.hubspot.com/contacts/47554685/record/0-3/46002974603"
        },
        {
          "id": "46002974604",
          "properties": {
            "billtoid": "10025",
            "createdate": "2025-10-15T14:10:20.298Z",
            "dealname": "PFIZER C/O RS INTEGRATED SUPPLY  -  S1543375",
            "e_sorder_id": "S1543375.0001",
            "hs_lastmodifieddate": "2025-12-20T18:50:58.475Z",
            "hs_object_id": "46002974604",
            "shiptoid": "18001"
          },
          "createdAt": "2025-10-15T14:10:20.298Z",
          "updatedAt": "2025-12-20T18:50:58.475Z",
          "archived": false,
          "url": "https://app.hubspot.com/contacts/47554685/record/0-3/46002974604"
        },
        {
          "id": "46002974605",
          "properties": {
            "billtoid": "14708",
            "createdate": "2025-10-15T14:10:20.298Z",
            "dealname": " PT Globalindo Simetrika -  S1555465",
            "e_sorder_id": "S1555465.0001",
            "hs_lastmodifieddate": "2025-12-20T17:58:27.594Z",
            "hs_object_id": "46002974605",
            "shiptoid": "14708"
          },
          "createdAt": "2025-10-15T14:10:20.298Z",
          "updatedAt": "2025-12-20T17:58:27.594Z",
          "archived": false,
          "url": "https://app.hubspot.com/contacts/47554685/record/0-3/46002974605"
        },
        {
          "id": "46002974606",
          "properties": {
            "billtoid": "183",
            "createdate": "2025-10-15T14:10:20.298Z",
            "dealname": "CAPP INCORPORATED SPEARS -  S1575023",
            "e_sorder_id": "S1575023.0001",
            "hs_lastmodifieddate": "2025-12-20T17:59:04.510Z",
            "hs_object_id": "46002974606",
            "shiptoid": "183"
          },
          "createdAt": "2025-10-15T14:10:20.298Z",
          "updatedAt": "2025-12-20T17:59:04.510Z",
          "archived": false,
          "url": "https://app.hubspot.com/contacts/47554685/record/0-3/46002974606"
        },
        {
          "id": "46002974607",
          "properties": {
            "billtoid": "8534",
            "createdate": "2025-10-15T14:10:20.298Z",
            "dealname": "IWM3 MECHANICAL  MAINTENANCE &  -  S1570110",
            "e_sorder_id": "S1570110.0001",
            "hs_lastmodifieddate": "2025-12-20T18:51:46.337Z",
            "hs_object_id": "46002974607",
            "shiptoid": "8534"
          },
          "createdAt": "2025-10-15T14:10:20.298Z",
          "updatedAt": "2025-12-20T18:51:46.337Z",
          "archived": false,
          "url": "https://app.hubspot.com/contacts/47554685/record/0-3/46002974607"
        },
        {
          "id": "46002974608",
          "properties": {
            "billtoid": "23921",
            "createdate": "2025-10-15T14:10:20.298Z",
            "dealname": "FERGUSON INDUSTRIAL C/O NXP USA Apollo 76-608-01A -  S1511842",
            "e_sorder_id": "S1511842.0001",
            "hs_lastmodifieddate": "2025-12-20T18:52:24.760Z",
            "hs_object_id": "46002974608",
            "shiptoid": "23947"
          },
          "createdAt": "2025-10-15T14:10:20.298Z",
          "updatedAt": "2025-12-20T18:52:24.760Z",
          "archived": false,
          "url": "https://app.hubspot.com/contacts/47554685/record/0-3/46002974608"
        },
        {
          "id": "46002974609",
          "properties": {
            "billtoid": "254",
            "createdate": "2025-10-15T14:10:20.298Z",
            "dealname": "TRI STATE TECHNICAL SALES CORP.  -  S1541635",
            "e_sorder_id": "S1541635.0001",
            "hs_lastmodifieddate": "2025-12-20T18:52:06.232Z",
            "hs_object_id": "46002974609",
            "shiptoid": "254"
          },
          "createdAt": "2025-10-15T14:10:20.298Z",
          "updatedAt": "2025-12-20T18:52:06.232Z",
          "archived": false,
          "url": "https://app.hubspot.com/contacts/47554685/record/0-3/46002974609"
        },
        {
          "id": "46002974610",
          "properties": {
            "billtoid": "14707",
            "createdate": "2025-10-15T14:10:20.298Z",
            "dealname": " Boss steel Caribbean limited -  S1535462",
            "e_sorder_id": "S1535462.0001",
            "hs_lastmodifieddate": "2025-12-20T17:55:55.600Z",
            "hs_object_id": "46002974610",
            "shiptoid": "14707"
          },
          "createdAt": "2025-10-15T14:10:20.298Z",
          "updatedAt": "2025-12-20T17:55:55.600Z",
          "archived": false,
          "url": "https://app.hubspot.com/contacts/47554685/record/0-3/46002974610"
        },
        {
          "id": "46002974611",
          "properties": {
            "billtoid": "26611",
            "createdate": "2025-10-15T14:10:20.298Z",
            "dealname": "SNSKAR SYSTEMS INDIA PVT LTD 71 -  S1496559",
            "e_sorder_id": "S1496559.0006",
            "hs_lastmodifieddate": "2025-12-20T17:56:57.976Z",
            "hs_object_id": "46002974611",
            "shiptoid": "26611"
          },
          "createdAt": "2025-10-15T14:10:20.298Z",
          "updatedAt": "2025-12-20T17:56:57.976Z",
          "archived": false,
          "url": "https://app.hubspot.com/contacts/47554685/record/0-3/46002974611"
        }
      ],
      "paging": {
        "next": {
          "after": "10"
        }
      }
    },
    "headers": {
      "date": "Tue, 30 Dec 2025 15:00:44 GMT",
      "content-type": "application/json;charset=utf-8",
      "content-length": "934",
      "connection": "close",
      "cf-ray": "9b626890898264a6-YYZ",
      "cf-cache-status": "DYNAMIC",
      "strict-transport-security": "max-age=31536000; includeSubDomains; preload",
      "vary": "origin, accept-encoding",
      "access-control-allow-credentials": "false",
      "x-content-type-options": "nosniff",
      "x-hubspot-correlation-id": "9dae5789-36de-4efd-99df-627cd702a18f",
      "x-hubspot-ratelimit-daily": "625000",
      "x-hubspot-ratelimit-daily-remaining": "621083",
      "set-cookie": [
        "__cf_bm=P8oVAywIP.ec_7tnk3z8itle2t3Q_rxpWpaTFhzFoCc-1767106844-1.0.1.1-lhT6RXP._tA_yPCBkg3s_GgrcVUB6ehHMYrJwv9Si4mRWZcKyly1LzyYxt9WMxd9H7D3V33i5utrFCXMyhYxt5IXeEuNjWyrs25cnM76K4k; path=/; expires=Tue, 30-Dec-25 15:30:44 GMT; domain=.hubapi.com; HttpOnly; Secure; SameSite=None"
      ],
      "report-to": "{\"endpoints\":[{\"url\":\"https:\\/\\/a.nel.cloudflare.com\\/report\\/v4?s=SAtWuCSi0h%2B4UZfk1DD%2FU8kiM5uorNjTarR2x9g7jk0kqJ11TaVB3EaMvTyjWHmHvrdELHBvfikVh881B3QGRONhvDrdGmOiv53Poer4f9VQYKqFZyZwpLLEbhNTee3W\"}],\"group\":\"cf-nel\",\"max_age\":604800}",
      "nel": "{\"success_fraction\":0.01,\"report_to\":\"cf-nel\",\"max_age\":604800}",
      "server": "cloudflare"
    },
    "statusCode": 200,
    "statusMessage": "OK"
  }
]

Here’s what the data looks like in the table:

Screenshot 2025-12-30 at 10.40.49 AM
Screenshot 2025-12-30 at 10.40.49 AM1920×951 217 KB

Information on your n8n setup

  • n8n version: 2.0.3
  • Database (default: SQLite): sqlite
  • n8n EXECUTIONS_PROCESS setting (default: own, main): default
  • Running n8n via (Docker, npm, n8n cloud, desktop app): npm (local install, i run “n8n start”)
  • Operating system: linux, ubuntu
2

If your hosting it locally, I would guess it’s because of your machine not being able to handle the data.

Do you need to send the set node output back to the IF node? If not, maybe you could just put a blank value in the set node, and that might stop the data cumulating together.

3

Hey, thanks for your reply. I know for sure that it isn’t my machine because I can get up to 600 records when I do batches of 200, but it only goes to 60 records when I do batches of 10. The issue is definitely n8n aggregating all data from previous loops.

I do need to loop the set node back to the if node because otherwise it won’t paginate through the rest of the results.

4

It’s likely doing this because when it sends it back, when it goes through the node outputs the entire database, which would cause it to get slower. (The postgres node might not actually be doing that, but if it is, that might be the problem.

5

Solution

Found a solution! For those in the future:

  • In the end, I was right about the memory bloating. Thanks to OP of this thread.
  • The solution was to loop via webhooks. see below
1 Like
6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.