How can I loop HTTP response correctly?

Questions
1

Hello guys,

I have an interesting situation, which I am trying to solve without any success.

I am sending HTTP request, which could return anything from 1 to 5000 items in an array form. The problem is that if there are more than 200 items, it returns only 200, and then adds a link to another response with additional parameter: URL$skip=200

Right now, I have a ton of IF conditions, checking if the line with $skip is present, and running another request if it is present, while checking for another $skip=400/600/…/5000 in the outcome.

What I am trying to do is to identify the number of requests, which skip previous values, and iterate it as many times as needed.

I can run another request, which will provide the number of items which would be returned in the array, so I can easily calculate the total number of requests, by dividing the number by 200.

How would I be able to loop the execution until the number of iterations is met?

Screen Shot 2021-07-12 at 1.47.53 PM
Screen Shot 2021-07-12 at 1.47.53 PM3726Ă—1274 421 KB

Right now, I am “recreating” the loop by numerous IF conditions, which isn’t the best way to address this problem.

2

I would do the following:

1 - Get the number of records

2 - In a function node create all URLs using the number of records

3 - Reference the URLs in the HTTP node using expressions

Example workflow 
  {
      "nodes": [
        {
          "parameters": {},
          "name": "Start",
          "type": "n8n-nodes-base.start",
          "typeVersion": 1,
          "position": [
            250,
            300
          ]
        },
        {
          "parameters": {
            "values": {
              "number": [
                {
                  "name": "total",
                  "value": 300
                }
              ]
            },
            "options": {}
          },
          "name": "Set",
          "type": "n8n-nodes-base.set",
          "typeVersion": 1,
          "position": [
            490,
            300
          ]
        },
        {
          "parameters": {
            "functionCode": "const total = items[0].json.total\n\nconst pageSize = 20;\n\nconst urls = [];\n\nconst iterations =  Math.ceil(total/pageSize);\n\nurls.push({ json: { url: `https://yoururl` } })\n\nfor (let i = 0; i < iterations; i++) {\n  urls.push({ json: { url: `https://yoururl?$skip=${pageSize * (i + 1)}` } })\n}\n\nreturn urls;"
          },
          "name": "Function",
          "type": "n8n-nodes-base.function",
          "typeVersion": 1,
          "position": [
            720,
            300
          ]
        },
        {
          "parameters": {
            "url": "={{$node[\"Function\"].json[\"url\"]}}\n",
            "options": {}
          },
          "name": "HTTP Request",
          "type": "n8n-nodes-base.httpRequest",
          "typeVersion": 1,
          "position": [
            970,
            300
          ]
        }
      ],
      "connections": {
        "Start": {
          "main": [
            [
              {
                "node": "Set",
                "type": "main",
                "index": 0
              }
            ]
          ]
        },
        "Set": {
          "main": [
            [
              {
                "node": "Function",
                "type": "main",
                "index": 0
              }
            ]
          ]
        },
        "Function": {
          "main": [
            [
              {
                "node": "HTTP Request",
                "type": "main",
                "index": 0
              }
            ]
          ]
        }
      }
    }
1 Like
3

As always, beautiful solution. Will need to adjust it a little bit, but the idea is golden. Thank you very much for your help!

4

Glad it worked, have fun.