Sequential Http requests

Hello Everyone!

I’m trying to send sequential HTTP requests and store the results of the calls to Airtable once a day.
The HTTP request I’m sending is to a website with the following format: www.testwebsite.com/id=1000, and I need to send requests from id 1000 to id 1050 (as an example) and then store the results of the calls, the next day I will then do requests from id 1050 to 1100 and so on.
I’m struggling to understand what would be the best way to do something like this.

All help is very much appreciated, thanks in advance.

Hi Spacewalker, I need a little more info to help but I think I can point you in the right direction.

To understand your goals, you are querying a website, and need to cycle through the id’s, which you are doing sequentially. What I’m trying to understand is are you doing

Call 1: www.testwebsite.com/id=1000
Call 2: www.testwebsite.com/id=1001
Call 3: www.testwebsite.com/id=1002
etc
Call 50: www.testwebsite.com/id=1050

then the workflow is complete? Not sure how you would track from day to day, other than setting the function variable check what the new starting number is, but otherwise your flow should look something like this, with something at the beginning to set the function to end at something other than 51.

{
  "nodes": [
    {
      "parameters": {},
      "name": "Start",
      "type": "n8n-nodes-base.start",
      "typeVersion": 1,
      "position": [
        200,
        460
      ]
    },
    {
      "parameters": {
        "conditions": {
          "boolean": [
            {
              "value1": "={{$node[\"SplitInBatches of 1\"].context[\"noItemsLeft\"]}}",
              "value2": true
            }
          ]
        }
      },
      "name": "Check if any calls left",
      "type": "n8n-nodes-base.if",
      "typeVersion": 1,
      "position": [
        1560,
        320
      ]
    },
    {
      "parameters": {
        "functionCode": "const newItems = [];\n\nfor (let i=1;i<51;i++) {\n  newItems.push({\n    json: {\n      loop: i\n    }\n  });\n}\n\nreturn newItems;"
      },
      "name": "Generate 50 items to loop",
      "type": "n8n-nodes-base.function",
      "typeVersion": 1,
      "position": [
        460,
        460
      ]
    },
    {
      "parameters": {
        "batchSize": 1,
        "options": {}
      },
      "name": "SplitInBatches of 1",
      "type": "n8n-nodes-base.splitInBatches",
      "typeVersion": 1,
      "position": [
        680,
        560
      ]
    },
    {
      "parameters": {
        "url": "=https://www.testwebsite.com/id={{$node[\"SplitInBatches of 1\"].json[\"loop\"]}}",
        "options": {}
      },
      "name": "Request Website",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 1,
      "position": [
        900,
        560
      ]
    },
    {
      "parameters": {
        "options": {}
      },
      "name": "Set Values you want to transfer ",
      "type": "n8n-nodes-base.set",
      "typeVersion": 1,
      "position": [
        1120,
        560
      ]
    },
    {
      "parameters": {},
      "name": "Send to Airtable",
      "type": "n8n-nodes-base.airtable",
      "typeVersion": 1,
      "position": [
        1340,
        560
      ]
    }
  ],
  "connections": {
    "Start": {
      "main": [
        [
          {
            "node": "Generate 50 items to loop",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Check if any calls left": {
      "main": [
        null,
        [
          {
            "node": "SplitInBatches of 1",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Generate 50 items to loop": {
      "main": [
        [
          {
            "node": "SplitInBatches of 1",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "SplitInBatches of 1": {
      "main": [
        [
          {
            "node": "Request Website",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Request Website": {
      "main": [
        [
          {
            "node": "Set Values you want to transfer ",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Set Values you want to transfer ": {
      "main": [
        [
          {
            "node": "Send to Airtable",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Send to Airtable": {
      "main": [
        [
          {
            "node": "Check if any calls left",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}
2 Likes

Hi djangelic, thanks a lot for helping.
What I’m trying to do is to basically get data from a website and save the result to airtable.
Given that the website is organized kind of as a database in which each article is saved on a web page and the web page links are sequential:

www.test.com/id=1200
www.test.com/id=1201
www.test.com/id=1202
www.test.com/id=1204
.
.
.
www.test.com/id=1250

I need to set 50 Url’s at a time automatically, and then send those 50 HTTP requests.
The requests don’t have to be sequential, in fact, I can pass all the 50 URLs altogether to the HTTP node, send the requests, and save the results.

So ideally the workflow would be like this:
Find the last Id I finished the search in the last iteration (for example 1250), create the next 50 URLS,
www.test.com/id=1251
.
.
www.test.com/id=1300
Pass the links to the HTTP node, send the requests, catch the answer, store the results in Airtable.
Store the last id used somewhere so that in the next iteration I can start from there and do the same thing.
I hope this is a bit clearer now, and I appreciate a lot the help.

Yes, that makes sense. Is it possible to call those ID’s from airtable so you can process those individually? For example call airtable and get back:
www.test.com/id=1200
www.test.com/id=1201
www.test.com/id=1202
www.test.com/id=1204
then it’s trivial to use the split in batches node to break each of these links into a loop to use the http request to call on save to Airtable then move on to the next one.

My question is basically, do you have those ID’s in airtable and are you calling those ID’s into your workflow or are you hoping the workflow generates those ID’s for you? In my experience, it’s easier if the links already exist in a spreadsheet or other database, so you can get all those ID’s and then pass them into your loop to pull the data from the page.

You’ll have to forgive me, I use Trello, not Airtable so I’m not super knowledgeable on how you get data back from there and if it’s even possible to generate those from airtable.