I’m trying to send sequential HTTP requests and store the results of the calls to Airtable once a day.
The HTTP request I’m sending is to a website with the following format: www.testwebsite.com/id=1000, and I need to send requests from id 1000 to id 1050 (as an example) and then store the results of the calls, the next day I will then do requests from id 1050 to 1100 and so on.
I’m struggling to understand what would be the best way to do something like this.
All help is very much appreciated, thanks in advance.
Hi Spacewalker, I need a little more info to help but I think I can point you in the right direction.
To understand your goals, you are querying a website, and need to cycle through the id’s, which you are doing sequentially. What I’m trying to understand is are you doing
Call 1: www.testwebsite.com/id=1000
Call 2: www.testwebsite.com/id=1001
Call 3: www.testwebsite.com/id=1002
Call 50: www.testwebsite.com/id=1050
then the workflow is complete? Not sure how you would track from day to day, other than setting the function variable check what the new starting number is, but otherwise your flow should look something like this, with something at the beginning to set the function to end at something other than 51.
Hi djangelic, thanks a lot for helping.
What I’m trying to do is to basically get data from a website and save the result to airtable.
Given that the website is organized kind of as a database in which each article is saved on a web page and the web page links are sequential:
I need to set 50 Url’s at a time automatically, and then send those 50 HTTP requests.
The requests don’t have to be sequential, in fact, I can pass all the 50 URLs altogether to the HTTP node, send the requests, and save the results.
So ideally the workflow would be like this:
Find the last Id I finished the search in the last iteration (for example 1250), create the next 50 URLS,
Pass the links to the HTTP node, send the requests, catch the answer, store the results in Airtable.
Store the last id used somewhere so that in the next iteration I can start from there and do the same thing.
I hope this is a bit clearer now, and I appreciate a lot the help.
Yes, that makes sense. Is it possible to call those ID’s from airtable so you can process those individually? For example call airtable and get back:
then it’s trivial to use the split in batches node to break each of these links into a loop to use the http request to call on save to Airtable then move on to the next one.
My question is basically, do you have those ID’s in airtable and are you calling those ID’s into your workflow or are you hoping the workflow generates those ID’s for you? In my experience, it’s easier if the links already exist in a spreadsheet or other database, so you can get all those ID’s and then pass them into your loop to pull the data from the page.
You’ll have to forgive me, I use Trello, not Airtable so I’m not super knowledgeable on how you get data back from there and if it’s even possible to generate those from airtable.