More of a question here. I am scraping webpages via the HTTP request node with data from Airtable. I perform a scrape, then upsert the information I do not currently have and if there is a “Next Max ID” (Another page) Then I update that in Airtable and then I wish to scrape that same user ID but with the “Next Max ID” included.
This brings me to my question.
How can I just pass the “UserID” and the “Next Max ID” to the second profile scrape? With the current workflow, I am passing all 100 previous datapoints which is just sending me around and around.
The profile in the example I am scraping has 118 followers. 100 done on the first scape. This leaves me with only 18 for the second scrape but I am still getting 100 returned because according the workflow I am doing another 100 requests.
After this long-winded question, I am open to any and all solutions you may have to manage this!
Apologies for the novel, I am still trying to wrap my head around how N8N workflows get laid out. Coming from Make has been a big learning curve!
- n8n version: Cloud
- Operating system: Windows