Multi Scrape Workflow

Questions
1

Describe the problem/error/question.

More of a question here. I am scraping webpages via the HTTP request node with data from Airtable. I perform a scrape, then upsert the information I do not currently have and if there is a “Next Max ID” (Another page) Then I update that in Airtable and then I wish to scrape that same user ID but with the “Next Max ID” included.
This brings me to my question.
How can I just pass the “UserID” and the “Next Max ID” to the second profile scrape? With the current workflow, I am passing all 100 previous datapoints which is just sending me around and around.
The profile in the example I am scraping has 118 followers. 100 done on the first scape. This leaves me with only 18 for the second scrape but I am still getting 100 returned because according the workflow I am doing another 100 requests.

After this long-winded question, I am open to any and all solutions you may have to manage this!

Apologies for the novel, I am still trying to wrap my head around how N8N workflows get laid out. Coming from Make has been a big learning curve!

What is the error message (if any)?

Please share your workflow

Information on your n8n setup

  • n8n version: Cloud
  • Operating system: Windows
3

How can I just pass the “UserID” and the “Next Max ID” to the second profile scrape? With the current workflow, I am passing all 100 previous datapoints which is just sending me around and around.

Hi @Zac_Magee, it seems to me your Set node would only pass on two values:

image
image474×650 26 KB

Is this not working as expected for you?

1 Like
4

Hey there!

No this isnt what happened, I was a little shocked when I checked the following API and saw I had 100 requests sent lol.

It passed through the correct values I assigned in the SET node but 100 of them from earlier in the workflow lol. Rather than just taking the “MAX ID” and “MAX ID PK” and sending just those 2 values to the last HTTP node, it is sending the 100 scraped “Followers” all the way through the workflow.

A better way to ask the question would be, how can I stop sending the 100 Followers through the workflow to the second HTTP node and only sending the params inside the set node?
image

I hope I explained that well.