Pagination Scraping html

Describe the issue/error/question

I want to scrape results from a page, eg. example.com/search?loc=86383&p=1&pp=50 and loop through the pagination.

p = page, pp = results

For a http request on other sites I get it managed, but here I’m somehow lost how to set p +1.

Information on your n8n setup

  • n8n version: latest
  • Database you’re using: SQLite
  • Running n8n with the execution process [own(default), main]:
  • Running n8n via desktop app

Hi @Kool_Baudrillard, I hope you are well?

I think if you separate the current page from the URL this could work (in this example I’ve also thrown in an IF node to prevent endless looping):

1 Like

Hey @Kool_Baudrillard,
I did something similar to @MutedJam but my IF node checks if more posts are available to continue looping :stuck_out_tongue_winking_eye:.

1 Like

@marcus & @MutedJam Thank you, guys! this is awesome. <3

2 Likes

just one curious questions, is it possible to limit the run? Like 200 runs and then finish the run?

Cheers

Hi @Kool_Baudrillard

Yes, you can use the page and limit it on the if node like this

2 Likes