How can I scrap infinite scroll content?

Help me Build my Workflow
1

I am using HTTP request node for web scrap but the problem is it is not scraping infinite scroll content. How can I do that?

1 Like
2

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
3

Sure,
n8n version

  • n8n version: Version 1.76.3
  • Database : SQLite
  • n8n EXECUTIONS_PROCESS setting (default: own, main): own
  • Running n8n via: npm
  • OS: ubuntu 20.0
4

We’ve created a new category for help with designing workflows, and I’ve moved your question there: Help me Build my Workflow.

5

To scrape a website that uses infinite scroll, you cannot directly use the HTTP request node because the HTTP request system works on a one-request-one-response basis. The response you receive is the initial HTML, but loading content that appears when scrolling requires executing JavaScript.

The solution is to use a headless browser like Puppeteer or Cloudflare Browser Rendering. These tools can simulate a real browser, including executing JavaScript and performing actions like auto-scrolling to load additional content. You can call an API for the headless browser, which will handle the auto-scrolling, load the content, and then send the resulting HTML back to your n8n workflow for further processing. This way, you can effectively scrape dynamically loaded content from infinite scroll websites.