Hi everyone,
I need some advice, please.
My goal is simple:
I want to collect daily stock data from our website and our competitor’s website, including:
Car make, model, year, price
Cars available in each branch
A daily summary comparing both companies
Then send a daily report.
What I tried so far
At first, I tried using the HTTP Request node to access both websites.
But both sites are large, with many pages, and load cars dynamically. The flow quickly became complex and unstable.
So, I tried a different approach:
I added an AI Agent inside n8n.
Through the Agent Tools, I added the HTTP tool and instructed the agent to fetch data from both sites.
It worked at first, but the data was inaccurate — the agent wasn’t returning the correct numbers.
ChatGPT suggested this was due to the websites being complex and heavily using JavaScript.
Next step: Firecrawl
Based on recommendations, I tried Firecrawl.
I tested all Firecrawl nodes:
Extract Data
Scrape URL and Get Content
Scrape Multiple URLs
Unfortunately, I still haven’t succeeded, and I ended up using almost all my monthly Firecrawl credits just testing.
What I need help with
Given this situation, what would you recommend?
Should I continue using Firecrawl?
Should I go back to using HTTP Request in n8n?
Is there a specific approach or pattern that works best for dynamic sites with pagination and “load more” buttons?
My use case is not complicated — I just need to extract car listings from two sites and build a daily comparison report.
I’ve been working on this nonstop for 5 days and feel like I’m not getting any closer.
Any guidance from the community would be greatly appreciated.
Thank you.
First of - your website data probably can be retrieved in more efficient way than scraping, worth checking if there’s any other option to access (git repos, database maybe?)
Using http request will not work well for scraping in many cases, many websites block n8n crawlers. I would probably go with custom crawler with python(so outside of n8n) with IP rotation, or some of the n8n nodes. Or possibly worth looking at selenium: e.g Ultimate Scraper Workflow for n8n | n8n workflow template
Thanks for your reply and advice.
Can I please ask why you recommend trying Docker instead of Firecrawl?
What are the benefits, and which one is more beginner-friendly?
I only started using n8n about a month ago and I’m still learning, so I’m wondering whether Docker would actually be easier for me to work with compared to Firecrawl.
Thanks again for your guidance.