Describe the problem/error/question
I am trying to scrape a web site with pagination. I use a loop over items to visit each “page” (from the pagination) and grab a list of links from it, then I use a second loop over items to visit each of the links scraped from the first page and save the text into MongoDB.
The workflow correctly grabs the pagination links and successfully runs the first loop over items once, which successfully grabs the links from that page, opens each link (via the second loop over items) and saves the text of the webpage to mongodb.
But then it fails to execute the rest of the iterations from the first loop.
What is the error message (if any)? No Error
Please share your workflow
I am trying to clean up a bunch of legal judgments prior to uploading to vector store for RAG - so eventually they will go into supabase not mongdb - I just want to check I can get the desired data quality before I chunk it.
Information on your n8n setup
- n8n version: Latest
- Database (default: SQLite): SQLite
- Running n8n via (Docker, npm, n8n cloud, desktop app): NPM
- Operating system: MacOS