Hey everyone,
I built a lead gen pipeline with two parallel AI branches per lead and I’m stuck at ~60% success rate. I’d love to get to 80%+ but can’t figure out where exactly things break.
Branch 1 — CEO Extraction:
AI Agent (GPT-4o-mini) uses Firecrawl map + scrape tools to find the managing director from German company imprint pages. Problems:
Agent sometimes outputs NOT_FOUND even when the imprint data is right there
Random timeouts on scrape calls (sites load fine in browser)
Can’t distinguish “genuinely no imprint” from “something broke silently”
Branch 2 — Screenshot Scoring:
Firecrawl HTTP takes full-page screenshots, Gemini Vision scores how badly the site needs a redesign. Problems:
Screenshots come back empty/null randomly on reachable sites
Some screenshots capture cookie banners or loading states instead of actual content (despite waitFor: 10000)
Both branches use continueRegularOutput on error, so failed items pass through silently with empty data — making debugging a nightmare.
What I need help with:
Better error visibility (why did it fail, not just that it failed)
Architecture tips to push reliability up
Anyone solved similar Firecrawl screenshot inconsistencies?
Stripped-down workflow JSON attached (just the two AI branches + merge logic). Sticky note has context.
Thanks What is the error message (if any)?
this is a solid use case — two parallel branches with ai agents + web scraping for lead quality. the reliability issues you’re hitting (agent false negatives, firecrawl timeouts, empty screenshots) are pretty common at scale. the big wins usually come from: (1) splitting the agent into two steps (scrape first, then extract vs trying to do both), (2) adding retry logic for firecrawl failures, and (3) proper error visibility instead of continueRegularOutput swallowing everything. happy to help debug the architecture if you want.
I’d add explicit status fields on both branches before the merge, like success, failure_stage, http_status, tool_used, and raw_response_present. Right now continueRegularOutput is hiding whether the issue is Firecrawl, the model, or the merge logic. Even a simple error taxonomy per item would make it much easier to see where your missing 40% is actually coming from.
Hey Benjamin, thanks for your time! The agent splitting makes total sense. Quick question though: how would you actually wire that up? My gut says two agents running ping-pong to achieve the Goal of a cheap and effective extractor isn’t really possible in n8n since execution is linear… so would you go the sub-workflow route, or something else entirely? Also curious if a stronger model would Help in this Case.
Hey Tamy, thanks for the quick reply this was super helpful and thats a good Point. One quick thing: if I add explicit status fields and route failures to a separate error branch, how do I make sure the leads still come back together in the correct order at the end? I’m worried about ending up with a jumbled mess where success leads and error leads lose their original sequence. Is there a specific merge pattern or node setup you’d recommend in n8n to keep everything tied together by lead ID?
no sub-workflow needed — two sequential agent nodes in the same flow is cleaner. agent 1 scrapes and returns raw markdown, agent 2 receives that as context and only does the extraction. easier to debug and you can tune each step independently.
on models: the split usually makes a bigger difference than upgrading. a focused “here’s the markdown, find the ceo” task is much easier for mini than combining scrape + extract in one shot — try the architecture change first.
What is the error message (if any)?