Automated niche-research pipeline: keyword expansion → live SERP checks → scored blueprint (workflow walkthrough)

Built with n8n
,
1

I’ve been building a pair of research workflows and figured the method might be useful to folks here — it’s a good example of n8n handling a genuinely messy multi-source pipeline.

agwegwegw
agwegwegw1588×616 105 KB

agwegweg
agwegweg1935×450 203 KB

352362462346
3523624623461698×585 129 KB

What it does:

Give it a niche (currently a catalogue of “boring” single-purpose tools — calculators, converters, generators), and it:

  1. Expands seed queries via a keyword-suggestion API (Twinword) — one HTTP request per seed, with batching tuned to the provider’s rate limit (2 req/10s — needed a 7s batchInterval, 5s sat exactly on the boundary and intermittently failed)
  2. Filters suggestions to tool-shaped queries, drops branded/app-shaped keywords (regex sets in Config), canonicalizes aliases (bmi calculator ≈ calculator bmi)
  3. Scores on demand (log volume), monetisation (CPC), build simplicity, cluster strength — with per-category quotas so one pillar can’t dominate the shortlist
  4. Checks live SERP top-10 per shortlisted keyword (Serper) against a curated strong-domain list — presence of calculator.net/nerdwallet/etc. penalizes the score; weak incumbents give it a bonus. Also runs a Chrome Web Store check via Apify for extension competition
  5. Pulls real market comps (Flippa scraper) for revenue anchoring
  6. Sends the scored cluster to OpenAI with response_format: json_schema (strict) to name the site and write per-tool briefs — then a validation node rejects any tool whose primary_keyword doesn’t exist in the real scored data (cheap grounding layer, works well)
  7. Writes everything to Notion with dedup keys (utl:{domain}:{date}, tool:{kw}:{date}) so reruns don’t duplicate rows

Numbers: ~$5/run, ~20 min, produces a ranked blueprint of 40+ tools with real volumes/CPCs/evidence strings.

What I learned building it:

  • continueRegularOutput + retries per node saved the run repeatedly — one dead source shouldn’t kill 45 minutes of upstream work
  • Keep the LLM at the end for synthesis only — everything quantitative is deterministic code. The model can’t hallucinate numbers it never sees.
  • Field-name validation matters more than expected: Twinword returns "search volume" with a literal space, and "paid competition" is an ad-density index, not SEO difficulty — almost shipped a scoring bug on that one
  • Dedup keys in the destination DB (queried before write) make the whole thing safely re-runnable

The other workflow (same architecture) scans Product Hunt/HN/Reddit/AppSumo/app stores for launch traction — same normalize→score→enrich→LLM-brief→Notion shape.

Happy to share the JSON or answer questions about any part — the filtering/scoring code nodes are where most of the interesting decisions live.