Built a RAG system in n8n + Supabase.......some things that actually tripped me up

Built with n8n
1

I built an internal knowledge copilot: ask a question in Telegram, get a cited answer from your company docs. If it doesn’t know, it says so.

Stack**:** n8n on Railway, Supabase pgvector, Gemini embeddings (768d), Groq LLaMA 3.3 70B, Google Drive as the source.

Three workflows:

  • Ingestion — chunks PDFs, embeds, upserts to Supabase. Skips unchanged files by diffing Drive modifiedTime vs stored timestamp.
  • Query — hybrid retrieval (pgvector + full-text, fused with RRF) → LLM re-rank → confidence gate → cited answer in Telegram
  • Eval — weekly run against a fixed Q&A set, scores groundedness, sends a digest of top unanswered questions

Things that actually cost me time:

Gemini cosine similarity for relevant chunks lands around 0.51–0.65, not 0.8+. I had my threshold at 0.72 and it was refusing good answers the whole time.

After any connection surgery in n8n, always re-validate for “Node not reachable” — removing one connection can silently drop both if two ports share the same target.

Anyone else using pgvector with n8n? Curious how your similarity scores look with other embedding models.

1 Like
2

The embedding similarity threshold issue is a real gotcha - Gemini’s text-embedding-004 returns compressed scores compared to OpenAI’s embeddings, so calibrating to 0.5-0.6 is the right move. One thing I’d add to your Eval workflow: track which questions consistently fall below threshold vs which return wrong answers - they need different fixes (retrieval tuning vs prompt/re-ranking tuning). The weekly digest is solid, but a per-failure-type breakdown makes it much faster to target improvements.

1 Like