Product Knowledge AI Assistant

Help me Build my Workflow
1

Hi everyone! :waving_hand:

Has anyone here built a Product Knowledge Assistant (or a similar AI agent) for answering employees’ questions about company products?

I’m particularly interested in how well these solutions perform with a medium to large knowledge base.

If you have experience with this, I’d really appreciate your insights:

  • Which vector database did you choose and why?

  • Which LLM are you using?

  • What architecture did you go with ?

  • What challenges did you encounter, and how did you address them?

Thanks in advance for sharing your experience! :folded_hands:

2

Hey @Arman_Shamyan1, here’s a summary of what we’ve learned in practice

DBs: pgvector or Qdrant have worked better for medium to large size knowledge bases. Hands down, they are easier and cheaper when compared to the costs of Pinecone at scale. Avoid Milvus/Weaviate unless you’re working with millions of vectors.

LLM: Both Claude and GPT-4o are great and you are free to use either. However, the quality of information retrieved is dependent more on the model selection.

Architecture: Appeasing RAG is not enough at scale. You’ll want vector + keyword hybrid search to achieve great results and also caught product codes, Hierarchical Reranking (which boosts accuracy), and metadata filtering.

At scale, you are more likely to encounter: stale docs (need to be re-ingested to clear out old data), conflicting information (reranking + recency metadata), vague user questions (add a clarifying step to the search), etc.

You can get in touch and I’ll be happy to help.

2 Likes
3

Hey @ShawnWilliams , thank you for answer !

How can I always refresh data in database automatically ?

4

I would avoid thinking of this as “refresh the vector database” in one big step.

For a product knowledge assistant, the safer pattern is a versioned ingestion workflow:

  1. pull from the source on a schedule or webhook;
  2. split docs into stable chunks;
  3. give each chunk a hash/version;
  4. upsert only changed chunks into the vector DB;
  5. mark missing chunks as stale instead of blindly deleting first;
  6. log the last successful refresh per source.

The important question is: where does your product knowledge live today? Google Drive / Notion / website pages / database / PDFs? The refresh design changes a lot depending on that source.

5

Hi Arman,

I’ve been building these types of Product Knowledge Assistants for several months now at Swapnil AI Labs, and the transition from a ‘working demo’ to a ‘production-ready’ internal tool is all about managing data entropy.

If you are aiming for high accuracy with a medium-to-large product knowledge base, I highly recommend moving away from the default RAG setup and focusing on these three pillars:

  1. Hybrid Search is Mandatory: Do not rely solely on vector search for products. SKU numbers and model codes often get lost in embeddings. Use Hybrid Search (Vector + BM25) to ensure that specific technical identifiers are retrieved with 100% precision.

  2. The Reranking Layer: Retrieval often grabs too much ‘noise.’ I use a Reranker (like Cohere) as a middle step. It scores the top results for actual relevance before they ever reach the LLM, which drastically reduces the chances of the agent answering based on an irrelevant snippet.

  3. Strict Guardrail Prompts: To solve the hallucination issue, your system prompt must explicitly state: 'You are a technical assistant for [Company Name]. You must only answer based on the provided product context. If the answer is not found in the context, inform the user you do not have that information

For the architecture, avoid one giant workflow. Build a modular system: one pipeline for Ingestion/Embedding that watches your product documentation for changes, and a completely separate, low-latency workflow for the Retrieval/Agentic logic.

It’s a powerful build gtting the retrieval accuracy right is 90% of the battle. Happy to dive deeper into how you’re handling the ingestion part if you’ve already started that

6

@Swapnil’s on the right track with isolating it. Here’s the fastest way to cut the problem in half before you touch any settings.

Open the Executions tab in n8n, then trigger the tool from ElevenLabs. One of two things happens, and they point in opposite directions:

  • An execution shows up → the webhook IS finding you, your connection is fine, and the problem is the response ElevenLabs gets back (timeout or empty body). Skip all the URL/header stuff.
  • Nothing shows up → the request never reached n8n, so it’s a URL, method, or activation problem.

If nothing shows up:

  1. Production URL + workflow Active. This is the #1 cause and it exactly matches “works in Postman, not from ElevenLabs.” n8n has two URLs. The /webhook-test/ one only fires while you’re in the editor with “Listen for test event” clicked, and only for a single call. The /webhook/ (Production) URL only works when the workflow toggle is flipped to Active. If Postman worked because you were listening in the editor, ElevenLabs calling a minute later hits nothing. Give ElevenLabs the Production URL and confirm the workflow is toggled Active (top-right, green).
  2. Method. Your node only accepts POST. Make sure the ElevenLabs tool is sending POST to exactly /webhook/elevenlabs-agent, not GET and not a trailing-slash variant.

If an execution DOES show up, the fix is on the response side:

  1. Your flow runs a Gemini agent, then sends a Gmail, then responds. That’s several seconds of work before ElevenLabs gets anything back. Agent tool calls are meant to be near-instant, so it’s very likely timing out on their end and marking the tool as failed even though n8n ran fine. For a voice agent, either respond first and send the email after, or set the Webhook’s response mode to “Immediately” so ElevenLabs gets an instant ack.
  2. Your Respond to Webhook node is returning an empty body. Give it an explicit JSON response the agent can read, like { “status”: “ok”, “message”: “Booking update sent” }. A lot of agent platforms treat a blank response as a failed call.

Do the Executions check first. It tells you which half of this list to even bother with.

7

Good thread already. Hybrid search + reranking + a strict grounding prompt is the right spine, so I won’t repeat it. Three things I’d add from building a handful of these that don’t get mentioned enough:

  1. Chunking is where most of your accuracy is actually won or lost. For product docs specifically, don’t use blind fixed-size character splitting, it shreds spec tables and separates a product from its own attributes. Chunk on document structure instead: by heading/section, keep tables intact, and try to keep one product (or one SKU’s spec block) per chunk. Bad chunking caps your accuracy no matter how good your reranker is.

  2. Pick your embedding model deliberately. Nobody named one. I default to OpenAI text-embedding-3-large for knowledge bases this size, it holds up well on technical/product language and the cost is fine at medium-to-large scale. Whatever you pick, decide it before you ingest, because switching embedding models later means re-embedding the entire base.

  3. Build a tiny eval set before you tune anything. This is the biggest gap I see. People wire up RAG and then adjust chunk size / rerank / top-k purely on vibes. Take 30 to 50 real questions your employees actually ask, write down the source chunk that should answer each, and measure your retrieval hit rate. Now every change (add hybrid, add reranking, change chunk size) is a number that moves instead of a guess. You’ll usually find the accuracy problem is retrieval, not the LLM.

On the refresh question you asked Shawn: Tony’s versioned hash-and-upsert answer is the right pattern. The one thing I’d add is bake a doc version + effective date into every chunk’s metadata at ingestion. That’s what lets you resolve conflicting info by recency, and just as important, it lets the agent cite the source doc and version in its answer. At scale, “here’s the answer, from Product Guide v4, updated March” is what makes employees actually trust it, and it’s your fastest debugging tool when an answer looks wrong.

Happy to go deeper on the chunking or the eval setup, that’s usually where these live or die.

8

The way we handle this, every source document goes through the same path in the same order:

  1. Convert to Markdown first. Whatever the source (PDF, Word, HTML, Notion, Drive export), the first move is normalizing it into clean Markdown. This is the unlock most people miss. Heterogeneous sources have wildly inconsistent structure, and Markdown gives you one consistent surface where headings become real section boundaries and tables survive as tables. That single step is what makes structure-aware chunking actually possible instead of falling back to blind character splitting.

  2. Meta-tag at the document level. Once it’s Markdown, tag the whole document up front with the fields you’ll filter on later: product line, SKU/model, doc type, source, version, effective date, audience/access level. Some of it is free from structure (folder, filename, source), and for the rest you run one cheap LLM pass at ingest to classify the doc type and pull the product line. Doing it before the split means every chunk inherits it automatically.

  3. Then chunk the Markdown on its structure, so each chunk carries the document-level metadata plus its own section context.

Why it’s worth the effort: it lets you metadata-filter before you ever run the vector search. Narrowing to the right product line first, then doing vector + BM25 over just that slice, is often a bigger accuracy jump than adding a reranker, because you deleted the noise before retrieval instead of trying to rank it out afterward. It also powers recency conflict resolution, source citation, and keeping internal-only docs away from the wrong person.

Ties straight into the versioned-ingestion pattern Tony described: the hash-and-upsert handles what changed, the Markdown-plus-metadata handles what each chunk is and whether it belongs in the candidate set at all. Put a version + effective date on every chunk and the agent can cite its source (“from Product Guide v4, updated March”), which is both what makes employees trust the answer and your fastest way to debug a bad one.

Setup checklist before it runs

  1. Credentials (I left the blocks off so the selectors stay clickable): attach Google Drive OAuth2 to the trigger + Download, and OpenAI to the Chat Model + Embeddings. Qdrant credential goes on the Qdrant Insert node.
  2. Folder ID: replace YOUR_DRIVE_FOLDER_ID on the trigger.
  3. MarkItDown: point the URL at your converter and confirm it returns JSON as { “markdown”: “…” }. If yours returns raw text, change the Information Extractor’s text and the Code node’s .json.markdown to .json.data (or whatever field it uses).
  4. Qdrant: set YOUR_QDRANT_HOST and a QDRANT_API_KEY env var. Create the product_knowledge collection at 3072 dims (matches text-embedding-3-large) before the first run.

Two design notes

  • Upsert = delete-then-insert. The HTTP delete clears any existing chunks for that file_id before re-inserting, so you never accumulate stale vectors when a doc changes. The content_hash rides along in metadata. To add the “skip if unchanged” gate we speced, drop an IF before the delete that compares content_hash against your log store, that’s the only piece needing an external table, so I left it out of the importable version.
  • Google Docs caveat: native Docs aren’t binary. If you’re watching a folder of Google Docs (not uploaded PDFs/Word files), add an export format in the Download node’s options, otherwise the download errors. Uploaded files work as-is.