RAG in Production: 7 Lessons from Shipping Knowledge Assistants

Chunking strategy matters more than the model

Semantic, structure-aware chunking with overlap beat every model upgrade we tested. Tables, headings, and metadata must survive ingestion.

Hybrid search wins

Combining vector similarity with keyword (BM25) retrieval and a reranker cut hallucinated answers by more than half versus pure vector search.

Evaluate retrieval separately from generation

Measure recall of the right passages before you judge answer quality. Most 'the model is wrong' bugs are actually 'the model never saw the right document'.

Why this matters in 2026

Most AI features fail in production not because the model is weak but because the surrounding engineering — retrieval, evaluation, observability, cost control — is missing. This guide covers the practices we apply on every LLM project so features are reliable, measurable, and affordable at scale.

Step-by-step: how to approach it

1. Start with a golden dataset. 100–500 real, anonymised inputs with expert-approved outputs. Nothing ships without passing it.
2. Separate retrieval and generation. Measure whether the right context was found before judging the answer.
3. Version prompts like code. Prompts, tools, and configs live in git; CI runs the eval suite on every change.
4. Instrument everything. Log prompts, retrieved chunks, tool calls, latency, and token cost with request IDs.
5. Set budgets. Per-request cost and p95 latency ceilings are release gates, not dashboards.

At a glance

Common mistakes to avoid

• Testing on a handful of hand-picked examples and calling it evaluation.
• Upgrading models without re-running evals — regressions are common.
• Chunking documents by fixed character count and losing tables and headings.
• No caching, so identical requests are paid for repeatedly.
• Treating hallucination as a model problem when it is usually a retrieval problem.

How FreshCodes can help

We are an AI-first development company that designs, builds, and operates AI agents, LLM-powered applications, and AI-native web and Flutter mobile products. Every engagement starts with a free discovery call where we map your process, assess your data, and give you a fixed-scope plan with a timeline and estimate — including projected running costs. From there we ship weekly, measure against an evaluation set, and support your product after launch.

• AI Agent Development — autonomous agents and copilots for support, sales, operations, and HR
• AI & ML Integration — add LLM features to your existing product without a rebuild
• Data Engineering & Analytics — AI-ready pipelines, warehouses, and conversational analytics
• AI Mobile App Development and AI-Powered Web Development — Flutter, React, Node.js, Laravel