— CASE 07 / PRODUCTION RAG · SILVER TOUCH
RAG CHATBOT
AT SCALE.
A production RAG platform serving thousands of clients — 50K+ queries a month across 10+ chatbot instances. Inherited a 45% MRR stack and rebuilt it into a 95% one.
50KQueries / month
45→95%MRR · FAISS → Milvus
16→8sResponse latency
10+Chatbot instances
— THE REBUILD
A 45% MRR
STACK ISN’T
PRODUCTION.
REBUILD
IT.
Inherited a RAG platform with Llama-2-7b loaded inline in the app and FAISS as the vector store — 45% MRR and a hard latency bottleneck. Re-architected it: the LLM became a standalone vLLM-served API (independent scaling, model swaps without redeploy), and the store moved to Milvus after a head-to-head benchmark on production data.
Benchmarked 6+ embedding models (BGE small/base/large, Stella, SPLADE) and settled on BGE-M3 + a reranker with custom query preprocessing and intent identification. Net result: MRR 45% → 95% and latency 16s → 8s.
- Benchmark. Milvus 95% vs Qdrant 85% vs FAISS 45% MRR on production-scale data (millions of vectors) — Milvus won.
- Serving. Llama-2-7b pulled out of app code into a vLLM API for independent scaling and hot model swaps.
- Quality. BGE-M3 + reranker, intent identification, query preprocessing — latency halved, MRR doubled.
PYTHONFASTAPIVLLMMILVUSFAISSQDRANTBGE-M3LLAMA-2RERANKER
— WANT THE FULL STORY?