A deliberately sanitised case study — it covers architecture, constraints, and trade-offs, not proprietary data or numbers. For the deeper retrieval write-up see the tech note Building a GenAI chatbot without a vector database.
The problem
The B2B retail platform needed an in-app shopping assistant that could answer product and commerce questions grounded in live catalogue data — at the traffic and latency profile of a major Indian retail platform. The hard constraints shaped every decision: responses had to be grounded (no hallucinated products), fast enough for an in-app chat, cost-bounded at scale, and operable by a small team.
Why no vector database
The reflexive answer to "RAG" is a vector DB. For this domain it wasn't the right trade-off. The catalogue is structured and bounded per query context, and introducing a vector store added operational surface (a service to run, index, shard, and pay for) that the retrieval quality didn't require. Instead, we convert product and commerce data into structured JSON embeddings and look them up with a lightweight semantic match. The result: fewer moving parts, lower latency on the hot path, and a smaller cost and ops footprint — the kind of "remove the component" decision that ages well.
The trade-off is honest: this approach suits a bounded, structured corpus, not an open-ended document store. Naming where a pattern stops applying is part of the design.
Architecture
- Orchestration — LangGraph. The assistant is a state machine, not a prompt: explicit nodes for intent, retrieval, tool-calling into commerce flows, grounding, and guardrails. State machines make an LLM system testable and debuggable instead of a black box.
- Retrieval. Structured-JSON embedding index over catalogue/commerce data; semantic lookup returns grounded context the model must cite.
- Performance. Caching on common intents and fallback paths keep p-tail latency and token cost predictable when traffic spikes.
- Integration. Wired into the existing commerce flows and a hybrid Next.js + Android surface so the chat bridges web and native in-app journeys.
Engineering for confidence
LLM features fail quietly, so I treated correctness as a first-class concern. A systematic Jest strategy reached 95%+ unit-test coverage on the front end. An evaluation suite checked grounding and answer quality so regressions surfaced before release, not in production.
What I'd revisit at scale
If the corpus grew open-ended or multi-tenant, a managed vector store would start to earn its complexity. An offline eval/feedback loop would graduate from a one-off test suite to a continuous quality pipeline. Knowing when a heavier component starts paying for itself is the actual design judgement here.