Overview
- RAG performs well on small corpora but degrades as document counts reach tens of thousands because fixed-size token chunking creates many fragmented vectors and makes nearest‑neighbor search noisy.
- Microsoft recommends restoring document structure with semantic chunking, deduplication, and hierarchical parent/child indexing to keep meaning intact and improve retrieval relevance.
- To make latency and cost predictable, teams must treat RAG like a distributed system by partitioning indexes, precomputing embeddings at ingest time, and using caching and reranking.
- Excess raw context hurts model attention so production systems should compress or summarize retrieved evidence and use the LLM mainly as a reasoning layer rather than the primary retriever.
- Some teams are pursuing alternatives such as persistent KV attention caches that keep full documents in model memory for high‑volume, focused workloads while tradeoffs remain for very large or highly dynamic collections.