Documentation Q&A (RAG)
Question answering over the FastAPI docs, grounded in the source with a citation on every claim, running fully offline. A 36-question golden set took hit@5 from 75% to 83.3% — and showed that two of three "obvious" retrieval upgrades were neutral or harmful.
- Year
- 2026
- Stack
- Python, LangChain, FAISS, BM25, bge-small-en-v1.5, Qwen2.5-3B, llama.cpp, FastAPI, Docker
Problem
Answer natural-language questions over technical documentation, grounded in the source text with a citation on every claim. Retrieval quality is the ceiling on answer quality — if the right passage never reaches the model, no prompt work recovers it — so it has to be measured, not eyeballed.
Approach
Documents are split with structure-aware chunking, and each chunk is prefixed with a contextual header like [Doc title - p.12]: “the timeout is 30s” is ambiguous alone and unambiguous with two words of context. Chunks are embedded locally with bge-small-en-v1.5 into FAISS, with a BM25 sidecar. Follow-up questions are rewritten into standalone queries, then hybrid retrieval fuses BM25 and dense rankings.
Answers come from a local Qwen2.5-3B via llama.cpp, so it runs offline with no quota — the Gemini free tier capped embeddings at 1,000/day against a corpus that needed 1,546. The model is told to answer only from the numbered passages. It turned out to ignore citation instructions entirely, so citations are computed: each sentence is matched against the retrieved passages, and a sentence with no confident match gets no marker rather than a misleading one.
The eval harness compares a baseline (fixed-width chunks, dense-only) against the tuned config, and ablates the retriever one component at a time. That showed MMR re-ranking was hurting — it cost 5.5 points of hit@5 and 20 of precision, so it is off by default — and that BM25 was neutral here.
Architecture
Eval setup
Eval set — FastAPI documentation, 141 documents. Golden set of 36 questions (eval/questions.yaml), 12 each of exact-identifier, paraphrase and multi-hop questions, each naming the source file(s) that should be retrieved.
| Metric | Method | n | Baseline | Result |
|---|---|---|---|---|
| hit@5 | hit = ≥1 relevant chunk (right file, and required text if set) in the top 5; baseline vs tuned config | 36 questions · 141 docs | 75% | 83.3% |
| MRR@5 | mean reciprocal rank of the first relevant chunk | 36 questions · 141 docs | 0.635 | 0.699 |
| precision@5 | share of the top 5 chunks that are relevant | 36 questions · 141 docs | 42.8% | 46.7% |
Results
hit@5: from 75% to 83.3%
- n
- 36 questions · 141 docs
- method
- hit = ≥1 relevant chunk (right file, and required text if set) in the top 5; baseline vs tuned config
MRR@5: from 0.635 to 0.699
- n
- 36 questions · 141 docs
- method
- mean reciprocal rank of the first relevant chunk
precision@5: from 42.8% to 46.7%
- n
- 36 questions · 141 docs
- method
- share of the top 5 chunks that are relevant
Limitations & next steps
- Every remaining miss is a paraphrase (10/12) or multi-hop (8/12) question; exact identifiers are 12/12. The headroom is in wording that shares nothing with the source.
- Hybrid BM25 adds nothing on this corpus — dense-only scores identically. It stays on as free insurance for corpora with rarer tokens.
- FAISS is rebuilt wholesale on ingest; there's no incremental update path.
- Fully local answers take about 60 s each on CPU; hosted models are a one-line swap.