Retrieval & Evaluation
Retrieval is only half the problem. The other half is proving the answer was grounded.
- Role
- Sole engineer
- Year
- 2026
- Type
- RAG system + evaluation harness
- Stack
- Python
- LangChain
- OpenAI API
- FAISS
- ChromaDB
- RAGAS
- Streamlit
- Gradio
Architecture
Building the pipeline took days. Learning to measure whether it was actually grounded is what changed how I build.
Overview
A retrieval-augmented generation system covering the full path: documents in, chunked, embedded, indexed, retrieved, answered — then evaluated with RAGAS.
Both FAISS and ChromaDB sit behind the retrieval interface, which made it possible to compare index behaviour on the same corpus and the same questions.
The problem
A RAG pipeline that returns fluent, plausible answers feels finished. Without evaluation there is no way to know whether an answer came from the retrieved context or from the model’s own weights — and those two cases are indistinguishable to a reader.
That is the gap the evaluation harness exists to close.
Document to answer
Each stage has parameters that materially change output quality, which is exactly why the measurement layer matters.
- 01ChunkSplit source documents with attention to size and overlap — the setting that changed retrieval quality most.
- 02EmbedEncode chunks into vectors for semantic rather than lexical matching.
- 03IndexStore in FAISS or ChromaDB behind a shared retrieval interface.
- 04RetrieveSelect the chunks most relevant to the question at query time.
- 05AnswerGenerate grounded in that retrieved context, with sources tracked.
- 06EvaluateScore the answer with RAGAS across four dimensions.
What RAGAS measures
Four metrics, each catching a different failure. Together they separate a retrieval problem from a generation problem — which tells you which half of the pipeline to fix.
Faithfulness
Is the answer actually supported by the retrieved context, or did the model fill a gap from memory?
Answer Relevancy
Does the answer address the question that was asked, rather than an adjacent one?
Context Precision
Of the chunks retrieved, how many were genuinely useful? Low precision means noisy retrieval.
Context Recall
Did retrieval find everything needed to answer, or did the right chunk never surface?
Technical decisions
Two vector stores behind one interface
FAISS and ChromaDB are interchangeable at the retrieval boundary, so comparing them was a config change rather than a rewrite.
Chunking treated as the primary variable
Chunk size and overlap moved answer quality more than prompt wording did. Making it easy to change was the point.
Evaluation in the loop, not at the end
The harness runs against a fixed question set, so a change to chunking or retrieval can be judged rather than guessed at.
Interface
Captures from the Streamlit and Gradio surfaces.
What I learned
Evaluation changes the way you build. Once there is a number attached to faithfulness, arguments about prompt wording end and experiments begin.
I also learned that most of what looks like a model problem is a retrieval problem. When context precision is poor, no amount of prompt engineering rescues the answer.