Capstone: a governed RAG feature
Grounded, access-aware, cited — production-shaped.
Build a small but production-shaped RAG feature over internal-style data: grounded answers with citations, access control at retrieval time, and an eval for retrieval quality. This exercises architecture, retrieval and governance together.
Key ideas
- 1
Done means: hybrid retrieval + grounded generation, citations on every answer, permission-aware retrieval, and a retrieval eval set.
- 2
Route the LLM call through a gateway-style wrapper (logging, cost, PII redaction) even if minimal.
- 3
Add graceful failure: if confidence/retrieval is poor, say 'I don't know' rather than hallucinate.
- 4
Document data residency and what data is indexed vs excluded.
Build steps
- Ingest a small corpus; chunk on structure with metadata (incl. an access tag).
- Implement hybrid retrieval (keyword + vector) + a re-ranker.
- Generate grounded answers WITH citations; refuse when retrieval is weak.
- Enforce access control at retrieval time (filter by the user's permissions).
- Add a retrieval eval (queries → expected sources) and a faithfulness check.
Stretch goals
- Wrap calls in a minimal gateway (log tokens/cost, redact PII).
- Add a cost-per-answer metric.
Watch
Do the work
0/5 · 0%Test yourself
Why enforce access control at RETRIEVAL time?
27 chapters · progress saves automatically