Overview
A Retrieval-Augmented Generation system that doesn't just retrieve and generate — it verifies every step. After retrieval, an LLM grades whether the documents are actually relevant. After generation, another LLM checks whether the answer is grounded in the documents or hallucinated. If retrieval fails, the system rewrites the query and tries again, with bounded retries to prevent infinite loops.
Built as a LangGraph state machine where the developer defines the execution flow. Each node calls the LLM for a specific, scoped task — classify, grade, rewrite, generate, verify — but the transitions between nodes are deterministic and developer-controlled.
Key Highlights
- Developer-Controlled Flow — LangGraph state machine with deterministic transitions. LLM makes focused decisions, not open-ended reasoning.
- Relevance Grading — Dedicated LLM node evaluates whether retrieved documents are actually relevant before generation.
- Hallucination Detection — Post-generation verification gate checks if the answer is grounded in the retrieved context.
- Self-Correcting Retrieval — Query rewriting with bounded retry loops (max 2) when documents fail grading.
- Full Execution Trace — Every node records its decision to a trace log, visible in the Streamlit UI sidebar.
- Graceful Degradation — When retries are exhausted, generates with whatever documents are available rather than failing.
Architecture
How the Graph Works
The graph is a LangGraph StateGraph compiled once at startup. A shared GraphState (TypedDict) flows through every node, carrying the query, documents, answer, trace log, and retry counters. Two conditional edge functions control branching: _after_grader routes to generator, rewriter, or force-generates after max retries; _after_hallucination_check accepts the answer or triggers regeneration.
Each node has a dedicated prompt tuned for its specific task. The LLM temperatures vary by node: 0.0 for Router, Grader, and Hallucination Check (deterministic decisions), 0.3 for Generator (controlled creativity), and 0.5 for Rewriter (encourages creative rephrasing).
6 Nodes, 5 LLM Calls
Router
Classifies the query as requiring vector store retrieval or web search. Explicit prompt: vectorstore is ONLY for internal/corporate documents; everything else routes to web. When in doubt, chooses web search.
Retriever
Fetches top-5 documents from ChromaDB (vector search) or Tavily (web search) based on the Router's decision. Transparently uses the rewritten query if one exists from a retry loop.
Grader
Evaluates retrieved documents as a whole — if at least some are relevant, it passes. Short-circuits to "irrelevant" without an LLM call if no documents were retrieved at all.
Rewriter
Rewrites the query to improve retrieval on retry. Uses higher temperature to encourage creative rephrasing. Builds on the previously rewritten query if one exists, not always the original.
Generator
Synthesizes an answer from retrieved context. Strict prompt: "Answer using ONLY the provided context. Do NOT make up information." If no documents exist, still produces an honest "not enough information" response.
Hallucination Check
Verifies the generated answer is grounded in the documents. Nuanced definition: correctly saying "I don't know" counts as faithful. Only fabricating specific facts counts as hallucination. Skipped entirely if no documents exist.
Self-Correction Loops
The graph implements two distinct self-correction mechanisms, each with bounded retries to prevent infinite loops:
Grader-Rewriter Loop (max 2 retries) — If documents are judged irrelevant, the Rewriter rewrites the query and the system loops back to the Retriever. After two failed attempts, the graph forces generation with whatever documents are available rather than looping indefinitely.
Hallucination-Regeneration Loop (max 1 retry) — If the answer is flagged as hallucinated, the Generator runs once more with the same context. After one retry, the graph accepts the output. This catches obvious hallucinations without endless regeneration.
Execution Trace & Observability
Every node appends its decisions to a trace field in the graph state. The Streamlit UI renders this trace in a sidebar with distinct visual treatment per node — icons, colors, and expandable details. You can see exactly which path the graph took, why the grader passed or failed, whether the rewriter was invoked, and whether the hallucination check caught anything. The trace makes this pipeline fully debuggable, not a black box.
Relationship to the RAG Portfolio
This is the third pattern in a trilogy, each using the same embedding model (nomic-embed-text-v1.5) and same LLM (Bedrock Claude 3 Haiku) to isolate the architectural pattern as the variable:
- VaultRAG (LangChain) — Traditional RAG with dual-mode privacy, multimodal support, hybrid retrieval
- Agentic RAG - ReAct (LlamaIndex) — Autonomous agent with tool selection loop
- This project (LangGraph) — Developer-controlled state machine with quality gates
The trade-off: ReAct gives the LLM full control (flexible but unpredictable). This graph gives the developer full control (predictable but rigid). See Evolution of RAG for the full comparison.