Answers With Receipts: Building an Advanced RAG Assistant That Verifies Its Own Citations

Most RAG tutorials stop at "embed your docs, do a similarity search, stuff the results into a prompt." That gets you a demo. It does not get you something you'd trust to answer a paying student's question and cite where the answer came from.

This post walks through an Advanced RAG Course Assistant — an app that answers questions about a video course using its .vtt transcripts, and cites the exact module, lesson, and timestamp for every claim. It routes across three data sources, transforms queries with HyDE, reranks with an LLM, and — the part I care about most — verifies its own citations with a check that doesn't trust the model at all.

The stack: Next.js (App Router) + TypeScript, OpenAI for embeddings and chat, Qdrant as the vector store, and an embedded SQLite DB standing in for an auth service.

Let's get into it.

---

The mental model: one scoped LLM call per stage

The single most important design decision in this project is that every stage of the pipeline is its own small, focused LLM call — not one giant prompt trying to do everything.

Here's the pipeline a single query flows through:

1. Input guardrail — is this on-topic and safe?

2. Query router — which data source(s) does this need?

3. HyDE — generate a hypothetical answer to search with

4. Rerank — score retrieved chunks against the real question

5. Text-to-SQL — (if needed) turn account questions into read-only SQL

6. Generate answer — final response with forced citation format

7. Output guardrail — verify every citation is real (no LLM call)

Here's the whole flow, including the corrective-retry loop and the branches:

Each of stages 1–6 is a separate prompt in its own file under lib/prompts/. Why not one mega-prompt? Because:

• Every call pays for every instruction. A guardrail classification shouldn't carry SQL rules and citation formats it will never use.

• Instructions bleed into each other. The model doing reranking gets distracted by citation rules; the guardrail starts trying to answer.

• Different calls want different output shapes — {"allowed": bool}, {"sources": [...]}, {"scores": [...]}, raw SQL. One prompt can't cleanly demand all of them.

• You can tune one stage without risking the others.

The tradeoff is more round-trips (5–7 sequential calls per query), which adds latency. For a RAG pipeline where correctness and citation integrity matter more than shaving 500ms, that's the right bet.

Key insight: the LLM never sees your file structure. Splitting prompts into files is code organization; the benefit lands at the call level — each call gets only the instructions it needs.

---

The three data sources

A query router decides which source(s) a question needs, so branches that aren't relevant never run:

• vector-store (Qdrant) — real transcript chunks. The default for almost every question.

• auth-db (SQLite) — student account data (enrollment, lesson progress). The LLM translates the question into SQL (text-to-SQL), which runs read-only.

• s3 (mock) — lesson attachments / downloadable resources, backed by a static manifest.

A single question can hit more than one source — "have I finished the lesson with the OAuth starter code, and can I get that file?" routes to both auth-db and s3.

The router prompt is deliberately opinionated about defaults:

// lib/prompts/queryRouter.ts
const system = `You are a query router for a course assistant. Decide which data
source(s) are needed to answer the student's question:

- "vector": the question is about course CONTENT — concepts, code, how something
  works. This is the default for almost all questions.
- "auth-db": the question is about the STUDENT'S OWN ACCOUNT — their progress...
- "s3": the question is asking to DOWNLOAD a FILE — attachments, starter code...

Respond ONLY with strict JSON, no markdown:
{"sources": ["vector" | "auth-db" | "s3", ...]}`;

Notice a design pattern that recurs throughout: fail-safe defaults chosen per stage. The router fails open — if parsing breaks, it defaults to ["vector"] (still try to answer). The input guardrail, as we'll see, fails closed.

---

HyDE: search with a fake answer, not the question

Here's a problem that quietly wrecks retrieval quality. A user's question and the transcript that answers it are written completely differently:

• Question: "How do I save data between app launches?"

• Transcript: "So we import AsyncStorage, call setItem with a key and a value, and that persists it to disk so when you reopen the app it's still there…"

Embedding the bare question and matching it against answer-style transcript vectors is an asymmetric search. Similarity is often weak.

HyDE (Hypothetical Document Embeddings) fixes this: instead of embedding the question, we first ask an LLM to write a plausible answer — in the voice of the course — and embed that. Now both sides of the search look like answers.

// lib/prompts/queryTransform.ts
const system = `You are an expert instructor for a course on: ${courseTopic}.
Given a student's question, write a short, plausible answer (3-5 sentences)
as if it came directly from the course transcript. Use natural spoken,
instructional language similar to how an instructor explains things out loud.
Do not hedge or say "I don't know" — just write your best hypothetical
explanation. This is used only to improve search retrieval, not shown to the user.`;

Two implementation details make this safe:

1. The HyDE output is used only for embedding — never for scoring. The vector branch passes the hypothetical answer as the search input, but the original question goes to the reranker:

// lib/rag/pipeline.ts
const hydePrompt = buildHydePrompt(query, COURSE_TOPIC);
searchInput = await callLLM(hydePrompt.system, hydePrompt.user);
relevantChunks = await retrieveAndRerank(searchInput, query);
//                                         ↑ HyDE       ↑ real question

So a hallucinated HyDE answer can only affect which candidates get pulled — not which ones survive. The reranker is the safety net.

2. Corrective retry deliberately drops HyDE. If the HyDE-driven search returns nothing that clears the relevance bar, the loop retries with the raw question:

if (attempt === 0) {
  searchInput = await callLLM(hydePrompt.system, hydePrompt.user); // HyDE
} else {
  searchInput = query; // corrective retry: fall back to the literal question
}

The prompt tells the model to write in spoken, instructional language and to never hedge — because the goal isn't a correct answer, it's a transcript-shaped one. Vocabulary matters more than accuracy; it's competing on cosine similarity against an actual spoken transcript.

---

Reranking: similarity is a rough filter

Vector search gives you the top-K by cosine similarity, but similarity ≠ relevance. So we retrieve the top 10, then have an LLM score each candidate 0–10 against the original question and keep only the top 4 scoring ≥ 5:

// lib/prompts/rerank.ts
const system = `You score how relevant each transcript excerpt is to a student's
question, on a scale of 0-10. 10 = directly and completely answers the question.
0 = completely unrelated.

Respond ONLY with strict JSON:
{"scores": [{"index": number, "score": number}, ...]}`;

If nothing clears the threshold, that's what triggers the corrective retry mentioned above. This is "corrective RAG" in its simplest, most useful form.

---

Text-to-SQL, where the prompt is not the security boundary

For account questions, the LLM translates natural language into a SQL query against the SQLite schema:

// lib/prompts/textToSql.ts
const system = `You translate a student's question into a single READ-ONLY SQL
query (SQLite dialect) against this schema:

${AUTH_DB_SCHEMA}

Rules:
- Only ever generate a SELECT statement. Never INSERT/UPDATE/DELETE/DROP/ALTER.
- Always scope results to the current student: student_id = ${studentId}...`;

Critical point: that prompt is not the security boundary. It just asks nicely. The real enforcement is in code — runReadOnlyQuery() hard-blocks anything that isn't a SELECT, and studentId is injected server-side so one student can never query another's data. Prompts request behaviour; code enforces it. Never confuse the two.

---

The output guardrail that doesn't trust the model

This is my favourite part of the whole design.

The generation prompt forces a rigid citation format:

(Module: <moduleName>, Lesson: <lessonName>, Timestamp: <startTimestamp>)

Because the format is rigid, we can verify citations with pure regex + a set lookup — no LLM call. The output guardrail pulls every citation out of the answer and checks each against the chunks that were actually retrieved. If the model cited a timestamp that wasn't in the context, it's flagged and a single repair pass is triggered.

// lib/prompts/outputGuardrail.ts
const validSet = new Set(
  usedChunks.map((c) => `${c.moduleName}|${c.lessonName}|${c.startTimestamp}`)
);
// ...for each citation found in the answer, check it's in validSet

This is the single most important anti-hallucination check in the pipeline, and it costs nothing. It catches the worst failure mode — an invented timestamp — deterministically.

A real bug this surfaced

While building the UI, I noticed the model was citing ranges (00:10:10-00:10:53) and sometimes grouping multiple citations in one parenthetical separated by ;. The original guardrail regex was anchored on a single timestamp immediately followed by ):

// BEFORE — misses ranges and grouped citations
/\(Module:\s*(.+?),\s*Lesson:\s*(.+?),\s*Timestamp:\s*(\d{2}:\d{2}:\d{2})\)/g

That regex silently matched zero citations on range-formatted answers — meaning the guardrail wasn't verifying anything on exactly the answers it was supposed to protect. The fix matches each citation entry independently (not anchored on )) and tolerates an optional range, capturing the start timestamp for validation:

// AFTER — matches single OR range timestamps, and grouped entries
/Module:\s*(.+?),\s*Lesson:\s*(.+?),\s*Timestamp:\s*(\d{2}:\d{2}:\d{2})(?:\s*[-–]\s*\d{2}:\d{2}:\d{2})?/g

Lesson: a guardrail that silently matches nothing is worse than no guardrail, because it looks like it's working. Always test the negative case — I verified a fabricated 09:99:99 still gets flagged.

---

Bringing ingestion into the browser

Ingestion started as a CLI script: parse every .vtt, embed in batches of 50, upsert to Qdrant. That's fine for a one-time setup, but I wanted to trigger it from the frontend.

The trick is realising the parser splits cleanly into two halves:

• Content parsing (parseVTT + chunkCues) works on a raw string — no filesystem needed.

• Metadata (moduleName, lessonName) comes from the folder structure — the file's parent folder is the lesson, the grandparent is the module.

So I extracted a shared helper both the CLI and the API route use:

// lib/parser.ts
export function chunksFromVtt(
  raw: string, moduleName: string, lessonName: string, sourceFile: string
): TranscriptChunk[] {
  const cues = parseVTT(raw);
  if (cues.length === 0) return [];
  return chunkCues(cues).map((chunk) => ({ moduleName, lessonName, ...chunk, sourceFile }));
}

Then two upload modes in the UI, hitting one POST /api/ingest:

• Folder / multi-file — pick whole course folders. Module & lesson are derived automatically from each file's webkitRelativePath.

• Single file — upload one .vtt and tag it manually (a lone file has no folder context).

One gotcha worth knowing: browsers drop folder paths when files hit FormData — the server only sees the basename. So the client sends each file's relative path alongside it, and the server derives module/lesson from that:

for (const f of folderFiles) {
  form.append("files", f);
  form.append("paths", relPath(f)); // webkitRelativePath, preserved explicitly
}

And because webkitdirectory only lets you pick one folder per dialog, I made selections accumulate across picks (deduped by relative path) so you can add multiple courses before ingesting.

---

Nicer citations in the UI

Raw model output dumps citations inline as noisy parentheticals:

Expo is recommended as the best way to start your mobile journey (Module: module 1, Lesson: 02_react-native-vs-expo, Timestamp: 00:10:10-00:10:53; Module: module 1, ...).

The UI now strips those out of the prose and turns them into numbered superscript chips (¹ ²) that map to a de-duplicated source list rendered as cards below the answer — module badge, lesson name, and a monospace timestamp pill. The prose stays readable; the evidence is one glance away.

---

When (and when not) to reach for Zustand

The app grew enough state that it's worth asking: do we need a state-management library?

My honest answer for most of the build was no. Zustand solves shared state across distant components and prop-drilling. If your state is local and shallow — useState in the component that owns it — a store is just indirection and a dependency you don't need. Classic YAGNI.

But one real need did emerge: when an ingest was running and you closed the ingest panel, the component unmounted and the running status was lost. That's a genuine state-sharing problem. The fix is either "lift state up" or a small store.

I went with a Zustand store because the app is set to grow (auth, more pages, persisted history), and putting the async actions in the store means an in-progress ingest survives the panel unmounting:

// lib/store.ts
export const useAppStore = create<AppState>((set, get) => ({
  studentId: 1,
  messages: [],
  chatLoading: false,
  sendQuery: async (query) => { /* fetch + set(...) */ },

  folderFiles: [],
  ingestStatus: { kind: "idle" },
  ingestFolders: async () => { /* build FormData, POST, set(...) */ },
}));

Now closing the panel mid-ingest keeps it running; reopening shows live status. Reach for a store when state is genuinely shared — not before.

---

Gotchas from the trenches

A couple of things that cost me time, in case they save you some:

• better-sqlite3 + bleeding-edge Node. On Node 25 (a non-LTS "Current" release) there were no prebuilt binaries, so it fell back to compiling from source and failed with node-gyp: command not found. Fix: npm rebuild better-sqlite3 (npm bundles its own node-gyp) once you have the compiler toolchain. Or just use Node 22 LTS and get prebuilds.

• One COURSE_TOPIC constant drives three prompts. The input guardrail, HyDE, and router all reference it. If it's wrong, the guardrail blocks legitimate questions and HyDE generates the wrong domain's vocabulary — silently tanking retrieval. It's the highest-leverage line in the codebase.

• Ingestion upserts, it doesn't replace. Re-ingesting the same transcripts creates new point IDs (UUIDs) rather than overwriting — so you get duplicates. Clear the collection first or dedup on a stable ID if that matters.

---

Takeaways

The through-line of this whole design:

• One scoped LLM call per stage beats one mega-prompt — cheaper, sharper, independently tunable.

• Fail-safe defaults, chosen per stage — the guardrail fails closed, the router fails open.

• HyDE improves recall but never gets to judge — the reranker uses the real question.

• The things that actually protect you are enforced in code, not prompts — read-only SQL and deterministic citation verification.

• Add libraries (like Zustand) when a real need appears — not preemptively.

RAG isn't "embed and pray." It's a pipeline of small, verifiable steps — and the most important step is the one that assumes the model is lying and checks its work.

---

Built with Next.js, TypeScript, OpenAI, and Qdrant. If you found this useful, drop a comment about which stage you'd tune first.