# Answers With Receipts: Building an Advanced RAG Assistant That Verifies Its Own Citations

Most RAG tutorials stop at "embed your docs, do a similarity search, stuff the results into a prompt." That gets you a demo. It does not get you something you'd trust to answer a paying student's question and cite where the answer came from.

This post walks through an **Advanced RAG Course Assistant** — an app that answers questions about a video course using its `.vtt` transcripts, and cites the exact **module, lesson, and timestamp** for every claim. It routes across three data sources, transforms queries with HyDE, reranks with an LLM, and — the part I care about most — verifies its own citations with a check that *doesn't trust the model at all*.

The stack: **Next.js (App Router) + TypeScript**, **OpenAI** for embeddings and chat, **Qdrant** as the vector store, and an embedded **SQLite** DB standing in for an auth service.

Let's get into it.

* * *

## The mental model: one scoped LLM call per stage

The single most important design decision in this project is that **every stage of the pipeline is its own small, focused LLM call** — not one giant prompt trying to do everything.

Here's the pipeline a single query flows through:

1.  **Input guardrail** — is this on-topic and safe?
    
2.  **Query router** — which data source(s) does this need?
    
3.  **HyDE** — generate a hypothetical answer to search with
    
4.  **Rerank** — score retrieved chunks against the real question
    
5.  **Text-to-SQL** — (if needed) turn account questions into read-only SQL
    
6.  **Generate answer** — final response with forced citation format
    
7.  **Output guardrail** — verify every citation is real (no LLM call)
    

Here's the whole flow, including the corrective-retry loop and the branches:

![](https://cdn.hashnode.com/uploads/covers/62d622ad3060d03288d9bd97/0c523aee-7026-499e-b837-1c1d4527b973.png align="center")

Each of stages 1–6 is a separate prompt in its own file under `lib/prompts/`. Why not one mega-prompt? Because:

*   **Every call pays for every instruction.** A guardrail classification shouldn't carry SQL rules and citation formats it will never use.
    
*   **Instructions bleed into each other.** The model doing reranking gets distracted by citation rules; the guardrail starts trying to *answer*.
    
*   **Different calls want different output shapes** — `{"allowed": bool}`, `{"sources": [...]}`, `{"scores": [...]}`, raw SQL. One prompt can't cleanly demand all of them.
    
*   **You can tune one stage without risking the others.**
    

The tradeoff is more round-trips (5–7 sequential calls per query), which adds latency. For a RAG pipeline where correctness and citation integrity matter more than shaving 500ms, that's the right bet.

> **Key insight:** the LLM never sees your file structure. Splitting prompts into files is *code organization*; the benefit lands at the **call** level — each call gets only the instructions it needs.

* * *

## The three data sources

A query router decides which source(s) a question needs, so branches that aren't relevant never run:

*   **vector-store (Qdrant)** — real transcript chunks. The default for almost every question.
    
*   **auth-db (SQLite)** — student account data (enrollment, lesson progress). The LLM translates the question into SQL (text-to-SQL), which runs **read-only**.
    
*   **s3 (mock)** — lesson attachments / downloadable resources, backed by a static manifest.
    

A single question can hit more than one source — *"have I finished the lesson with the OAuth starter code, and can I get that file?"* routes to both `auth-db` and `s3`.

The router prompt is deliberately opinionated about defaults:

```ts
// lib/prompts/queryRouter.ts
const system = `You are a query router for a course assistant. Decide which data
source(s) are needed to answer the student's question:

- "vector": the question is about course CONTENT — concepts, code, how something
  works. This is the default for almost all questions.
- "auth-db": the question is about the STUDENT'S OWN ACCOUNT — their progress...
- "s3": the question is asking to DOWNLOAD a FILE — attachments, starter code...

Respond ONLY with strict JSON, no markdown:
{"sources": ["vector" | "auth-db" | "s3", ...]}`;
```

Notice a design pattern that recurs throughout: **fail-safe defaults chosen per stage.** The router *fails open* — if parsing breaks, it defaults to `["vector"]` (still try to answer). The input guardrail, as we'll see, *fails closed*.

* * *

## HyDE: search with a fake answer, not the question

Here's a problem that quietly wrecks retrieval quality. A user's question and the transcript that answers it are written completely differently:

*   **Question:** *"How do I save data between app launches?"*
    
*   **Transcript:** *"So we import AsyncStorage, call setItem with a key and a value, and that persists it to disk so when you reopen the app it's still there…"*
    

Embedding the bare question and matching it against answer-style transcript vectors is an **asymmetric** search. Similarity is often weak.

**HyDE (Hypothetical Document Embeddings)** fixes this: instead of embedding the question, we first ask an LLM to *write a plausible answer* — in the voice of the course — and embed **that**. Now both sides of the search look like answers.

```ts
// lib/prompts/queryTransform.ts
const system = `You are an expert instructor for a course on: ${courseTopic}.
Given a student's question, write a short, plausible answer (3-5 sentences)
as if it came directly from the course transcript. Use natural spoken,
instructional language similar to how an instructor explains things out loud.
Do not hedge or say "I don't know" — just write your best hypothetical
explanation. This is used only to improve search retrieval, not shown to the user.`;
```

Two implementation details make this safe:

**1\. The HyDE output is used *only* for embedding — never for scoring.** The vector branch passes the hypothetical answer as the *search* input, but the **original question** goes to the reranker:

```ts
// lib/rag/pipeline.ts
const hydePrompt = buildHydePrompt(query, COURSE_TOPIC);
searchInput = await callLLM(hydePrompt.system, hydePrompt.user);
relevantChunks = await retrieveAndRerank(searchInput, query);
//                                         ↑ HyDE       ↑ real question
```

So a hallucinated HyDE answer can only affect *which candidates get pulled* — not which ones survive. The reranker is the safety net.

**2\. Corrective retry deliberately drops HyDE.** If the HyDE-driven search returns nothing that clears the relevance bar, the loop retries with the **raw question**:

```ts
if (attempt === 0) {
  searchInput = await callLLM(hydePrompt.system, hydePrompt.user); // HyDE
} else {
  searchInput = query; // corrective retry: fall back to the literal question
}
```

The prompt tells the model to write in *spoken, instructional language* and to **never hedge** — because the goal isn't a *correct* answer, it's a *transcript-shaped* one. Vocabulary matters more than accuracy; it's competing on cosine similarity against an actual spoken transcript.

* * *

## Reranking: similarity is a rough filter

Vector search gives you the top-K by cosine similarity, but similarity ≠ relevance. So we retrieve the top 10, then have an LLM **score each candidate 0–10 against the original question** and keep only the top 4 scoring ≥ 5:

```ts
// lib/prompts/rerank.ts
const system = `You score how relevant each transcript excerpt is to a student's
question, on a scale of 0-10. 10 = directly and completely answers the question.
0 = completely unrelated.

Respond ONLY with strict JSON:
{"scores": [{"index": number, "score": number}, ...]}`;
```

If nothing clears the threshold, that's what triggers the corrective retry mentioned above. This is "corrective RAG" in its simplest, most useful form.

* * *

## Text-to-SQL, where the prompt is *not* the security boundary

For account questions, the LLM translates natural language into a SQL query against the SQLite schema:

```ts
// lib/prompts/textToSql.ts
const system = `You translate a student's question into a single READ-ONLY SQL
query (SQLite dialect) against this schema:

${AUTH_DB_SCHEMA}

Rules:
- Only ever generate a SELECT statement. Never INSERT/UPDATE/DELETE/DROP/ALTER.
- Always scope results to the current student: student_id = ${studentId}...`;
```

Critical point: **that prompt is not the security boundary.** It just asks nicely. The real enforcement is in code — `runReadOnlyQuery()` hard-blocks anything that isn't a `SELECT`, and `studentId` is injected server-side so one student can never query another's data. Prompts *request* behaviour; code *enforces* it. Never confuse the two.

* * *

## The output guardrail that doesn't trust the model

This is my favourite part of the whole design.

The generation prompt forces a rigid citation format:

```plaintext
(Module: <moduleName>, Lesson: <lessonName>, Timestamp: <startTimestamp>)
```

Because the format is rigid, we can verify citations with **pure regex + a set lookup — no LLM call.** The output guardrail pulls every citation out of the answer and checks each against the chunks that were *actually retrieved*. If the model cited a timestamp that wasn't in the context, it's flagged and a single repair pass is triggered.

```ts
// lib/prompts/outputGuardrail.ts
const validSet = new Set(
  usedChunks.map((c) => `${c.moduleName}|${c.lessonName}|${c.startTimestamp}`)
);
// ...for each citation found in the answer, check it's in validSet
```

This is the single most important anti-hallucination check in the pipeline, and it costs nothing. It catches the worst failure mode — an invented timestamp — deterministically.

### A real bug this surfaced

While building the UI, I noticed the model was citing **ranges** (`00:10:10-00:10:53`) and sometimes **grouping** multiple citations in one parenthetical separated by `;`. The original guardrail regex was anchored on a single timestamp immediately followed by `)`:

```ts
// BEFORE — misses ranges and grouped citations
/\(Module:\s*(.+?),\s*Lesson:\s*(.+?),\s*Timestamp:\s*(\d{2}:\d{2}:\d{2})\)/g
```

That regex silently matched **zero** citations on range-formatted answers — meaning the guardrail wasn't verifying anything on exactly the answers it was supposed to protect. The fix matches each citation entry independently (not anchored on `)`) and tolerates an optional range, capturing the **start** timestamp for validation:

```ts
// AFTER — matches single OR range timestamps, and grouped entries
/Module:\s*(.+?),\s*Lesson:\s*(.+?),\s*Timestamp:\s*(\d{2}:\d{2}:\d{2})(?:\s*[-–]\s*\d{2}:\d{2}:\d{2})?/g
```

Lesson: a guardrail that silently matches nothing is worse than no guardrail, because it *looks* like it's working. Always test the negative case — I verified a fabricated `09:99:99` still gets flagged.

* * *

## Bringing ingestion into the browser

Ingestion started as a CLI script: parse every `.vtt`, embed in batches of 50, upsert to Qdrant. That's fine for a one-time setup, but I wanted to trigger it from the frontend.

The trick is realising the parser splits cleanly into two halves:

*   **Content parsing** (`parseVTT` + `chunkCues`) works on a raw string — no filesystem needed.
    
*   **Metadata** (`moduleName`, `lessonName`) comes from the *folder structure* — the file's parent folder is the lesson, the grandparent is the module.
    

So I extracted a shared helper both the CLI and the API route use:

```ts
// lib/parser.ts
export function chunksFromVtt(
  raw: string, moduleName: string, lessonName: string, sourceFile: string
): TranscriptChunk[] {
  const cues = parseVTT(raw);
  if (cues.length === 0) return [];
  return chunkCues(cues).map((chunk) => ({ moduleName, lessonName, ...chunk, sourceFile }));
}
```

Then two upload modes in the UI, hitting one `POST /api/ingest`:

*   **Folder / multi-file** — pick whole course folders. Module & lesson are derived automatically from each file's `webkitRelativePath`.
    
*   **Single file** — upload one `.vtt` and tag it manually (a lone file has no folder context).
    

One gotcha worth knowing: **browsers drop folder paths when files hit** `FormData` — the server only sees the basename. So the client sends each file's relative path alongside it, and the server derives module/lesson from that:

```ts
for (const f of folderFiles) {
  form.append("files", f);
  form.append("paths", relPath(f)); // webkitRelativePath, preserved explicitly
}
```

And because `webkitdirectory` only lets you pick *one* folder per dialog, I made selections **accumulate** across picks (deduped by relative path) so you can add multiple courses before ingesting.

* * *

## Nicer citations in the UI

Raw model output dumps citations inline as noisy parentheticals:

> Expo is recommended as the best way to start your mobile journey **(Module: module 1, Lesson: 02\_react-native-vs-expo, Timestamp: 00:10:10-00:10:53; Module: module 1, ...)**.

The UI now **strips those out of the prose** and turns them into numbered superscript chips (¹ ²) that map to a **de-duplicated source list** rendered as cards below the answer — module badge, lesson name, and a monospace timestamp pill. The prose stays readable; the evidence is one glance away.

* * *

## When (and when not) to reach for Zustand

The app grew enough state that it's worth asking: do we need a state-management library?

My honest answer for most of the build was **no**. Zustand solves *shared state across distant components* and *prop-drilling*. If your state is local and shallow — `useState` in the component that owns it — a store is just indirection and a dependency you don't need. Classic YAGNI.

But one real need did emerge: when an ingest was running and you **closed the ingest panel**, the component unmounted and the running status was lost. *That's* a genuine state-sharing problem. The fix is either "lift state up" or a small store.

I went with a Zustand store because the app is set to grow (auth, more pages, persisted history), and putting the **async actions in the store** means an in-progress ingest survives the panel unmounting:

```ts
// lib/store.ts
export const useAppStore = create<AppState>((set, get) => ({
  studentId: 1,
  messages: [],
  chatLoading: false,
  sendQuery: async (query) => { /* fetch + set(...) */ },

  folderFiles: [],
  ingestStatus: { kind: "idle" },
  ingestFolders: async () => { /* build FormData, POST, set(...) */ },
}));
```

Now closing the panel mid-ingest keeps it running; reopening shows live status. Reach for a store when state is genuinely shared — not before.

* * *

## Gotchas from the trenches

A couple of things that cost me time, in case they save you some:

*   `better-sqlite3` **\+ bleeding-edge Node.** On Node 25 (a non-LTS "Current" release) there were no prebuilt binaries, so it fell back to compiling from source and failed with `node-gyp: command not found`. Fix: `npm rebuild better-sqlite3` (npm bundles its own node-gyp) once you have the compiler toolchain. Or just use Node 22 LTS and get prebuilds.
    
*   **One** `COURSE_TOPIC` **constant drives three prompts.** The input guardrail, HyDE, and router all reference it. If it's wrong, the guardrail blocks legitimate questions and HyDE generates the wrong domain's vocabulary — silently tanking retrieval. It's the highest-leverage line in the codebase.
    
*   **Ingestion upserts, it doesn't replace.** Re-ingesting the same transcripts creates new point IDs (UUIDs) rather than overwriting — so you get duplicates. Clear the collection first or dedup on a stable ID if that matters.
    

* * *

## Takeaways

The through-line of this whole design:

*   **One scoped LLM call per stage** beats one mega-prompt — cheaper, sharper, independently tunable.
    
*   **Fail-safe defaults, chosen per stage** — the guardrail fails *closed*, the router fails *open*.
    
*   **HyDE improves recall but never gets to judge** — the reranker uses the real question.
    
*   **The things that actually protect you are enforced in code, not prompts** — read-only SQL and deterministic citation verification.
    
*   **Add libraries (like Zustand) when a real need appears** — not preemptively.
    

RAG isn't "embed and pray." It's a pipeline of small, verifiable steps — and the most important step is the one that assumes the model is lying and checks its work.

* * *

*Built with Next.js, TypeScript, OpenAI, and Qdrant. If you found this useful, drop a comment about which stage you'd tune first.*