# I Couldn't Prove My Prompt Was Better, So I Built a Lab to Measure It

* * *

The assignment was one line: **build a news classifier and improve its consistency using few-shot prompting.**

I read that as two tasks. The first — classify an article as Sports, Politics, Business or Technology — is an afternoon's work. The second one is the one I got stuck on, because of a single word.

*Improve.*

Improve compared to **what**? You can't put "improved" in a README and call it done. Improvement is a comparative claim, and a comparative claim needs a baseline, a metric, and a number. The moment I took that seriously, the shape of the whole project changed: it stopped being a classifier that happens to use few-shot examples, and became a **measurement rig** that ships a classifier as its default configuration.

This post is the whole thing, start to finish — what zero-shot and few-shot prompting actually are, how I built both, how I compared them, the experiment that failed, and what the numbers said once I fixed it.

* * *

## Table of contents

1.  [The brief and the first decision](#the-brief-and-the-first-decision)
    
2.  [What "consistency" actually means](#what-consistency-actually-means)
    
3.  [Zero-shot prompting, explained](#zero-shot-prompting-explained)
    
4.  [Few-shot prompting, explained](#few-shot-prompting-explained)
    
5.  [Making bad output structurally impossible](#making-bad-output-structurally-impossible)
    
6.  [One class, two switches](#one-class-two-switches)
    
7.  [The first experiment failed](#the-first-experiment-failed)
    
8.  [Fixing the experiment](#fixing-the-experiment)
    
9.  [The results](#the-results)
    
10.  [Shipping it: three surfaces and a lab](#shipping-it-three-surfaces-and-a-lab)
     
11.  [Five things that bit me](#five-things-that-bit-me)
     
12.  [Testing an LLM app without an API key](#testing-an-llm-app-without-an-api-key)
     
13.  [What this does not prove](#what-this-does-not-prove)
     
14.  [What I'd take to the next project](#what-id-take-to-the-next-project)
     

* * *

## The brief and the first decision

The classifier takes a short news article and returns exactly one of four categories:

*   **Sports**
    
*   **Politics**
    
*   **Business**
    
*   **Technology**
    

Plus a confidence score and a one-line rationale, so a human can see *why* it chose what it chose.

The first real decision was architectural, and I made it before writing any prompt: **the ability to turn few-shot examples off had to be a first-class feature, not a debugging hack.** If I couldn't run the exact same classifier with the examples disabled, I could never produce a before/after — and the before/after *is* the assignment.

So `NewsClassifier` was born with two independent switches:

```python
class NewsClassifier:
    def __init__(
        self,
        client: openai.OpenAI | None = None,
        model: str = MODEL,
        use_few_shot: bool = True,   # replay the worked examples?
        use_guide: bool = True,      # include the category boundary rules?
    ) -> None:
        ...
```

Two booleans give a 2×2. That grid turned out to be the single most useful thing in the project, and I'll come back to why.

The stack, for anyone who wants to replicate it:

| Piece | Choice |
| --- | --- |
| Language | Python 3.12 |
| Package/venv manager | `uv` |
| Model API | OpenAI, via `responses.parse()` |
| Validation & schema | Pydantic v2 |
| Web layer | FastAPI + plain HTML/CSS/JS (no build step) |
| Tests | pytest — 65 tests, all offline |
| Deploy | Vercel |

* * *

## What "consistency" actually means

Before I could improve consistency, I had to define it in a way a script could compute. I landed on two numbers that must **always be reported together**:

**Agreement** — classify the *same* article N times and see what fraction of the runs land on the most common ("modal") label.

```plaintext
5 runs → Sports, Sports, Sports, Technology, Sports
modal label = Sports (4 of 5)
agreement   = 0.80
```

An agreement of 1.0 means the classifier never wavers on that article. This is the consistency metric.

**Accuracy** — is that modal label the one we expected?

Here's the crucial part, and it took me a failed experiment to internalise it: **agreement on its own is not a quality metric.** A prompt that confidently returns "Business" for every article ever written scores 100% agreement. It is perfectly consistent and perfectly useless. Consistency is only meaningful next to correctness, so nothing in this project — not the CLI, not the API response, not the UI — reports one without the other.

The other thing baked in from the start: **a single run cannot measure consistency.** One run gives you one label. Consistency is a property of *repeated* runs, so every measurement path in the codebase takes a `runs` parameter, defaults it to 3, and caps it at 10.

* * *

## Zero-shot prompting, explained

**Zero-shot prompting** means you describe the task to the model in instructions and ask it to perform the task cold — with no worked examples of the input/output pair. Zero examples. Hence, zero-shot.

It's what most people write first, and it's a genuinely strong baseline with modern models. My zero-shot prompt has three parts.

### Part 1 — the header, which fixes the task

```python
_HEADER = (
    "You classify short news articles into exactly one of four categories: "
    "Sports, Politics, Business, Technology.\n"
)
```

### Part 2 — the category guide, which fixes the boundaries

This is where most of my prompt-writing effort went. It's easy to write "Sports — athletic competition, teams, players." That's fine until an article about a football club's takeover arrives and the model has to decide between Sports and Business.

So every definition states **what it excludes as well as what it includes**:

```python
CATEGORY_GUIDE = """\
Sports — athletic competition, teams, players, matches, transfers, tournaments,
  scores, injuries, coaching. A club's finances or broadcast-rights deal is
  Sports only when the article is framed around the sport; if it is framed as a
  corporate transaction, it is Business.
Politics — government, elections, legislation, courts, diplomacy, policy debate,
  political parties, protests, public officials acting in office. Regulation of
  companies is Politics when the article is about the policy or the lawmakers,
  and Business when it is about the effect on the company.
Business — companies, markets, earnings, mergers and acquisitions, funding
  rounds, jobs, trade, commodities, the economy, consumer prices. A tech
  company's earnings or IPO is Business, not Technology.
Technology — products, software, hardware, AI systems, research breakthroughs,
  cybersecurity, space and scientific engineering. Choose Technology when the
  article is about what the technology does or how it works, rather than about
  the money around it.
"""
```

Every one of those cross-references is a boundary I watched the model get wrong before I wrote it down. Overlapping categories are where classifiers become inconsistent, because the model re-decides the tie fresh on every call unless you tell it how the tie breaks.

### Part 3 — the decision rules

```python
_RULES = """\
Rules:
- Choose the single category the article is *about*, not every topic it mentions.
  Decide by the framing of the lede, not by keywords appearing later.
- Every article gets one of the four categories. There is no "Other" -- pick the
  closest fit and lower the confidence instead.
- Confidence reflects how cleanly the article sits inside one category: use 0.95+
  for unambiguous articles, 0.7-0.9 when it straddles a boundary, below 0.7 when
  the article could reasonably be filed either way.
- Keep the rationale to one sentence naming the decisive signal.
"""
```

Two of those rules exist purely to kill inconsistency:

*   **"Decide by the framing of the lede, not by keywords appearing later"** is the single rule the whole classifier turns on. An article can mention a chip fabrication process nine times and still be a story about a government export ban.
    
*   **"There is no Other"** removes an escape hatch. Without it, a model faced with an ambiguous article invents a fifth category or hedges — and it does so unpredictably.
    

Assembling them is one function, and the `use_guide` switch is right there in the signature:

```python
def build_system_prompt(use_guide: bool = True) -> str:
    parts = [_HEADER]
    if use_guide:
        parts.append("\n" + CATEGORY_GUIDE)
    parts.append("\n" + _RULES)
    return "".join(parts)

SYSTEM_PROMPT = build_system_prompt()
```

### Where zero-shot falls down

Zero-shot is *telling*. And telling has a specific failure mode: the model agrees with your rule in principle and then applies it differently from one call to the next, because your rule is prose and prose has slack in it.

"Framed around the sport" is clear to you and me. To a model reading an article about a £1.2 billion consortium buying a Premier League club, it's a judgement call — and a judgement call made twice can come out twice differently. That wobble *is* the inconsistency the assignment asked me to fix.

* * *

## Few-shot prompting, explained

**Few-shot prompting** means you include a handful of worked examples — actual input/output pairs — in the prompt, so the model infers the task from demonstrations instead of (only) from description. A few examples. Hence, few-shot.

The difference is telling versus **showing**. Instead of describing how to resolve a boundary case, you show the boundary case already resolved, with the reasoning attached.

### Design decision 1: examples as conversation turns, not pasted text

The naive approach is to paste examples into the system prompt as a wall of text. I replayed them as **alternating user/assistant turns** instead:

```python
def _few_shot_turns() -> list[dict[str, str]]:
    turns: list[dict[str, str]] = []
    for article, result in FEW_SHOT_EXAMPLES:
        turns.append({"role": "user", "content": article})
        turns.append({"role": "assistant", "content": result.model_dump_json()})
    return turns
```

So the request the model actually receives looks like this:

```plaintext
system    : header + category guide + rules
user      : "Manchester City edged past Arsenal 2-1 at the Etihad..."
assistant : {"rationale":"A match report centred on the result...","category":"Sports","confidence":0.99}
user      : "The Senate voted 58-42 on Thursday to advance..."
assistant : {"rationale":"Legislative process in the Senate...","category":"Politics","confidence":0.99}
...  (10 pairs total)
user      : <the article you actually want classified>
```

This matters for three reasons:

1.  The model sees the **exact input format** it will receive.
    
2.  It sees the **exact output shape** it should produce, and where it should put it.
    
3.  It sees a conversation it is already participating in, which is a much stronger frame than "here are some examples, now do a different thing."
    

### Design decision 2: four easy examples, six hard ones

I ship **10 examples**. The split is deliberate.

**Four unambiguous ones** — a match report, a Senate vote, a share price drop, a battery research result. These aren't there to teach anything subtle. They fix the label vocabulary and the output shape, one per category, so the classifier never returns "Tech" or "sports" or a two-sentence rationale.

**Six that sit exactly on the boundaries** — these are the ones doing real work:

| Example article | Label | Why it's hard |
| --- | --- | --- |
| Nvidia's quarterly revenue and market cap | **Business** | A chipmaker, but framed as earnings |
| OpenAI ships a new video-understanding capability | **Technology** | Same kind of company, opposite framing |
| Sovereign fund buys 70% of a Premier League club | **Business** | The sport is the asset, not the subject |
| European Parliament votes on platform algorithm rules | **Politics** | Technology is the *object* of the law |
| National coach resigns amid a ministry inquiry | **Sports** | Government is involved; the squad is the story |
| Central bank holds interest rates at 4.5% | **Business** | Policy decision read through economic framing |

Notice the pairs. Nvidia-earnings and OpenAI-capability are the *same type of company* pointed in opposite directions. That contrast teaches "framing, not keywords" far more sharply than either example alone. The same is true of the club takeover (Sports keywords → Business) against the coach resignation (Politics keywords → Sports).

Here's one in full so that you can see the shape:

```python
(
    "Nvidia reported quarterly revenue of $60.9 billion, up 22% from a year "
    "earlier and ahead of analyst estimates. The chipmaker's stock rose 8% "
    "in after-hours trading and its market capitalisation now exceeds $3 trillion.",
    Classification(
        rationale="Framed around earnings, estimates and market cap rather than the technology itself.",
        category=Category.BUSINESS,
        confidence=0.92,
    ),
),
```

Two details in there are intentional:

*   The **rationale is written in the style I want back** — one sentence, naming the decisive signal. Few-shot examples teach tone and format as much as they teach labels.
    
*   The **confidence is 0.92, not 0.99.** If every example is maximally confident, the model learns to always be maximally confident, and the confidence score stops carrying information. The boundary examples deliberately sit between 0.82 and 0.95 so the score stays calibrated to how genuinely ambiguous the article is.
    

### Design decision 3: keep the examples balanced

A skewed example set biases the classifier toward the over-represented category. That's a quiet failure — everything looks fine until your Sports recall is suspiciously high. So there's a test that fails the build if the set gets lopsided, and another asserting every category is still demonstrated at all.

### Design decision 4: the prefix must be byte-identical

The system prompt and the 10 example turns are identical on every single call. That's ~1.3K tokens of prefix that never changes.

This is not just tidiness — it's what makes OpenAI's **automatic prompt caching** kick in. There's no `cache_control` to set; the API caches long identical prefixes on its side. But a prefix that varies between calls gets no discount at all. So the prompt is built **once** in `__init__` and reused:

```python
self._system = build_system_prompt(use_guide)
self._prefix = _few_shot_turns() if use_few_shot else []
```

Build one `NewsClassifier` and reuse it. Rebuilding it per request throws the cache away. There's a test that asserts the prefix is byte-stable across calls, because that property is load-bearing and would rot silently.

* * *

## Making bad output structurally impossible

Before comparing prompts, I closed off an entire category of inconsistency that has nothing to do with prompting.

If the model replies in free text, you have to parse it, and it can return `Tech`, `sports`, `Business/Technology`, `"I'd say Politics, though..."` or a paragraph of prose. Every one of those is an inconsistency, and none of them is a *prompting* problem — they're a *schema* problem.

So the output is a Pydantic model, and the category is an enum:

```python
class Category(str, Enum):
    SPORTS = "Sports"
    POLITICS = "Politics"
    BUSINESS = "Business"
    TECHNOLOGY = "Technology"


class Classification(BaseModel):
    rationale: str = Field(
        description="One short sentence citing the decisive signal in the article."
    )
    category: Category = Field(description="The single best-fitting category.")
    confidence: float = Field(
        ge=0.0, le=1.0, description="Confidence in the chosen category, 0.0-1.0."
    )
```

Passed to the API as `text_format`, this becomes a JSON schema where `category` is an enum of exactly four strings. The model **cannot** return anything else — it's constrained at the API boundary, not cleaned up afterwards.

### The field order is a design decision

Look at the order: `rationale`, **then** `category`, then `confidence`.

That's not alphabetical or aesthetic. Structured outputs are generated in field order, so putting `rationale` first means the model has to **name the decisive signal before it commits to a label**. Put `category` first and the rationale becomes a post-hoc justification for a label it already picked — which is a different, and worse, computation.

It's a one-line change with a real effect on the reasoning, and it's the kind of thing that's invisible unless you're thinking about the generation order.

* * *

## One class, two switches

Here's the classify path, with the ablation switches doing their work:

```python
def classify(self, article: str) -> Classification:
    article = article.strip()
    if not article:
        raise ValueError("Article text is empty.")

    request = {
        "model": self.model,
        "max_output_tokens": MAX_OUTPUT_TOKENS,
        "input": [
            {"role": "system", "content": self._system},   # guide on/off
            *self._prefix,                                  # examples on/off
            {"role": "user", "content": article},
        ],
        "text_format": Classification,
    }
    ...
```

The four configurations live in one file as data, not as scattered `if` statements:

```python
CONFIGS: list[PromptConfig] = [
    PromptConfig("zero_shot", "Zero-shot", True, False,
        "The full system prompt, with no worked examples. The baseline.", "zero-shot"),
    PromptConfig("full", "Few-shot", True, True,
        "The same prompt plus 10 worked examples. What this project ships.", "few-shot"),
    PromptConfig("minimal", "Zero-shot, bare prompt", False, False,
        "Category names only -- no boundary rules, no examples.", "zero-shot"),
    PromptConfig("examples_only", "Few-shot, bare prompt", False, True,
        "No boundary rules; the examples alone have to teach them.", "few-shot"),
]
```

The CLI, the JSON API and the browser UI all read this same list. Adding a fifth configuration means adding a tuple — not touching three call sites. This is the highest-leverage 20 lines in the repo.

* * *

## The first experiment failed

I wrote six held-out boundary articles — deliberately *not* reused from the few-shot set, so the examples would have to generalise rather than be recalled — ran the comparison, and got this:

| Configuration | Agreement | Accuracy |
| --- | --- | --- |
| Zero-shot | 100% | 100% |
| Few-shot | 100% | 100% |

Perfect scores everywhere. Which looks like a triumph and is actually a **broken experiment**.

If both configurations score 100%, the measurement has no headroom. It can't tell me the examples helped; it can't tell me they didn't. It tells me nothing at all, and reporting "few-shot is 100%!" would have been a lie of omission.

Two causes, and it's worth separating them:

1.  **My eval articles were too easy.** They straddled categories in theory, but the framing was clear enough that any competent prompt got them right.
    
2.  **The category guide was already doing the examples' job.** This is the subtle one. `CATEGORY_GUIDE` explicitly states the boundaries — "a tech company's earnings is Business, not Technology" — which is *exactly* what the Nvidia example teaches. My baseline wasn't a naive prompt. It was a carefully engineered zero-shot prompt that had already absorbed the lessons. I'd optimised my control group into the ground.
    

That second point is a genuinely useful lesson about ablation studies: **if your baseline already contains the treatment, you will measure nothing.**

* * *

## Fixing the experiment

Two changes.

### 1\. A harder evaluation set — and it lives in a file, not in code

I wrote 12 articles engineered to remove the ceiling, built around three specific traps:

*   **Lede/body conflicts** — the story opens in one category and the details live in another.
    
*   **Vocabulary borrowed from the wrong category** — dense semiconductor jargon in an article that is fundamentally about a government rule-making action.
    
*   **Genuinely overlapping events** — a broadcast-rights auction is honestly both a sports story and a business story; only the framing decides.
    

Each case carries a `hard_because` field, so a miss reads as *"fell for the semiconductor vocabulary"* rather than just a red cell:

```json
{
  "text": "The Commerce Department added four more chip-design firms to its export blacklist on Wednesday, barring them from selling advanced lithography components abroad without a licence. ... Industry groups warned the measure would cost domestic suppliers an estimated $4 billion in annual revenue.",
  "expected": "Politics",
  "hard_because": "Dense semiconductor vocabulary and a revenue figure, but the article is a government rule-making action.",
  "source": null
}
```

Critically, the eval set is **data, not code**. It loads from JSON or CSV, resolved in this order: an explicit path → the `NEWS_CLASSIFIER_CASES` environment variable → the bundled default. Anyone can point it at real articles:

```bash
NEWS_CLASSIFIER_CASES=my_articles.csv uv run news-classifier-eval --hard
```

And there's one loader decision I'm particularly happy with. **A broken user file must not stop the app from starting — but it must not silently fall back to synthetic data either**, or you'd believe you had measured on real articles when you hadn't:

```python
try:
    return load_cases(), None
except CaseSetError as exc:
    # fall back to the bundled set AND keep the error
    return load_cases(DEFAULT_PATH), str(exc)
```

That error then surfaces in the CLI, in `/api/eval/config`, and as a banner in the UI. The app runs; it just refuses to let you think you measured something you didn't.

### 2\. Ablate the guide, not just the examples

Since the guide was masking the effect, I made it a variable too. That's the second switch, and it turns a single before/after into a 2×2:

|  | **Guide OFF** | **Guide ON** |
| --- | --- | --- |
| **Examples OFF** | `minimal` | `zero_shot` *(the real baseline)* |
| **Examples ON** | `examples_only` | `full` *(shipped)* |

Now I could ask two different questions:

*   `zero_shot` vs `full` → **do examples help a prompt that already explains the boundaries?**
    
*   `minimal` vs `examples_only` → **can examples teach the boundaries on their own?**
    

The second question is the one the first experiment couldn't reach.

* * *

## The results

12 articles × 3 runs, per configuration:

| Configuration | Guide | Few-shot | Agreement | Accuracy |
| --- | --- | --- | --- | --- |
| **Guide + few-shot** *(shipped)* | on | on | **94%** | **83%** |
| Guide only | on | off | 89% | 83% |
| Few-shot only | off | on | 94% | 75% |
| Neither | off | off | 97% | 67% |

Three findings.

### Finding 1 — examples genuinely do teach the boundaries

Compare the bottom two rows. With **no rules in the prompt at all**, adding the examples moves accuracy from **67% → 75%**. Nobody told the model that a chipmaker's earnings report is Business; it inferred that boundary from six worked demonstrations.

That's the few-shot contribution in isolation, and it's the cleanest result in the set.

### Finding 2 — on top of a good prompt, examples buy *consistency*, not accuracy

Compare the top two rows. Accuracy is **83% either way** — the examples didn't make it more correct. But agreement goes **89% → 94%**: the same article gets the same label more often across repeated runs.

That's not a disappointment. That is precisely the thing the assignment asked me to improve. The guide had already taught the model *what the right answer is*; the examples made it **stop wavering** on the way there. Rules tell you where the boundary is. Demonstrations make you land on the same side of it twice in a row.

### Finding 3 — the row that justifies the whole methodology

Look at "Neither": the **highest agreement in the table (97%)** and the **lowest accuracy (67%)**.

The bare prompt is *consistently wrong*. It has settled into a confident, stable, incorrect habit. If I had only measured consistency — the literal ask — the winning configuration would have been the worst classifier in the experiment.

This is the entire argument for reporting both numbers everywhere, and it's why the failed first experiment was worth having. I'd have shipped a number that flattered me and meant nothing.

> **Caveat, stated up front:** 12 articles × 3 runs is a small sample and a few points sit inside the noise. I'd treat the 67% → 75% accuracy gap as solid and the 89% → 94% agreement gain as suggestive — worth re-running at `-n 10` before quoting as settled. Publishing the caveat alongside the result costs nothing and is the difference between a measurement and a marketing number.

* * *

## Shipping it: three surfaces and a lab

The measurement is the interesting part, but it still had to be usable.

### The CLI

```bash
uv run news-classifier --demo                        # four built-in samples
uv run news-classifier "Arsenal beat Spurs 2-0..."   # one article
uv run news-classifier -f article.txt                # from a file
cat article.txt | uv run news-classifier             # from stdin
uv run news-classifier --demo --json                 # JSON Lines
```

```plaintext
Sports       (confidence 0.99)
  why: A match report centred on the result and the scorer.
```

And the evaluation runner:

```bash
uv run news-classifier-eval -n 5             # shipped config
uv run news-classifier-eval -n 5 --compare   # few-shot vs zero-shot
uv run news-classifier-eval --hard           # hard set
uv run news-classifier-eval --ablate         # all four configurations
```

### The JSON API

FastAPI, with Pydantic models on both request and response, and generated docs at `/docs`.

| Endpoint | Purpose |
| --- | --- |
| `POST /api/classify` | One article (max 20,000 chars) |
| `POST /api/classify/batch` | Up to 25 articles in one call |
| `POST /api/compare` | One article, zero-shot vs few-shot, N runs each |
| `POST /api/eval/run` | One hard case × N runs × chosen configurations |
| `GET /api/eval/config` | The configurations and the eval set |
| `GET /api/health` | Model, categories, credential status |

```bash
curl -X POST http://127.0.0.1:8000/api/classify \
  -H 'Content-Type: application/json' \
  -d '{"article": "The Senate voted 58-42 to advance the bill."}'
```

```json
{"category":"Politics","confidence":0.99,
 "rationale":"A floor vote on a bill.","elapsed_ms":812}
```

Errors are real HTTP status codes, not a `200` with an error body: `400` empty article, `401` bad key, `422` failed validation, `429` rate limited, `502` upstream error, `503` no credentials.

### The web UI, and the Consistency Lab

The main page is what you'd expect — paste an article, get a card with the category, a confidence meter and the rationale.

The button I actually care about is **Compare prompts**. It takes *your* article, runs it N times with the examples and N times without, everything else held identical, and leads with the delta:

```plaintext
Few-shot was more consistent: +25 pts agreement

           Configuration   Label     Agreement                Confidence
ZERO-SHOT  Zero-shot       Sports    50%  Sports×2, Tech×2     0.88
FEW-SHOT   Few-shot        Sports    75%  Sports×3, Tech×1     0.81
```

That's a real run on the camera-tracking article from the hard set. The zero-shot prompt was a coin flip on it. The few-shot prompt wasn't.

Two details I'd defend:

*   **When the runs disagree, the UI shows the dissenting label and its reasoning too.** Showing only the winner hides the fact that the model wavered, which is the entire finding.
    
*   **When the delta is zero, it says so plainly.** Unambiguous articles score 100% under both prompts. A tool that only reports wins isn't a measurement tool.
    

Then there's `/eval` — the **Consistency Lab** — which runs the full ablation in the browser as a matrix: one row per article, one column per configuration. Green for correct, red for wrong, and a **striped** cell when a configuration disagreed with *itself* between runs. Hovering shows the full label spread. Each row carries its `hard_because`.

Two implementation notes there. The browser drives the loop **one article per request**, so progress renders live and no single request approaches a serverless timeout. And both tables render *before* you run anything, with cells reading `not run` — so you can see exactly what's about to be compared, and the estimated API call count, before spending a rupee.

* * *

## Five things that bit me

### 1\. The `reasoning` parameter isn't universal

Reasoning models accept `reasoning={"effort": "low"}`. The `gpt-4o` family rejects it with a 400. The obvious fix is a hardcoded list of which models support it — which goes stale every release.

So instead: send it optimistically, catch that **one specific** rejection, drop it, and remember on the instance.

```python
if self._supports_reasoning is False:
    response = self.client.responses.parse(**request)
else:
    try:
        response = self.client.responses.parse(
            **request, reasoning={"effort": REASONING_EFFORT})
        self._supports_reasoning = True
    except openai.BadRequestError as exc:
        if not _is_unsupported_reasoning(exc):
            raise                       # never swallow a real 400
        self._supports_reasoning = False
        response = self.client.responses.parse(**request)
```

Cost: one extra request on the first call, nothing after. The `raise` is the important line — it would be very easy to write a bare `except BadRequestError` here and silently swallow genuine errors like a malformed schema. There's a test that specifically asserts other 400s still propagate.

### 2\. Reasoning tokens eat your output budget

I set `max_output_tokens=256` at first. It's a classification — how many tokens could it need?

The answer is that **reasoning tokens count against** `max_output_tokens`. The model burned the budget thinking and got truncated before emitting any JSON. And the failure is quiet: `response.output_parsed` comes back as `None` rather than raising.

Fix: a 2048 ceiling, and an explicit error rather than a silent `None`:

```python
parsed = response.output_parsed
if parsed is None:
    raise RuntimeError(
        f"Model returned no parsed result (status={response.status!r})."
    )
```

### 3\. Serverless has a hard clock

Vercel caps a request at 60 seconds. One classification is 1–3s, which is fine. A **batch of 25** run sequentially is ~40s and can time out. So batches fan out over a bounded thread pool:

```python
workers = min(BATCH_CONCURRENCY, len(request.articles))
with ThreadPoolExecutor(max_workers=workers) as pool:
    results = list(pool.map(lambda a: _classify_one(classifier, a), request.articles))
```

`pool.map` preserves input order — which matters enormously when you're returning results the caller has to line up with their inputs, and which has its own test. Concurrency defaults to 5 to stay inside rate limits, overridable via `BATCH_CONCURRENCY`. A 25-article batch drops from ~25s to ~5s.

### 4\. Error messages should say what to do

The common failures in an LLM app aren't bugs. They're an unset key, an empty billing account, or a rate limit — and each has a completely different fix. Generic error handling makes all three look identical.

```python
if isinstance(exc, openai.RateLimitError):
    # OpenAI returns 429 for both throttling and an exhausted quota.
    message = getattr(exc, "message", str(exc))
    if "quota" in message.lower() or "billing" in message.lower():
        return ("The OpenAI account is out of quota. Add credit at "
                "https://platform.openai.com/settings/organization/billing")
    return "Rate limited by the OpenAI API. Wait a moment and retry."
```

That 429 disambiguation cost me twenty minutes of confusion the first time I hit it, so it's now permanently encoded.

Related: a missing API key does **not** crash the server. It's caught at startup, reported through `/api/health`, and returned as a `503` on classify — so the UI can say something useful instead of failing blank.

### 5\. The `src/` layout vs. the serverless cold start

The project uses a `src/` layout, so `news_classifier` is only importable once the project itself is installed. If a build installs only *dependencies*, you get `ModuleNotFoundError` at cold start — in production, on a Friday.

```python
_SRC = Path(__file__).parent / "src"
if _SRC.is_dir() and str(_SRC) not in sys.path:
    sys.path.insert(0, str(_SRC))

from news_classifier.api import app
```

Three defensive lines, and a test that imports `app.py` the way Vercel would.

> One more Vercel gotcha worth flagging: don't copy the `excludeFiles` example from Vercel's docs verbatim. It excludes `static/**`, which would strip the entire UI out of your bundle.

* * *

## Testing an LLM app without an API key

**65 tests, 4.4 seconds, no network, no key.** A stub classifier stands in for the model everywhere.

```python
class StubClassifier:
    """Records what it was asked and returns a fixed answer."""
```

The tests I'd point at in a code review aren't the "does it return a value" ones — they're the ones pinning down properties the design depends on:

*   the few-shot prefix is **byte-stable across calls** — the premise of the caching claim
    
*   every example's assistant turn **still parses back into the current schema** (so changing the schema can't silently invalidate the examples)
    
*   **no hard case is reused from the few-shot set** — so the eval measures generalisation, not recall
    
*   concurrent batching **preserves input order**
    
*   the reasoning capability is probed **exactly once** per instance, and other 400s aren't swallowed
    
*   the rationale displayed next to a label **comes from a run that actually produced that label** (with N runs and disagreement, it's easy to show a mismatched pair)
    
*   `app.py` imports **without the package installed** — the Vercel path, tested locally
    

The unifying idea: test the *invariants your design leans on*, not just the happy path. Every one of those would rot silently otherwise.

* * *

## What this does not prove

**Every article in the repo is synthetic.** The 10 few-shot examples, the 12 hard cases, the 6 first-pass eval articles, the 5 UI samples — all written for this project. None is real journalism; names, figures and events are invented.

That was deliberate: an article that reliably puts its lede in one category and its body in another isn't something you find on demand, and the hard set needs exactly that to discriminate. But it costs something, and the cost should be named:

*   **The same person wrote the examples and the eval set**, so they're stylistically related. A test forbids literal reuse, but stylistic correlation isn't something a test can rule out. The examples may be unfairly well-matched to these cases.
    
*   So the measured gain is a **lower-confidence result** than the same measurement on independently sourced articles. It should be reported as "measured on a synthetic benchmark", not as a general claim about news classification.
    

Which is exactly why the eval set is data and not code, and why the `synthetic: true` flag travels with the file, through the API, and into a banner above the results in the UI. The weakness is carried in the payload rather than buried in a README, because the person reading the numbers is the person who needs to know.

Swapping in real articles is one environment variable. The plumbing to do it honestly is already built.

* * *

## What I'd take to the next project

**"Improve X" is a request for two numbers, not one.** The baseline is half the deliverable. If you can't switch your improvement off, you can't claim it works.

**Ablate more than one variable.** My first experiment measured nothing because my baseline had already absorbed the treatment. The 2×2 rescued it — and the extra cell answered a question I hadn't thought to ask.

**A metric alone can be gamed by accident.** 97% agreement at 67% accuracy is a perfectly consistent, perfectly wrong classifier. Consistency needed correctness next to it to mean anything.

**Constrain the output before you tune the prompt.** An enum-typed schema deleted a whole class of inconsistency that no amount of prompt-wording would have fully fixed.

**Field order in a structured output is a reasoning decision.** `rationale` before `category` makes the model think before it commits. One line; real effect.

**A failed experiment that you diagnose is worth more than a passing one you don't understand.** The 100%-everywhere run was the most useful thing that happened in this build.

* * *

*The classifier ships as* `guide + few-shot` *— the configuration that scored 94% agreement and 83% accuracy. But the switch to turn the examples off is still there, still tested, still exposed in the CLI, the API and the UI. Anyone who doubts the claim can re-run the measurement on their own articles, and that's rather the point.*