I Couldn't Prove My Prompt Was Better, So I Built a Lab to Measure It
The assignment was one line: build a news classifier and improve its consistency using few-shot prompting.
I read that as two tasks. The first — classify an article as Sports, Politics, Business or Technology — is an afternoon's work. The second one is the one I got stuck on, because of a single word.
Improve.
Improve compared to what? You can't put "improved" in a README and call it done. Improvement is a comparative claim, and a comparative claim needs a baseline, a metric, and a number. The moment I took that seriously, the shape of the whole project changed: it stopped being a classifier that happens to use few-shot examples, and became a measurement rig that ships a classifier as its default configuration.
This post is the whole thing, start to finish — what zero-shot and few-shot prompting actually are, how I built both, how I compared them, the experiment that failed, and what the numbers said once I fixed it.
Table of contents
The brief and the first decision
The classifier takes a short news article and returns exactly one of four categories:
Sports
Politics
Business
Technology
Plus a confidence score and a one-line rationale, so a human can see why it chose what it chose.
The first real decision was architectural, and I made it before writing any prompt: the ability to turn few-shot examples off had to be a first-class feature, not a debugging hack. If I couldn't run the exact same classifier with the examples disabled, I could never produce a before/after — and the before/after is the assignment.
So NewsClassifier was born with two independent switches:
class NewsClassifier:
def __init__(
self,
client: openai.OpenAI | None = None,
model: str = MODEL,
use_few_shot: bool = True, # replay the worked examples?
use_guide: bool = True, # include the category boundary rules?
) -> None:
...
Two booleans give a 2×2. That grid turned out to be the single most useful thing in the project, and I'll come back to why.
The stack, for anyone who wants to replicate it:
| Piece | Choice |
|---|---|
| Language | Python 3.12 |
| Package/venv manager | uv |
| Model API | OpenAI, via responses.parse() |
| Validation & schema | Pydantic v2 |
| Web layer | FastAPI + plain HTML/CSS/JS (no build step) |
| Tests | pytest — 65 tests, all offline |
| Deploy | Vercel |
What "consistency" actually means
Before I could improve consistency, I had to define it in a way a script could compute. I landed on two numbers that must always be reported together:
Agreement — classify the same article N times and see what fraction of the runs land on the most common ("modal") label.
5 runs → Sports, Sports, Sports, Technology, Sports
modal label = Sports (4 of 5)
agreement = 0.80
An agreement of 1.0 means the classifier never wavers on that article. This is the consistency metric.
Accuracy — is that modal label the one we expected?
Here's the crucial part, and it took me a failed experiment to internalise it: agreement on its own is not a quality metric. A prompt that confidently returns "Business" for every article ever written scores 100% agreement. It is perfectly consistent and perfectly useless. Consistency is only meaningful next to correctness, so nothing in this project — not the CLI, not the API response, not the UI — reports one without the other.
The other thing baked in from the start: a single run cannot measure consistency. One run gives you one label. Consistency is a property of repeated runs, so every measurement path in the codebase takes a runs parameter, defaults it to 3, and caps it at 10.
Zero-shot prompting, explained
Zero-shot prompting means you describe the task to the model in instructions and ask it to perform the task cold — with no worked examples of the input/output pair. Zero examples. Hence, zero-shot.
It's what most people write first, and it's a genuinely strong baseline with modern models. My zero-shot prompt has three parts.
Part 1 — the header, which fixes the task
_HEADER = (
"You classify short news articles into exactly one of four categories: "
"Sports, Politics, Business, Technology.\n"
)
Part 2 — the category guide, which fixes the boundaries
This is where most of my prompt-writing effort went. It's easy to write "Sports — athletic competition, teams, players." That's fine until an article about a football club's takeover arrives and the model has to decide between Sports and Business.
So every definition states what it excludes as well as what it includes:
CATEGORY_GUIDE = """\
Sports — athletic competition, teams, players, matches, transfers, tournaments,
scores, injuries, coaching. A club's finances or broadcast-rights deal is
Sports only when the article is framed around the sport; if it is framed as a
corporate transaction, it is Business.
Politics — government, elections, legislation, courts, diplomacy, policy debate,
political parties, protests, public officials acting in office. Regulation of
companies is Politics when the article is about the policy or the lawmakers,
and Business when it is about the effect on the company.
Business — companies, markets, earnings, mergers and acquisitions, funding
rounds, jobs, trade, commodities, the economy, consumer prices. A tech
company's earnings or IPO is Business, not Technology.
Technology — products, software, hardware, AI systems, research breakthroughs,
cybersecurity, space and scientific engineering. Choose Technology when the
article is about what the technology does or how it works, rather than about
the money around it.
"""
Every one of those cross-references is a boundary I watched the model get wrong before I wrote it down. Overlapping categories are where classifiers become inconsistent, because the model re-decides the tie fresh on every call unless you tell it how the tie breaks.
Part 3 — the decision rules
_RULES = """\
Rules:
- Choose the single category the article is *about*, not every topic it mentions.
Decide by the framing of the lede, not by keywords appearing later.
- Every article gets one of the four categories. There is no "Other" -- pick the
closest fit and lower the confidence instead.
- Confidence reflects how cleanly the article sits inside one category: use 0.95+
for unambiguous articles, 0.7-0.9 when it straddles a boundary, below 0.7 when
the article could reasonably be filed either way.
- Keep the rationale to one sentence naming the decisive signal.
"""
Two of those rules exist purely to kill inconsistency:
"Decide by the framing of the lede, not by keywords appearing later" is the single rule the whole classifier turns on. An article can mention a chip fabrication process nine times and still be a story about a government export ban.
"There is no Other" removes an escape hatch. Without it, a model faced with an ambiguous article invents a fifth category or hedges — and it does so unpredictably.
Assembling them is one function, and the use_guide switch is right there in the signature:
def build_system_prompt(use_guide: bool = True) -> str:
parts = [_HEADER]
if use_guide:
parts.append("\n" + CATEGORY_GUIDE)
parts.append("\n" + _RULES)
return "".join(parts)
SYSTEM_PROMPT = build_system_prompt()
Where zero-shot falls down
Zero-shot is telling. And telling has a specific failure mode: the model agrees with your rule in principle and then applies it differently from one call to the next, because your rule is prose and prose has slack in it.
"Framed around the sport" is clear to you and me. To a model reading an article about a £1.2 billion consortium buying a Premier League club, it's a judgement call — and a judgement call made twice can come out twice differently. That wobble is the inconsistency the assignment asked me to fix.
Few-shot prompting, explained
Few-shot prompting means you include a handful of worked examples — actual input/output pairs — in the prompt, so the model infers the task from demonstrations instead of (only) from description. A few examples. Hence, few-shot.
The difference is telling versus showing. Instead of describing how to resolve a boundary case, you show the boundary case already resolved, with the reasoning attached.
Design decision 1: examples as conversation turns, not pasted text
The naive approach is to paste examples into the system prompt as a wall of text. I replayed them as alternating user/assistant turns instead:
def _few_shot_turns() -> list[dict[str, str]]:
turns: list[dict[str, str]] = []
for article, result in FEW_SHOT_EXAMPLES:
turns.append({"role": "user", "content": article})
turns.append({"role": "assistant", "content": result.model_dump_json()})
return turns
So the request the model actually receives looks like this:
system : header + category guide + rules
user : "Manchester City edged past Arsenal 2-1 at the Etihad..."
assistant : {"rationale":"A match report centred on the result...","category":"Sports","confidence":0.99}
user : "The Senate voted 58-42 on Thursday to advance..."
assistant : {"rationale":"Legislative process in the Senate...","category":"Politics","confidence":0.99}
... (10 pairs total)
user : <the article you actually want classified>
This matters for three reasons:
The model sees the exact input format it will receive.
It sees the exact output shape it should produce, and where it should put it.
It sees a conversation it is already participating in, which is a much stronger frame than "here are some examples, now do a different thing."
Design decision 2: four easy examples, six hard ones
I ship 10 examples. The split is deliberate.
Four unambiguous ones — a match report, a Senate vote, a share price drop, a battery research result. These aren't there to teach anything subtle. They fix the label vocabulary and the output shape, one per category, so the classifier never returns "Tech" or "sports" or a two-sentence rationale.
Six that sit exactly on the boundaries — these are the ones doing real work:
| Example article | Label | Why it's hard |
|---|---|---|
| Nvidia's quarterly revenue and market cap | Business | A chipmaker, but framed as earnings |
| OpenAI ships a new video-understanding capability | Technology | Same kind of company, opposite framing |
| Sovereign fund buys 70% of a Premier League club | Business | The sport is the asset, not the subject |
| European Parliament votes on platform algorithm rules | Politics | Technology is the object of the law |
| National coach resigns amid a ministry inquiry | Sports | Government is involved; the squad is the story |
| Central bank holds interest rates at 4.5% | Business | Policy decision read through economic framing |
Notice the pairs. Nvidia-earnings and OpenAI-capability are the same type of company pointed in opposite directions. That contrast teaches "framing, not keywords" far more sharply than either example alone. The same is true of the club takeover (Sports keywords → Business) against the coach resignation (Politics keywords → Sports).
Here's one in full so that you can see the shape:
(
"Nvidia reported quarterly revenue of $60.9 billion, up 22% from a year "
"earlier and ahead of analyst estimates. The chipmaker's stock rose 8% "
"in after-hours trading and its market capitalisation now exceeds $3 trillion.",
Classification(
rationale="Framed around earnings, estimates and market cap rather than the technology itself.",
category=Category.BUSINESS,
confidence=0.92,
),
),
Two details in there are intentional:
The rationale is written in the style I want back — one sentence, naming the decisive signal. Few-shot examples teach tone and format as much as they teach labels.
The confidence is 0.92, not 0.99. If every example is maximally confident, the model learns to always be maximally confident, and the confidence score stops carrying information. The boundary examples deliberately sit between 0.82 and 0.95 so the score stays calibrated to how genuinely ambiguous the article is.
Design decision 3: keep the examples balanced
A skewed example set biases the classifier toward the over-represented category. That's a quiet failure — everything looks fine until your Sports recall is suspiciously high. So there's a test that fails the build if the set gets lopsided, and another asserting every category is still demonstrated at all.
Design decision 4: the prefix must be byte-identical
The system prompt and the 10 example turns are identical on every single call. That's ~1.3K tokens of prefix that never changes.
This is not just tidiness — it's what makes OpenAI's automatic prompt caching kick in. There's no cache_control to set; the API caches long identical prefixes on its side. But a prefix that varies between calls gets no discount at all. So the prompt is built once in __init__ and reused:
self._system = build_system_prompt(use_guide)
self._prefix = _few_shot_turns() if use_few_shot else []
Build one NewsClassifier and reuse it. Rebuilding it per request throws the cache away. There's a test that asserts the prefix is byte-stable across calls, because that property is load-bearing and would rot silently.
Making bad output structurally impossible
Before comparing prompts, I closed off an entire category of inconsistency that has nothing to do with prompting.
If the model replies in free text, you have to parse it, and it can return Tech, sports, Business/Technology, "I'd say Politics, though..." or a paragraph of prose. Every one of those is an inconsistency, and none of them is a prompting problem — they're a schema problem.
So the output is a Pydantic model, and the category is an enum:
class Category(str, Enum):
SPORTS = "Sports"
POLITICS = "Politics"
BUSINESS = "Business"
TECHNOLOGY = "Technology"
class Classification(BaseModel):
rationale: str = Field(
description="One short sentence citing the decisive signal in the article."
)
category: Category = Field(description="The single best-fitting category.")
confidence: float = Field(
ge=0.0, le=1.0, description="Confidence in the chosen category, 0.0-1.0."
)
Passed to the API as text_format, this becomes a JSON schema where category is an enum of exactly four strings. The model cannot return anything else — it's constrained at the API boundary, not cleaned up afterwards.
The field order is a design decision
Look at the order: rationale, then category, then confidence.
That's not alphabetical or aesthetic. Structured outputs are generated in field order, so putting rationale first means the model has to name the decisive signal before it commits to a label. Put category first and the rationale becomes a post-hoc justification for a label it already picked — which is a different, and worse, computation.
It's a one-line change with a real effect on the reasoning, and it's the kind of thing that's invisible unless you're thinking about the generation order.
One class, two switches
Here's the classify path, with the ablation switches doing their work:
def classify(self, article: str) -> Classification:
article = article.strip()
if not article:
raise ValueError("Article text is empty.")
request = {
"model": self.model,
"max_output_tokens": MAX_OUTPUT_TOKENS,
"input": [
{"role": "system", "content": self._system}, # guide on/off
*self._prefix, # examples on/off
{"role": "user", "content": article},
],
"text_format": Classification,
}
...
The four configurations live in one file as data, not as scattered if statements:
CONFIGS: list[PromptConfig] = [
PromptConfig("zero_shot", "Zero-shot", True, False,
"The full system prompt, with no worked examples. The baseline.", "zero-shot"),
PromptConfig("full", "Few-shot", True, True,
"The same prompt plus 10 worked examples. What this project ships.", "few-shot"),
PromptConfig("minimal", "Zero-shot, bare prompt", False, False,
"Category names only -- no boundary rules, no examples.", "zero-shot"),
PromptConfig("examples_only", "Few-shot, bare prompt", False, True,
"No boundary rules; the examples alone have to teach them.", "few-shot"),
]
The CLI, the JSON API and the browser UI all read this same list. Adding a fifth configuration means adding a tuple — not touching three call sites. This is the highest-leverage 20 lines in the repo.
The first experiment failed
I wrote six held-out boundary articles — deliberately not reused from the few-shot set, so the examples would have to generalise rather than be recalled — ran the comparison, and got this:
| Configuration | Agreement | Accuracy |
|---|---|---|
| Zero-shot | 100% | 100% |
| Few-shot | 100% | 100% |
Perfect scores everywhere. Which looks like a triumph and is actually a broken experiment.
If both configurations score 100%, the measurement has no headroom. It can't tell me the examples helped; it can't tell me they didn't. It tells me nothing at all, and reporting "few-shot is 100%!" would have been a lie of omission.
Two causes, and it's worth separating them:
My eval articles were too easy. They straddled categories in theory, but the framing was clear enough that any competent prompt got them right.
The category guide was already doing the examples' job. This is the subtle one.
CATEGORY_GUIDEexplicitly states the boundaries — "a tech company's earnings is Business, not Technology" — which is exactly what the Nvidia example teaches. My baseline wasn't a naive prompt. It was a carefully engineered zero-shot prompt that had already absorbed the lessons. I'd optimised my control group into the ground.
That second point is a genuinely useful lesson about ablation studies: if your baseline already contains the treatment, you will measure nothing.
Fixing the experiment
Two changes.
1. A harder evaluation set — and it lives in a file, not in code
I wrote 12 articles engineered to remove the ceiling, built around three specific traps:
Lede/body conflicts — the story opens in one category and the details live in another.
Vocabulary borrowed from the wrong category — dense semiconductor jargon in an article that is fundamentally about a government rule-making action.
Genuinely overlapping events — a broadcast-rights auction is honestly both a sports story and a business story; only the framing decides.
Each case carries a hard_because field, so a miss reads as "fell for the semiconductor vocabulary" rather than just a red cell:
{
"text": "The Commerce Department added four more chip-design firms to its export blacklist on Wednesday, barring them from selling advanced lithography components abroad without a licence. ... Industry groups warned the measure would cost domestic suppliers an estimated $4 billion in annual revenue.",
"expected": "Politics",
"hard_because": "Dense semiconductor vocabulary and a revenue figure, but the article is a government rule-making action.",
"source": null
}
Critically, the eval set is data, not code. It loads from JSON or CSV, resolved in this order: an explicit path → the NEWS_CLASSIFIER_CASES environment variable → the bundled default. Anyone can point it at real articles:
NEWS_CLASSIFIER_CASES=my_articles.csv uv run news-classifier-eval --hard
And there's one loader decision I'm particularly happy with. A broken user file must not stop the app from starting — but it must not silently fall back to synthetic data either, or you'd believe you had measured on real articles when you hadn't:
try:
return load_cases(), None
except CaseSetError as exc:
# fall back to the bundled set AND keep the error
return load_cases(DEFAULT_PATH), str(exc)
That error then surfaces in the CLI, in /api/eval/config, and as a banner in the UI. The app runs; it just refuses to let you think you measured something you didn't.
2. Ablate the guide, not just the examples
Since the guide was masking the effect, I made it a variable too. That's the second switch, and it turns a single before/after into a 2×2:
| Guide OFF | Guide ON | |
|---|---|---|
| Examples OFF | minimal |
zero_shot (the real baseline) |
| Examples ON | examples_only |
full (shipped) |
Now I could ask two different questions:
zero_shotvsfull→ do examples help a prompt that already explains the boundaries?minimalvsexamples_only→ can examples teach the boundaries on their own?
The second question is the one the first experiment couldn't reach.
The results
12 articles × 3 runs, per configuration:
| Configuration | Guide | Few-shot | Agreement | Accuracy |
|---|---|---|---|---|
| Guide + few-shot (shipped) | on | on | 94% | 83% |
| Guide only | on | off | 89% | 83% |
| Few-shot only | off | on | 94% | 75% |
| Neither | off | off | 97% | 67% |
Three findings.
Finding 1 — examples genuinely do teach the boundaries
Compare the bottom two rows. With no rules in the prompt at all, adding the examples moves accuracy from 67% → 75%. Nobody told the model that a chipmaker's earnings report is Business; it inferred that boundary from six worked demonstrations.
That's the few-shot contribution in isolation, and it's the cleanest result in the set.
Finding 2 — on top of a good prompt, examples buy consistency, not accuracy
Compare the top two rows. Accuracy is 83% either way — the examples didn't make it more correct. But agreement goes 89% → 94%: the same article gets the same label more often across repeated runs.
That's not a disappointment. That is precisely the thing the assignment asked me to improve. The guide had already taught the model what the right answer is; the examples made it stop wavering on the way there. Rules tell you where the boundary is. Demonstrations make you land on the same side of it twice in a row.
Finding 3 — the row that justifies the whole methodology
Look at "Neither": the highest agreement in the table (97%) and the lowest accuracy (67%).
The bare prompt is consistently wrong. It has settled into a confident, stable, incorrect habit. If I had only measured consistency — the literal ask — the winning configuration would have been the worst classifier in the experiment.
This is the entire argument for reporting both numbers everywhere, and it's why the failed first experiment was worth having. I'd have shipped a number that flattered me and meant nothing.
Caveat, stated up front: 12 articles × 3 runs is a small sample and a few points sit inside the noise. I'd treat the 67% → 75% accuracy gap as solid and the 89% → 94% agreement gain as suggestive — worth re-running at
-n 10before quoting as settled. Publishing the caveat alongside the result costs nothing and is the difference between a measurement and a marketing number.
Shipping it: three surfaces and a lab
The measurement is the interesting part, but it still had to be usable.
The CLI
uv run news-classifier --demo # four built-in samples
uv run news-classifier "Arsenal beat Spurs 2-0..." # one article
uv run news-classifier -f article.txt # from a file
cat article.txt | uv run news-classifier # from stdin
uv run news-classifier --demo --json # JSON Lines
Sports (confidence 0.99)
why: A match report centred on the result and the scorer.
And the evaluation runner:
uv run news-classifier-eval -n 5 # shipped config
uv run news-classifier-eval -n 5 --compare # few-shot vs zero-shot
uv run news-classifier-eval --hard # hard set
uv run news-classifier-eval --ablate # all four configurations
The JSON API
FastAPI, with Pydantic models on both request and response, and generated docs at /docs.
| Endpoint | Purpose |
|---|---|
POST /api/classify |
One article (max 20,000 chars) |
POST /api/classify/batch |
Up to 25 articles in one call |
POST /api/compare |
One article, zero-shot vs few-shot, N runs each |
POST /api/eval/run |
One hard case × N runs × chosen configurations |
GET /api/eval/config |
The configurations and the eval set |
GET /api/health |
Model, categories, credential status |
curl -X POST http://127.0.0.1:8000/api/classify \
-H 'Content-Type: application/json' \
-d '{"article": "The Senate voted 58-42 to advance the bill."}'
{"category":"Politics","confidence":0.99,
"rationale":"A floor vote on a bill.","elapsed_ms":812}
Errors are real HTTP status codes, not a 200 with an error body: 400 empty article, 401 bad key, 422 failed validation, 429 rate limited, 502 upstream error, 503 no credentials.
The web UI, and the Consistency Lab
The main page is what you'd expect — paste an article, get a card with the category, a confidence meter and the rationale.
The button I actually care about is Compare prompts. It takes your article, runs it N times with the examples and N times without, everything else held identical, and leads with the delta:
Few-shot was more consistent: +25 pts agreement
Configuration Label Agreement Confidence
ZERO-SHOT Zero-shot Sports 50% Sports×2, Tech×2 0.88
FEW-SHOT Few-shot Sports 75% Sports×3, Tech×1 0.81
That's a real run on the camera-tracking article from the hard set. The zero-shot prompt was a coin flip on it. The few-shot prompt wasn't.
Two details I'd defend:
When the runs disagree, the UI shows the dissenting label and its reasoning too. Showing only the winner hides the fact that the model wavered, which is the entire finding.
When the delta is zero, it says so plainly. Unambiguous articles score 100% under both prompts. A tool that only reports wins isn't a measurement tool.
Then there's /eval — the Consistency Lab — which runs the full ablation in the browser as a matrix: one row per article, one column per configuration. Green for correct, red for wrong, and a striped cell when a configuration disagreed with itself between runs. Hovering shows the full label spread. Each row carries its hard_because.
Two implementation notes there. The browser drives the loop one article per request, so progress renders live and no single request approaches a serverless timeout. And both tables render before you run anything, with cells reading not run — so you can see exactly what's about to be compared, and the estimated API call count, before spending a rupee.
Five things that bit me
1. The reasoning parameter isn't universal
Reasoning models accept reasoning={"effort": "low"}. The gpt-4o family rejects it with a 400. The obvious fix is a hardcoded list of which models support it — which goes stale every release.
So instead: send it optimistically, catch that one specific rejection, drop it, and remember on the instance.
if self._supports_reasoning is False:
response = self.client.responses.parse(**request)
else:
try:
response = self.client.responses.parse(
**request, reasoning={"effort": REASONING_EFFORT})
self._supports_reasoning = True
except openai.BadRequestError as exc:
if not _is_unsupported_reasoning(exc):
raise # never swallow a real 400
self._supports_reasoning = False
response = self.client.responses.parse(**request)
Cost: one extra request on the first call, nothing after. The raise is the important line — it would be very easy to write a bare except BadRequestError here and silently swallow genuine errors like a malformed schema. There's a test that specifically asserts other 400s still propagate.
2. Reasoning tokens eat your output budget
I set max_output_tokens=256 at first. It's a classification — how many tokens could it need?
The answer is that reasoning tokens count against max_output_tokens. The model burned the budget thinking and got truncated before emitting any JSON. And the failure is quiet: response.output_parsed comes back as None rather than raising.
Fix: a 2048 ceiling, and an explicit error rather than a silent None:
parsed = response.output_parsed
if parsed is None:
raise RuntimeError(
f"Model returned no parsed result (status={response.status!r})."
)
3. Serverless has a hard clock
Vercel caps a request at 60 seconds. One classification is 1–3s, which is fine. A batch of 25 run sequentially is ~40s and can time out. So batches fan out over a bounded thread pool:
workers = min(BATCH_CONCURRENCY, len(request.articles))
with ThreadPoolExecutor(max_workers=workers) as pool:
results = list(pool.map(lambda a: _classify_one(classifier, a), request.articles))
pool.map preserves input order — which matters enormously when you're returning results the caller has to line up with their inputs, and which has its own test. Concurrency defaults to 5 to stay inside rate limits, overridable via BATCH_CONCURRENCY. A 25-article batch drops from ~25s to ~5s.
4. Error messages should say what to do
The common failures in an LLM app aren't bugs. They're an unset key, an empty billing account, or a rate limit — and each has a completely different fix. Generic error handling makes all three look identical.
if isinstance(exc, openai.RateLimitError):
# OpenAI returns 429 for both throttling and an exhausted quota.
message = getattr(exc, "message", str(exc))
if "quota" in message.lower() or "billing" in message.lower():
return ("The OpenAI account is out of quota. Add credit at "
"https://platform.openai.com/settings/organization/billing")
return "Rate limited by the OpenAI API. Wait a moment and retry."
That 429 disambiguation cost me twenty minutes of confusion the first time I hit it, so it's now permanently encoded.
Related: a missing API key does not crash the server. It's caught at startup, reported through /api/health, and returned as a 503 on classify — so the UI can say something useful instead of failing blank.
5. The src/ layout vs. the serverless cold start
The project uses a src/ layout, so news_classifier is only importable once the project itself is installed. If a build installs only dependencies, you get ModuleNotFoundError at cold start — in production, on a Friday.
_SRC = Path(__file__).parent / "src"
if _SRC.is_dir() and str(_SRC) not in sys.path:
sys.path.insert(0, str(_SRC))
from news_classifier.api import app
Three defensive lines, and a test that imports app.py the way Vercel would.
One more Vercel gotcha worth flagging: don't copy the
excludeFilesexample from Vercel's docs verbatim. It excludesstatic/**, which would strip the entire UI out of your bundle.
Testing an LLM app without an API key
65 tests, 4.4 seconds, no network, no key. A stub classifier stands in for the model everywhere.
class StubClassifier:
"""Records what it was asked and returns a fixed answer."""
The tests I'd point at in a code review aren't the "does it return a value" ones — they're the ones pinning down properties the design depends on:
the few-shot prefix is byte-stable across calls — the premise of the caching claim
every example's assistant turn still parses back into the current schema (so changing the schema can't silently invalidate the examples)
no hard case is reused from the few-shot set — so the eval measures generalisation, not recall
concurrent batching preserves input order
the reasoning capability is probed exactly once per instance, and other 400s aren't swallowed
the rationale displayed next to a label comes from a run that actually produced that label (with N runs and disagreement, it's easy to show a mismatched pair)
app.pyimports without the package installed — the Vercel path, tested locally
The unifying idea: test the invariants your design leans on, not just the happy path. Every one of those would rot silently otherwise.
What this does not prove
Every article in the repo is synthetic. The 10 few-shot examples, the 12 hard cases, the 6 first-pass eval articles, the 5 UI samples — all written for this project. None is real journalism; names, figures and events are invented.
That was deliberate: an article that reliably puts its lede in one category and its body in another isn't something you find on demand, and the hard set needs exactly that to discriminate. But it costs something, and the cost should be named:
The same person wrote the examples and the eval set, so they're stylistically related. A test forbids literal reuse, but stylistic correlation isn't something a test can rule out. The examples may be unfairly well-matched to these cases.
So the measured gain is a lower-confidence result than the same measurement on independently sourced articles. It should be reported as "measured on a synthetic benchmark", not as a general claim about news classification.
Which is exactly why the eval set is data and not code, and why the synthetic: true flag travels with the file, through the API, and into a banner above the results in the UI. The weakness is carried in the payload rather than buried in a README, because the person reading the numbers is the person who needs to know.
Swapping in real articles is one environment variable. The plumbing to do it honestly is already built.
What I'd take to the next project
"Improve X" is a request for two numbers, not one. The baseline is half the deliverable. If you can't switch your improvement off, you can't claim it works.
Ablate more than one variable. My first experiment measured nothing because my baseline had already absorbed the treatment. The 2×2 rescued it — and the extra cell answered a question I hadn't thought to ask.
A metric alone can be gamed by accident. 97% agreement at 67% accuracy is a perfectly consistent, perfectly wrong classifier. Consistency needed correctness next to it to mean anything.
Constrain the output before you tune the prompt. An enum-typed schema deleted a whole class of inconsistency that no amount of prompt-wording would have fully fixed.
Field order in a structured output is a reasoning decision. rationale before category makes the model think before it commits. One line; real effect.
A failed experiment that you diagnose is worth more than a passing one you don't understand. The 100%-everywhere run was the most useful thing that happened in this build.
The classifier ships as guide + few-shot — the configuration that scored 94% agreement and 83% accuracy. But the switch to turn the examples off is still there, still tested, still exposed in the CLI, the API and the UI. Anyone who doubts the claim can re-run the measurement on their own articles, and that's rather the point.
