How to Build an LLM Eval: Testing Prompts Like Code

Why “it looks right” isn’t a test for a prompt

You tighten the wording of a ticket-routing prompt, run it against the three examples open in your scratch notebook, and ship it. A week later, billing complaints that used to land in the right queue start landing in “question” instead. Not the three cases you checked: the other four hundred you didn’t. Nothing crashed. There’s no stack trace. The prompt still reads fine out loud.

Code review and a few manual checks miss this because the same prompt against the same input doesn’t return the same text twice. You can’t diff two runs byte for byte and call a mismatch a bug, the way you would with a pure function. What you can check is whether each output has the properties you care about: the right category, a summary under the length limit, no field the source never mentioned. An eval runs those checks over a set of real inputs every time the prompt or the model changes, so the regression shows up in your terminal instead of in a customer’s ticket.

What are the parts of an LLM eval?

Every eval, whatever tool runs it, has three parts: a dataset of representative inputs, an assertion that checks each output against what that input should produce, and a pass rate you track from run to run.

Think of it as a test suite for a function that returns a slightly different sentence every time. The assertion can’t check for an exact match, so it checks a property of the sentence instead: does it name the right category, does the number in it parse. Once you accept that the test is “does this hold” rather than “is this identical”, writing one is ordinary engineering.

How to build a golden dataset for an LLM eval

Take a prompt that turns a raw support email into a structured ticket:

1
2
3
4
5
6
7
8
9

TRIAGE_PROMPT = """Read this support email and return JSON with exactly these keys:
- category: one of "billing", "bug", "question"
- summary: the complaint in 12 words or fewer
- urgent: true if the customer can't use the product right now, else false

Email:
{email_text}

Return only the JSON object, no other text."""

The golden dataset is a set of real emails paired with the answer you already know is correct. Real, not invented to make the prompt look good:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

golden_set = [
    {
        "email": "I was charged twice for my October invoice, can you refund the duplicate?",
        "category": "billing",
        "urgent": False,
    },
    {
        "email": "The app crashes every time I try to export a PDF, I have a deadline in an hour.",
        "category": "bug",
        "urgent": True,
    },
    {
        "email": "Does the Pro plan include API access, or do I need Enterprise?",
        "category": "question",
        "urgent": False,
    },
]

Five to ten cases is enough to start. Size matters less than where the later cases come from: real production failures.

How to write deterministic assertions for LLM output

Most of what goes wrong with this prompt has nothing to do with taste. The model returns malformed JSON, or invents a fourth category, or writes forty words when you asked for twelve. Plain code catches all of it, no second model required:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

import json

def run_triage_case(case, call_model):
    raw = call_model(TRIAGE_PROMPT.format(email_text=case["email"]))
    try:
        output = json.loads(raw)
    except json.JSONDecodeError:
        return {"passed": False, "reason": "not valid JSON"}

    if output.get("category") not in {"billing", "bug", "question"}:
        return {"passed": False, "reason": f"bad category: {output.get('category')!r}"}

    if len(output.get("summary", "").split()) > 12:
        return {"passed": False, "reason": "summary over 12 words"}

    if not isinstance(output.get("urgent"), bool):
        return {"passed": False, "reason": "urgent is not a boolean"}

    if output["category"] != case["category"]:
        return {"passed": False, "reason": f"expected {case['category']}, got {output['category']}"}

    return {"passed": True, "reason": None}

Run it over the whole dataset and you get a pass rate, not a single yes or no:

1
2

results = [run_triage_case(c, call_model) for c in golden_set]
pass_rate = sum(r["passed"] for r in results) / len(results)

These checks cost nothing and never flake: the same bad output fails the same way every time. Write them first. They catch most real regressions, like a schema change or a summary that quietly doubled in length.

When to use LLM-as-judge, and how to calibrate it

Category and urgency are objective, and the dataset already holds the right answer. Summary quality isn’t. “Refund needed, double charge” and “Customer has a billing question” are both valid JSON, both under 12 words, and only one of them describes the email. No regex catches that, so a second model grades it against a rubric:

1
2
3
4
5
6
7
8
9

JUDGE_PROMPT = """Email: {email}
Summary: {summary}

Does the summary accurately capture the main complaint in the email?
Answer with only "yes" or "no"."""

def judge_summary(email, summary, judge_model):
    verdict = judge_model(JUDGE_PROMPT.format(email=email, summary=summary), temperature=0)
    return verdict.strip().lower() == "yes"

Before you trust that verdict on the full dataset, run the judge on 15-20 cases a person has already graded by hand and count how often they agree. A judge that disagrees with people a third of the time is a weak assertion wearing a model’s authority.

Two biases are worth watching for. Judges tend to rate longer, more elaborate answers higher even when a shorter one is equally correct. And a model grading its own outputs rates them more favorably than an independent model would. Use a different model as judge when you can, and keep the prompt narrow: one yes/no question per call, not five scores in one request.

In test-suite terms, deterministic checks are your unit tests: fast, exact, run on every change. The judge is closer to an acceptance check, the one a person would otherwise make by reading every output. The tradeoffs between those levels are the same ones covered in unit vs. integration vs. end-to-end tests.

How to run LLM evals in CI to catch regressions

An eval that runs only when someone remembers to run it catches nothing. Commit the script next to the prompt it tests (evals/triage.py alongside prompts/triage.py) and run it in CI on every change to either file, before merge. Track the pass rate over time and fail the build when it drops below a baseline you set. A slide from 95% to 88% is a regression worth blocking, even though most cases still pass.

Keep a regression set inside the dataset: every case that was once a reported bug, pinned permanently after the fix. For a prompt, this gives you the loop described in what TDD actually is. Change the prompt, rerun the suite, see exactly which assertions flipped, fix, repeat. Feed new production failures into the dataset as they happen, so you aren’t only testing the inputs the prompt was designed around.

How to evaluate tool-calling and RAG pipelines

The same three parts carry over. For a tool-calling loop, the dataset pairs each input with the correct tool name and arguments, and the assertion checks that the model picked that tool with the required fields filled in. The mechanics of the loop itself are in LLM function calling and tool use.

A RAG pipeline needs two separate scores. Did retrieval pull the chunk that contains the answer? Did generation use that chunk correctly? Score them together and you can’t tell which half broke. See how RAG works for the pipeline you’re splitting.

Common LLM eval mistakes

Building the dataset from cases that already pass. Five examples picked because the prompt handles them well will pass forever and tell you nothing about next month’s breaking input. Every real failure that reaches you belongs in the dataset the same day.

Asserting exact text on variable output. summary == "Refund needed for duplicate charge" fails the moment the model swaps one word, even when the new phrasing is equally correct. Assert the property instead: the word count, the category, whether a required fact appears.

Skipping judge calibration. A judge you never checked against even a dozen human-graded examples hands you its biases as ground truth, including the one that favors its own output.

Rerunning only the case you just fixed. That’s how an old bug comes back unnoticed. The regression set exists so it can’t.

When is an LLM eval suite not worth building?

A prompt you’ll run once, or one a person reviews on every call, doesn’t need a harness. Put the JSON-validity check inline if it’s cheap and move on. The suite pays for itself once a prompt ships to users repeatedly or you expect to edit it more than a couple of times. That’s when “it looked fine on my three examples” stops being good enough.

To start: five real inputs, one deterministic assertion per property you care about, and an LLM judge only for the dimension code can’t check. Run it before your next prompt edit, not after the complaint.

Related articles

• LLM Function Calling and Tool Use, Explained
• What Is RAG: Retrieval-Augmented Generation Explained
• What Is TDD and How the Red-Green-Refactor Cycle Works
• Unit vs Integration vs End-to-End Tests Explained