How to Test LLM Features: A Practical Guide to Evals for Software Engineers

The short answer: You test LLM features with evals: a dataset of realistic inputs plus automated checks that score the outputs. Start by reading real outputs and categorizing the failures, then write cheap code-based assertions for anything objective, use an LLM judge with narrow pass/fail criteria (validated against human labels) for anything subjective, and keep a human reviewing samples. Run the suite on every prompt, model, or tool change, just like unit tests.

Let me describe a release process that's far more common than anyone admits.

A customer complains that the AI assistant gave a rambling answer. An engineer opens the prompt, adds "Be concise." to the end, tries the customer's question, and sees a nice short answer. They try two more questions they happen to remember. Looks good. Ship it.

A week later, someone notices the assistant has stopped including the setup steps that users actually needed, because "be concise" made it drop them. Another person notices it now refuses to answer longer questions at all in some cases. Nobody connects either problem to the one-line change, because there's no record of what the outputs looked like before.

That's vibes-based development. It works for a demo and quietly falls apart in production, because a change that fixes one example can break ten others you didn't look at. The cure is the same one we found decades ago for ordinary code: automated tests. In the LLM world they're called evals, and this guide explains how to build them in a way that's practical for a normal engineering team, without a research department or a big budget.

Unit tests feel wrong here, and that's fine

When software engineers first try to test LLM output, two things feel off.

First, the outputs aren't deterministic. Even with temperature at zero, many providers don't guarantee identical outputs across calls, and any model update can change behavior. assertEqual(expected, actual) doesn't work when there are a thousand valid ways to phrase a correct answer.

Second, "correct" is often a judgment call. Is this summary good? Is the tone right? Did it answer the question the user meant? These don't reduce neatly to a boolean.

Both are real, but neither means testing is impossible. It means the tests look a bit different:

• Instead of asserting exact output, you assert properties of the output: it's valid JSON, it mentions the order number, it doesn't promise a refund, it cites a source.
• Instead of a single pass or fail, you track a pass rate across a dataset. "92% of cases pass" is a meaningful, comparable number.
• For subjective qualities, you use a grader (sometimes a model, sometimes a person) with written criteria, and you check that the grader itself is trustworthy.

The familiar testing pyramid still applies. Lots of cheap, fast, deterministic checks at the bottom. Fewer, more expensive, model-graded checks in the middle. A small amount of careful human review at the top.

Start with error analysis, not metrics

This is the step almost everyone skips, and it's the most important one. Before you write a single eval, look at real outputs and figure out how your system actually fails.

The temptation is to grab a generic metric off the shelf: "helpfulness", "coherence", "relevance", scored from 1 to 10. These sound rigorous and mean almost nothing. A system can score 8.2 on helpfulness while consistently making the one mistake that's costing you customers. Generic metrics measure what's easy to name, not what's going wrong for you.

Hamel Husain has written about this extensively (his post "Your AI Product Needs Evals" is a great place to start), and the process he and others recommend borrows from qualitative research. It goes like this:

1. Gather around 100 real traces. A trace is the full record of one interaction: input, any retrieved context, tool calls, and final output. Use production logs if you have them. If you're pre-launch, have teammates use the feature realistically.
2. Read them one by one and write a quick note on each failure. Don't use a predefined list. Just describe what's wrong in plain words: "ignored the date in the question", "made up a feature we don't have", "correct answer but buried in paragraph four", "called the search tool with an empty query".
3. Group the notes into categories. After 50 or so traces, patterns appear. Maybe 30% of failures are hallucinated features, 20% are wrong dates, 15% are formatting issues.
4. Count. Now you know what to test and what to fix first.

A spreadsheet is perfectly adequate for this. So is a simple internal page that shows one trace at a time with a notes box. What matters is that someone who understands the product reads actual outputs. The person should ideally be a domain expert, not only an engineer. If you're building a legal tool, a lawyer will spot failures an engineer would miss entirely.

This step usually takes a few hours and is consistently the most valuable few hours in the whole process. Teams that skip it end up with dashboards full of numbers and no idea why users are unhappy.

Layer 1: code-based assertions

Once you know your failure modes, sort them by how they can be checked. A surprising number can be checked with plain code, and those checks should always come first. They're fast, free, deterministic, and never argue with you.

Good candidates for code-based checks:

• Output is valid JSON and matches the expected schema
• Required fields or identifiers are present (order number, customer name, citation markers)
• Forbidden content is absent (competitor names, internal URLs, "as an AI language model")
• Length stays within bounds
• The right tool was called, with arguments that pass validation
• Every cited source ID actually exists in the provided context
• Links point to allowed domains

Here's what this looks like with plain pytest. The dataset is a list of cases, each describing the input and the properties a good output must have:

import json
import pytest

from app.assistant import answer_support_question

CASES = [
    {
        "id": "refund-annual",
        "question": "Can I get a refund on my annual plan after 3 weeks?",
        "must_include": ["30 days"],
        "must_not_include": ["14 days"],
    },
    {
        "id": "order-lookup",
        "question": "Where is order #A-10293?",
        "must_include": ["A-10293"],
        "must_call_tool": "get_order_status",
    },
    {
        "id": "out-of-scope",
        "question": "Can you write my college essay?",
        "must_include_any": ["can't help with that", "only help with"],
    },
]


@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_support_assistant(case):
    result = answer_support_question(case["question"])
    text = result.text.lower()

    for phrase in case.get("must_include", []):
        assert phrase.lower() in text, f"missing {phrase!r}"
    for phrase in case.get("must_not_include", []):
        assert phrase.lower() not in text, f"should not say {phrase!r}"
    if options := case.get("must_include_any"):
        assert any(o.lower() in text for o in options)
    if tool := case.get("must_call_tool"):
        assert tool in [call.name for call in result.tool_calls]

Nothing fancy, and that's a feature. Any engineer on the team can read it, add a case, and run it locally. Many teams get a large share of their total eval value from checks like these alone, especially for structured-output and tool-calling features.

One tip: keep the test cases in a data file (JSON, YAML, CSV) rather than in code once the list grows. Then product managers and domain experts can contribute cases without touching Python.

Layer 2: LLM-as-judge, done carefully

Some failure modes can't be checked with string matching. "The answer is factually consistent with the retrieved documents." "The tone is appropriate for an upset customer." "The summary includes every action item from the meeting." For these, you can use a language model as a grader.

Research has shown that strong models agree with human preferences reasonably well in many settings. The paper that popularized the approach, "Judging LLM-as-a-Judge" (Zheng et al., 2023), found that strong LLM judges could reach agreement levels with humans similar to the agreement between humans themselves on the tasks they studied. It also documented the judges' biases, which we'll come to in a moment.

The difference between a useful LLM judge and a random number generator comes down to a few rules.

Rule 1: one judge per failure mode

Don't write a single judge prompt that scores "overall quality". Write a separate, narrow judge for each important failure mode you found during error analysis. "Does the answer contain any claim not supported by the provided sources?" is a question a model can answer consistently. "Is this a good answer?" isn't.

Rule 2: binary pass/fail, not 1-to-10 scales

Numeric scales feel more informative, but they're hard to calibrate. What's the difference between a 6 and a 7? Neither the model nor your teammates will apply that distinction consistently. A pass/fail decision with a clear definition is easier to validate, easier to act on, and easier to aggregate into a pass rate. If you need nuance, add more binary judges rather than more scale points.

Rule 3: ask for reasoning before the verdict

Have the judge explain its assessment first and give the verdict last. You'll get more consistent judgments, and when you review disagreements, the reasoning tells you whether the judge misunderstood the criteria or spotted something you missed.

import json

GROUNDEDNESS_JUDGE = """You are checking an answer from a support assistant.

Sources the assistant was given:
<sources>
{sources}
</sources>

The assistant's answer:
<answer>
{answer}
</answer>

Task: decide whether EVERY factual claim in the answer is supported by the
sources. Ignore greetings and phrasing. A claim that is plausible but not
stated in the sources counts as unsupported.

Respond with JSON only:
{{"reasoning": "<list each claim and whether it is supported>",
  "verdict": "pass" or "fail"}}"""


def judge_groundedness(client, sources: str, answer: str) -> bool:
    response = client.complete(
        prompt=GROUNDEDNESS_JUDGE.format(sources=sources, answer=answer),
        temperature=0,
    )
    return json.loads(response)["verdict"] == "pass"

(The client.complete call stands in for whichever SDK you use. Use your provider's structured-output or JSON mode if it has one, so you don't have to parse free text.)

Rule 4: validate the judge against humans

This is the step that makes the difference, and it's the one people skip. An unvalidated LLM judge is just another opinion. To trust it, you need to know how often it agrees with a careful human.

Take 50 to 100 outputs and have a domain expert label each one pass or fail for the specific criterion. Run the judge on the same outputs. Then look at two numbers separately rather than a single accuracy figure:

• True positive rate: of the outputs the human marked as passing, how many did the judge pass?
• True negative rate: of the outputs the human marked as failing, how many did the judge fail?

The second number usually matters more, because catching failures is the whole point. A judge that passes everything will score high accuracy on a dataset where most outputs are fine, while being completely useless. Read every disagreement. Often the fix is a clearer definition in the judge prompt, or a few examples of borderline cases. Sometimes you'll find the human label was wrong, which is useful too.

Revalidate whenever you change the judge prompt or the judge model. An upgrade to the model behind your judge can silently shift its standards.

Rule 5: know the biases

LLM judges have predictable quirks, and some are easy to correct for:

• Position bias: when comparing two outputs side by side, judges tend to favor one position. Run each comparison twice with the order swapped, and only count consistent verdicts.
• Verbosity bias: longer answers often get rated higher even when they're not better. If length matters, make it an explicit criterion.
• Self-preference: a model may favor outputs that sound like its own. Using a different model family as the judge can reduce this.

Layer 3: humans, sampled

Automated evals don't eliminate human review. They make it affordable. Instead of reading everything, a person reviews a sample: a few dozen production traces a week, plus every case where the automated evals disagree with each other or with user feedback.

Keep the review interface dead simple: show the trace, offer pass/fail buttons and a notes field. The notes feed back into error analysis, which produces new failure categories, which become new evals. That loop is the engine of steady improvement. Without it, your eval suite gradually drifts away from what users actually experience.

Building the dataset

Your evals are only as good as the inputs you run them on. A few sources, in rough order of value:

1. Production traces, especially ones that got negative feedback, a regeneration, or an escalation to a human.
2. Every bug report. When something goes wrong, add the input to the dataset before you fix it. This is the LLM version of writing a failing test first.
3. Edge cases from domain experts: the tricky questions they know users ask.
4. Adversarial cases, such as attempts to extract the system prompt or plant instructions in documents. Our guide on prompt injection has a list of starter attacks worth turning into eval cases.
5. Synthetic cases to fill gaps.

Synthetic data deserves a word of caution. If you just ask a model to "generate 200 test questions", you'll get 200 polite, well-formed, oddly similar questions that look nothing like what real users type. A better approach is to define the dimensions that matter for your product (for example user type × task × difficulty × input style) and generate cases for specific combinations, like "a frustrated non-technical user, asking about billing, with a typo-heavy one-line message". Then review the generated cases and throw away anything unrealistic. Use synthetic data to cover gaps, not as the foundation.

As for size: start with 50 to 100 well-chosen cases. That's enough to catch regressions and compare prompt versions. Grow it organically as you find new failures. A small dataset you actually read beats a large one you don't.

If you're building a retrieval feature, keep separate cases for retrieval quality (did we fetch the right documents?) and answer quality (did we use them correctly?). The guide on why RAG chatbots give bad answers shows how to measure the retrieval half with recall@k.

Running evals in CI

Evals pay off when they run automatically. Treat prompt files, model configuration, tool definitions, and retrieval settings as code, and run the suite whenever any of them change.

A few practical decisions:

• Tiers. Run the fast, code-based checks on every pull request. Run the slower, model-graded suite on changes to prompts or models, or nightly. This keeps CI quick and costs predictable.
• Thresholds, not perfection. Fail the build if the pass rate for any category drops below its baseline by more than a small margin. Requiring 100% just teaches everyone to ignore the suite.
• Per-category reporting. An overall pass rate of 91% hides a lot. Report "groundedness 97%, tool use 88%, refusals 100%" so a regression in one area is visible.
• Handle flakiness honestly. For borderline cases, run them several times and use the majority result, or track the pass rate across repetitions. Don't quietly delete cases because they're flaky; that's often where the real problems are.
• Pin model versions. Use a specific dated model identifier in production and in evals, so an upstream update doesn't change results without you knowing. When you upgrade, run the full suite first.

A minimal GitHub Actions job is all it takes to get started:

name: llm-evals
on:
  pull_request:
    paths:
      - "prompts/**"
      - "app/assistant/**"
      - "evals/**"

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest evals/ --junitxml=eval-results.xml
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}

Later you can add result comparisons against the main branch and post a summary as a pull-request comment. Start simple. A suite that runs is worth infinitely more than a sophisticated one that's still being designed.

Monitoring in production

Offline evals tell you whether a change is safe to ship. Production monitoring tells you how the system behaves with real users, who will always find inputs your dataset didn't anticipate.

• Run cheap checks on live traffic. Your code-based assertions can run on every response as guardrails: block or retry invalid JSON, strip disallowed links, flag missing citations.
• Run judges on a sample. Grading every production response with an LLM gets expensive. Grade a random sample plus every response with negative feedback, and track pass rates over time.
• Watch implicit feedback. Thumbs-down clicks are rare. Regenerations, copy events, follow-up questions like "no, I meant...", abandoned sessions, and escalations to a human are much more common and often more honest.
• Feed it back. Interesting failures from production become new dataset cases. That's how the loop closes.

Comparing prompts and models

Evals also turn "should we switch to the new model?" from a debate into an experiment. Run your full suite on both options and compare per-category pass rates, along with cost and latency. For subjective qualities, use pairwise comparison: show a judge (or a human) both outputs for the same input and ask which better meets a specific criterion, swapping the order to cancel position bias.

Be suspicious of tiny differences. With 100 cases, a gap of two or three percentage points is often noise. Look at which specific cases changed, and read them. Sometimes a "worse" score turns out to be the new model correctly refusing something your old expected output got wrong.

Traps to avoid

• Overfitting to the eval set. If you tune a prompt by repeatedly tweaking it until every eval case passes, you may end up with a prompt that's great at those exact cases and worse at everything else. Keep a held-out set you don't look at while iterating.
• Goodhart's law. When a measure becomes a target, it stops being a good measure. If the team is rewarded for pass rates, cases that fail will mysteriously become "out of scope". Keep reading the traces.
• Buying a tool before having a process. Eval platforms are genuinely useful (promptfoo, Inspect, Braintrust, LangSmith, and Langfuse are all popular options) but none of them can tell you what your failure modes are. Do error analysis in a spreadsheet first, then pick a tool that fits the workflow you already know you need.
• Trusting the judge forever. Revalidate it periodically and whenever its prompt or model changes.
• Only testing the happy path. Include ambiguous inputs, out-of-scope requests, adversarial inputs, and empty or malformed data. Production will send them all.

A one-week plan to get started

If you have an LLM feature in production with no evals, here's a realistic way to go from zero to a working suite in a week, alongside normal work.

1. Day 1: Make sure you're logging complete traces. Export 100 recent ones.
2. Day 2: Error analysis. Read the traces, write notes, group them into failure categories, and count.
3. Day 3: Turn the top failure categories into a dataset of 50 to 100 cases. Write code-based assertions for everything objective.
4. Day 4: Write one narrow LLM judge for the most important subjective failure. Label 50 outputs by hand and measure agreement.
5. Day 5: Wire the suite into CI with a baseline threshold, and set up a weekly sample of production traces for human review.

By the end of the week you'll know your system's actual pass rates, and the next "be concise" change will tell you what it broke before your users do.

Frequently asked questions

What are LLM evals?

LLM evals are automated tests for features built on language models. They run the feature on a dataset of realistic inputs and score the outputs, using code-based checks for objective properties and model or human graders for subjective ones. The result is a pass rate you can track across prompt, model, and code changes.

How many examples do I need in an eval set?

Start with 50 to 100 carefully chosen, realistic cases that cover your main failure modes. That's enough to catch regressions and compare versions. Grow the set over time by adding every production failure and bug report, rather than generating thousands of synthetic cases up front.

Can I trust an LLM to grade another LLM?

Only after you've validated it. Have a domain expert label a sample of outputs, run the judge on the same sample, and measure how often they agree, especially on failures. Narrow pass/fail criteria, reasoning before the verdict, and order-swapping in comparisons all make LLM judges more reliable.

How often should I run evals?

Run fast code-based checks on every pull request, and the full model-graded suite whenever you change a prompt, model, tool definition, or retrieval setting, plus on a regular schedule such as nightly. In production, run cheap checks on all traffic and model-graded checks on a sample.

Do I need a special tool to run LLM evals?

No. A test framework like pytest, a data file of cases, and a spreadsheet for error analysis are enough to start. Dedicated eval tools become useful as your suite grows, mainly for tracking results over time, comparing runs, and making traces easy to review.

The takeaway

Evals aren't a research discipline you need a PhD for. They're testing, adapted to software that answers in sentences instead of return values. Read your outputs, name your failures, check what you can in code, judge the rest carefully, and run it all every time something changes.

The teams that do this ship prompt changes with confidence and upgrade models in an afternoon. The teams that don't keep adding "be concise" to the end of the prompt and hoping.