Evaluation is how a prompt earns its release

A prompt change is a code change whose behaviour you cannot read off the diff. The same sentence of instruction may help on one kind of input and harm on another, so the only way to know is to run it. Evaluation turns that run into a repeatable routine, and monitoring extends the routine into production, where the inputs are no longer ones you chose.

A comparison is only meaningful if the thing being compared stays put. That is one more reason to keep prompts in a registry, where a version cannot change underneath your results.

Prompt versions in MLflow are immutable, providing strong guarantees for reproducibility.

Every evaluation result should therefore be stored with the prompt version, the model, and the settings that produced it. A score without that record cannot be reproduced and so cannot be trusted.

Build the test set first

The introduction to Promptfoo describes evaluation as a short sequence of steps, and the first one concerns the test set.

Define test cases by identifying core use cases and failure modes.

Core use cases are the requests the prompt exists to handle. Failure modes are the ways it is likely to go wrong: missing context, ambiguous questions, hostile input, answers that must be refused. A set built only from happy paths will pass nearly every candidate and teach you nothing.

Prepare a set of prompts and test cases that represent the scenarios.

Representative is the important word. Sample real requests where you can, add the awkward ones on purpose, and tag each case with the behaviour it protects. When a production incident teaches you something, the input that caused it becomes a permanent member of the set.

Run it the same way every time

Configure the evaluation by specifying prompts, test cases, and API providers.

Configuration is where reproducibility is won or lost. Keep it in a file under version control, pin the settings that affect randomness, and run each candidate against exactly the same cases. Differences in the results should come from the prompt, not from the harness.

Use the command-line tool or library to execute the evaluation, then record model outputs for each prompt.

The sketch below shows the shape of such a harness without depending on any vendor. It receives a function that renders the prompt, a function that calls whatever model you use, and a list of cases that each carry a checking function.

def run_suite(render, call_model, cases, version):
    """Run every case once and keep enough context to compare runs later."""
    results = []
    for case in cases:
        output = call_model(render(case["input"]))
        results.append({"version": version, "case": case["id"], "output": output, "passed": bool(case["check"](output))})
    return results


def pass_rate(results):
    return sum(r["passed"] for r in results) / len(results)


cases = [
    {"id": "refund-basic", "input": "I want my money back", "check": lambda out: "refund" in out.lower()},
    {"id": "empty-input", "input": "", "check": lambda out: len(out) > 0},
]
stub = lambda prompt: "We can start a refund for you."  # replace with a real model call
print(pass_rate(run_suite(lambda text: f"Support agent. User says: {text}", stub, cases, "v3")))

Simple string checks like these are cheap, fast and deterministic, so run them first. They cannot judge tone or helpfulness, which is where a second kind of scoring comes in.

Scoring with a grader model

Many qualities of an answer cannot be tested with a substring. For those, a second model can act as grader. The testing guide cited here gives advice on how to ask for a grade that you can use.

Be specific: for example, tell the LLM to output only correct or incorrect, or to judge on a scale of 1–5.

A constrained answer is easy to parse and easy to aggregate. Free-form commentary is not, and the same guide explains why leaning on it alone is a problem.

Purely qualitative evaluations are hard to assess quickly and at scale.

The guide also recommends letting the grader think before it scores.

Encourage reasoning by using a grader model with thinking on, so it reasons before producing an evaluation score.

Whether that extra reasoning is worth its cost is something to measure on your own cases, not to assume.

This increases evaluation performance, particularly for tasks requiring complex judgment.

Putting both ideas together gives a grader that reasons in free text and then ends with a strict, machine-readable verdict. The function below validates that verdict instead of trusting it, because a grader that drifts from the format would otherwise corrupt your averages silently.

import json

JUDGE_PROMPT = """You are grading an assistant's answer.
Question: {question}
Answer: {answer}
Expected behaviour: {expected}
Reason briefly, then end with one JSON object on the last line:
{{"accuracy": <integer 1-5>, "clarity": <integer 1-5>, "verdict": "correct" or "incorrect"}}"""


def judge(call_model, question, answer, expected):
    reply = call_model(JUDGE_PROMPT.format(question=question, answer=answer, expected=expected))
    scores = json.loads(reply.strip().splitlines()[-1])
    if scores.get("verdict") not in ("correct", "incorrect"):
        raise ValueError(f"unexpected verdict: {scores.get('verdict')!r}")
    for key in ("accuracy", "clarity"):
        if not 1 <= scores[key] <= 5:
            raise ValueError(f"{key} out of range: {scores[key]!r}")
    return scores


def fake_grader(prompt):
    return 'The answer names the right city.\n{"accuracy": 5, "clarity": 4, "verdict": "correct"}'


print(judge(fake_grader, "What is the capital of France?", "Paris", "Names Paris"))

Treat the grader as an instrument that needs calibrating. Before relying on it, have people score a sample of answers, compare their verdicts with the grader's, and read every disagreement. A grader that is easy to fool will happily approve the confident, fluent answers you most need to catch.

Reading the results

Raw outputs are hard to compare, so tools present them as a grid with one row per test case and one column per prompt.

Promptfoo produces matrix views, which let you quickly evaluate outputs across many prompts.

Scan the grid for rows where the candidate differs from the baseline, not just for the average. A candidate that gains on average but breaks a case you care about is a trade-off a person should decide on, not a number.

Some tools also test the safety side of a prompt.

Promptfoo also produces high-level vulnerability and risk reports.

Whether you use such a report or your own adversarial cases, keep a separate section of the suite for inputs that try to subvert the prompt. See the glossary entry on prompt injection for the kinds of attack worth covering.

Monitoring in production

Offline evaluation can only test the inputs you thought of. Production brings the rest, so the same prompt needs a second layer of checks once real users arrive. Group what you track into three families.

• Quality: user ratings, the share of replies that follow the required format, and a sampled estimate of how often answers contain unsupported claims, sometimes called hallucination.
• Performance: response latency, looking at the slow tail and not only the average, and how quickly tokens stream back.
• Cost: spend per request and per day, since a longer prompt or a retry loop can raise the bill without changing a single quality metric.

None of this is useful unless each measurement can be traced to the prompt version that produced it.

Langfuse's overview describes one way to make that traceable.

You can link prompts to traces to analyze performance by prompt version.

With that link in place, a regression stops being a vague feeling and becomes a question you can answer: which version, since when, and on which kinds of requests. A simple alert compares a recent window with a baseline measured while the previous version was healthy.

from statistics import mean


def percentile(values, pct):
    ordered = sorted(values)
    index = max(0, min(len(ordered) - 1, round(pct / 100 * (len(ordered) - 1))))
    return ordered[index]


def regression_alert(baseline_scores, recent_scores, tolerance=0.05):
    drop = mean(baseline_scores) - mean(recent_scores)
    return drop > tolerance, round(drop, 3)


print(regression_alert([0.91, 0.93, 0.92], [0.84, 0.86, 0.85]))  # (True, 0.07)
print(percentile([120, 180, 200, 240, 900], 95))  # the slow tail, not the average

Pick the tolerance from the natural wobble you see between healthy windows, because an alert that fires on noise teaches people to ignore it. When it does fire, the first response is usually to move the production label back to the previous version, which is exactly what a registry makes cheap.

Closing the loop

Evaluation and monitoring feed each other. A monitored failure becomes an evaluation case, the case guards the next candidate, and the candidate is released behind a label so the cycle can repeat. Teams that skip the last step end up with a test set that describes last quarter's problems instead of today's.