Why prompt changes deserve an experiment

A new prompt version always looks better on the three examples its author tried. That is a weak basis for shipping it to everyone, because the examples were chosen after the fact and the real traffic is far more varied. An experiment replaces the impression with a comparison.

A/B testing is a quantitative method that compares performance against a baseline model or earlier version.

In prompt work the baseline is usually the version currently labelled for production, and the challenger is the candidate you want to promote. Everything else, from the model to the retrieval step to the traffic, should stay the same, so that a difference in outcomes can be traced to the wording and nothing else.

Choosing what to measure

Before you split any traffic, decide which numbers will count as success. Picking the metric after you see the data lets you find a winner in almost any experiment, which is exactly the habit the test is meant to prevent.

Numbers provide clarity and scalability, but qualitative measures can also be valuable if consistently applied along with quantitative measures.

A useful experiment therefore carries one primary metric, a small set of guardrail metrics, and room for human review of a sample. The primary metric answers whether the change helped. The guardrails tell you what it may have cost, and the reviewed sample catches problems that no counter can see.

The guide cited here groups quantitative measures by what they describe.

• Task-specific: F1 score, BLEU score, and perplexity.
• Generic: accuracy, precision, and recall.
• Operational: response time and uptime.

Map your own situation onto those groups. A summarizer might be judged on a task-specific score, a classifier on precision and recall, and both on the operational side, where the latency of the slower variant can quietly erase a quality gain. If the work involves wording changes as well as measurement, the prompt engineering entry is a good companion.

Even topics that sound hazy can be turned into a countable rule, as the same guide shows for safety.

Less than 0.1% of outputs out of 10,000 trials should be flagged for toxicity by the content filter.

Notice the shape of that rule: a rate, a trial count and a stated detector. Any guardrail you write for your own prompts should have the same three parts, otherwise nobody can tell later whether it was met.

Assigning users to variants

An experiment is only fair if each user always sees the same variant. Otherwise one person might read the control tone in the morning and the treatment tone in the afternoon, and your metrics would mix the two. The standard solution is to derive the variant from a stable hash of the experiment name and the user id.

Avoid the language's built-in string hash for this. In Python its values for strings can change from one process to the next, so a user could move between variants after a restart. A cryptographic digest is stable everywhere and spreads users evenly enough for traffic splitting.

import hashlib


def assign_variant(experiment, user_id, split):
    """Return a variant name; identical inputs always give the same answer."""
    digest = hashlib.sha256(f"{experiment}:{user_id}".encode()).hexdigest()
    bucket = int(digest[:8], 16) % 10_000 / 10_000  # a number in [0, 1)
    cumulative = 0.0
    for variant, share in split.items():
        cumulative += share
        if bucket < cumulative:
            return variant
    return next(iter(split))  # guards against shares that do not sum to one


split = {"control": 0.9, "treatment": 0.1}
print(assign_variant("tone-test", "user-42", split))
print(assign_variant("tone-test", "user-42", split))  # same user, same variant

Including the experiment name in the hashed text matters. Without it, the same users would land in the treatment group of every experiment you run, and the results of different tests would be entangled.

Start the treatment on a small share of traffic. If the candidate is badly broken, only a few users meet it, and you can widen the split once the guardrails look healthy.

Reading the result honestly

Two groups will never produce identical numbers, even when the prompts are equivalent, so the question is whether the gap is bigger than chance would explain. For a yes-or-no outcome such as a resolved ticket or a clicked suggestion, a two-proportion test answers that with a p-value, the probability of seeing a gap at least this large if the two prompts were really equal.

from math import erf, sqrt


def two_proportion_p_value(wins_a, total_a, wins_b, total_b):
    pooled = (wins_a + wins_b) / (total_a + total_b)
    spread = sqrt(pooled * (1 - pooled) * (1 / total_a + 1 / total_b))
    z = (wins_b / total_b - wins_a / total_a) / spread
    return 2 * (1 - 0.5 * (1 + erf(abs(z) / sqrt(2))))


print(round(two_proportion_p_value(120, 1000, 150, 1000), 3))  # 0.05: right at the usual line
print(round(two_proportion_p_value(120, 1000, 135, 1000), 3))  # 0.315: indistinguishable from noise

The first call shows why a threshold is a convention, not a cliff: a result sitting almost exactly on the line deserves another round of data rather than a celebration. Three habits protect you from fooling yourself.

• Fix the sample size and the stopping rule before launch, because checking the p-value every day and stopping at the first good-looking value inflates false positives.
• Report the size of the effect next to the p-value, since a tiny but significant gain may not be worth a rewrite.
• Look at the guardrails and a reviewed sample before promoting, even when the primary metric wins.

Letting the traffic adapt

A fixed split keeps sending users to the weaker variant until the experiment ends. A multi-armed bandit trades some statistical tidiness for less wasted traffic: it shifts requests toward the variant that is currently doing better while still exploring the others. Thompson sampling is a compact way to do it, because each variant keeps a count of successes and failures and you draw a random plausibility score from each.

import random


class ThompsonBandit:
    def __init__(self, variants):
        self.wins = {v: 1 for v in variants}
        self.losses = {v: 1 for v in variants}

    def choose(self):
        draws = {v: random.betavariate(self.wins[v], self.losses[v]) for v in self.wins}
        return max(draws, key=draws.get)

    def update(self, variant, success):
        (self.wins if success else self.losses)[variant] += 1


bandit = ThompsonBandit(["v12", "v15"])
for _ in range(200):
    chosen = bandit.choose()
    bandit.update(chosen, success=random.random() < (0.30 if chosen == "v15" else 0.20))
print(bandit.wins, bandit.losses)

The bandit suits low-risk, high-volume decisions where you mostly want the better variant to take over. Keep a classic fixed split when you need a clean estimate of the effect, such as before a costly rollout, because adaptive allocation makes the final numbers harder to interpret.

Running the loop with registry and eval tools

Experiments go much more smoothly when variants are named versions in a registry instead of strings pasted into branches.

Aliasing also lets you perform tasks such as A/B testing and roll-backs.

Your serving code then asks for the control or treatment alias, and ending the experiment means moving an alias, not shipping code. The registry documentation also describes how a stored prompt reaches your application.

To use a prompt in an LLM application, you can load it using the mlflow.genai.load_prompt() API, and then fill in the variables using the format() method of the prompt object.

An offline evaluation tool can run before any user sees the candidate. The introduction to Promptfoo describes the step that follows a test run.

Analyze results by setting up automatic requirements, or by reviewing results in a structured format or web UI.

A pre-launch run like this filters out candidates that fail obvious cases, so live traffic is spent only on versions that deserve it.

Use these results to select the best model and prompt for the use case.

Offline results rank the candidates, and the live experiment confirms the winner with real users. The same source ends with a reminder about keeping the loop going.

As you gather more examples and user feedback, continue to expand the test cases.

Each experiment that surprises you is a new test case. Add the failing inputs to the evaluation set, and the next candidate has to clear a harder bar than the last one did.