Standard benchmarks tell you about general capabilities. Custom evaluations tell you whether a model works for your specific use case.

Designing Your Eval Suite

1. Define success criteria: What does a "good" output look like for your task? Be specific.
2. Collect test cases: 50-200 representative examples covering edge cases and common scenarios.
3. Create golden answers: Expert-written reference outputs for comparison.
4. Choose metrics: Select metrics that align with your success criteria.
5. Automate: Make evaluation runnable with a single command.

Metric Categories

Exact match metrics (deterministic):

• Accuracy, F1, precision, recall for classification
• Exact match (EM) for extraction tasks
• Code execution pass rate (compile + run + test)

Similarity metrics (fuzzy):

• BLEU, ROUGE for text generation
• BERTScore for semantic similarity
• Edit distance for structured outputs

LLM-as-Judge (flexible):

• Use a strong model (GPT-4, Claude) to evaluate outputs against criteria
• Provide clear rubrics with scoring guidelines
• Use pairwise comparison ("Which response is better?") for more reliable judgments

LLM-as-Judge Template

You are evaluating an AI assistant's response. Score from 1-5 on each criterion:

**Accuracy** (1-5): Are all facts correct? Any hallucinations?
**Completeness** (1-5): Does it address all parts of the question?
**Clarity** (1-5): Is it well-organized and easy to understand?
**Helpfulness** (1-5): Would this actually help the user?

Question: {question}
Reference answer: {reference}
Model response: {response}

Provide scores and brief justification for each.

Running Evals at Scale

Use frameworks like:

• OpenAI Evals: Open-source evaluation framework
• RAGAS: Specialized for RAG pipeline evaluation
• DeepEval: Pytest-like framework for LLM evaluation
• Promptfoo: CLI tool for comparing prompts across models

CI/CD Integration

Run evals on every prompt or model change:

1. Define minimum quality thresholds (e.g., accuracy > 85%)
2. Run evals in CI pipeline on each commit
3. Block deployment if quality drops below thresholds
4. Track metrics over time in a dashboard