LLM Evaluation Playbook

Step 1 — Match your use cases to task types

Pick the row that matches each of your LLM-powered features. Use the suggested metrics and tools as your starting point; refine per the charter you write in Phase 1.

Step 2 — Work through the six phases in order

Each phase is a maturity step. Don't skip ahead: CI gates are worthless without stable golden sets; drift alerts are noisy without calibrated judges. Complete the success signals before moving on.

1. 1

   Choose task types and metrics

   Goal. Match each production use case to a task family and pick the right metrics for that family.

   Actions

   • Inventory every LLM-powered feature and classify each by task type (QA, RAG, summarization, code, safety, agent).
   • For each, pick 2–4 metrics: at least one automated, one judge-based, and one safety-oriented where applicable.
   • Write a one-page metric charter per feature — what it measures, why, threshold to ship.

   Success signals

   • Every LLM feature has a documented metric charter.
   • No two teams disagree on what "good" means for the same task type.
2. 2

   Build and version golden datasets

   Goal. Assemble representative, versioned evaluation sets that reflect production traffic.

   Actions

   • Sample 100–500 examples from real production traffic (PII-scrubbed).
   • Add adversarial cases: known hallucination triggers, injection attempts, edge distributions.
   • Version datasets in Git or DVC; tag with release, dataset hash, and refresh date.
   • Set a refresh cadence: quarterly for stable domains, monthly for fast-changing ones.

   Success signals

   • Every eval run points to a specific dataset version.
   • Datasets refreshed on cadence; a review log exists.
3. 3

   Wire evaluation into CI

   Goal. Make evaluation a blocking gate on prompt and model changes, not a manual step.

   Actions

   • Add an eval job to your CI pipeline triggered on any prompt or model config change.
   • Fail the PR if any metric drops below its charter threshold.
   • Cache eval results by prompt+model+dataset hash to keep CI fast.
   • Publish a per-PR eval report as a Markdown comment; link to full run in your eval tool.

   Success signals

   • A prompt regression is blocked in CI before merge — verified by a synthetic bad PR.
   • CI eval runtime under 10 minutes for common changes.
4. 4

   Red-team continuously

   Goal. Actively probe for hallucination, prompt injection, jailbreaks, and safety failures — not just on release day.

   Actions

   • Adopt an automated red-team suite (Giskard, Inspect AI) and integrate into CI.
   • Rotate manual red-team days: quarterly for consumer-facing systems, monthly for high-risk.
   • Track a red-team scoreboard by system, with severity classification and time-to-fix.

   Success signals

   • Automated red-team runs on every release candidate.
   • Severity-1 findings have a defined SLA to remediation.
5. 5

   Observe drift in production

   Goal. Detect quality regressions in the field, not just from user complaints.

   Actions

   • Instrument LLM traces with an observability tool (Langfuse, TruLens).
   • Sample production outputs and score with an LLM-as-judge, tracked over time.
   • Set alerts on 7-day rolling quality drop or judge-score divergence.
   • Wire alerts to on-call; define playbook for auto-rollback vs. investigate.

   Success signals

   • A drift alert triggers a real on-call page within 24 hours of a real regression.
   • Auto-rollback is a documented, rehearsed capability.
6. 6

   Calibrate LLM-as-judge and close the loop

   Goal. Keep judge scoring aligned to what humans actually care about; feed learning back into prompts.

   Actions

   • Label a sample of judge outputs with human review each cycle.
   • Track inter-annotator agreement and judge–human agreement over time.
   • Version rubrics; roll back judge prompts that drift out of calibration.
   • Feed high-signal failures into your prompt-optimization pipeline (e.g., DSPy) with human sign-off.

   Success signals

   • Judge–human agreement above 0.8 (Cohen's kappa) for critical metrics.
   • Rubric versions are auditable.