LLM-as-Judge
An evaluation pattern where one LLM (typically a strong model) grades outputs from another. Cheaper and faster than human eval, but subject to positional bias, sycophancy toward the model being evaluated, and mode collapse — mitigate with rubric prompts and reference answers.
Related terms
- Eval Harness
The full pipeline that runs a set of evaluation prompts through one or more models, scores the outputs (via reference match, LLM-as-judge, or human review), and reports results. Common open-source harnesses: lm-eval-harness, HELM, OpenAI evals. Enterprise programs usually build a custom harness against a golden-dataset backend.
- Guardrails
Runtime controls that constrain what an LLM system can produce or do — input validators, output filters, topic classifiers, tool allowlists, cost budgets, and human-approval checkpoints. Distinct from model-level safety training; guardrails run at inference time.
Related on this site
Where this fits
LLM-as-Judge is part of the Evaluation & Governance vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 149 defined terms, or take the free maturity assessment to see where your organisation stands.