AI Observability
The practice of instrumenting LLM applications to capture prompts, completions, tool calls, latencies, token counts, costs, feedback, and quality metrics — then analyzing that stream for drift, regressions, and abuse. Distinct from traditional APM; tools include Langfuse, Arize, Datadog LLM Obs, Braintrust.
Related terms
- Eval Harness
The full pipeline that runs a set of evaluation prompts through one or more models, scores the outputs (via reference match, LLM-as-judge, or human review), and reports results. Common open-source harnesses: lm-eval-harness, HELM, OpenAI evals. Enterprise programs usually build a custom harness against a golden-dataset backend.
- LLM-as-Judge
An evaluation pattern where one LLM (typically a strong model) grades outputs from another. Cheaper and faster than human eval, but subject to positional bias, sycophancy toward the model being evaluated, and mode collapse — mitigate with rubric prompts and reference answers.
Related on this site
Where this fits
AI Observability is part of the Evaluation & Governance vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 149 defined terms, or take the free maturity assessment to see where your organisation stands.