Do we need agent evals if we already do prompt evals?

Yes — and they measure different things. Prompt evals score a single LLM call on a static dataset. Agent evals score an entire trajectory: which tools were chosen, in what order, whether the state was correctly propagated, whether the final answer was reached, whether guardrails fired. An agent that scores 95% on prompt evals can still fail 40% of end-to-end tasks. Both layers are necessary for anything running in production.

Last reviewed: 2026-09-02