SWE-Bench Verified
A benchmark of real GitHub issues from popular Python repos where the model must produce a passing patch. The "Verified" subset filters for well-specified problems. In 2026 the leading indicator of coding-agent capability, with frontier models scoring 60-75.
Related terms
- Eval Harness
The full pipeline that runs a set of evaluation prompts through one or more models, scores the outputs (via reference match, LLM-as-judge, or human review), and reports results. Common open-source harnesses: lm-eval-harness, HELM, OpenAI evals. Enterprise programs usually build a custom harness against a golden-dataset backend.
- Reasoning Model
A class of LLMs (OpenAI o-series, Claude with extended thinking, DeepSeek-R1) trained to spend significant inference-time compute on internal reasoning before producing the final answer. Substantially better on math, coding, and multi-step problems at the cost of higher latency and token consumption.
Related on this site
Where this fits
SWE-Bench Verified is part of the Evaluation & Governance vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 149 defined terms, or take the free maturity assessment to see where your organisation stands.