Promptfoo vs OpenAI Evals: A/B Testing and Regression for LLM Apps
Promptfoo and OpenAI Evals both let you compare prompts and models programmatically, but they optimise for different intents: Promptfoo is a CLI + web UI for local A/B testing, matrix comparisons, and red-teaming across providers; OpenAI Evals is a registry-based framework for writing durable evals that any OpenAI-compatible model can run at scale.
Last reviewed:
What we're comparing
CLI + web UI for prompt/model A/B testing, red-teaming, and CI regression across providers. YAML-configured.
Registry-based eval framework; author custom evals in YAML + Python and run against any OpenAI-compatible model.
Side-by-side
| Attribute | Promptfoo | OpenAI Evals |
|---|---|---|
| Primary optimisation | Prompt / model comparison and red-teaming with a fast local loop | Durable, versioned custom evals runnable at scale by any team member |
| Config format | YAML; matrix of prompts × providers × test cases inline | YAML registry + Python for custom eval logic |
| Provider coverage | OpenAI, Anthropic, Google, Cohere, Bedrock, Azure, Ollama, Hugging Face, local (30+) | OpenAI-compatible APIs primarily; others via custom completion_fn |
| Web UI | Local web viewer for side-by-side comparison + shareable HTML reports | CLI only; results as JSON logs or plot via separate tooling |
| Red-teaming | First-class: `promptfoo redteam` generates adversarial cases across OWASP LLM Top 10 | Not built-in; you can encode adversarial cases as evals but no generator |
| CI ergonomics | GitHub Action + exit codes; fails PR on regression | CLI produces JSON; wire into CI yourself |
| Custom metrics | Built-in asserts (contains, regex, LLM-rubric, similarity) + custom JS/Python | Python subclasses; deeper flexibility for complex scoring |
| Best-fit team | Engineers iterating on prompts daily who want fast local feedback + a share button | ML teams building a durable eval library that outlives any single prompt change |
When to use which
- Use Promptfoo
You compare prompts and models weekly, want a shareable HTML report to send to stakeholders, or need to red-team a system before release.
- Use OpenAI Evals
You want to build an eval library that lives with your codebase for years, with the flexibility of Python for complex scoring logic.
FAQs
Which one gates my CI?
Promptfoo is friendlier — a single GitHub Action + exit code. OpenAI Evals runs cleanly in CI too, but you write the harness that decides pass/fail.
Do I still need Ragas or DeepEval?
Yes for the metric layer. Promptfoo and OpenAI Evals orchestrate runs and comparisons; they call metric implementations underneath (either built-in or from Ragas/DeepEval).
What about vendor red-teaming?
Giskard and the UK AI Safety Institute's Inspect AI are stronger dedicated tools. Promptfoo's red-team mode is a solid starting point but not a replacement for a full adversarial program.