LLM Observability Platforms: Langfuse vs LangSmith vs Arize Phoenix vs Datadog LLM Observability
LLM observability is converging on OpenTelemetry: all four platforms here accept OTLP traces using the GenAI semantic conventions, so instrumentation is becoming portable and the real decision is about where trace data lives and who runs it. Langfuse (MIT core, now owned by ClickHouse) and Arize Phoenix (Elastic License 2.0) are the self-host defaults, LangSmith is the natural fit for LangChain / LangGraph shops with an Enterprise path to self-host, and Datadog LLM Observability wins when you already run Datadog and want LLM traces next to APM. Instrument with OTel first, then pick the backend.
Last reviewed:
What we're comparing
Open-source LLM engineering platform (tracing, evals, prompt management, datasets). MIT core; acquired by ClickHouse in January 2026 with no planned licensing changes.
LangChain's commercial observability and evaluation platform. Native LangChain / LangGraph integration, but also ingests plain OpenTelemetry traces from any framework.
Source-available (ELv2) tracing and eval tool built on OpenTelemetry and OpenInference. Free to self-host; Arize AX is the managed enterprise platform using the same instrumentation.
LLM / agent tracing module inside the Datadog SaaS platform. Natively ingests OTel GenAI semantic conventions (v1.37+) and correlates LLM spans with existing APM and infrastructure data.
Side-by-side
| Attribute | Langfuse | LangSmith | Arize Phoenix | Datadog LLM Observability |
|---|---|---|---|---|
| License | MIT core; enterprise governance features (SCIM, audit logs, project RBAC) under a commercial license | Proprietary | Elastic License 2.0 (source-available, not OSI); no feature gates | Proprietary |
| Deployment | Cloud (US / EU / Japan / HIPAA regions) or self-host (Docker Compose for small setups, Kubernetes / Terraform for production) | Cloud (US or EU); hybrid and self-hosted (Kubernetes / Helm) as an Enterprise add-on with license key | Self-host anywhere (pip, Docker, Kubernetes / Helm); managed via Arize AX | Datadog SaaS only; not available on US government (ddog-gov) sites |
| OpenTelemetry ingestion | OTLP backend (HTTP JSON / protobuf; no gRPC yet); maps OTel GenAI semconv attributes | OTLP endpoint; maps GenAI semconv, OpenInference, Traceloop and Logfire attributes | Built on OTel + OpenInference instrumentation | Native GenAI semconv v1.37+ and OpenInference; direct OTLP, Collector, or Datadog Agent |
| Framework coupling | Framework-agnostic | Deepest with LangChain / LangGraph; OTel and SDK paths for everything else | Framework-agnostic | Framework-agnostic; auto-instrumentation in the Datadog SDK for common providers |
| Evaluation | LLM-as-judge on live traces and experiments, datasets, annotation queues, CI via GitHub Action | Datasets, offline and online LLM-as-judge, annotation queues | LLM-based response and retrieval evals, versioned datasets, experiments | Managed evaluators plus custom LLM-as-judge templates; experiments SDK |
| Prompt management | Versioned prompts with labels, client-side SDK caching, playground | Prompt hub with commits, tags, webhooks; playground | Prompt versioning, tagging, playground with trace replay | Prompt tracking linking template versions to LLM calls |
| Compliance claims (managed offering) | SOC 2 Type II, ISO 27001, GDPR, HIPAA-ready region | SOC 2 Type II, HIPAA (BAA on Enterprise), GDPR | Self-host: your controls. Arize AX lists SOC 2 Type II, HIPAA, PCI DSS 4.0; EU residency | Datadog platform programs apply; confirm product-level HIPAA / BAA coverage with Datadog |
| Pricing model | Self-host free; cloud usage-based tiers | Per seat plus per-trace usage; Enterprise custom | Free self-host; Arize AX commercial | Metered on LLM spans ingested, on top of Datadog contract |
| Best for | Self-hosted, OSS-first platform with full eval + prompt workflow | LangChain / LangGraph teams wanting one vendor for build, eval, deploy | OTel-native tracing and evals in your own environment; path to Arize AX | Orgs already standardized on Datadog for APM and on-call |
When to use which
- Use Langfuse
You need trace data inside your own boundary (data residency, regulated workloads) and want tracing, evals and prompt management in one OSS stack. Budget for running ClickHouse, Postgres, Redis and blob storage in production.
- Use LangSmith
Your agents are built on LangChain / LangGraph and you want the tightest integration across tracing, evaluation and deployment. Self-hosting is possible but sits on the Enterprise plan.
- Use Arize Phoenix
You want a free, self-hosted, OTel / OpenInference-native tool for tracing and evals during build and early production, with the option to move to Arize AX later without re-instrumenting. Check ELv2 terms if you plan to offer it to third parties as a service.
- Use Datadog LLM Observability
Datadog is already your system of record for APM, logs and incidents, and you want LLM spans in the same traces, dashboards and monitors. Not a fit if trace data must stay self-hosted.
FAQs
Which LLM observability platform should we start with?
Instrument with OpenTelemetry (GenAI semantic conventions or OpenInference) first — all four platforms here ingest it, so the backend becomes a reversible choice. At early maturity, a self-hosted Phoenix or Langfuse is enough. As you move to production at scale, pick based on where trace data must live and which team owns on-call: platform-engineering-owned Datadog, or an AI-team-owned Langfuse / LangSmith.
What happened to Helicone?
Mintlify acquired Helicone in March 2026. Per the announcement, Helicone continues in maintenance mode (security updates, bug fixes, new models) and Mintlify is helping customers migrate to other platforms. Do not adopt it for new enterprise workloads; existing users should plan a migration.
Does the ClickHouse acquisition change Langfuse licensing?
Not according to the January 2026 announcement: Langfuse stays open source and self-hostable, with no planned licensing changes and Langfuse Cloud continuing as-is. Langfuse already stores traces in ClickHouse, so the acquisition aligns with its existing architecture. As with any acquired vendor, track the license file and roadmap at each renewal.
Do we need a separate observability tool if we already use an LLM gateway?
Usually yes for mature programs. Gateways capture request-level logs, cost and latency, but agent workflows need nested spans (retrieval, tool calls, sub-agents), evaluation scores attached to traces, and dataset curation from production. Many teams export gateway and app spans via OTel into one of the platforms above.
Get updates when this comparison changes
We re-review comparisons as tools and pricing move. One practical GenAI note per month, plus alerts on material changes. Unsubscribe anytime.
Related comparisons
- LLM Evaluation Frameworks: HELM vs lm-eval-harness vs OpenAI Evals vs Braintrust vs Ragas
- Ragas vs DeepEval: Choosing an LLM Evaluation Framework
- Direct LLM API vs LLM Gateway: Portkey vs Kong AI vs LiteLLM vs Custom
- LLM Gateway Comparison: LiteLLM vs Portkey vs Kong AI Gateway vs Cloudflare AI Gateway