LLM Observability & Evals

What is LLM observability?

LLM observability is the discipline of instrumenting your LLM application so you can see inside every request: the full prompt, the model's response, token counts, latency, cost, and the chain of tool calls or retrievals that produced the answer. It combines tracing (request-level spans showing the full execution path), evaluation (automated scoring of output quality), and monitoring (aggregate dashboards for cost, latency, and error rates). Without it, production LLM failures are opaque — you cannot debug what you cannot see.

The three platforms that matter in 2026

LangSmith is LangChain's proprietary observability platform with the deepest LangChain integration, but it started charging per-trace in 2024. Langfuse is fully open-source (MIT, 23,000+ GitHub stars), self-hostable, and vendor-neutral — the default for cost-conscious teams. Arize Phoenix offers an open-source core with a commercial Arize AX tier, strong on ML-native evaluation. The 2026 LLM observability market is projected at $2.69B growing to $9.26B by 2030 (Birjob).

Why evaluation is the hard part

Tracing tells you what happened; evaluation tells you whether it was good. Traditional software has unit tests; LLM outputs are open-ended, so evaluation requires either expensive human labeling or LLM-as-a-judge — using a strong model to score a weaker model's outputs. The arXiv bias literature (2506.22316) finds GPT-4.1 scores correlate highest with human judgment, but every LLM judge carries biases (position bias, verbosity bias) that must be measured before trusting the scores.

Related clusters

• LLM Gateways & Cost — LiteLLM and Portkey integrate with observability platforms for tracing
• Inference Engines — vLLM and SGLang export Prometheus metrics for the monitoring layer
• Cost Optimization — observability is the measurement layer that makes optimization possible