AI Agent Observability Guide 2026: Monitoring, Tracing, Evaluation

What is AI agent observability in 2026?

In short

AI agent observability is the practice of capturing every LLM call, tool invocation, retrieval, and reasoning step from an autonomous agent as structured traces, then layering evaluation scores, cost, and latency metrics on top. It combines OpenTelemetry GenAI semantic conventions with OpenInference span kinds — LLM, CHAIN, TOOL, RETRIEVER, EMBEDDING, AGENT — and platforms like LangSmith, Langfuse, Arize Phoenix, and Braintrust.

AI agents are non-deterministic. The same input can produce different reasoning trajectories, different tool call sequences, and different final answers on separate runs. That single property invalidates the mental model classical APM was built on — request in, response out, count the errors, chart the p95. For agents, one user turn fans out into dozens of LLM calls, tool invocations, retrievals, and reasoning steps. You cannot debug that from a flat log line.

The discipline in 2026 rests on four signals, and every serious production stack emits all four:

• Structured traces — a nested span tree per user turn, with each LLM, tool, and retrieval call captured as its own span.
• Evaluation scores — offline (Ragas, DeepEval, custom judges) and online (user feedback, thumbs, task completion) — attached to the same trace IDs.
• Cost per run — input tokens, output tokens, cache read/creation tokens, aggregated by user, tenant, and agent version.
• Tail latency — first-token, total-trace, and per-tool durations, not just entrypoint p95.

Two open standards do most of the heavy lifting. OpenTelemetry GenAI semantic conventions, maintained in the open-telemetry/semantic-conventions-genai repository, define the attribute names every backend should recognize — gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, agent and tool span kinds. OpenInference, from Arize AI, defines the six span kinds every agent should emit: LLM, CHAIN, TOOL, RETRIEVER, EMBEDDING, and AGENT.

Alice Labs runs this stack across 100+ production agent deployments. This guide documents what we standardize on, what we alert on, and which platform we reach for when the trade-offs matter. For a broader comparison of the frameworks these agents run on, see our best AI agent frameworks guide for 2026.

Why traditional APM breaks for LLM agents

In short

Datadog and New Relic without GenAI extensions miss the failure modes that matter for agents: silent quality regressions never raise HTTP 5xx, token cost variance of 10-100x within a single endpoint invalidates flat p95 SLOs, tool-loop failures manifest as latency spikes not error rates, and prompt/model changes require diffable replayable spans that APM stacks do not store. GenAI-native platforms (LangSmith, Langfuse, Phoenix, Braintrust) fill these gaps.

APM was built for deterministic request-response services. Every failure mode it is good at — 5xx spikes, latency regressions, memory leaks — is defined by a clean binary or numeric signal at a service boundary. LLM agents break all of those assumptions, and the failure modes that actually kill agent deployments are invisible to a stock Datadog or New Relic setup.

The four gaps we see most often:

• Silent quality regressions never trip HTTP 5xx. A model swap turns a helpful summariser into a lazy one. Every response is a 200 OK. Every latency chart looks fine. Users churn. Classical APM tells you nothing until support tickets spike a week later.
• Token cost variance destroys flat SLOs. Cost per request in an agent workload can vary 10–100× on the same endpoint depending on tool-loop count and retrieved context size. Averaging or p95-ing it hides the runaway loops that generate four-figure bills.
• Tool-loop failures show up as latency, not errors. An agent stuck retrying a broken tool never raises an exception — it just takes longer. p95 of the entrypoint moves by seconds, not the orders of magnitude you would expect from a real regression. You need per-tool-call spans and loop-count alerts to see it.
• Prompt and model changes require replayable spans. When quality drops after a prompt tweak, the debugging move is to rerun the exact captured call against the old prompt. Datadog stores enough to know an event happened; it does not store enough to rerun it. Arize Phoenix explicitly builds this capability.

None of this makes classical APM obsolete — you still need it for the containers, the queues, the databases. But for the agent-shaped part of the stack, you need a GenAI-native layer alongside. That is what LangSmith, Langfuse, Arize Phoenix, and Braintrust are built for.

OpenTelemetry GenAI semantic conventions explained

In short

OpenTelemetry GenAI semantic conventions, maintained in the open-telemetry/semantic-conventions-genai repository, define the standard span, attribute, metric, and event names for LLM and agent telemetry. Key attributes: gen_ai.system, gen_ai.request.model, gen_ai.operation.name, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens. The spec covers agent spans, MCP integration, and provider-specific attributes for Anthropic, Bedrock, Azure AI Inference, and OpenAI.

OpenTelemetry's GenAI working group split the semantic conventions for AI workloads into a dedicated repository in 2026: open-telemetry/semantic-conventions-genai. Standardizing on it early is the single cheapest reliability investment available — it makes your traces portable across backends and future-proofs the schema before it gets locked in.

The core attribute set is small and worth memorising:

• gen_ai.system — the provider (openai, anthropic, aws.bedrock, azure.ai_inference, vertex_ai).
• gen_ai.request.model — the model ID requested (claude-opus-5, gpt-5, gemini-2.5-pro).
• gen_ai.response.model — the model that actually served the response, which can differ under server-side fallbacks.
• gen_ai.operation.name — chat, text_completion,embeddings, tools.
• gen_ai.usage.input_tokens, gen_ai.usage.output_tokens — the atomic cost signal.

Beyond the base attributes, the spec covers three things that matter for agents: agent and tool span kinds are first-class (not bolted on as string attributes), MCP integration has dedicated span semantics, and provider-specific attributes are standardized for Anthropic (prompt caching, tool_use), Bedrock (Converse API), Azure AI Inference, and OpenAI (Responses API, tool_calls).

The practical benefit: your instrumentation code emits standard OTel spans; any OTel-compatible backend can ingest them; you can switch from a hosted vendor to self-hosted Phoenix or Langfuse by changing an environment variable, not rewriting SDK calls. Alice Labs' rule is that any framework-specific attribute goes on top of the standard schema, never in place of it.

OpenInference span kinds every agent must emit

In short

OpenInference defines six span kinds: LLM (a model call), CHAIN (composite step), TOOL (function or API invocation), RETRIEVER (search/vector lookup), EMBEDDING (vector generation), and AGENT (top-level autonomous run). Instrumentors exist for OpenAI, Anthropic, Claude Agent SDK, LangChain, LlamaIndex, DSPy, Bedrock, CrewAI, and PydanticAI in Python; Vercel AI, LangChain.js, OpenAI, Anthropic, Bedrock, and Claude Agent SDK in TypeScript. Any OpenTelemetry-compatible collector accepts OpenInference.

OpenInference is the concrete instrumentation contract that turns the abstract OTel GenAI schema into working code. It defines six span kinds and provides instrumentors for every major framework and provider. Alice Labs standardizes on it because the span kinds map cleanly to what agents actually do:

• AGENT — the top-level autonomous run. Root span for a user turn. Attributes: session_id, user_id, agent_version, model_id, tool_registry_hash.
• CHAIN — a composite step (RAG pipeline, plan-and-execute stage). Groups related child spans.
• LLM — a single model call. Attributes cover input, output, tokens, temperature, and tool_calls.
• TOOL — a function or external API invocation. Attributes: tool_name, tool_input, tool_output, tool_success.
• RETRIEVER — a search or vector lookup. Attributes: query, retrieved_documents, retrieval_score, chunk_count.
• EMBEDDING — a vector generation call. Attributes: input text, embedding vector (or hash), model.

Python instrumentors ship for OpenAI, Anthropic, Claude Agent SDK, LangChain, LlamaIndex, DSPy, Bedrock, CrewAI, PydanticAI, Mistral, Groq, LiteLLM, and Guardrails. TypeScript coverage includes Vercel AI, LangChain.js, OpenAI, Anthropic, Bedrock, and Claude Agent SDK. Java support covers the LangChain4J and Spring AI ecosystem.

The critical property: any OpenTelemetry-compatible collector accepts OpenInference. Arize Phoenix (native), Arize AX (native), and Langfuse (via attribute mapping) ingest it directly. A generic OTel Collector routes it anywhere else. This is what lets us instrument once and choose the backend based on cost, data residency, or compliance needs — never on which SDK we happened to install.

# Python — instrument the Claude Agent SDK with OpenInference
from openinference.instrumentation.anthropic import AnthropicInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="https://cloud.langfuse.com/api/public/otel/v1/traces")
    )
)
trace.set_tracer_provider(provider)

AnthropicInstrumentor().instrument()
# every anthropic.messages.create() call now emits an LLM span

LangSmith deep dive: tracing, evals, and Engine

In short

LangSmith is LangChain's commercial hosted observability platform. Pricing (2026): free Developer tier with 5k base traces/mo, 1 seat, 14-day base retention; Plus at $39/seat with 10k base traces, unlimited seats, and access to the Engine and Deployment products; extended trace retention up to 400 days for an additional fee. Priced in LCU ($1.50) and LSU ($1.00) units. Engine auto-detects recurring issues and diagnoses root causes. Native integrations with OpenAI, Anthropic, CrewAI, Vercel AI SDK, and Pydantic AI.

LangSmith is what LangChain, Inc. ships as its commercial observability product. It is the natural default for teams whose codebase already lives inside LangChain and LangGraph — instrumentation is one environment variable, and the tracing UI knows how to render chain and graph structure directly.

The 2026 pricing model has two dimensions to know about. Seats are per-user (Developer free at 1 seat, Plus at $39/seat/mo unlimited). Traces are counted in LangSmith Charge Units (LCUs) at $1.50 and LangSmith Storage Units (LSUs) at $1.00 — the base plans include a monthly allowance (5k and 10k base traces) and everything over it consumes credits. Extended trace retention up to 400 days is priced separately and matters for teams under regulatory retention requirements.

The two features most teams underinvest in:

• Engine — LangChain's auto-diagnosis product. Detects recurring issues across traces, groups them by root cause, and points at the offending prompt or tool. Available on Plus and up.
• Deployment — hosted LangGraph runtime with per-run tracing wired in by default. Useful when you want managed hosting for a LangGraph agent and one bill instead of two.

LangSmith's native integrations cover OpenAI, Anthropic, CrewAI, Vercel AI SDK, and Pydantic AI in addition to every LangChain-family framework. OpenTelemetry support exists but is partial — you can send LangSmith spans over OTel, but the richest experience is still through the native SDK. Self-hosting is Enterprise-tier only. If open source or full self-host is a hard requirement, this is the wrong platform and Langfuse or Phoenix are the answer.

Alice Labs' rule: LangSmith wins when the team already lives inside LangChain, wants managed hosting, and values the Engine auto-root-cause product. Everywhere else, we reach for a self-hostable OTel-native platform first.

Langfuse deep dive: OSS, OpenTelemetry-native, self-hostable

In short

Langfuse is the open-source, OpenTelemetry-native observability platform. Cloud regions: EU, US, Japan, with a HIPAA-certified endpoint. Hobby free tier: 50k units/month, 30-day retention, 2 users. Pro at $199/month: 100k units/month, 3-year retention, SOC2 and ISO 27001. OTLP endpoint at /api/public/otel accepts JSON and protobuf. Native support for OpenLIT, OpenLLMetry, and OpenInference attribute mapping. Self-hosting under an OSS licence is a first-class shipping mode.

Langfuse is the reference for what a self-hostable, OpenTelemetry-native agent observability platform looks like in 2026. It ships as an OSS project with a production-grade Docker Compose and Kubernetes story, and a managed cloud with EU, US, and Japan regions plus a HIPAA-certified endpoint for US healthcare workloads.

The core commercial hooks:

• Hobby (free) — 50k units/month, 30-day retention, 2 users. Real enough for a startup to run production on.
• Pro ($199/month) — 100k units/month, 3-year retention, SOC2 and ISO 27001. This is the tier most enterprise teams land on for cloud.
• Team and Enterprise — higher unit ceilings, SLA, and dedicated support. Both cloud and self-host paths.

The OTLP endpoint at /api/public/otel accepts both HTTP JSON and protobuf, and Langfuse maps OpenInference, OpenLIT, and OpenLLMetry attributes on ingest automatically. This is the property that makes framework-agnostic instrumentation actually portable — you can point any OTel-compatible tracer at Langfuse and it renders as a first-class trace, not a raw span dump.

Beyond tracing, Langfuse ships an evaluation module, dataset management, and a prompt management surface with versioning. The prompt management piece is the one most teams adopt slower than they should — versioned prompts stored in Langfuse and pulled at runtime turn "we tweaked the prompt and quality dropped" into a diff you can actually roll back.

# Environment variables — any OTel SDK can ship to Langfuse
OTEL_EXPORTER_OTLP_ENDPOINT=https://cloud.langfuse.com/api/public/otel
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64(public:secret)>"
OTEL_SERVICE_NAME=my-agent-service

Alice Labs runs self-hosted Langfuse in customer VPCs across the Nordics for EU AI Act-scoped deployments. The combination of OTel-native ingest, EU data residency, and a real OSS licence covers the compliance surface that hosted-only vendors cannot.

Arize Phoenix deep dive: OSS observability plus OpenInference

In short

Arize Phoenix is Arize AI's OSS observability platform, built directly on OpenTelemetry and OpenInference (Arize authors the OpenInference spec). Self-hostable on Docker, Kubernetes, and major clouds. Span replay lets you rerun a captured LLM call against a new prompt or model. Native integrations with Ragas, DeepEval, and Cleanlab for evaluator plug-ins. Supports LlamaIndex, LangChain, DSPy, Mastra, Vercel AI SDK, OpenAI, Bedrock, and Anthropic in Python, TypeScript, and Java.

Arize Phoenix is the OSS observability platform from Arize AI, and it is the one we reach for when the debugging problem is deep. Two properties make it distinctive: Arize authors the OpenInference specification, so Phoenix's ingest is native rather than an adapter; and Phoenix ships span replay, which lets you rerun a captured LLM call against a new prompt, a new model, or a modified tool schema without leaving the observability UI.

Span replay is a bigger deal than it sounds. When quality drops after a prompt tweak, the debugging move most teams cannot do is "rerun this exact span against yesterday's prompt". Phoenix captures enough of the original context (messages, tool_calls, attachments) that this becomes a UI action, not a code change and redeployment. Alice Labs uses this for every non-obvious quality regression — you find the failing span, you rerun it three ways, you have the root cause in minutes instead of hours.

Evaluation is the second area where Phoenix punches above its weight. Native plug-ins for Ragas, DeepEval, and Cleanlab let you score traces without exporting data. Custom evaluators plug in as Python functions. LLM-as-judge is built in.

Framework coverage in 2026: LlamaIndex, LangChain, DSPy, Mastra, Vercel AI SDK, OpenAI, Bedrock, Anthropic in Python; expanded coverage in TypeScript and Java. If you use a framework OpenInference has an instrumentor for — and that is almost every serious framework — Phoenix ingests it out of the box.

Deployment is Docker, Kubernetes, or major-cloud managed services. There is no metered pricing on the OSS project itself; Arize AX is the paid managed platform that shares the same trace format. Alice Labs uses Phoenix in EU customer VPCs where we want a fully open, no-vendor-lock-in observability layer alongside Langfuse.

Braintrust deep dive: eval-first observability

In short

Braintrust treats evaluations as the primary artifact — the six-stage workflow is instrument, observe, annotate, evaluate, deploy, administer. Consumption pricing: Starter free ($10 model credits, 1GB/month, 10k scores, 14-day retention); Pro $249/month ($249 model credits, 5GB/month, 50k scores, 30-day retention, RBAC). Unlimited users on every tier including free. The bt CLI runs evals, instruments code, queries logs, syncs data, and manages functions.

Braintrust reframes the observability problem: evals come first, everything else hangs off them. The workflow it enforces is six stages — instrument, observe, annotate, evaluate, deploy, administer — and every action in the platform slots into one of them. For teams whose culture is CI-first and regression-test-first, this is the observability platform that matches how you already work.

Pricing is consumption-based, and the free tier is real:

• Starter (free) — $10 model credits, 1 GB/month, 10k scores, 14-day retention, unlimited users.
• Pro ($249/month) — $249 model credits, 5 GB/month, 50k scores, 30-day retention, RBAC, unlimited users.
• Enterprise — SSO, custom retention, private deployment, SLA.

The unlimited-users property on every tier including free is the one that matters at scale — you do not pay per engineer to look at eval results, which unblocks the cultural adoption problem that per-seat platforms create.

The bt CLI is where Braintrust earns its keep for engineering teams. It runs evals against captured logs, instruments code, queries logs, syncs local dataset files, and manages the Braintrust "functions" concept (prompts and eval logic versioned as first-class artifacts). Wiring bt eval into CI as a regression gate before every prompt or model change is the pattern Alice Labs uses most: no prompt ships until the eval suite passes.

# Braintrust — gate every prompt change on eval regression
npx bt eval ./evals/answer-quality.eval.ts \
  --dataset production_regression_set \
  --score-threshold 0.85 \
  --fail-below-threshold

Braintrust prefers its own SDK over OpenTelemetry ingest. If OTel-native is a hard requirement (typically for teams that already run a shared collector), Langfuse or Phoenix are better matched. If eval-first culture is the top priority and OTel is optional, Braintrust wins.

LangSmith vs Langfuse vs Arize Phoenix vs Braintrust

In short

Self-host: Langfuse (OSS) yes, Phoenix (OSS) yes, LangSmith enterprise-only, Braintrust enterprise-only. OpenTelemetry-native ingest: Langfuse yes, Phoenix yes, LangSmith partial, Braintrust proprietary SDK first. Pricing model: LangSmith per-seat + per-trace, Langfuse per-unit, Phoenix free OSS, Braintrust consumption. Best fit: LangSmith for LangChain-heavy stacks; Langfuse or Phoenix for OSS/self-host; Braintrust for eval-first teams.

No single platform wins on every axis. The four dimensions that actually change the answer:

Comparison — LangSmith, Langfuse, Arize Phoenix, Braintrust (August 2026)

The Alice Labs decision tree in practice: if EU data residency is required, start with Langfuse EU cloud or self-hosted Phoenix. If the team is already deep in LangChain and wants managed hosting, LangSmith. If the culture is regression-test- first and CI-heavy, add Braintrust for eval gating on top of whatever tracing backend you picked. For fully OSS, multi-framework production stacks, Phoenix plus OpenInference is the pattern we default to. See our companion piece on the best AI agent frameworks for 2026 for how the underlying agent frameworks fit alongside these choices.

What to instrument in every AI agent

In short

The Alice Labs instrumentation contract: (1) root AGENT span with session_id, user_id, agent_version, model_id, tool_registry_hash; (2) LLM span per model call with gen_ai.usage.input_tokens/output_tokens/cache_read_tokens; (3) TOOL span per tool call with tool_name, tool_input_hash, tool_latency_ms, tool_success; (4) RETRIEVER span with query, retrieved_doc_ids, retrieval_score, chunk_count; (5) structured feedback events tied to trace_id for online evaluation.

The instrumentation contract below is what Alice Labs enforces in code review for every production agent, regardless of framework. It is deliberately boring — no clever tricks, just the five spans and one event stream that make every other piece of observability tractable.

Root AGENT span. One per user turn. Attributes: session_id, user_id, tenant_id, agent_version, model_id, tool_registry_hash, prompt_hash. This is the span every downstream analysis groups by — miss any of these attributes and you can't segment by cohort, agent version, or prompt version later.

LLM span per model call. gen_ai.system, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read_tokens, gen_ai.usage.cache_creation_tokens, plus retry_count and retry_reason when applicable. Prompt caching attribution is the number one cost signal most teams get wrong.

TOOL span per tool call. tool_name, tool_input_hash (never the raw input if it contains PII), tool_latency_ms, tool_success, tool_error_class. The tool_success flag is what turns "the agent is slow" tickets into "the Postgres MCP is timing out" tickets.

RETRIEVER span for every search call. query, retrieved_doc_ids, retrieval_score, chunk_count, rerank_score if reranked. This is what makes RAG debugging tractable — without retrieved_doc_ids you cannot answer "did the retriever find the right document?".

Structured feedback events. Emit user thumbs-up/down, task-success flags, and any qualitative annotation as events tied to trace_id. This is what feeds online evaluation and the offline eval datasets. Do not put feedback in a separate table with no join key — you will regret it inside a month.

For a deeper look at the tool-use side of this contract across different frameworks, our guide on AI agent tool-use patterns covers the archetypes each of these spans instruments.

Evaluation frameworks: Ragas, DeepEval, LLM-as-judge

In short

Ragas metrics cover faithfulness, context precision, context recall, response relevancy, agent goal accuracy, and tool call accuracy. DeepEval offers G-Eval (criteria-based chain-of-thought), DAG (conditional multi-step), QAG (reference-grounded), plus hallucination and task completeness — and reports 100M+ daily evaluations across 150k+ developers and >50% of the Fortune 500. LangSmith adds code evaluators for deterministic checks, LLM-as-judge for subjective, and pairwise for A/B comparison.

Evaluation is where observability graduates from "we see what happened" to "we know whether it was good". Three frameworks own the space in 2026:

• Ragas — the RAG-focused eval library. Its agent-mode metrics (agent goal accuracy, tool call accuracy) extend beyond retrieval into agent-shaped workflows. Faithfulness, context precision, context recall, and response relevancy remain the reference RAG scores.
• DeepEval — general-purpose LLM evaluation with four evaluator families: G-Eval (criteria-based chain-of-thought scoring), DAG (conditional multi-step), QAG (reference-grounded question answering), and specialised evaluators for hallucination and task completeness. DeepEval reports 100M+ daily evaluations across 150k+ developers and more than 50% of the Fortune 500 — meaning its scoring definitions have become de-facto industry defaults.
• LangSmith evaluators — three shapes ship in LangSmith itself. Code evaluators for deterministic checks (regex, structural, schema). LLM-as-judge for subjective quality. Pairwise for A/B comparison of two agent versions.

The two practical decisions to make early:

Where does the eval dataset live? Braintrust and LangSmith both offer versioned dataset storage. Phoenix and Langfuse both let you tag traces into datasets. If your team already has a source-controlled test dataset (JSON files in the repo), keep it there and load into the platform — do not fragment the golden data across two systems.

Which evals gate CI? Not every metric belongs in CI. Deterministic structural checks (JSON schema, tool-call correctness, output length) always. LLM-as- judge checks with a stable rubric and a locked judge model, yes. Free-form "helpfulness" scoring from an unlocked judge, no — the score drift when the judge model updates will break your build for reasons unrelated to your agent. Alice Labs gates on the deterministic and rubric-locked half, tracks the free-form half in a dashboard.

For the broader picture on how evaluation fits into a production agent lifecycle, see our AI agent evaluation metrics guide covering online, offline, and production A/B measurement patterns.

Cost monitoring: token accounting done right

In short

Emit gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on every LLM span. Track cache_creation_tokens and cache_read_tokens separately for prompt caching accuracy — cached reads are 10-90% cheaper depending on the provider. Alert on p95 cost-per-conversation, not average; long-tail loops dominate spend. Attribute cost to user_id, tenant_id, and agent_version for chargeback and cohort analysis.

Cost is where the difference between "we have observability" and "we do not" shows up first, because runaway agent loops turn into four-figure bills over a single hour. The two failure modes we see most often:

• The team charts average cost per request. Averages hide the tail. A single agent stuck in a tool-retry loop consumes 100× the tokens of a normal run and disappears into the average of a thousand well-behaved sibling requests. You do not see it until the monthly bill arrives.
• Prompt caching is not attributed. Anthropic prompt caching, OpenAI cached inputs, and Gemini implicit caching all price cached reads at 10-90% cheaper than fresh reads depending on provider. If your span emits only a single "tokens" field, cost attribution is wrong by up to an order of magnitude on cache-heavy workloads.

The Alice Labs cost-instrumentation contract:

• Emit gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_creation_tokens, gen_ai.usage.cache_read_tokens on every LLM span.
• Compute cost per span at ingestion, not at query time — model prices change and you want the cost frozen at the moment the call happened.
• Aggregate by user_id, tenant_id, agent_version, model_id. All four axes matter for chargeback and regression detection.
• Alert on p95 cost-per-conversation and on tool-loop count above a per-agent threshold. Both catch runaway agents before the bill does.

The production catch that saves the most money: alert on tool-loop count, not just cost. Alice Labs has caught runaway agent loops that would have cost four figures in a single hour by paging on a threshold of 15 tool calls in one agent run — long before the cost signal itself crossed our alert band.

For a deeper treatment of AI economics beyond the observability layer, see our build vs buy AI guide covering model spend forecasting and per-agent unit economics.

Latency monitoring: tail-first SLOs for tool-using agents

In short

Track first-token latency and total-trace latency separately. Alert on tool-loop count exceeding a per-agent threshold, not just wall-clock. Streamed responses need dedicated time-to-first-token spans. Retry attributes (retry_count, retry_reason) are mandatory or root-cause analysis is impossible. Alice Labs pages on-call when tool-loop count exceeds a per-agent SLO — this catches misbehaving agents before latency alone does.

Latency for agents is not one number. Users perceive first-token latency (how long until anything appears on screen); infrastructure cares about total-trace latency (how long until the container recycles); on-call cares about tool-loop count (how many spans deep the agent went before finishing). All three need their own signal.

The Alice Labs latency-instrumentation contract:

• First-token latency span. For streamed responses, emit a dedicated time_to_first_token_ms attribute on the root LLM span. This is the user-perceived latency and the only one that correlates with abandonment.
• Total-trace latency. Duration of the root AGENT span. This is what infrastructure alerting cares about.
• Per-tool latency. Duration on every TOOL span. Slow MCP servers, database timeouts, and network glitches all surface here.
• Retry attributes. retry_count, retry_reason on every LLM and TOOL span that retried. Without these, root-cause analysis for intermittent slowness is impossible.

The alert we care about most is tool-loop count, not latency itself. A healthy agent finishes in 3-8 tool calls. An unhealthy one loops through 40+ tool calls, each within its individual timeout, and hits total latency in the minutes. Alerting on tool-loop count > per-agent threshold catches this failure mode before p95 latency does — and it correlates directly with the cost blow-up alert from the previous section.

One production trap: streaming APIs mask latency problems. Your agent starts emitting tokens fast, so first-token latency looks fine, and total-trace latency is "big number" because there's a lot to stream. Neither tells you the agent looped 30 times through a broken retrieval before generating the answer. This is why the tool- loop-count signal exists.

For a deeper treatment of agent architectural patterns that affect latency (parallel fan-out, supervisor/worker, reflection loops), see our AI agent architecture patterns guide.

The Alice Labs production observability stack

In short

Alice Labs' reference stack across 100+ production deployments: OpenInference SDK for framework-agnostic instrumentation across LangGraph, CrewAI, and Claude Agent SDK; Langfuse or Arize Phoenix self-hosted in EU for GDPR and EU AI Act compliance; Braintrust for eval regression gates in CI before every prompt or model change; PagerDuty on cost-per-conversation, tool-loop count, and eval-score regression — not just error rate.

The Alice Labs default across 100+ deployed agent workloads is deliberately layered — one component per job, no single platform trying to do everything:

• OpenInference SDK for instrumentation. Framework-agnostic. Runs across LangGraph, CrewAI, Claude Agent SDK, OpenAI Agents SDK, and every other framework we ship on. Instrumentation code is written once and stays stable when the backend changes.
• Langfuse or Arize Phoenix for tracing. Both self-hosted in the customer's EU VPC for GDPR and EU AI Act compliance. Langfuse when the team wants prompt management and evaluation in the same UI as traces; Phoenix when the debugging workload benefits from span replay.
• Braintrust for CI eval gates. bt eval runs the regression suite on every prompt or model change; a score below threshold fails the pipeline. This is what stops silent quality regressions from shipping.
• PagerDuty alerts on three signals. p95 cost-per-conversation, tool-loop count per agent, and eval-score regression on the production regression set. Error-rate alerts stay too — but they are the least useful of the four.

The one non-negotiable piece: a PreToolUse equivalent hook that stamps every tool call into an immutable ledger for EU AI Act Article 12 compliance. In the Claude Agent SDK this is the built-in PreToolUse hook (see our Claude Agent SDK guide); in LangGraph it is a middleware wrapper around every tool node; in CrewAI it is a tool-level decorator. Six months' retention minimum for high-risk deployments.

For the broader compliance picture that this stack enables, our EU AI Act compliance checklist for 2026 covers Article 12 tool logging alongside the rest of the requirements. And for architectural context on the frameworks that this observability layer sits above, best AI agent frameworks 2026 compares LangGraph, CrewAI, and the Claude Agent SDK head to head.

How to choose an observability platform in 2026

In short

Alice Labs' decision framework: (1) EU data residency required — Langfuse EU cloud or self-hosted Phoenix; (2) Enterprise SSO/audit and LangChain-heavy stack — LangSmith Enterprise; (3) eval-first culture with strong CI discipline — Braintrust for gating, plus a tracing backend; (4) fully OSS and multi-framework — Arize Phoenix plus OpenInference. Combining platforms (Langfuse tracing + Braintrust eval gates) is the pattern that ships most often.

No single platform wins every deployment. The four questions we walk clients through in order:

1. Is EU data residency required? If the answer is yes — because the deployment is EU AI Act-scoped, or because customer data is GDPR-restricted, or because contracts require it — Langfuse EU cloud or self-hosted Phoenix are the answers. LangSmith is self-host on Enterprise only; Braintrust is Enterprise only. Do not defer this question — it constrains everything downstream.

2. Does the team live inside LangChain? LangChain and LangGraph shops get more value from LangSmith than any other platform because the native rendering of chains and graphs matches how the code is organised. Engine (LangSmith Plus) auto-diagnoses recurring issues in a way generic tracing UIs cannot. If the answer is yes and EU residency is not required, LangSmith wins.

3. Is the culture regression-test-first? Teams that already run eval suites in CI, treat prompts as versioned artifacts, and want to gate deploys on quality get the most out of Braintrust. This is orthogonal to the tracing choice — Braintrust for eval gates plus Langfuse or Phoenix for traces is the combination that ships most often at Alice Labs.

4. Is OSS a hard requirement? If the answer is yes — because the company runs everything self-hosted, or because vendor lock-in is a boardroom risk, or because engineering culture demands it — Arize Phoenix plus OpenInference is the multi-framework OSS stack. Langfuse is also OSS but has more managed-cloud emphasis; Phoenix's OSS is fully first-class.

The pattern to avoid: picking a platform because a specific engineer likes it, and then discovering three months later that the compliance team, the finance team, or the framework choice all constrain the answer differently. Walk the four questions above with every stakeholder in the room before you install anything.