Production AI Agents Deployment Guide 2026: Best Practices

What does deploying AI agents to production actually mean in 2026?

In short

Deploying an AI agent to production means giving it durable state, observability, guardrails, human-in-the-loop gates, cost caps, and a rollback path so it can serve real users under real load. Alice Labs treats an agent as production-ready only when all six are present. Prototypes without persistence, tracing, or safety layers are demos, not deployments, and consistently generate more incidents than value.

Shipping agents to production is nothing like shipping traditional software. Production agents behave as distributed systems with probabilistic outputs, not deterministic services — non-determinism, long-horizon failures, and tool-use risk make them structurally different from any request/response backend most teams have shipped before (MLflow, 2026).

The 2026 working definition Alice Labs uses across 100+ production deployments: an agent is production-grade when it has (1) durable state that survives restarts, (2) observability at trace granularity, (3) guardrails enforced at the architectural layer, (4) human-in-the-loop gates for high-impact actions, (5) a rollback plan tied to a single artifact, and (6) cost caps enforced upstream of model calls. Anything missing one of the six is a prototype, and prototypes in production generate more incidents than value.

The other thing that changed in 2026: the regulatory clock started. August 2, 2026 is the binding EU AI Act deadline for high-risk AI system obligations covering logging, human oversight, and cybersecurity resilience. If your agent classifies as high-risk under Annex III, you cannot ship without the compliance scaffolding — audit logs, dataset lineage, human-oversight interface, Article 15 adversarial resilience — from day one. The rest of this guide is organised around what it actually takes to hit that bar without slowing down.

For a broader comparison of the frameworks you deploy on top of this stack — LangGraph, Claude Agent SDK, CrewAI, AutoGen, MAF — see our best AI agent frameworks guide for 2026.

What infrastructure do production AI agents need?

In short

The 2026 reference stack has six layers: model provider (Anthropic/OpenAI/Bedrock/Vertex), agent framework (LangGraph, Claude Agent SDK, CrewAI, AutoGen), durable state (Postgres for checkpointers, Redis for pub/sub), tool sandbox (Firecracker microVMs or gVisor containers), LLM gateway (LiteLLM or custom proxy) for routing and cost caps, and OpenTelemetry-native observability. Any agent with access to external APIs, files, or databases needs a sandboxed execution context.

The infrastructure question is where most rollouts go wrong. Teams pick a framework, wire it to a model, and skip the other four layers. Six months later they are debugging production incidents caused by missing pieces of exactly that stack. Alice Labs runs the same six-layer reference architecture on every deployment because each layer solves a failure mode the others cannot.

The 2026 production AI agent reference stack

Two rules matter more than they look. First, any agent with access to external APIs, file systems, or databases needs a sandboxed execution context — container-based execution is the practical default in 2026 (MLflow). Second, the LLM gateway is a security and cost layer, not just a routing layer: it is where per-tenant spend caps, model refusals, prompt caching, and the audit log live. Teams that put those controls inside the agent framework instead of upstream lose the ability to enforce them globally.

How do you persist agent state so it survives restarts?

In short

LangGraph checkpointers serialize the full graph state after every node execution, enabling pause/resume and crash recovery. PostgresSaver is the recommended production checkpointer; MemorySaver loses all data on restart and must never leave prototypes. Thread IDs must be under 255 characters and old checkpoints should be pruned periodically. Combine short-term checkpointers (thread-scoped) with long-term stores (cross-thread) for user preferences, facts, and shared knowledge.

Durable state is the single biggest thing that separates an agent you can deploy from an agent that dies the moment its container restarts. In LangGraph, this is a checkpointer — an object that serializes the full graph state (messages, node outputs, intermediate variables) after every node execution. The agent can pause, resume, and recover from a crash exactly where it left off (LangChain docs).

Two checkpointers ship in LangGraph. MemorySaver stores state in process memory — good for tests and local demos, catastrophic in production because a restart wipes every in-flight run. PostgresSaver writes to Postgres and is the recommended production checkpointer. Alice Labs makes this a non-negotiable requirement: any agent that spans more than a single turn ships with PostgresSaver or an equivalent durable store, not MemorySaver.

# Python — LangGraph with PostgresSaver
from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.graph import StateGraph

with PostgresSaver.from_conn_string(POSTGRES_URL) as checkpointer:
    checkpointer.setup()  # first-run only — creates schema

    graph = builder.compile(checkpointer=checkpointer)
    config = {"configurable": {"thread_id": user_conversation_id}}

    for event in graph.stream({"messages": [msg]}, config):
        print(event)

Two operational rules trip up teams that skip the docs. thread_id values must be under 255 characters — use a stable business ID (user ID + conversation ID hash) not a raw natural-language string. And old checkpoints grow unboundedly if you never prune; schedule a nightly job that deletes checkpoints for completed threads older than your retention window.

Short-term checkpointers scope per thread. For anything that has to survive across threads — user preferences, facts learned from a customer, shared team knowledge — you need a long-term store. LangGraph exposes InMemoryStore and PostgresStore for this. The right shape is both: a checkpointer for turn-by-turn resume, plus a store for durable cross-conversation memory. For deeper coverage see our AI agent memory systems guide.

How do you monitor and trace AI agents in production?

In short

Emit OpenTelemetry GenAI spans (agent, workflow, tool, model) and pipe them into a tracing backend such as LangSmith, Arize Phoenix, Langfuse, or Braintrust. Capture the reasoning trace, tools considered, tools invoked, arguments, responses, tokens per step, and per-hop latency stitched into one hierarchical trace. The OpenTelemetry GenAI semantic conventions (v1.41) define the span structure but most gen_ai.* attributes still carry Development stability badges — names can change without a major version bump.

Observability is what makes an agent debuggable. A blind agent generates support tickets that engineers cannot resolve — they can see the final answer was wrong but not why, which tool failed, or what the model reasoned about before making the bad call. The 2026 industry answer is OpenTelemetry, specifically the GenAI semantic conventions (v1.41), which define spans for four entity types: agent, workflow, tool, and model — plus latency and token-usage metrics that stitch across them.

Effective agent traces capture the full reasoning chain: which tools Claude or GPT considered, which it actually invoked, the arguments it passed, the responses it got back, tokens per step, and latency of every hop. Rendered as a hierarchical trace, this is what your on-call engineer opens when the pager fires at 3am. Rendered as a table of top-level requests, it is what tells you which tenant is burning 90% of your monthly budget.

Backends are competitive and mostly interchangeable in 2026 — LangSmith, Arize Phoenix, Braintrust, and Langfuse all now emit OTel-compatible traces. Alice Labs standardizes on OTel export to stay backend-portable and avoid vendor lock-in across client environments where compliance teams sometimes require a specific vendor.

# Python — OTel GenAI span for a tool call
from opentelemetry import trace
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("gen_ai.tool.execute") as span:
    span.set_attribute("gen_ai.tool.name", "search_customers")
    span.set_attribute("gen_ai.tool.type", "function")
    span.set_attribute("gen_ai.tool.input", json.dumps(args))
    result = tool.run(**args)
    span.set_attribute("gen_ai.tool.output.tokens", count_tokens(result))
    return result

One caveat worth naming: nearly all gen_ai.* attributes still carry Development stability badges in the OTel semantic conventions v1.41. Names can change without a major version bump. Alice Labs wraps the raw attributes in a thin adapter layer so a spec change becomes a one-line library update instead of a codebase-wide refactor.

How do you control cost for production AI agents?

In short

Agents burn roughly 50× more tokens per session than a chat because of tool loops and re-reads. Anthropic, OpenAI, and Bedrock prompt caching charges cached input at 10–25% of normal input cost. Model routing to smaller models for routine tasks delivers 30–70% cost reductions in production; caching plus routing routinely delivers 70–85% savings on unoptimized baselines. Alice Labs enforces per-run token budgets, max_turns caps, and per-tenant spend limits at the LLM gateway before requests reach a model.

Agent cost economics are different from chat. A chat call sends the user turn plus some history; an agent call fans out across tool loops, re-reads context, retries, and replans — burning roughly 50× more tokens per session than a chat (LeanOps, 2026). This is why unoptimized production agents cost $10–100 per session and why teams get surprise invoices two weeks after launch.

Three levers move the number:

• Prompt caching. Anthropic, OpenAI, and Bedrock all now charge cached input at 10–25% of normal input token cost. Long system prompts, tool schemas, and static context should always be cache-anchored. This alone often halves total spend for agents with sizeable system context.
• Model routing. Route routine tasks to smaller models — read-only exploration to Haiku 4.5, main work to Sonnet, planning and critique to Opus. In production this pattern delivers 30–70% cost reductions on its own.
• Budgets and caps. Per-run token budgets, max_turns caps, and per-tenant spend limits enforced at the LLM gateway — before the request ever reaches a model provider. This is what stops a single misbehaving agent from burning your monthly budget on one runaway session.

Combined, prompt caching plus routing plus budgets routinely deliver 70–85% savings versus unoptimized baselines across Alice Labs' production portfolio. The single highest-ROI action in most audits is not switching models — it is anchoring the system prompt to a cache breakpoint and routing exploration to a cheaper model. Both take an afternoon.

# Alice Labs pattern — per-tenant budget enforced at gateway
async def gateway_middleware(request, tenant_id):
    spent_today = await budget_store.get_spend(tenant_id, "today")
    if spent_today >= BUDGET_PER_TENANT_DAY:
        raise BudgetExceeded(tenant_id)
    await budget_store.reserve(tenant_id, request.estimated_cost)
    response = await forward_to_model(request)
    await budget_store.commit(tenant_id, response.actual_cost)
    return response

For a broader treatment of build-vs-buy economics on production AI agents, including model spend forecasting, see our build vs buy AI guide.

What safety guardrails belong at the architecture layer?

In short

Real enforcement lives at the architectural callback layer — callback functions, MCP gateways, and LLM gateways that intercept requests before they reach a tool. Guardrails must make an agent structurally unable to reach a forbidden domain or action, not merely instructed against it. OWASP LLM01:2025 lists prompt injection as the top LLM vulnerability for the third year running and it now appears in over 73% of production AI deployments (2026 data). Alice Labs deploys five defense layers: input filtering, tool allowlists, output validators, sandboxed execution, and human confirmation on high-impact actions.

The most expensive mistake in agent safety is putting the guardrail in the system prompt. Prompt-level instructions like "do not delete files" or "never send email" fail against adversarial inputs, tool-poisoning attacks, and the model's own probabilistic drift. Real enforcement has to live in the architecture — the callback layer that runs before the tool executes, regardless of what the model decided.

Three architectural surfaces do the enforcing:

• LLM gateway. Sits between your agent and the model provider. Filters inputs, strips known injection patterns, applies per-tenant policy, and logs everything. This is where you enforce "this tenant cannot call the medical-advice model".
• MCP gateway. Sits between your agent and every MCP server. Applies per-tenant tool allowlists, terminates OAuth, and rate-limits per user. Structurally prevents the agent from reaching servers it should not have.
• Callback layer. The framework-level hook that fires before every tool call (LangGraph interrupts, Claude Agent SDK canUseTool, tool middleware). Applies per-call logic — if amount > $10K, require human approval.

Prompt injection is the exemplar failure mode this architecture defends against. It appears in over 73% of production AI deployments in 2026 (OWASP LLM01) and instructing the model to ignore attacks does not work — the attack is aimed at the layer above the instruction. Alice Labs deploys five defense layers on every high-risk deployment:

1. Input filtering at the LLM gateway (known-bad patterns, encoded payload detection).
2. Tool allowlists so the agent structurally cannot reach forbidden tools even if convinced to try.
3. Output validators that check tool results before they re-enter the model context (blocks tool poisoning).
4. Sandboxed execution so even a successful attack cannot exfiltrate from the host.
5. Human confirmation on any action above a defined impact threshold (spend, PII, external comms).

For deeper coverage of the risks and controls specific to agentic systems, our AI agent security risks guide covers prompt injection, tool misuse, exfiltration paths, and the containment patterns for each.

How do you build human-in-the-loop gates for production agents?

In short

Use confidence-based escalation: the agent runs autonomously below a threshold and requires human approval above it. LangGraph's interrupt() primitive and Claude Agent SDK's permission system (canUseTool) are the two dominant implementation patterns Alice Labs uses across Nordic client deployments. HITL cannot be the only defense — a 2026 study of AI coding agents found humans still only caught a bad action 9–26% of the time even with plan approval — but it satisfies EU AI Act Article 14's mandate for human-machine interfaces enabling effective oversight by natural persons.

Human-in-the-loop (HITL) is where agent design meets the EU AI Act. Article 14 mandates that high-risk AI systems be designed with human-machine interfaces enabling effective oversight by natural persons — not an on/off toggle, an interface that a supervising human can actually use to intervene, override, or halt the system in flight.

The 2026 production-ready pattern is confidence-based escalation. The agent runs autonomously when a signal (self-reported confidence, model uncertainty, action-impact score) is below a threshold; it requires human approval above it. This gives you the throughput of an autonomous agent for the 80% of cases where it is right, and the safety of a human review for the 20% where the cost of being wrong is high.

Two implementation primitives dominate:

• LangGraph interrupt(). Pauses the graph, persists state via the checkpointer, and yields control to the caller. The human reviews, approves or rejects, and the graph resumes from the interrupt point. Fully durable across restarts.
• Claude Agent SDK canUseTool. Fires before every tool call, receives the tool name and full input, and returns { behavior: "allow" | "deny", reason? }. Alice Labs wires it to Slack, Jira, or ServiceNow for enterprise approval flows.

# Python — LangGraph interrupt for high-value approvals
from langgraph.types import interrupt

def approval_node(state):
    if state["proposed_action"]["amount"] > 10_000:
        decision = interrupt({
            "action": state["proposed_action"],
            "reason": "amount exceeds autonomous threshold",
        })
        if not decision["approved"]:
            return {"status": "rejected", "reviewer": decision["user_id"]}
    return {"status": "approved"}

One caveat that has to be named: HITL is not a substitute for architectural guardrails. A 2026 study of AI coding agents found humans still only caught a bad action 9–26% of the time even with plan approval — attention fatigue and rubber-stamping are real. HITL sits on top of the five-layer defense pattern above, not in place of it.

For the EU AI Act treatment specifically, our EU AI Act compliance checklist for 2026 covers Article 14 oversight design end to end.

How should you sandbox agent tool execution?

In short

Four dominant 2026 platforms: E2B (Firecracker microVMs — each sandbox has its own kernel, deepest isolation), Modal (gVisor + GPU — faster cold start, shallower boundary), Daytona (agent runtime), and Fly Machines (low-level primitive). Alice Labs picks Firecracker whenever an agent runs untrusted code paths (customer-uploaded scripts, tool-called shell), and gVisor when latency and GPU access dominate. The four tradeoffs to size against your workload: cold-start latency, isolation depth, developer experience, and cost.

Sandboxing is the layer most teams put off and later regret. The moment an agent can execute code, call arbitrary APIs, or read files on behalf of a user, the host machine becomes an attack surface. In 2026 the choice is not whether to sandbox — it is which sandbox technology matches your workload.

Four platforms dominate the 2026 landscape:

• E2B — Firecracker microVMs. Each sandbox has its own kernel; the isolation boundary is a full hypervisor. Deepest isolation available, correspondingly higher cold-start latency (hundreds of milliseconds).
• Modal — gVisor plus GPU support. gVisor is a user-space kernel that intercepts syscalls; the isolation boundary is shallower than a microVM but cold start is significantly faster and GPU access is first-class.
• Daytona — purpose-built agent runtime with pre-warmed workers and a code-focused developer experience.
• Fly Machines — low-level primitive. Full VM boot on demand. Maximum flexibility, more infrastructure to own.

The Alice Labs rule of thumb is a simple two-way split. If the agent runs untrusted code paths — customer-uploaded scripts, model-generated shell commands, RCE-shaped tool calls — pick Firecracker for the deeper kernel boundary. If latency and GPU access dominate and the code paths are trusted (internal tools, controlled APIs) — pick gVisor for faster cold starts and GPU support.

The four tradeoffs to size against your workload are always the same:

1. Cold-start latency. Firecracker: hundreds of ms. gVisor: tens of ms with pre-warming.
2. Isolation depth. Hypervisor (Firecracker) > user-space kernel (gVisor) > container (raw Docker — do not do this).
3. Developer experience. SDK ergonomics, filesystem access, secret injection, log capture.
4. Cost. Firecracker microVMs are more expensive per second; the gap closes if you can pool and reuse.

One anti-pattern is worth naming explicitly: raw Docker containers on shared hosts, with no seccomp profile and no user namespace, running untrusted code. This is not a sandbox — it is a fig leaf. If Docker is your only option, run it inside a Firecracker microVM (Kata Containers) or accept the risk in writing.

How do you version and CI/CD deploy AI agents?

In short

LLMOps artifacts include prompts, retrieval configs, model provider settings, and evaluation thresholds — all need version control, peer review, and automated testing. Semantic evaluation with an LLM-as-judge is now standard in CI/CD because deterministic string comparison does not work for agent outputs. Braintrust, Latitude, Langfuse, and LangSmith all provide eval-gated deployment workflows; Alice Labs treats a failing offline eval as a blocking CI check. Organizations that use evaluation tools move nearly 6× more AI systems to production than those that do not (OneReach, 2026).

An agent is not one artifact — it is a bundle. Prompts, retrieval configs, model provider settings, tool definitions, and evaluation thresholds all move together and all need version control. Treating them as separate concerns is how teams end up debugging a regression in staging and discovering the prompt changed six weeks ago with no PR trail.

The Alice Labs rule: every agent is versioned as a single artifact — a manifest that captures the prompt hash, model ID, tool schema hash, and eval-threshold set. A deployment is a single manifest promoted from staging to production. Rollback is one command: point the runtime at the previous manifest.

Semantic evaluation with an LLM-as-judge is now standard in CI/CD because deterministic string comparison does not work for agent outputs. Two answers can be semantically equivalent and lexically different; two can be lexically similar and semantically wrong. The 2026 pattern is a scored eval set of representative inputs, with expected-behaviour rubrics graded by a judge model, run on every PR that touches the agent bundle.

Four vendor-neutral platforms — Braintrust, Latitude, Langfuse, and LangSmith — all provide eval-gated deployment workflows. Alice Labs treats a failing offline eval as a blocking CI check, exactly the way a failing unit test blocks a merge.

# Python — Braintrust eval gate in CI
from braintrust import Eval

result = Eval(
    "customer-support-agent",
    data=lambda: load_eval_dataset("prod_regression_set"),
    task=lambda inp: run_agent(inp),
    scores=[semantic_match, tool_use_correctness, safety_pass],
    threshold=0.85,  # blocks merge if score < 0.85
)
result.raise_if_below_threshold()  # exits 1 on regression

The 2026 industry data on this is decisive: organizations that use evaluation tools move nearly 6× more AI systems to production than those that do not (OneReach, 2026). The bottleneck to shipping agents is not model quality — it is the confidence to deploy without an eval floor breaking under you.

How do you roll out agent changes safely: shadow, canary, percentage, full?

In short

Agent rollout is a four-stage gate: shadow (duplicate real traffic to a candidate without user impact), canary (1–5% of real traffic with a shared correlation ID), percentage (10–50% with A/B eval metrics), full. Skipping one ships a production incident. AI-specific canaries need longer soak times than classical software — a 10-minute stage for a web service may need 2 hours for an agent because hallucination and coherence signals are noisy (FutureAGI, 2026). Alice Labs runs 24–48h shadow soaks before any canary on customer-facing agents; rollback is automated on eval-metric regression.

Every mature production discipline eventually converges on staged rollouts, and agents in 2026 are no exception. The four stages are unambiguous:

1. Shadow. The new agent version receives duplicated production traffic but its output goes nowhere — it is compared to the current version's output offline. Zero user impact. This is where you catch behaviour drift that unit tests missed.
2. Canary. 1–5% of real traffic is routed to the new version with a shared correlation ID for A/B analytics. Users see the new agent; you see the metrics. Cost, latency, and eval-score deltas should be visible in the first hour.
3. Percentage. 10–50% traffic split, longer soak, business-metric A/B (task success, resolution rate, CSAT) rather than just eval-set scores.
4. Full. 100% cutover with automatic rollback on regression.

The AI-specific twist is soak duration. FutureAGI's 2026 write-up on eval, shadow, and canary practices makes the point clearly: a 10-minute canary stage that would be plenty for a classical web service may need 2 hours or more for an agent because the signals you care about — hallucination rate, coherence, tool-use correctness — are noisy and low-frequency. A 10-minute window on 1% traffic often does not accumulate enough events to distinguish signal from noise.

Alice Labs' operating rule on customer-facing agents: 24–48h shadow soak before any canary; automated rollback tied to eval-metric regression detected in the tracing backend. When a metric drifts more than two standard deviations from the trailing 7-day baseline, the deployment rolls back to the previous manifest without human intervention.

The trap teams fall into is skipping shadow. Shadow feels expensive — you are paying for two agents to serve every request — but it is the cheapest incident-prevention available. Every production incident Alice Labs has audited that was caused by an agent change would have been caught in a 24-hour shadow window.

How do you secure MCP servers in production?

In short

28% of Fortune 500 companies had deployed MCP by early 2026 (WorkOS, 2026); the protocol is now a required-scope compliance boundary under the EU AI Act's action layer. The NSA and joint government agency guidance (June 2026) recommend OAuth 2.1 with per-user attribution, explicit allowlists, and centralized MCP gateways. The 2026-07-28 MCP release candidate adds a stateless core, Extensions framework, Tasks, MCP Apps, and authorization hardening. Alice Labs runs all customer MCP connections through a WorkOS-style gateway that terminates OAuth and applies per-tenant tool allowlists before the agent process sees them.

Model Context Protocol went from experimental in 2025 to production-critical in 2026. By early 2026, 28% of Fortune 500 companies had deployed MCP (WorkOS), and the protocol is now a required-scope compliance boundary under the EU AI Act's action layer — any tool an agent calls via MCP is inside the compliance perimeter.

The June 2026 joint NSA and government-agency guidance on MCP security establishes the 2026 minimum bar. Three requirements matter most:

• OAuth 2.1 with per-user attribution. Every MCP call must carry an identity, not just a service token. Auditors need to trace which user's agent made which tool call, not just the agent service made this call.
• Explicit tool allowlists. The agent's manifest declares which MCP tools it may call. Anything else is denied by default at the gateway.
• Centralized MCP gateways. All MCP traffic passes through a single policy-enforcement point (WorkOS, custom proxy) that terminates OAuth, applies allowlists, rate-limits, and logs.

The 2026-07-28 MCP release candidate hardens all of this: stateless core (better for horizontally scaled gateways), Extensions framework (structured spec for extensions without proto-level forks), Tasks primitive for long-running operations, MCP Apps, and specific authorization hardening. The protocol is stabilising fast — teams that were on 2025 spec should schedule a migration.

The Alice Labs deployment pattern for enterprise MCP: every customer MCP connection passes through a WorkOS-style gateway that terminates OAuth and applies per-tenant tool allowlists before the agent process ever sees the connection. From the agent's perspective, MCP looks the same as any other client library call; from the security team's perspective, every call is authenticated, authorized, logged, and rate-limited at a chokepoint they own.

How do you evaluate agents beyond simple accuracy?

In short

Agent evaluation must score the trajectory (which tools were considered vs invoked, in what order) not only the final answer. LLM-as-judge produces varying grades run-to-run; use ensemble judging and human calibration to keep the eval itself stable. Alice Labs' four-metric scorecard for production agents: task success rate, cost per resolved case, p95 latency, and safety-violation rate per 10K runs. Regression alarms fire when any of the four metrics drifts more than 2 standard deviations from the trailing 7-day baseline.

Accuracy alone is a bad metric for agents. Two runs can produce the same final answer through radically different trajectories — one calling three tools cleanly, one thrashing through fifteen. The second is more expensive, more fragile, and more likely to hit a rate limit or exceed budget. If your eval only scores the last message, you have no way to distinguish them.

The 2026 pattern is trajectory evaluation: score which tools the agent considered, which it invoked, in what order, and how many steps it took. The eval treats the tool-use trace as a first-class output, not just the final answer. This is what catches a regression where the answer stayed correct but the agent now takes 10 tool calls instead of 3.

LLM-as-judge is the standard evaluator for open-ended agent output, but it has a well- documented failure mode: judges are themselves stochastic and produce varying grades run-to-run. The 2026 mitigation is ensemble judging (multiple judge models, vote or average) and periodic human calibration to catch judge drift. Alice Labs recalibrates every judge model against a human-graded gold set quarterly.

The Alice Labs four-metric scorecard for every production agent:

• Task success rate. Trajectory-scored, ensemble-judged. Baseline target: 92% for customer-facing, 85% for exploratory.
• Cost per resolved case. Total tokens × per-token cost, divided by successful resolutions. The metric that turns cost from an infra concern into a product concern.
• p95 latency. End-to-end, user-perceived. Long tail matters more than mean — the 95th-percentile user is the one who calls support.
• Safety-violation rate per 10K runs. Prompt-injection successes, PII leaks, guardrail bypasses. Rare events but catastrophic — needs a rate, not a count.

Regression alarms fire when any of the four metrics drifts more than 2 standard deviations from the trailing 7-day baseline. The alarm routes to on-call with a filtered dashboard link (traces from the affected time window) rather than a generic paging message.

What are the EU AI Act obligations for production AI agents?

In short

August 2, 2026 is the binding enforcement date for high-risk AI system obligations covering Articles 9–17 (provider) and Article 26 (deployer). Article 15 requires resilience against adversarial attacks across the entire action layer — not just the model output — pulling MCP servers, tool calls, and third-party APIs into scope. In a chain of AI agents, the compliance boundary extends to every agent that performs a high-risk function. Alice Labs' EU AI Act-native architecture ships every high-risk deployment with a risk file, dataset lineage, tamper-evident logs, and a human-oversight panel from day one.

August 2, 2026 is the binding EU AI Act enforcement date for high-risk AI system obligations. The relevant articles are 9–17 for providers and 26 for deployers. If your agent classifies as high-risk under Annex III — recruitment, credit scoring, biometrics, critical infrastructure, education, law enforcement, migration, justice — these obligations bite, whether the agent is a Nordic pilot or a global rollout.

The article most teams misjudge is Article 15. It requires resilience against adversarial attacks across the entire action layer, not just the model output. This pulls MCP servers, tool calls, and third-party APIs into scope. If a prompt injection can trick the agent into calling a downstream API that leaks customer data, Article 15 has been breached — even if the model itself was faultless.

The Alice Labs interpretation across 100+ deployments: in a chain of AI agents, the compliance boundary extends to every agent that performs a high-risk function. If a classifier agent triages high-risk applications and a follow-up agent processes them, both agents are inside the perimeter. Downstream agents cannot inherit compliance from upstream — they need their own logs, oversight, and audit trail.

The compliance scaffolding that ships with every Alice Labs high-risk deployment:

• Risk file. Article 9 risk management: identified risks, mitigations, residual risk, review cadence.
• Dataset lineage. Article 10: sources, licences, filtering, bias testing. Machine-readable and reviewer-friendly.
• Tamper-evident logs. Article 12: append-only, cryptographically signed, six-month minimum retention.
• Human-oversight panel. Article 14: real UI (not a config flag) that a designated supervisor can use to review, override, or halt.
• Adversarial-resilience log. Article 15: attack-surface catalogue, red-team findings, mitigations.

For the full compliance treatment, our EU AI Act compliance checklist for 2026 walks through each article deployer by deployer. Always consult qualified legal counsel for compliance determinations — this section is design guidance, not legal advice.

What does the operational runbook for a production AI agent look like?

In short

Alice Labs runs three SLOs per production agent: task success rate ≥ 92%, p95 end-to-end latency, and cost per resolved case within budget. Six standard incident classes: prompt injection detected, tool-call loop, cost budget breach, model provider outage, checkpointer corruption, guardrail bypass. Every deployment must have a one-command rollback to the previous prompt+model+tool-config bundle — agents are versioned as a single artifact, not per-file. On-call playbooks tie each incident class to a specific dashboard filter and a canned mitigation (kill-switch, model swap, tool disable).

Production agents need the same runbook discipline as any other service — SLOs, incident classes, mitigations, rollback. The pattern below is what Alice Labs actually ships to on-call teams; it is battle-tested across the 100+ deployment portfolio and has been simplified aggressively over three years.

Three SLOs per production agent:

• Task success rate ≥ 92% (customer-facing) / ≥ 85% (exploratory). Measured by the trajectory-scored eval set on live traffic samples.
• p95 end-to-end latency within the product's stated SLA (typically < 20s for chat-shaped, < 2min for research-shaped).
• Cost per resolved case within the per-tenant budget with a 10% safety margin.

Six standard incident classes. Each maps to a specific dashboard filter and a canned mitigation:

1. Prompt injection detected. Filter: guardrail-bypass counter spike. Mitigation: enable strict input filter at gateway, page security.
2. Tool-call loop. Filter: turns-per-run p99. Mitigation: enforce max_turns, disable the looping tool.
3. Cost budget breach. Filter: per-tenant spend. Mitigation: pause tenant, notify.
4. Model provider outage. Filter: 5xx rate on model calls. Mitigation: flip to fallback model via LLM gateway.
5. Checkpointer corruption. Filter: resume errors. Mitigation: restore from last-good backup, notify affected users.
6. Guardrail bypass. Filter: policy-violation counter. Mitigation: kill-switch, escalate.

Rollback discipline. Every deployment must have a one-command rollback to the previous prompt + model + tool-config bundle. Agents are versioned as a single artifact, not per-file — a rollback is a manifest swap, not a git bisect. Alice Labs' target time to rollback is under 60 seconds from decision to fully cutover.

The alarms-per-severity discipline matters too. Not every incident class pages on-call at night — cost budget breaches route to Slack during business hours; a guardrail bypass pages immediately. Under-paging is a recruitment risk; over-paging is an attrition risk. Both are worth tuning.

How does Alice Labs deploy production AI agents in practice?

In short

Alice Labs' standard reference stack: LangGraph or Claude Agent SDK for the framework, Postgres for checkpointers, LiteLLM gateway for routing and budgets, E2B or Modal for sandboxed tools, LangSmith + OTel for tracing, Braintrust for CI evals. Every project ships with an EU AI Act readiness pack. The team is senior-only, workflow-embedded, and prices agent deployments transparently rather than by seat, reflecting Alice Labs' Stockholm-founded, Nordic and European delivery model.

The stack Alice Labs actually runs across production deployments is deliberately unglamorous — the same tools, wired the same way, from Stockholm to Copenhagen to Amsterdam. The default reference architecture:

• Framework. LangGraph when graph-based state control matters, Claude Agent SDK when Anthropic's harness is the fit. Occasionally both in the same product, at different layers.
• State. Postgres for checkpointers (LangGraph PostgresSaver) and long-term stores. Redis for pub/sub and cache.
• Gateway. LiteLLM for routing, prompt caching, and per-tenant budgets. Custom proxy where compliance demands EU data residency inspection.
• Sandbox. E2B (Firecracker) for untrusted code paths; Modal (gVisor) for latency-sensitive trusted paths with GPU.
• Observability. LangSmith + native OTel export. Backend-portable so a customer's compliance team can move the destination without a rewrite.
• CI/CD. Braintrust eval gates on every PR; four-stage rollout (shadow, canary, percentage, full); automated rollback on eval regression.

The first 90 days of any engagement follow the OneReach pattern: 30 days to assess and prepare (data audit, use-case selection, risk classification), 30 days to build and validate (agent, eval set, gateway policy), 30 days to deploy and scale with measured traffic. This cadence has been the single biggest predictor of deployments that stick versus prototypes that stall.

Every high-risk project ships with an EU AI Act readiness pack: Article 14 oversight design, Article 15 attack-surface log, tamper-evident audit trail, dataset lineage, and risk-management file — all built by day one, not bolted on before a regulator visit. Alice Labs' Stockholm founding and Nordic + European delivery model make this the default posture, not a special mode.

The team is senior-only and workflow-embedded — meaning engineers ship next to your team on your infrastructure, not from a black-box vendor portal. Pricing is transparent per deployment, not per seat, which aligns incentives on outcome instead of expansion. If you are evaluating whether to build in-house or engage external support, our build vs buy AI guide documents the same decision framework we use across the portfolio.