AI Agent Observability and Tracing: Making Autonomous Systems Debuggable
Executive Summary
An LLM call is easy to observe: one request, one response, a token count, a latency number. An agent is not. A single user-facing task can fan out into dozens of model calls, nested tool invocations, sub-agent delegations, and hours-long pauses for human approval — and by the time something goes wrong, the reasoning that produced the wrong answer is buried three or four levels deep in a call graph that no traditional APM tool was built to represent. Through 2025 and into 2026, the industry appears to be converging on a shared answer to "what does a trace of an agent even look like": the OpenTelemetry GenAI semantic conventions, a dedicated specification repository covering spans, metrics, and events for GenAI clients, MCP, and provider-specific conventions, which introduces dedicated span types — invoke_agent, execute_tool, create_agent, invoke_workflow — layered on top of the classic chat/text_completion model-call span. As of mid-2026 the spec is reportedly still under active, pre-stabilization development (specific version numbers below are as tracked in industry write-ups current to this writing and should be treated as approximate rather than independently re-verified), and coverage from observability vendors including Datadog, Honeycomb, and Elastic is commonly cited as growing — a trend that, if it holds, would make switching observability backends increasingly a matter of changing an exporter rather than rewriting instrumentation. Reports of OpenTelemetry's broader 2026 CNCF project-maturity milestones circulate widely but are not independently confirmed here.
A second, narrower problem reportedly got its own fix this year: agents that call tools over the Model Context Protocol used to produce two disconnected traces — one for the agent process, one for the MCP server — because there was no shared correlation mechanism. MCP-specific semantic conventions (fields commonly referenced as mcp.session.id, mcp.method.name, propagated W3C Trace Context) are described in industry write-ups as closing that gap by nesting the MCP server's span under the calling tool span instead of duplicating it into an orphan trace.
Around this substrate, a market of roughly six serious platforms has formed, and — per vendor documentation and third-party write-ups, rather than independent benchmarking — they appear to differentiate less on "can it capture a trace" (most now can, via OTel) and more on what they do with the trace afterward. Langfuse is commonly described as leading on self-hosted data residency and turning failed traces directly into eval regression tests. Arize Phoenix is commonly cited for evaluation depth (vendor materials claim 50+ research-backed metrics) via its own OpenInference attribute layer, which the project itself describes as complementary to, rather than competing with, OTel GenAI semconv. LangSmith is often positioned as deepest for LangGraph-specific state diffs and replay. Datadog and Honeycomb are extending existing infra-observability incumbencies into agent-shaped data — Honeycomb notably storing everything as unified "wide events" rather than the classic logs/metrics/traces split, which it argues (a vendor claim, not an independently verified benchmark) handles the high cardinality of agent telemetry better than a Prometheus/Grafana stack. Braintrust and AgentOps are generally positioned toward the eval-first and lifecycle-hook ends of the spectrum, respectively — these are broad vendor-positioning characterizations rather than results of a controlled comparison.
What remains genuinely unsolved, and is drawing fresh 2026 academic attention, is attribution and debugging when full step-level tracing either wasn't captured or is too voluminous to read by hand — the "only the final text survives" problem for multi-agent systems (arXiv:2603.17445), and counterfactual replay as a way to find the one causally decisive event buried in a thousand-span trace (arXiv:2606.14805). For a fleet operator, the practical takeaway is that instrumentation is now a comparatively standardized layer; the open frontier has moved to what you do with the resulting flood of spans.
Why Agent Tracing Is a Different Problem
Classic distributed tracing (the kind OpenTelemetry was originally built for) assumes a call graph that is mostly deterministic and mostly synchronous: service A calls service B calls service C, each span's duration is dominated by network and compute time, and the shape of the graph doesn't change much from request to request. Agent execution violates every one of those assumptions.
The reasoning process is a black box by default. Without agent-specific instrumentation, a trace of an agentic task looks like one long, opaque LLM call — you see that it took 40 seconds and cost $0.30, but not how many tool round-trips happened inside, or which of them the model chose to make and why. The entire motivation for the invoke_agent/execute_tool span types is to open that box: a typical trace tree now looks like an invoke_agent root, alternating chat spans (the model deciding what to do next) and execute_tool spans (the tool actually running), terminating in a final chat span that produces the answer.
Delegation is hierarchical and often parallel. When a supervisor agent calls a sub-agent as if it were a tool, that sub-agent's entire span tree needs to nest under the supervisor's span for the trace to reconstruct the true flow — otherwise you get a pile of isolated traces per agent with no way to tell which supervisor request triggered which sub-agent run. Fan-out makes this worse: a single root request can spawn many concurrent children (exactly the shape of a background-agent dispatch), and correlating them requires trace context to propagate consistently across process and service boundaries, including into things like MCP tool servers that may run as entirely separate processes.
Failures cascade and don't repeat. A multi-step plan that goes wrong rarely fails at a single, isolable point the way a crashed microservice does. One widely circulated 2026 failure taxonomy for agents lists cascading errors across multi-step plans, tool misuse, prompt injection, context-window decay, planning failures, hallucinated grounding, and — tellingly — "observability gaps" and "context-blind deployments" as failure modes in their own right, not just causes of other failures; this framing is repeated across secondary industry sources rather than a single audited study, so treat it as illustrative of practitioner consensus rather than a formal measurement. Because the model's decisions are non-deterministic, the same trace won't recur on retry, which breaks the classic "reproduce the bug" debugging loop.
Execution can be async and long-running, sometimes for days. Frameworks increasingly build durable execution: a workflow can pause indefinitely — for a human approval gate, for example — and resume later, possibly on a different worker entirely. Vendor documentation describes LangGraph's interrupt() handing control back to the caller and checkpointing state, the OpenAI Agents SDK serializing a RunState via to_state() so a run resumes where it left off, and Google's Agent Development Kit doing something similar with Cloud Trace integration — these are framework-provided mechanisms as documented by their respective vendors, not independently verified by this article. Observability has to represent a trace that stays legitimately "open" for hours or days without either timing out spuriously or looking like a hung process.
Token and cost accounting is a session-level property, not a per-call one. A single coding-agent session can run to well over a hundred turns and millions of tokens. Mid-session context compaction — auto-triggering summarization above a token threshold is one pattern vendors including Anthropic have shipped in some form — rewrites history partway through, which breaks naive per-call token counting. Some industry commentary suggests teams that don't account for this underprice multi-step agentic workflows by a factor of several times, because the cost of a task is not simply the sum of the individually-priced calls inside it; it's shaped by how much context each call had to carry and how often that context was compacted or reconstructed. That multiplier is a rounded, illustrative figure from secondary sources rather than a benchmarked result.
And at fleet scale, you need to correlate across agents, not just within one. A single well-instrumented agent trace is necessary but not sufficient; a fleet operator wants throughput, error rate, latency distribution, and cost roll-ups across dozens of heterogeneous agents without building a bespoke dashboard for each one — which only works if every agent emits telemetry in the same shape.
The OpenTelemetry GenAI Semantic Conventions
The spec now lives in its own repository, open-telemetry/semantic-conventions-genai, separate from the core OTel semantic conventions — a sign of how much surface area agent telemetry now covers; that repository confirms it covers spans, metrics, and events for GenAI clients, MCP, and provider-specific conventions. The specific version numbering below (roughly v1.41, with content-capture events said to have arrived at v1.37 and MCP conventions at v1.39) reflects industry write-ups current to mid-2026 rather than a version history independently re-derived for this article, and a fast-moving pre-1.0 spec is exactly the kind of detail likely to have moved by the time you're reading this. Instrumentations reportedly default to a conservative attribute set unless an opt-in environment variable enables the newest experimental fields — a pattern OTel has used before during earlier semconv transitions (HTTP, database conventions) — and secondary sources describe the spec's own documentation as not having set a stabilization date, though that specific framing wasn't independently re-confirmed here.
The span model has two layers. The base layer covers individual model calls: operation names chat, text_completion, and generate_content, all CLIENT-kind spans carrying gen_ai.provider.name (e.g. openai, anthropic, aws.bedrock), gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens/output_tokens, and gen_ai.response.finish_reasons. This layer is a direct extension of what a non-agentic LLM app would already emit.
The second layer is specific to agents and reportedly didn't exist in earlier semconv revisions: create_agent for provisioning a remote agent (OpenAI Assistants, Bedrock Agents), invoke_agent for a single agent's task execution — described in recent semconv discussion as split into CLIENT kind for a remote agent service versus INTERNAL kind for a local in-process framework like LangGraph — invoke_workflow for deterministic (non-autonomous) workflow execution, and execute_tool for a tool call, with the tool's name said to be required in the span name itself as of a recent revision. (The version number at which each of these details landed is drawn from secondary write-ups, not re-derived from the spec's own changelog here.) A representative trace tree:
invoke_agent (INTERNAL)
├── chat gpt-4o (CLIENT) — model decides to search
├── execute_tool web_search — search executed
├── chat gpt-4o (CLIENT) — reasoning over results
├── execute_tool summarize
└── chat gpt-4o (CLIENT) — final answer
Provider-specific detail layers on top of this generic shape rather than polluting it: an OpenAI reasoning-model call might carry gen_ai.usage.reasoning.output_tokens or cache-hit fields like gen_ai.usage.cache_read.input_tokens, none of which apply to a Bedrock or Anthropic span, so the spec discriminates by gen_ai.provider.name first and layers vendor-specific attributes second.
Two features of the spec are worth calling out because — per secondary write-ups describing it — they answer real operational questions rather than just naming spans. First, content capture reportedly has several explicit modes, chosen deliberately rather than left to each team to improvise: not recorded at all, recorded inline as span attributes (size-limited), or externalized — the span holds only a reference URL to content stored elsewhere (S3, a database), so sensitive payloads can sit under independent IAM policy from the trace metadata itself. Second, evaluation is described as a first-class citizen of the trace, not a separate system bolted on afterward: an evaluation-result event attaching a score value and label directly to the span it evaluates is the kind of mechanism that would let a platform like Langfuse turn a failed production trace into a regression-test case with one click, using vendor-neutral data rather than a proprietary schema — though the exact attribute names above should be treated as representative rather than verbatim-confirmed from the current spec.
The MCP-specific conventions exist, per project documentation and secondary coverage, to fix a concrete, previously broken behavior: an agent process calling a tool exposed via the Model Context Protocol used to produce two entirely disconnected traces, because the agent side and the MCP server side had no shared correlation ID. The described fix adds session, method-name, protocol-version, and transport-type fields; when both sides propagate W3C Trace Context, the MCP server's span is meant to nest under the calling tool span rather than duplicating it — the intent, as described, is to enrich the existing tool span, not create a parallel trace.
Metrics reportedly get similar treatment to spans: duration and token-usage histograms dimensioned by operation, model, and provider, with the spec said to mandate that instrumentations use provider-reported billable token counts and omit the metric entirely rather than estimate when billable counts aren't available — a small but telling detail if accurate, since estimated token counts are a common source of silently wrong cost dashboards.
A parallel, complementary schema exists alongside OTel GenAI semconv: OpenInference, maintained by Arize, which the project's own repository describes as "a set of conventions and plugins that is complimentary to OpenTelemetry" rather than a rival standard. OpenInference reportedly layers a richer LLM-specific type vocabulary (spanning categories such as Generation, Tool, Retriever, Agent, Chain) on top of the same OTel span transport, and its repository confirms instrumentor coverage across a wide framework list including LangChain, LangGraph, CrewAI, AutoGen, the Claude Agent SDK, the OpenAI Agents SDK, and the Vercel AI SDK. The practical choice for a team isn't "OTel or OpenInference" so much as "raw OTel GenAI semconv for maximum backend portability" versus "OpenInference for richer built-in semantics, at the cost of leaning more on the Arize ecosystem" — both transport over OTel, so neither choice is a dead end.
The Platform Landscape
By mid-2026 roughly six platforms are commonly cited as dominating production conversations, with several credible alternates — this ranking reflects a synthesis of vendor marketing and secondary industry write-ups rather than an independent market survey conducted for this article. The differentiation is now mostly described as downstream of tracing — what a platform does with spans once it has them — rather than whether it can capture them at all, since OTel-based instrumentation is commonly described as table stakes.
Langfuse is frequently described as a leading self-hosted, data-residency-first option — the open-source self-host tier is reportedly free with no usage caps, a meaningful factor for teams with contractual or regulatory reasons to keep trace data (which routinely contains prompts and completions) off a third-party SaaS. Its observation taxonomy is described as richer than a raw span, and vendor materials describe an official integration for the Claude Agent SDK. Its most operationally notable feature, per its own documentation, is turning any trace, pass or fail, into an eval dataset entry with one click, so a production failure becomes a permanent regression test rather than a one-off incident.
LangSmith, LangChain's own platform, is generally positioned as deepest specifically for LangGraph: node-by-node state diffs, full execution graph visualization, and the ability to replay a captured trace against a different model version to see whether an upgrade would have avoided a given failure. One 2026 benchmark write-up (not independently reproduced here) credits it with near-zero instrumentation overhead, a real consideration at fleet scale where a heavy tracing SDK can itself become a latency source — but treat that specific figure as a single secondary-source claim rather than a verified measurement.
Arize Phoenix (open source) and Arize AX (managed) are marketed as differentiating on evaluation depth — vendor materials cite over 50 research-backed metrics covering faithfulness, relevance, safety, toxicity, and hallucination, built OTel-native via the OpenInference layer described above. Its instrumentor coverage is described as actively maintained, though a specific claim like a named instrumentor shipping in a given month is the kind of detail that ages quickly and wasn't independently re-verified here.
Traceloop/OpenLLMetry is often described as the most vendor-neutral option: Apache-2.0, pure OTel, exports to any OTel-compatible backend (Datadog, New Relic, Sentry, Honeycomb), with SDKs across Python, TypeScript, Go, and Ruby. It's commonly positioned as the right choice for a team that already runs OTel for conventional APM and doesn't want a second, parallel observability stack just for LLM traffic — the trade-off, per this positioning, is giving up the LLM-specific UX (prompt versioning, deep eval tooling) that dedicated platforms build on top of the same underlying spans.
Helicone is proxy-based — point your API base URL at it and get logging with what vendor materials describe as essentially a one-line integration change, no SDK required. That speed reportedly comes at the cost of depth: it is described as logging requests and responses rather than modeling structured spans and tool calls the way Langfuse or Phoenix do, making it, per this positioning, a better fit for quick visibility than for debugging complex agent trajectories.
Datadog is extending its infrastructure-monitoring incumbency into agent telemetry, and reported additions of automatic instrumentation for frameworks including Google's Agent Development Kit, Bedrock Agents, LangChain, CrewAI, Pydantic AI, and LiteLLM are commonly cited as an adoption signal — teams that already trust Datadog for infra may not have to stand up a second vendor relationship for agent observability, though the specific timing of any individual integration announcement was not independently re-verified for this article. Vendor materials describe an "Agent Observability Insights" feature auto-flagging duration and error-rate anomalies against built-in evals, and sensitive-data scanning and redaction as an enterprise control.
Honeycomb takes a distinct architectural bet, per its own materials: rather than the classic logs/metrics/traces (LGTM) split that Grafana-style stacks use, it stores everything as unified "wide events" in one datastore and derives all three views from the same data. Its argument — a vendor claim, not an independently benchmarked comparison — is that agent telemetry is high-cardinality in a way Prometheus-style stacks handle poorly, and it markets an "Agent Timeline" view showing relationships between user input, model calls, tool calls, and agent invocations directly.
Braintrust is best characterized, based on its own positioning, as an eval-first platform with tracing attached, rather than a tracing platform with evals attached — a fit for teams whose primary operational bottleneck is regression gating before release, since production traces and pre-release eval datasets are described as sharing the same scorer infrastructure. AgentOps is generally positioned at the lifecycle-hook end of the spectrum; one secondary-source benchmark reportedly put its instrumentation overhead meaningfully higher than LangSmith's — a potentially material difference if overhead compounds across a large fleet, but again a single cited figure rather than a reproduced measurement. Weights & Biases Weave is commonly recommended for teams already living in the W&B ecosystem for experiment tracking, with vendor materials describing preserved parent-child relationships across multi-agent calls including per-agent and per-trace token usage.
Patterns Worth Copying
A handful of concrete patterns recur across the platforms and the framework integrations, independent of which vendor a team ultimately picks.
Sampling generally needs to be tail-based, not head-based, because you don't know a trace is "interesting" until it's finished — a trace that ends in an error or exceeds some cost threshold (illustrative examples in secondary sources cite figures on the order of tens of cents per trace) is a candidate to keep at close to 100%, with the uninteresting remainder sampled down to roughly single-digit-to-low-double-digit percentages. Deciding whether to keep a trace before it starts, the way classic head-based sampling does, doesn't work when the entire point is to catch the expensive or failing cases. Treat the specific numbers here as illustrative rather than a prescribed standard.
PII redaction should be cost-tiered, not uniform. The pattern several sources converge on is cheap regex-based detection on every request, named-entity recognition on free-text-heavy surfaces on every request, and expensive LLM-based redaction — which catches indirect or contextual identifiers that regex and NER miss — reserved for flagged samples or explicitly high-risk pathways. Running the expensive check on every span is both slow and unnecessary given how rarely it changes the outcome.
Cost attribution belongs at the gateway, propagated as span attributes. Pricing every LLM call at a single AI-gateway layer, where model and token counts are unambiguously known, and then propagating agent.id/session.id onto a single root span per agent run is enough to let a fleet-wide dashboard roll cost up by agent, task type, or end user — one cited framing is that even a bare root span carrying just those two IDs plus start/end timestamps "beats no attribution" at all, which is a useful floor to aim for before building anything more elaborate.
Durable execution needs the trace to stay legitimately open across a pause. Whether via LangGraph's interrupt(), the OpenAI Agents SDK's serialized RunState, or Google ADK's checkpoint model, the trace has to represent a run that's waiting — on a human, on a rate limit, on a scheduled retry — for hours or days without that gap reading as a hang. Heartbeat checkpointing, where an agent periodically emits progress and a crashed run resumes from the last heartbeat on a new worker, depends directly on this durability.
What's Still Genuinely Unsolved
Two 2026 arXiv papers are worth flagging precisely because they push back against the "this is a solved, standardized problem" framing the rest of this article has largely supported. "When Only the Final Text Survives: Implicit Execution Tracing for Multi-Agent Attribution" (arXiv:2603.17445) addresses the case where intermediate agent actions and inter-agent communications are lost or were never captured, and only the final output text remains — a real scenario for any fleet where full step-level tracing wasn't enabled at the time, whether for cost, storage, or oversight reasons — and proposes reconstructing which agent contributed what without the execution log that the rest of this article assumes exists. "Knowledge-Based Zero-Replay Debugging of Multi-Agent LLM Traces" (arXiv:2606.14805) tackles a related but distinct problem: even when full tracing was captured, the handful of causally decisive events in a long trace are buried in what is, in practice, unstructured noise, and the proposed remedy is a form of counterfactual analysis — rewinding to a specific span, editing an input or tool output, and re-running downstream to measure that one decision's actual causal effect on the final outcome, rather than relying on a human to spot it by reading spans top to bottom (the paper's own framing emphasizes predicting high-effect events to avoid the cost of full replay, which is a refinement worth reading the abstract for rather than taking this summary as exhaustive).
Both papers point at the same underlying gap: instrumentation has become standardized and cheap to adopt, but turning a large volume of well-formed spans into an actual explanation of why an agent did what it did remains a research problem, not a product feature. For an operator running a fleet rather than a single agent, that's the honest state of the art — get the OTel GenAI semconv layer in place because it's now close to free and vendor-portable, but don't expect it to answer "why did this happen" on its own once trace volume gets large enough that no human is reading most of it.
Notes on Sourcing and Fast-Moving Claims
This article's specific version numbers, product feature claims, benchmark figures, and adoption/timing details trace to vendor announcements, product documentation, press coverage, and aggregated secondary industry reporting current to mid-2026, supplemented by direct verification (this session) of the OpenTelemetry GenAI semantic-conventions repository, the OpenInference repository, and the two cited arXiv papers. They are illustrative of the trend and the shape of the landscape rather than independently audited facts, and given how fast this space moves, some of the specifics (version numbers, named integrations, benchmark percentages) may already be dated by the time you're reading this — treat product comparisons in particular as a snapshot of vendor positioning, not a controlled evaluation.
Sources: OpenTelemetry GenAI semantic conventions (open-telemetry/semantic-conventions-genai), OpenTelemetry blog "Inside the LLM Call" (2026), Greptime "How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools" (May 9, 2026), Arize OpenInference, CrewAI Arize Phoenix integration docs, Datadog LLM/Agent Observability docs and InfoQ coverage of the Google ADK integration (reported Feb 2026), Honeycomb AI/LLM observability use-case page, Langfuse integration docs and Medium write-up on the Langfuse gateway playbook (June 2026), Braintrust articles on agent observability and LLM cost tracking (2026), Sentry "Scaling observability for multi-agent AI systems," Maxim "Agent Tracing for Debugging Multi-Agent AI Systems," MLflow "Seeing Inside Your Multi-Agent System," Google Developers Blog on long-running ADK agents, LangChain durable-execution docs, OpenAI Agents SDK tracing/hooks docs, Claude Code observability-with-OpenTelemetry docs, Laminar's Claude Agent SDK instrumentation write-up (Dec 2025), arXiv:2603.17445, arXiv:2606.14805.