Agentic Workflow Replay and Time-Travel Debugging

Executive Summary

A production AI agent fails after 47 steps. The error message is unhelpful. The logs show the final tool call returned a 429, but re-running from scratch takes 12 minutes and $0.80 in LLM costs — and may not reproduce the failure at all, because LLMs are non-deterministic. Traditional debuggers cannot rewind an agent's probabilistic state. print statements cannot reconstruct a 6,000-token context window that was assembled and discarded 40 steps ago.

This is the debugging problem unique to agentic systems, and it has no equivalent in traditional software engineering. A bug at step 12 — a malformed API response that silently corrupted the agent's working memory — may only manifest as a failure at step 47. By then, the evidence is gone.

Agentic workflow replay addresses this by treating every agent execution as an event log that can be rewound, forked, and re-executed from any intermediate point. It draws from three disciplines that have independently arrived at similar solutions: event sourcing (append-only logs as the source of truth), durable execution (memoizing side effects for fault tolerance), and interactive debuggers (state inspection and modification at arbitrary breakpoints). The emerging synthesis — time-travel debugging for agents — is not a single product but a set of patterns that teams can apply today with available tooling.

This article covers the non-determinism problem that makes agent replay hard, the architectural patterns for recording and replaying executions, the current tool landscape, and the cost dynamics that make replay financially worthwhile beyond its debugging benefits.

The Non-Determinism Problem

The naive fix for agent non-determinism is temperature=0. It does not work. The sources of non-determinism in LLM inference are deeper than the sampling step:

Batch composition. Cloud providers combine requests on shared GPUs. Different batches produce slightly different floating-point results due to different reduction orders in the matrix operations.

Hardware heterogeneity. H100s, A100s, and older architectures implement matrix operations with different numerical precision. A model served from different hardware may return different tokens even with identical inputs and temperature=0.

Model design. Anthropic's own documentation states: "Even with temperature=0.0, results will not be fully deterministic."

Measured impact: accuracy variations up to 15% across identically-configured runs, with performance gaps reaching 70% between best and worst outcomes. In multi-step agent loops, this non-determinism compounds. The planner picks different tools based on sampling. Tools return different results given different invocation timing. The next plan depends on previous outputs. Two runs from the same starting state can take completely different trajectories.

This compounding is why replay cannot simply mean "re-run the agent from step 1 with the same initial inputs." True replay means returning the same outputs for every external interaction — LLM calls, tool calls, API requests — that the original run encountered. Anything short of this is re-execution, not replay.

The distinction matters because replay and re-execution serve different purposes:

• Replay (deterministic): investigate what actually happened, with bit-for-bit identical behavior. Useful for debugging and regression testing.
• Re-execution (non-deterministic): run again and see what happens differently. Useful for testing prompt changes or confirming a fix.

Both are valuable. The mistake is conflating them.

Core Architectural Patterns

Pattern 1: Checkpoint-Based State Replay

LangGraph implements the most widely-used version of this pattern. Every node execution is automatically persisted to a Saver backend (InMemorySaver for development, a database-backed saver for production). The checkpoint history forms a branching tree — like git commits for agent state.

# Access full execution history
states = list(agent.get_state_history(config))

# Select a checkpoint to branch from
target_checkpoint = states[12]  # Step 12, where things went wrong

# Modify state at that checkpoint
new_config = agent.update_state(
    target_checkpoint.config,
    values={'income_verification': corrected_value}
)

# Re-execute from that point
result = agent.invoke(None, new_config)

The critical caveat that LangChain's own documentation states explicitly: "Replay re-executes nodes — it doesn't just read from cache, so LLM calls, API requests, and interrupts fire again and may return different results."

This means LangGraph's "time travel" is actually re-execution with state injection, not true deterministic replay. It is still highly useful — you can isolate which steps were affected by injecting corrected state and observing downstream behavior — but you must account for LLM non-determinism when interpreting results. Two branches from the same checkpoint may diverge simply due to temperature.

Pattern 2: Trace-Based Deterministic Replay

The gold standard for reproducibility. Record every external interaction into an append-only log. On replay, return recorded responses instead of executing live calls.

This requires a two-mode architecture:

• Record mode: intercepts all LLM calls, tool calls, timestamps, and external state. Logs them as structured events in JSONL format.
• Replay mode: uses a ReplayLLMClient and ReplayToolClient that return recorded outputs token-for-token, in recorded sequence.

The key insight is dependency injection for zero-intrusion agent code:

class Agent:
    def __init__(self, llm_client, tool_client):
        self.llm = llm_client    # RecordingLLMClient or ReplayLLMClient
        self.tools = tool_client  # RecordingToolClient or ReplayToolClient

# In production (record mode)
agent = Agent(
    llm_client=RecordingLLMClient(live_client, trace_writer),
    tool_client=RecordingToolClient(live_tools, trace_writer)
)

# In replay mode
agent = Agent(
    llm_client=ReplayLLMClient(trace_reader),
    tool_client=ReplayToolClient(trace_reader)
)

The agent logic is identical in both modes. No conditional logging. No debugging scaffolding embedded in business logic.

What to record per LLM call:

• Complete prompt (all messages, system prompt, tool schemas)
• Model ID and exact version string
• All sampling parameters: temperature, top_p, max_tokens, seed
• Exact response tokens and finish reason
• Latency metrics: TTFT and total generation time
• Token counts: input, output, cached separately

What to record per tool call:

• Tool name and version
• Exact serialized arguments
• Complete response payload
• Success/failure status and full error details

An important subtlety: clock virtualization. If agents embed timestamps in prompts ("summarize events from the last 24 hours") or use wall-clock time for decision logic, replaying on a different day will produce different LLM behavior even with recorded responses — because the prompt content differs. Replay engines must intercept system clock calls and substitute recorded timestamps.

Pattern 3: Durable Execution Memoization

Durable execution frameworks (Temporal, Restate) arrive at replay from a different direction. Their primary goal is fault tolerance, but the mechanism they use — storing every activity result in an immutable event history — also enables debugging replay as a side effect.

In Temporal's model, a workflow consists of:

• Orchestration code (deterministic): must make the same decisions on every replay
• Activities (non-deterministic OK): results are stored in the Event History

When a Temporal workflow restarts after a crash, it replays the Event History. Because LLM call results are stored as activity outputs, the workflow does not re-invoke the LLM during replay — it returns the cached result from the log. The LLM sees the same response it saw originally. From the workflow's perspective, execution is perfectly deterministic.

From Temporal's engineering blog: "if a Workflow has to recover from a crash, it 'replays' your agent's progress to date, but the Workflow does not ask the agent for a new plan for decisions it has already made — the agent instead uses Temporal's Event History as a record of past decisions."

This has a powerful implication for experimentation. If you want to test 100 prompt variants on step 92 of a 100-step workflow, you can replay steps 1–91 from the Event History at zero cost (no LLM re-invocation), then branch into 100 parallel experiments for steps 92–100. Temporal has quantified this: "91% reduction in steps" in a representative example. The economics of prompt experimentation change fundamentally when historical steps can be replayed for free.

Restate implements the same concept through a Journal — a sequenced log of steps per workflow invocation. From Restate's documentation: "the engine restarts the function and replays the journal: each previously-completed step returns its recorded result instantly, until execution catches up to the point of failure." Developers wrap external calls in ctx.run(...) blocks; the framework handles retries, idempotency, and recovery.

A subtle but important point from Jack Vanlightly's deep-dive on durable execution: frameworks provide deterministic primitives for dates, random numbers, and UUIDs. If your agent generates a UUID to label an artifact, and you re-run the workflow, the framework ensures the same UUID is generated — not a different one. Without this, workflow "identity" breaks down during replay.

Pattern 4: ACID Transactions with Git-Backed Rollback

The agent-vcr open-source library introduces a pattern specific to coding agents and file-system-mutating workflows. When an agent fails mid-run, the problem is not just bad in-memory state — it is files written to disk that need deletion, not just state reversion.

agent-vcr wraps agent sessions in ACID-style transactions backed by git:

• BEGIN: creates an isolated git branch per session
• SAVEPOINT: checkpoints both in-memory state and the filesystem snapshot
• ROLLBACK: git reset --hard physically deletes files created during the failed run
• COMMIT: clean merge to main

Its "Ghost Replay" feature caches successful runs by task-description fingerprinting. On replay, outputs stream from the recording — no LLM calls occur. The cost ledger reports: CostLedger(saved=100% | $0.00 | 0 tokens). This is useful for regression testing: after a code or prompt change, verify that previously-passing task types still produce the same outcomes without paying LLM costs.

The Replay Fidelity Ladder

Not all replay is equal. A useful mental model is a fidelity ladder with increasing reproducibility guarantees:

Level	What Is Recorded	Replay Behavior
0	Logs only	Human forensics; no programmatic replay
1	Tool responses only	Tools are mocked; LLM calls are live (non-deterministic)
2	Full state snapshots	Replay from any snapshot; LLM calls are live
3	Tool responses + state + branching for parallel tools	Deterministic tool behavior; LLM still live
4	LLM responses + tool responses + state + timestamps	True deterministic replay; diff-based experiments

Level 0 is where most teams start; it requires no infrastructure but provides no programmatic reproducibility. Level 4 is where production debugging becomes reliable, but it requires a recording infrastructure and careful handling of non-determinism edge cases.

Most teams find Level 2 or 3 sufficient for debugging — full state snapshots let you isolate which state transition caused a failure, even without bit-perfect LLM replay.

The Tool Landscape

LangGraph / LangSmith

LangGraph provides native checkpoint-based time travel (Pattern 1). LangSmith provides the observability layer: step-by-step trace timelines, token counts per node, and a "Polly" AI assistant for trace analysis. LangSmith can load any production trace into the Playground for prompt iteration. LangGraph is the only major orchestration framework that enables branching from an intermediate checkpoint rather than restarting from scratch.

Temporal

Temporal is the industrial-strength choice for Pattern 3 (durable execution memoization). It supports L1–L5 workflow complexity levels, from conversational agents (seconds to minutes) to fully autonomous long-running workflows (indefinite). Production users include OpenAI, Replit, and Hebbia. Temporal's 2026 releases added Serverless Workers and Workflow Streams (durable streaming via Signal & Update for token delivery and live monitoring).

Restate

Restate implements a lighter-weight version of the same concept with a simpler programming model. SDKs for TypeScript, Python, Java/Kotlin, Go, and Rust. The ctx.run(...) primitive wraps non-deterministic operations; the framework handles journal management transparently.

AgentOps

AgentOps provides session replay with time-travel debugging for CrewAI, AutoGen, and LangChain agents. Over 400 LLM and framework integrations. It functions as an observability layer rather than a replay infrastructure — you can inspect what happened, but re-execution requires triggering the agent again.

Vellum

Vellum's Workflow Sandbox supports replay from any step with mocked integrations — no live API calls during debugging. This is Level 1 fidelity with good developer UX.

Braintrust

Braintrust allows converting production failures into permanent test cases. Its Playground loads any production trace for prompt iteration. The Brainstore query engine is optimized for AI trace data.

Open-Source: agent-replay

The agent-replay TypeScript CLI (clay-good/agent-replay on GitHub) stores 9 distinct step types (thought, tool_call, llm_call, retrieval, output, decision, error, guard_check) in SQLite. AI-powered evaluation presets (ai-root-cause, ai-quality-review, ai-security-audit) use small, cheap models to analyze replays at under $0.01 per evaluation. This makes systematic regression testing economically viable for teams without enterprise observability budgets.

Framework Comparison: Debugging Ergonomics

The three major orchestration frameworks take meaningfully different approaches to debugging failures:

LangGraph is the most capable for stateful debugging. It persists every node's state automatically. The get_state_history() API returns the complete checkpoint tree. You can modify state at any checkpoint and branch from it. The limitation is that re-execution is not deterministic — LLM calls fire again with potentially different results.

CrewAI provides task-level error boundaries. If an executor agent fails, the manager can reassign without restarting the entire crew. This is recovery, not debugging — you cannot inspect intermediate state or replay from a checkpoint.

AutoGen uses conversational retry: the agent reflects on its mistake in the conversation, revises the plan, and tries again. There is no persistent checkpoint mechanism; recovery happens through conversation, not state rollback. This works for short interactions but breaks down for long-horizon workflows where the conversation history grows large.

The pattern that emerges: for complex, long-running, multi-step agents where debugging failures is a real operational need, LangGraph with a persistent Saver backend (or Temporal/Restate for production-grade durability) is the appropriate choice.

Cost Dynamics

Replay is often framed as a debugging feature. Its cost implications are equally important.

Re-execution costs accumulate quickly. A 20-step agent that takes 2 minutes and $0.50 to run is expensive to debug via re-execution. If you reproduce a failure 5 times while investigating, you have spent $2.50 before writing a single fix. For agents with longer horizons or more expensive model calls, this compounds rapidly.

Memoized replay changes the economics. With Temporal/Restate, replaying steps 1–19 of a 20-step workflow costs $0.00 — results are returned from the event history without invoking the LLM. Only the step being modified or tested incurs new costs. The "91% reduction" that Temporal cites for prompt experimentation translates directly to debugging: investigating a failure at step 18 means paying for 1 step of LLM inference, not 20.

Trace storage is cheap. A typical 10-step agent run with complete prompt recording is 100–200 KB. At cloud storage prices (~$0.023/GB/month for S3), storing a year of daily agent runs for a small team costs less than $1/month. The overhead that matters is indexing and query infrastructure for the observability layer, not raw storage.

Regression testing changes character. Without replay, verifying that a prompt change did not break existing behavior requires re-running the full agent against test cases — real LLM cost for every test run. With cached recordings (Ghost Replay or Temporal memoization), regression tests can run at near-zero marginal cost. Teams can afford to run comprehensive regression suites on every pull request, not just before major releases.

AI-powered trace analysis. Tools like agent-replay use small models (Claude Haiku, Gemini Flash, GPT-4o-mini) to automatically classify failures from replay data — root cause analysis, security audits, quality reviews — at under $0.01 per evaluation. This makes systematic post-mortem analysis affordable at scale.

The Seven Primitives for Trustworthy Replay

The Sakura Sky engineering team published a systematic framework for building replay infrastructure around seven primitives:

1. Structured Execution Trace: append-only JSONL, every LLM call and tool invocation recorded with full fidelity
2. Stable Model and Tool Metadata: exact model version strings and tool version tags, without which replay output may diverge silently
3. Replay Engine: deterministic event ordering via monotonically increasing step counters; independent cursors per event category; exhaustion detection to catch truncated recordings
4. Deterministic Stubs: ReplayLLMClient and ReplayToolClient that return recorded outputs and validate metadata consistency, raising errors on version mismatches
5. Agent Harness: dependency injection so the same agent code runs in record and replay modes without modification
6. Governance Integration: replay integrates with audit logs; kill switch activations, breaker trips, and policy violations are recorded as trace events
7. Deterministic Regression Testing: historical traces become "golden files" for snapshot-style testing — the same concept as VCR cassettes in HTTP testing, applied to full agent executions

The governance integration is often overlooked but becomes important at production scale: if your replay infrastructure does not record policy enforcement events, you cannot audit whether an agent that was later found to have caused harm was operating within its declared permission boundaries at the time.

Practical Recommendations

Start with trace logging before replay infrastructure. Comprehensive structured logging (everything recorded in JSONL with run IDs and step IDs) delivers significant value before you build full replay. You can do forensic debugging manually. The data is there when you are ready to build replay on top of it.

Use dependency injection from day one. Injecting LLM and tool clients rather than instantiating them directly in agent code costs nothing and enables replay later. Teams that embed LLM calls directly in business logic have to refactor when they add observability — dependency injection avoids this.

Choose your replay model based on your failure modes. If your agents fail during tool calls (API errors, network timeouts), LangGraph checkpoints with state injection are sufficient. If your agents fail due to subtle reasoning errors that are sensitive to LLM non-determinism, you need trace-based deterministic replay with recorded LLM responses.

Treat recordings as regression assets. When an agent fails in production and you fix it, record the fixed run and add it to your test suite. The next prompt or code change that causes a regression will manifest in the replay comparison before it reaches production.

Account for clock virtualization. If your agents use wall-clock time in any prompt or decision, add clock interception to your recording layer from the start. This is much harder to retrofit than to build in early.

For multi-step agents in production: use durable execution. The choice between Temporal and Restate is mostly about operational complexity versus programming model simplicity. Both provide replay as an intrinsic property of their execution model — not as an add-on feature.

Conclusion

The debugging crisis for long-running AI agents has a structural solution: event sourcing applied to agent executions. The same append-only log that enables forensic investigation enables deterministic replay, which enables regression testing, which enables safe iteration on prompts and agent logic.

The tooling is immature but available. LangGraph provides checkpoint-based state branching. Temporal and Restate provide memoized replay as a reliability primitive. Open-source libraries like agent-vcr and agent-replay provide lighter-weight alternatives. Observability platforms are converging on OTel GenAI conventions that make trace data portable across tools.

The teams shipping reliable long-horizon agents in 2026 share a common characteristic: they treat agent executions as data, not as ephemeral processes. Every run is a record. Every record is a potential debugging artifact, regression test, or training signal. The shift from "run the agent and hope" to "record every run and replay when needed" is the difference between shipping with confidence and shipping with anxiety.

As one engineering team put it: "If you can't replay it, you can't ship it."

---

Sources: JumpCloud - What Is Time-Travel Debugging for Agent Traces, TianPan.co - Deterministic Replay: Debugging AI Agents That Never Run the Same Way Twice, Sakura Sky - Trustworthy AI Agents: Deterministic Replay, DEV Community - Debugging Non-Deterministic LLM Agents: LangGraph Time Travel, LangChain Docs - Use time-travel, Temporal - Durable Execution meets AI, Temporal - Replay 2026 product announcements, Restate - What is Durable Execution, Jack Vanlightly - Demystifying Determinism in Durable Execution, agent-vcr GitHub, Galileo - AutoGen vs. CrewAI vs. LangGraph vs. OpenAI Agents Framework, arxiv - AgentRR: Get Experience from Practice, ACM CHI 2025 - Interactive Debugging and Steering of Multi-Agent AI Systems