Session Resumption and Conversation Continuity in Realtime Voice AI Agents

Executive Summary

Native mid-session resumption is rare. Only Gemini Live API offers a first-class resumption token (session_resumption handle, valid 2 hours, unlimited reuse). OpenAI Realtime, Vapi, and Deepgram Voice Agent have no model-level resume — a dropped or expired session is simply gone, and the application must rebuild context on a fresh connection.
Every vendor imposes a hard session ceiling: OpenAI Realtime 60 minutes, Gemini Live 15 min audio-only / 2 min audio+video without context compression, Amazon Nova Sonic ~8 minutes (AWS-imposed connection limit). LiveKit and Pipecat are orchestration layers, not model providers, so their limits are really "how long can you keep the underlying model session alive," which inherits the vendor cap above.
The dominant continuity pattern across the ecosystem is hybrid re-injection: keep the last 1-2 turns verbatim, summarize everything older into a single system-role item, and re-insert that summary into the new session. OpenAI's own cookbook example triggers this at 20k-32k tokens and keeps only the last 2 turns raw.
Reconnection engineering converges on the same recipe everywhere: exponential backoff (roughly 100ms → 5s ceiling), periodic ping/keepalive frames (8-30s depending on vendor), and buffering microphone audio locally during the outage so nothing is lost before the socket comes back.
Async ASR (input transcription running as a side-channel) reliably arrives out of order relative to the model's own audio reply because transcription is decoupled from generation. OpenAI's documented fix is to correlate everything by item_id / previous_item_id rather than arrival order — there is no guaranteed-ordering guarantee in the event stream itself.
Multi-day continuity (standup bots, coaching agents) is handled almost universally as an external memory layer, not a model session concept: extract durable facts after each call, merge into a profile, and re-inject a condensed profile (not a raw transcript) at the start of the next call.

1. How production stacks handle drops, and native resumption

OpenAI Realtime API. As of mid-2026, the documented hard limit is a flat 60-minute maximum session duration (raised from an earlier 30-minute cap) with no built-in resume mechanic; the guide is explicit that connections are treated as continuous and doesn't describe a recovery path for a dropped or expired session [1][4]. The model gpt-realtime has a 32,768-token context window, with responses capped at 4,096 output tokens by default, and the server auto-truncates from the oldest messages once the window fills — silently dropping early context unless the app manages it [23]. Community threads describe practitioners proactively reconnecting every 5-10 minutes rather than waiting for the ceiling, and refreshing ephemeral tokens roughly every 29 minutes when running two overlapping sessions to paper over the cutoff [1][2]. A LiveKit GitHub issue is a good illustration of how brittle this gets in practice: LiveKit's auto-reconnect logic (added to handle generic disconnects) does not trigger when OpenAI closes the socket specifically with a "session hit the maximum duration of 30 minutes" error — the error is logged but no retry fires, an issue open since August 2025 with no confirmed fix [3].

Gemini Live API. This is the only stack in the set with genuine server-side session resumption. Enabling SessionResumptionConfig makes the server emit SessionResumptionUpdate messages containing a handle; the client stores the latest handle and passes it into SessionResumptionConfig(handle=...) on the next connection to resume with context intact, with no limit on how many times you can do this and tokens valid for 2 hours after termination [5]. Before cutting a connection the server sends a GoAway message with a timeLeft field — 60 seconds of advance warning — giving the client a window to gracefully reconnect using the stored handle before the socket is closed as ABORTED [5][22]. Session ceilings without compression are short: 15 minutes for audio-only, 2 minutes for audio+video; enabling context-window compression (a sliding-window mechanism with a configurable token trigger) extends sessions "to an unlimited amount of time" by compressing older turns server-side [5]. A transparent resumption mode exists specifically to let the client replay any unconsumed server messages after reconnecting, minimizing lost audio.

LiveKit Agents. LiveKit is transport/orchestration, not a model — it wraps whichever realtime API you choose. Its own WebRTC layer has two reconnection tiers: a lightweight signaling-socket reconnect + ICE restart for transient network blips (near-invisible to the user), and a full reconnection when ICE restart fails, which fires a Reconnecting/Reconnected room event pair so the app can update UI state [7][8]. The client SDK's ReconnectPolicy takes a list of backoff delays it iterates through, giving up once the list is exhausted [8]. Separately, keeping an agent session alive across a participant's reconnect requires explicit configuration: AgentSession.close_on_disconnect=False (default is True, i.e., the agent closes the moment the user drops) plus a room departureTimeout (default 20s) extended to outlast a real reconnect — LiveKit's own blog suggests 300 seconds for web apps. The rejoining client must reuse the same participant identity so RoomIO re-links to the same agent session and its in-memory ChatContext, which is what actually preserves conversation history — but only until departureTimeout expires, at which point the room (and context) is destroyed [7].

Pipecat. WebSocketTransport will retry a broken connection twice by default, but continuity quality is only as good as the weakest link in the pipeline — a real production bug report shows a third-party STT service's socket dying silently after ~60-70 seconds of call silence because it had no keepalive ping (unlike the paired TTS service, which pinged every 20 seconds), and the client-side guard failed to detect the dead socket [9]. The recommended continuity pattern is entirely application-managed: persist the conversation context object to a database keyed by user ID, and reload it into a fresh OpenAILLMContext (or equivalent) on reconnect — Pipecat itself has no session-resumption primitive.

Vapi. Distinguishes chat "sessions" from phone calls. Text/chat sessions expire after 24 hours by default and simply require a new session afterward — no resumption [13]. For phone calls specifically, Vapi documents that it does not support resuming a dropped call from where it left off; once a call drops and a new one is placed, the conversation starts over. The recommended pattern is to save conversation state via the end-of-call-report webhook, then on the caller's next inbound call look up their prior transcript and inject it into the new assistant's configuration as context [13][14]. Every call ends with a structured endedReason code for diagnosing why the drop happened.

Retell AI. Recent platform updates add ping-pong keepalive events and an auto-reconnect mechanism on the LLM websocket leg, including server round-trip latency tracking exposed via the get-call API, plus a config flag to toggle reconnection behavior [15]. Cross-call (not mid-call) continuity is handled via CRM integration: returning callers are recognized and the agent can pick up from where a previous call left off, which is memory continuity rather than session resumption.

Deepgram Voice Agent API. Idle WebSocket connections are closed after ~10 seconds of silence; if the client is intentionally quiet (e.g., between turns), it must send a KeepAlive message roughly every 8 seconds to hold the socket open [16]. There's no session-duration cap analogous to OpenAI/Gemini/Nova Sonic in the docs consulted, but every reconnection is a fresh connection with timestamps resetting to 00:00:00, so the client must realign audio timestamps manually after a reconnect [17].

Amazon Nova Sonic. The InvokeModelWithBidirectionalStream API carries an AWS-imposed connection limit of roughly 8 minutes [11]. AWS's own reference architecture treats this as something to engineer around rather than a resumable session: proactively open a new connection before the limit hits, pass the prior session's chat history into the new session as context, and buffer user audio during the handoff so there's no audible gap — described as a "seamless session rotation" pattern rather than a native resume feature [11][12].

2. Continuity when the model session can't be resumed

Three patterns recur, in increasing sophistication:

Full transcript re-injection. Simplest and safest for fidelity, but scales linearly with call length and hits context-window ceilings fast on 15-32k-token windows, especially since voice/audio tokens cost roughly 10x the equivalent text tokens (cited directly in OpenAI's cookbook example) [18].
Summarization-only. Compress the entire prior conversation into a short paragraph and drop the turn-by-turn detail. Cheapest on tokens, but loses specific phrasing, numbers, and commitments the user made — risky for anything transactional.
Hybrid (dominant pattern). Keep the last N turns verbatim, summarize everything before that into a single compressed item. OpenAI's cookbook implementation triggers on response.done when tokens exceed a SUMMARY_TRIGGER (2,000 in the demo, 20k-32k realistic for production) and history exceeds KEEP_LAST_TURNS (2). The summary is re-injected via conversation.item.create with role: "system" rather than "assistant", specifically to avoid the model interpreting it as something it "said" and shifting modality; the original condensed items are then removed with conversation.item.delete, positioned using previous_item_id [18].

Trade-offs. Token cost scales with option 1 and shrinks with 2-3, but summarization requires an extra LLM call on the critical or near-critical path. Model re-anchoring is real: a fresh session has no memory of how it was speaking (tone, in-progress task state), so summaries need to encode not just facts but conversational state ("we were mid-way through collecting the user's shipping address; last field asked was ZIP code"). Greeting awkwardness — the new session's model re-introducing itself or re-greeting the user after a reconnect that felt seamless to them — is a known UX failure mode; none of the vendor docs consulted describe a built-in suppression mechanism, so this is best treated as an inference from the general pattern requiring an explicit system-prompt instruction such as "you are continuing an existing call after a brief technical interruption; do not re-introduce yourself."

3. Client-side reconnection engineering

The engineering pattern is consistent across blog posts and vendor guidance: exponential backoff starting aggressively (100ms → 200 → 400 → 800, capped around 5s), combined with periodic ping/pong frames (15-30s generally; Deepgram wants a KeepAlive every 8s during silence [16]) to detect a half-dead socket before the user notices. Keeping the microphone/audio graph alive independent of the transport matters — one pattern is capturing mic input before the agent connection finishes establishing ("instant connect"), buffering that pre-connection audio so nothing spoken in the first moment is lost. During an outage the client keeps recording into a local buffer (commonly 24kHz PCM16, ~100ms frames) and flushes it once the socket re-opens, rather than dropping audio. LiveKit's WebRTC layer exposes two severities via Reconnecting/Reconnected room events — a benign "reconnecting…" indicator for ICE-restart-class blips, a harder state for full reconnects [7][8].

On the UX-state question, real product behavior diverges: LiveKit-style browser apps favor silent auto-retry with a UI indicator, while ChatGPT's Advanced Voice Mode is documented (via user bug reports) to surface an explicit "Connection failed, tap to retry" manual-retry prompt when reconnection isn't immediate [21], suggesting OpenAI's client intentionally avoids indefinite silent retry loops that could strand a user mid-call without feedback. Community reports also link some ChatGPT voice failures to IPv6/carrier-path issues resolved by forcing IPv4 — a reminder that "reconnection" failures are sometimes network-layer, not application-layer [21].

4. Async ASR ordering and archive consistency

A frequently reported production gotcha: with OpenAI Realtime, conversation.item.input_audio_transcription.completed events are not guaranteed to arrive in speaking order relative to other events, including the model's own spoken reply — a user's transcript can show up after the agent has already responded to it. OpenAI's guidance is explicit: do not rely on arrival order; instead correlate each transcription event to its source input item via item_id, and use input_audio_buffer.committed's previous_item_id field to reconstruct true conversational ordering [19]. The root cause, per community analysis (not officially confirmed by OpenAI staff in the thread reviewed), is that server-side VAD only commits the input buffer after a detected pause (silence_duration_ms), and transcription only starts after commit — so during continuous speech, no incremental transcript deltas stream at all, and the "completed" transcript can lag well behind the audio response that the model already generated directly from raw audio (bypassing the transcript entirely) [19]. Practical workarounds reported: manually commit the audio buffer every 300-500ms instead of relying on VAD-triggered commits, use gpt-4o-transcribe/mini (which support streaming deltas) instead of whisper-1, or switch to semantic VAD (better turn-boundary punctuation, at the cost of occasional truncation at segment edges) [19].

The practical ordering strategy consistent with this design (an inference drawn from the documented item_id/previous_item_id correlation mechanism, not a directly cited "best practice" doc) is to render a placeholder slot in the UI transcript keyed by item_id the moment the item is created, then fill in its text asynchronously whenever the transcription event lands — rather than appending transcript lines in event-arrival order. For durable storage/archival, the same principle applies: persist events with their item_id/previous_item_id chain and sort by that logical chain at read time, not by ingestion timestamp, so a slow ASR result doesn't get archived in the wrong position relative to the reply it was actually a response to.

5. Multi-day continuity across sessions

None of the realtime voice APIs surveyed carry state across independent calls/days by default — Gemini's resumption handle is capped at 2 hours, Nova Sonic's session rotation is intra-call only, and OpenAI/Deepgram sessions are fully stateless once closed. Multi-day continuity is therefore handled as an external memory layer above the realtime session, documented well by Mem0's voice-agent architecture guide [20]. Two write patterns recur: per-round writes (extract and store facts after every exchange, off the latency-critical path) versus per-session writes (batch extraction once the call ends, cheaper and gives the summarizer the full call) [20]. Retrieval at call-start commonly uses a sliding window plus memory-layer lookup: keep only the most recent literal transcript in context, and rely on the memory layer — not raw replay — for anything older, described as the most common production pattern since it avoids per-call summarization [20]. For recurring check-in agents (standup bots, coaching agents), the equivalent is a dynamic profile: after each session, extraction pulls durable facts (goals, blockers, recurring themes) and merges them into a standing profile injected at the next call's start — not the prior day's raw transcript. This bounds injected context regardless of history length, at the cost of losing exact earlier phrasing (the same fidelity-vs-token trade-off as in-session summarization, at day-granularity). Amazon's Bedrock AgentCore advertises "persistent memory across sessions" as a platform feature here, though the material reviewed stayed at a marketing level without exposing storage/retrieval mechanics [12].