Production Realtime Voice Agents: Relay Architecture, Barge-In, and the Sample-Rate Trap

Executive Summary

OpenAI's Realtime API and Google's Gemini Live API make a working voice demo trivial: open a socket, stream PCM, get PCM back. Getting that demo to survive real phones, real networks, and real interruptions for months without silent quality regressions is a different problem, and almost none of it is covered in the vendor quickstarts. The failure modes cluster around five places: the browser's audio capture pipeline lying about its own sample rate, the model never producing an input transcript unless a second model is explicitly asked for one, the client and model disagreeing about how much of an interrupted answer was actually heard, turn-detection defaults that chop sentences mid-thought, and a client that — if unconstrained — can send arbitrary control events to a paid, stateful upstream session. None of these show up in a five-minute demo on a laptop with a wired connection. All of them show up in week two of production, on Android Chrome, over LTE, mid-interruption. This note walks through each pitfall, what the current external ecosystem (OpenAI, Google, LiveKit, Pipecat) says about it, and the mitigations Zylos's Rounds voice-standup product had to build to hold up in production.

The Architecture Baseline: Why You Need a Relay at All

Both major streaming speech-to-speech APIs are stateful, long-lived, bidirectional connections: OpenAI's Realtime API over WebSocket or WebRTC, Gemini Live over WebSocket. Neither offers a story for authenticating a browser directly with a long-lived secret. As one detailed technical write-up on the API puts it, the Realtime API "currently lacks client-side authentication, making it insecure to connect to the API directly from the user's browser" — and OpenAI's own WebRTC guide confirms the mitigation: either (a) WebRTC with a short-lived ephemeral token minted server-side, or (b) a relay, where the browser talks to your server and your server holds the real key and talks upstream. Gemini Live's guidance converges on the same shape — Google's own architecture writeup describes a backend that "manages the persistent WebSocket connection" to the Live API while the browser only ever talks to your own domain.

This is exactly the shape Rounds uses: browser ↔ relay server ↔ upstream realtime API. The API key never reaches client JS, which matters because a browser tab is untrusted attacker-controlled code the moment it's open in someone's phone. But the relay earns its keep beyond key custody in two other ways that are easy to underweight until you've been burned:

1. Event allowlisting. The Realtime/Live protocols are chatty, symmetric event buses — the client can send session.update to rewrite the system prompt or tools mid-call, or conversation.item.create to inject fake turns into the transcript. If client code is compromised, malicious, or just buggy, an unfiltered relay lets it do all of that on a paid session. A production relay validates every inbound client→upstream event against a strict allowlist (audio append, commit, response.create, truncate, cancel — not attacker-supplied session.update) before forwarding it.
2. Egress and cost control. Centralizing the upstream connection lets you enforce per-session token/time budgets, log every event, and swap providers without touching client code.

The tradeoff against WebRTC-direct-to-vendor is real: WebRTC gives a shorter media path (browser → vendor edge, no extra hop), and OpenAI's own docs explicitly recommend it for browser and mobile clients "for more consistent performance." Reported end-to-end latency for these APIs commonly falls somewhere in the low-hundreds-of-milliseconds-to-roughly-a-second range depending on turn, network path, and measurement method — figures vary across vendor posts and independent write-ups, so treat any specific number, including ones in this note, as illustrative rather than a benchmark to design against; if latency budget is load-bearing for your product, measure it yourself against the current API. A relay adds a hop, and therefore latency and infrastructure. Most production deployments accept that cost because the alternative — an ephemeral-token WebRTC connection with no server in the loop — loses the allowlist and centralized observability, and still needs a server to mint the ephemeral token in the first place. The pattern that's emerged across the ecosystem (LiveKit Agents, Pipecat, Rounds, all independently) is: keep the server in the media path, accept the extra hop, get control back in return.

Pitfall 1: The Browser Lies About Its Sample Rate

This is the pitfall that costs the most debugging time because the symptom is deceptive: the model receives audio and responds, but transcription quality degrades and the model's own voice sounds subtly wrong to some users, specifically on Android. The cause is a browser bug, not a protocol issue.

The naive approach: force new AudioContext({ sampleRate: 24000 }) to match the model's expected input rate and skip resampling entirely. This works reliably on desktop Chrome and Safari. It silently fails on mobile Chromium: "not all browsers or devices will honor your request for a specific sample rate — some will stick to the system default," and on Android in particular the hardware layer frequently overrides the request. AudioContext.sampleRate then reports 24000 (or whatever was requested) while the actual PCM samples delivered were captured at the device's native rate — commonly 48kHz. The result is audio pitch-shifted and time-compressed by exactly the requested/actual ratio before it ever reaches the model: transcription and VAD components try, and partially fail, to make sense of it, and any playback processed under the same false assumption sounds wrong.

The fix Rounds settled on, which matches the general Web Audio guidance to resample rather than fight the platform: never force a rate. Open the AudioContext with no sampleRate option, let it report whatever the device's native rate actually is, capture at that rate, and downsample client-side before sending to the relay:

device native rate → capture (no forced AudioContext rate)
  → if 48000: exact 2:1 decimation via pair-averaging (cheap, alias-safe for 2:1)
  → else: linear interpolation resample to 24000,
          carrying fractional-sample phase across packet boundaries
  → send 24kHz PCM16 to relay

The pair-averaging special case for 48k→24k matters because it's the overwhelmingly common native rate on both desktop and mobile, and averaging adjacent sample pairs is cheap and a reasonable anti-alias filter for an exact 2:1 ratio — consistent with the DSP result that linear interpolation itself acts as a triangular-pulse lowpass filter appropriate for downsampling. For other native rates (44.1kHz being the other common one), linear interpolation is used, but naively resetting the interpolation phase at each audio-callback boundary reintroduces exactly the discontinuity-driven noise that appears "when pieces of PCM data with different sampling frequencies are switched from one to another" — audible clicks or roughness at chunk boundaries. Carrying the fractional-sample phase remainder across packet boundaries, treating the whole stream as one continuous resample rather than many independent ones, removes that. It's a small amount of code, but the single highest-leverage fix in the stack: the difference between "works on my MacBook" and "works on the phone the user is actually holding."

Pitfall 2: There Is No Free Input Transcript

A natural assumption going in: a speech-to-speech model that hears audio and produces audio must obviously "know what was said" in text form too, so surely there's an event for it. There generally isn't, by default. OpenAI's Realtime API only emits an input transcript if you explicitly configure session.input_audio_transcription with a transcription model — whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe, or similar — at which point it runs that model as a separate transcription pass alongside the conversational model and emits conversation.item.input_audio_transcription.completed events once it finishes. This is confirmed directly in OpenAI's docs: it's architecturally a second model bolted onto the session, not a byproduct of the speech-to-speech model itself. Gemini Live is similar — the low-latency native-audio conversational path and a text transcript are not the same computation, and the practical rule is the same: don't assume transcripts, request them explicitly.

This is exactly the invariant Rounds encoded: run a dedicated ASR sidecar with a language hint, rather than relying on incidental output from the conversational model. Beyond needing text for logging, this matters for two easy-to-miss reasons: the transcription model can lag noticeably behind the audio, since it's a separate inference pass and its completed events arrive out of lockstep with the conversation; and it needs its own per-conversation language hint — a model with no language signal will happily mis-transcribe non-English speech into English-shaped nonsense, especially in code-switching scenarios common in bilingual teams. Any system that needs a durable, searchable transcript of what a user actually said, not what the model decided to respond to, has to treat transcription as its own pipeline stage with its own failure modes, not an afterthought of the voice loop.

Pitfall 3: Barge-In Needs Exact Milliseconds, Not "Cancel"

Interruption handling is where the gap between demo and production is widest, because the naive implementation looks correct in testing and silently corrupts context in the field. The naive version: when the user starts talking over the model, call response.cancel and stop local playback. This is necessary but not sufficient. OpenAI's own developer community has documented the failure repeatedly: canceling the response stops future audio, but the model's conversation history still contains the full, uninterrupted response it had generated — as far as the model's context is concerned, the user heard the complete answer, including the part cut off and never actually played. Next turn, the model reasons from a false premise and the conversation quietly desyncs from what actually happened. One community thread frames the open question precisely: "are we really meant to delete the entire response text, or just what had not been heard by the client?" — the answer is the latter, and getting it right requires client-tracked state the API cannot infer on its own.

The correct sequence, confirmed across OpenAI's docs and community troubleshooting threads: on input_audio_buffer.speech_started from the server, stop local playback immediately and measure exactly how many milliseconds of the assistant's audio were actually played before it stopped; send conversation.item.truncate with that item's ID, content index, and audio_end_ms set to the actual played-back duration (not the full response length, not zero); send response.cancel to stop in-flight generation; and guard against a known race where audio deltas already in flight can still arrive and briefly resume playback after truncation — production implementations mute-and-drop any further deltas tagged with the truncated item's ID.

This is precisely Zylos invariant #3, and it's corroborated rather than contradicted by the external ecosystem — one of the most-repeated gotchas in OpenAI's own developer community, suggesting it's a near-universal first production bug on this API rather than a Rounds-specific quirk. Gemini Live handles the symmetric problem differently: some implementations run client-side VAD in an AudioWorklet thread so local playback can be muted at effectively zero latency before waiting for a server-side interruption signal, trading a bit of false-positive risk for eliminating the round-trip delay of server-detected barge-in. Either way, the lesson is the same: the model's belief about what the user heard has to be actively synchronized by the client, not assumed.

Pitfall 4: Turn Detection Defaults Split Sentences

Server-side voice activity detection ships in two flavors on OpenAI's Realtime API: server_vad, a threshold-driven detector with tunable threshold, prefix_padding_ms, and silence_duration_ms; and semantic_vad, which uses a semantic classifier over the words spoken so far to estimate the probability the user has actually finished their thought, extending the wait when the utterance sounds incomplete (e.g., trailing off on "ummm…") and cutting the wait short when it sounds definitively finished. Semantic VAD exposes an eagerness parameter (low / medium / auto / high) that tunes the maximum wait timeout before the model is allowed to respond.

The default (auto, equivalent to medium) is tuned for generic English conversational cadence, and it under-serves languages and sentence structures where the grammatically decisive information lands late in the sentence — Chinese topic-comment structures, subordinate clauses that resolve at the end, or speakers who simply pause mid-thought. A model that jumps in at the first plausible pause doesn't just interrupt rudely; it responds to a fragment as if it were the complete input, guessing at intent instead of waiting for it. This is Zylos invariant #4, and it's a direct, documented consequence of how eagerness is specified, not a Rounds-specific edge case — the parameter exists precisely because OpenAI anticipated that different content and languages need different tolerance for how long a pause can be before it's actually a turn boundary. The fix isn't a single global constant: eagerness needs per-language tuning, and for languages where sentence-final particles or verbs carry the meaning, biasing toward low (or a longer silence_duration_ms under server_vad) trades responsiveness for not truncating half the user's thought.

Pitfall 5: Make the Model Admit When It Didn't Hear Well

The generic advice on voice-agent hallucination — lower temperature, tight role/boundary prompts, escalate on low confidence — is well established, but it treats hallucination purely as a quality-of-response problem. Zylos invariant #5 reframes it as a diagnostic tool: an explicit system instruction telling the model to say "I didn't catch that" and ask the user to repeat rather than guessing at a plausible-sounding response, given ambiguous or garbled audio.

The reframe matters because a model that confidently answers a question it never actually understood is the worst failure mode operationally — it produces a coherent-looking transcript and a coherent-sounding answer, so nothing in the logs flags it as broken. It looks like a successful turn. Only a careful listen, or a user who circles back confused, reveals the model answered a question no one asked. An instruction to admit uncertainty converts these into a directly observable signal: a spike in "I didn't catch that" responses for a specific user, device, or network condition is a leading indicator of a capture-quality regression — often precisely the sample-rate bug from Pitfall 1 — well before it shows up as a downstream complaint. Cheap to implement, a few sentences in the system prompt, and disproportionately useful as an early-warning system, which is why it belongs next to the audio-pipeline fixes rather than filed under generic prompt-engineering advice.

Cost and Observability at Scale

Realtime audio is priced and metered very differently from text, and the gap between naive and optimized cost is large enough to change architecture decisions. Pricing for these APIs moves often, so treat the figures here as on the order of, as of mid-2026, rather than a current rate card. OpenAI's published pricing for its flagship realtime model runs $32 per million audio-input tokens and $64 per million audio-output tokens; converting that to a per-minute estimate requires an assumption about how many audio tokens a minute of speech consumes, and published/community estimates for that conversion vary, so any "$X per minute" figure — including ones derived here — is a back-of-envelope estimate, not a vendor-quoted rate, and is notably output-heavy since the model "talks" more tokens than it "listens" to per second. Gemini's Live audio pricing is structured similarly at different absolute rates (see Google's own pricing page for current figures). The single biggest lever is prompt/audio caching — OpenAI's rate card lists cached audio input at roughly 98–99% cheaper than fresh input for its realtime models — which rewards architectures that keep session context (system instructions, prior turns) stable rather than rewriting it every turn.

On observability, the practical minimum is per-session logging of: full audio (to debug capture-quality issues like Pitfall 1), the full ASR-sidecar transcript (Pitfall 2), every tool/function call, latency percentiles per turn, and token usage split by input/output/cached — enough to reconstruct, after the fact, both what happened on the wire and what the user experienced. Because most of the pitfalls above manifest as subtle quality regressions rather than hard errors, a system with no audio/transcript retention has no way to detect them until a user complains.

Framework Landscape: Build vs. Adopt

For teams evaluating whether to hand-roll a relay or adopt an existing framework, two open-source options that are widely used as of mid-2026 are Pipecat and LiveKit Agents. Pipecat models the problem as a pipeline of frame processors (audio in → STT → LLM → TTS → audio out) with a large, actively maintained library of provider integrations — a strong fit for teams wanting vendor flexibility, comfortable in Python. LiveKit Agents models the agent as a participant in a WebRTC "room" alongside humans, and treats SIP/telephony as a first-class participant type rather than a bolt-on — a natural fit for real concurrency or phone calling without a separate bridge. Both are substrates for implementing the pitfalls above, not alternatives to solving them: none of the five disappear by adopting a framework, which mainly saves you from re-deriving relay/allowlist/transport plumbing, not from getting sample-rate handling, transcription, truncation, turn-detection, or anti-hallucination framing right. In our own deployment (an internal observation about our own build choice, not a benchmarked comparison against the frameworks above), Rounds' relay was built directly against the vendor APIs rather than through either framework, on the reasoning that invariants this specific — mobile sample-rate handling, per-language eagerness tuning — benefited more from owning the relay layer than from a generic pipeline abstraction.

None of the five pitfalls above are exotic — each is individually documented somewhere in a vendor doc or a community forum thread. What makes them a production concern rather than a footnote is that they compound: a sample-rate bug degrades both ASR transcript quality and the model's own turn-detection accuracy at once, and a barge-in truncation bug corrupts every subsequent turn, not just the interrupted one. Building a realtime voice agent that survives real users, real phones, and real networks is less about the initial integration — which the vendor quickstarts make genuinely easy — and almost entirely about closing these five gaps between what the protocol promises and what actually happens at the edges: mobile audio hardware, network jitter, half-finished sentences, and users who talk over the agent because that's what humans actually do in conversation.

Notes on Sourcing and Fast-Moving Claims

Several figures in this note — realtime API pricing, token-per-minute conversions, end-to-end latency, and the exact capabilities of third-party frameworks — trace to vendor announcements, vendor pricing pages, and aggregated community/press reporting current to mid-2026. Where a claim was independently checked against a primary source at the time of writing, it's linked inline; where it wasn't, the language above has been hedged accordingly. These numbers are illustrative of the trend (realtime audio is expensive relative to text, caching materially reduces it, WebRTC-direct is faster but gives up relay control) rather than independently audited benchmarks, and pricing in particular should be expected to have moved by the time you're reading this — check the vendor's current rate card before making a cost decision on it.