Server-Relay Architectures for Realtime Voice Agents: When the Browser Can't (or Shouldn't) Talk to the Model Directly

Executive Summary

Speech-to-speech LLM APIs (OpenAI Realtime, Gemini Live, Qwen-Omni-Realtime, GLM-Realtime) present builders with an architectural fork that looks like a transport detail but is actually a control-plane decision: connect the browser directly to the provider over WebRTC using short-lived ephemeral tokens, or route all audio through a server relay that holds the real API key. Several major providers' browser quickstarts steer toward the first option — OpenAI's docs call client-side use of a durable key "a dangerous and insecure practice" and position WebSocket for "server-to-server integration" (Gemini Live is the exception, discussed below: Google recommends a relay for browser clients rather than a first-party browser WebRTC path) — yet a large class of real deployments ends up on the relay anyway, for reasons the quickstart guides don't dwell on: the client network can't reach the provider at all (regional blocking, corporate egress policy), the application needs the raw transcript and audio in its own custody rather than reconstructed from provider webhooks, moderation or business logic must sit in the audio path, or the team wants provider swappability behind a stable client contract.

The relay's costs are quantifiable and, for conversational use cases, tolerable. The extra hop adds roughly 10–100ms according to one widely-cited WebRTC infrastructure vendor's engineering writeup; TCP head-of-line blocking replaces WebRTC's loss-tolerant UDP behavior, which matters on bad networks but is invisible on good ones. Against the empirically verified human conversational turn-gap of ~208ms (Stivers et al. 2009, PNAS, ten languages) and the industry's practical 500–800ms voice-to-voice target, a well-placed relay fits inside that budget: in our own overnight MVP — browser AudioWorklet capture at 24kHz PCM16, WebSocket to a relay process, upstream WebSocket through a forward proxy to gpt-realtime-2.1 — we measured roughly 800ms from request to first audio delta (an internal observation from one build, not an independently benchmarked figure), comfortably conversational.

What the quickstarts also under-document is that the relay topology transfers several responsibilities from the provider's WebRTC stack onto your code: barge-in truncation (the client must stop playback unilaterally and report heard-samples, not generated-samples), echo cancellation (browser AEC cannot see audio you schedule manually through AudioContext, so speaker-and-mic setups will feed the assistant's voice back into its own ears), and audio-format discipline (OpenAI rejects wrong sample rates outright rather than resampling). Meanwhile, the protocol layer is quietly converging: Qwen-Omni-Realtime and GLM-Realtime both mirror OpenAI's Realtime event taxonomy almost name-for-name, and xAI's Grok Voice explicitly claims OpenAI-compatibility — which makes a thin in-house provider abstraction (session lifecycle, audio I/O normalization, event translation, non-blocking tool dispatch) a realistic weekend-scale investment rather than a framework commitment.

The Fork: Ephemeral-Token WebRTC vs. Server Relay

A common provider-recommended browser topology (OpenAI's is the clearest example) works like this: your backend mints a short-lived token (OpenAI: POST /v1/realtime/client_secrets, expiring in about a minute if unused), hands it to the browser, and the browser negotiates WebRTC directly with the provider. Audio flows as Opus media tracks with forward error correction; JSON events ride a data channel. Your server never touches the media path — which is precisely the point, and precisely the problem.

What direct connection optimizes away is also what it gives up:

Reachability. The browser must be able to reach the provider's edge. If your users sit behind a network boundary that blocks the provider (an entire national market, a locked-down enterprise network), the recommended topology simply does not work. This is not an exotic edge case; it is the default condition for any China-market deployment of a US-hosted model API, and no amount of ephemeral-token engineering fixes it.
Custody. In the direct topology, raw audio and transcripts live with the provider; your server sees what webhooks and session-completion callbacks choose to tell it. If the application is the system of record — an interview tool, a compliance-logged support line, a daily-standup collector — reconstructing the conversation after the fact is strictly worse than owning it in-line.
Control. Moderation, budget enforcement, per-user session caps, and business-logic injection all want a chokepoint. OpenAI's own docs acknowledge the trade: the server-mediated alternative "puts your application server in the critical path for session initialization" — which is exactly where a lot of applications want it.

The relay inverts the trade: browser ↔ your server over WebSocket (or WebRTC-to-your-edge, in heavier setups), your server ↔ provider over WebSocket with the durable key. OpenAI documents this as the server-to-server pattern; Google goes further and officially recommends a relay for browser Gemini Live clients, since Gemini Live has, as far as we could confirm as of mid-2026, no first-party browser WebRTC endpoint (Gemini does issue ephemeral session tokens with a roughly 1-minute session-start window, but not tied to a confirmed first-party browser WebRTC path) — its partners (LiveKit, Daily/Pipecat) exist partly to supply that missing leg. Alibaba's Qwen-Omni-Realtime appears to be the only one of the four offering symmetric first-party WebRTC and WebSocket, per its own documentation, though we were not able to independently cross-check every provider's current feature matrix in this pass.

The Latency Ledger

The relay's reputation for latency cost deserves numbers rather than vibes:

The floor is biological. Human turn-gaps average +208ms across languages (modal offset 0–200ms; Japanese fastest at +7ms, Danish slowest at +469ms). Humans hit this by predicting the end of the incoming turn, not by generating faster — which is the empirical argument for semantic turn-detection over raw speed.
The provider dominates the budget. One widely-read practitioner writeup reports OpenAI's own API time-to-first-byte at about 500ms for US clients against an 800ms voice-to-voice target, leaving roughly 300ms for everything else; OpenAI's July 2026 gpt-realtime-2.1 release is reported to have cut p95 latency by at least 25% across its Realtime voice models on top of that.
The relay hop is the small line item. The same 10–100ms relay-hop figure shows up in practitioner writeups. Tuned production stacks are commonly reported in the 500–900ms end-to-end voice-to-voice range, with untuned default configurations reportedly landing around 1.2–1.7s median regardless of topology — figures we're citing as representative industry reporting rather than numbers we independently re-measured, and which suggest most latency problems are configuration problems, not transport problems.
Our data point. In our own build — a single-relay setup with an additional forward-proxy egress hop, about as unfavorable as a relay gets — we observed roughly 780–800ms from response request to first audio delta, with WebSocket session establishment around 1.3–2.2s (one-time, per conversation). This is an internal observation from one MVP, not an independently benchmarked figure; subjectively it read as conversational, with no user-visible lag complaints in our own end-to-end tests.

The honest caveat is loss behavior, not added latency. WebRTC's UDP transport discards late packets and lets Opus's error concealment paper over the gap ("a brief gap sounds far better than a stutter"); a TCP WebSocket under packet loss stalls the whole stream behind the missing segment. On stable networks this never surfaces. On hostile last-mile networks, it is the strongest technical argument for keeping WebRTC between the browser and your edge, even when the provider leg must remain a relayed WebSocket.

What the Relay Makes You Own

Three provider-stack behaviors quietly disappear in a raw-WebSocket relay, and each has a documented failure mode:

Barge-in truncation. When the user interrupts, the provider's WebRTC stack normally handles buffer truncation. In a relay, the client must stop local playback immediately and unilaterally — waiting for a server round-trip means the speaker keeps talking over the user. The subtle part: the app must then tell the server how much audio the user actually heard (conversation.item.truncate), tracked as played-samples, not received-samples. Getting this wrong corrupts the model's conversation state — it "remembers" saying things the user never heard. Semantic VAD helps upstream of this: OpenAI's semantic_vad judges whether the user finished a thought rather than merely paused, which matters in reporting-style conversations where people stop to think mid-sentence.

Echo cancellation. Browser AEC works by modeling the acoustic path between a known far-end signal and the mic — and it only knows about audio routed through <audio> elements or RTCPeerConnection. PCM deltas scheduled manually through AudioContext are invisible to it. The reported symptom is memorable: on speaker-and-mic setups, the assistant hears its own voice and "starts answering itself." Mitigations, in descending order of dignity: headsets; application-level half-duplex gating (mute or raise the mic gate while the agent speaks, plus a decay window after); a standalone software AEC; or accepting push-to-talk as the fallback — LiveKit's position is that robust barge-in without functioning client-side AEC is not reliably achievable.

Audio plumbing. The format contract is strict: OpenAI wants mono PCM16 at 24kHz and rejects wrong rates outright (a filed issue confirms 8kHz input simply fails); Gemini wants 16kHz but resamples server-side. Capture belongs in an AudioWorklet (128-sample Float32 frames, chunked to ~50–200ms before base64-encoding into JSON events); playback belongs in a self-scheduled queue keyed off AudioContext.currentTime with an accumulating next-start-time, or clicks appear at buffer seams. And the classic trap for anyone base64-encoding audio in JS: String.fromCharCode.apply(null, bigArray) overflows the call stack around 65k elements — chunk it.

Structured output. For conversations that must end in structured data (a summary, a form, an order), in-session function calling works and has improved: OpenAI's own reporting has ComplexFuncBench-audio rising from 49.7% to 66.5% across model generations. But a May 2026 paper introducing a text-to-voice evaluation framework reports that on the "Confetti" tool-calling benchmark, the voice-native gap below text-based extraction on the same content ranges from about 1.8 to 4.8 points depending on model, dominated by argument-value errors — misheard dates, names, numbers. The pragmatic pattern: use an end-of-session function call for the structured artifact (it lives inside the conversation's context, which is the whole advantage), keep the full server-side transcript as ground truth, and regenerate from transcript with a text model when the voice-native extraction looks wrong. In our own MVP (an internal observation, not an independently benchmarked result) the end-of-call function call produced correct four-field summaries in our end-to-end tests — but we still keep the transcript on disk precisely because the external benchmark above says voice-native extraction sometimes won't be correct.

Provider Abstraction: Convergence Makes It Cheap

The strongest recent development for relay builders is protocol convergence. Laid side by side, Qwen-Omni-Realtime's and GLM-Realtime's documented event schemas (session.update, input_audio_buffer.append, input_audio_buffer.speech_started, response.create, response.audio.delta, response.done, …) are near-1:1 matches for OpenAI's Realtime taxonomy — unmistakably modeled on it, the same way the industry converged on OpenAI-compatible chat-completions endpoints. Only xAI states the compatibility explicitly — "The Grok Speech to Speech API is compatible with the OpenAI Realtime API", with a documented event-naming delta; Alibaba and Zhipu present theirs as self-contained products, so the convergence is structural evidence, not a vendor promise — code against it with a translation layer, not a shared client.

This changes the framework calculus. Pipecat (frames/processors, five realtime providers, transport-owned VAD) and LiveKit Agents (a dedicated RealtimeModel base class parallel to its STT/TTS/LLM abstractions) are excellent, and the right answer for multi-party rooms or telephony. But for a single-user voice session behind a relay, the minimal in-house abstraction — a session-lifecycle interface, audio I/O normalization, event-name translation, and background tool dispatch — is a few hundred lines against a converged protocol family. One independent practitioner's advice matches our experience: build directly against one provider's raw API first, and "don't reach for the framework until you have a second provider you actually want to support." The relay topology makes the eventual swap a server-side change the client never sees — which is exactly the property you want when the model landscape shifts quarterly (this year alone, per vendor/press announcements we did not each independently re-verify: gpt-realtime-2.1 and its reasoning-capable mini in July, and reported spring releases of Gemini's and Qwen's own realtime models).

Security Posture of the Relay

The relay concentrates risk and control in the same place. The durable API key never leaves the server — which matters more every year: as of a 2024 GitGuardian report, OpenAI API keys were the fastest-growing leaked-secret category the firm tracked on GitHub (a reported 1,212x surge), with many found hardcoded in client-side JS; more recent GitGuardian reporting shows AI-service secrets broadly (not OpenAI keys specifically) continuing to grow sharply through 2026. Exact resale/monetization pricing for a stolen key is not something we could independently verify — treat any specific dollar figure here as illustrative rather than audited, but the underlying risk (an attacker running unmetered inference on your account) is well documented. In exchange, the relay must implement what the provider's token infrastructure would otherwise cover:

Per-user session tokens minted by the application (opaque, revocable, scoped to one member/one purpose), never the provider key in any form.
Concurrency as the rate-limit primitive. A voice session holds a capacity slot for minutes; count active sessions, not requests. Cap concurrent sessions and hard-cap session duration server-side (some deployment guides suggest capping around 15 minutes; conversation history is re-billed each turn, so runaway sessions compound cost).
Transcript custody as a feature. Everything can stay on infrastructure you control — relevant given that provider-side residency guarantees for realtime endpoints reportedly lag their text counterparts (OpenAI's EU data-residency documentation, as of our last check, notes tracing is not yet residency-compliant for /v1/realtime — worth re-confirming directly before relying on it, since compliance documentation changes without much notice).

What We'd Tell a Team Starting Tomorrow

If your users can reach the provider and you don't need the audio path, do what the docs say: ephemeral tokens, browser-direct WebRTC. It is less code and lower latency.
The moment reachability, custody, moderation, or swappability enters the requirements, stop fighting for the direct path. The relay reportedly costs on the order of 10–100ms and a day of careful audio plumbing; it buys you the control plane.
Budget the conversation, not the hop: semantic VAD for turn detection, preamble phrases or async tool calls to kill dead air, 500–800ms voice-to-voice as the target, and instrument first-audio-delta latency from day one.
Treat barge-in truncation and echo cancellation as first-class features, not polish. They are the two places a relay demo embarrasses you in a real room.
Write the structured summary inside the session (function call), keep the transcript outside it (your database), and let the second regenerate the first when needed.
Abstract the provider at the relay, thinly. The event-schema convergence means the second provider costs a translation table, not a rewrite — and pricing/quality across OpenAI, Google, Alibaba, and Zhipu differ enough (premium reasoning-first pricing vs. markedly cheaper domestic-market tokens, English-optimized vs. native-Chinese voices) that you will want the option.

Notes on Sourcing and Fast-Moving Claims

Where a specific figure or claim above links directly to a page, we fetched and confirmed that page supports the claim as stated. Where a figure is instead attributed generally ("reportedly," "on the order of," "one practitioner writeup") without a link, we were not able to independently re-verify it against a primary source in this pass, and it should be treated as illustrative of the trend rather than an audited number. Provider names, model versions, release dates, and pricing/latency figures throughout this piece trace to vendor announcements, engineering blogs, press coverage, and aggregated industry reporting current to mid-2026; this is a genuinely fast-moving space, and any given figure may already be stale by the time you read it — re-check anything load-bearing for a real decision directly against the provider's current documentation. Figures explicitly marked as internal (our MVP's latency, function-calling results) are our own single-build observations, not benchmarked or peer-reviewed results.

Sources: OpenAI Realtime API documentation (WebRTC/WebSocket/SIP guides, VAD, conversations, costs, EU data residency); Google Gemini Live API docs (live API, ephemeral tokens, capabilities, logs policy); Alibaba Cloud Model Studio Qwen-Omni-Realtime docs; Zhipu GLM-Realtime API reference (docs.bigmodel.cn); xAI Grok Voice Agent API docs; Stivers et al. 2009, PNAS (turn-taking universals); webrtcHacks (Realtime API latency measurements, unofficial WebRTC guide); Daily.co engineering blog (transport trade-offs, relay hop costs); LiveKit blog and docs (turn detection taxonomy, realtime plugins, latency tuning); Pipecat documentation; Vercel AI Gateway realtime announcements; latent.space Realtime API analysis; workadventu.re (interruption truncation pattern); GoNoGo and OpenAI community forum (echo cancellation reports); "Confetti" voice tool-calling benchmark (arXiv 2605.15104); Deepgram conversation-intelligence guidance; GitGuardian/vicarius API-key-leak reports; MarkTechPost and TechCrunch coverage of 2026 realtime model releases; hands-on measurements from an internal voice-standup MVP built on gpt-realtime-2.1 (July 2026).