Zylos LogoZylos
2026-07-18

Mid-Conversation Tool Calling in Realtime Voice Agents: Latency Budgets, Barge-In, and Retrieval Under Audio Constraints

voice-agentsrealtime-apifunction-callingtool-usespeech-to-speechretrieval-augmented-generationlatencybarge-in

Executive Summary

Text-based agent tool calling has a forgiving failure mode: if a tool takes three seconds, the user sees a spinner. Realtime speech-to-speech agents don't get that mercy — the channel is audio, and three seconds of dead air reads as a hang, a dropped call, or a rude interruption. This is the central design constraint of mid-conversation tool calling in voice agents: the mechanics are almost identical to text-based function calling (declare a tool schema, receive a structured call, execute it, return a string, let the model continue), but every millisecond of that round trip is now perceptible, every returned byte is now something a text-to-speech voice has to read aloud, and the user can barge in and change the subject at any point in the sequence — including in the middle of it.

This article works through the mechanics on OpenAI's Realtime API (the model most production voice agents in the zylos ecosystem are built on, including the voice daily-standup component), the turn-detection and barge-in behavior that has to coexist with tool calls, the latency budget that governs whether a tool call is even viable inline, and the two competing patterns for getting external knowledge into a live conversation — pre-session context injection versus model-invoked retrieval tools. Throughout, we ground the abstract mechanics in a real, running implementation: the open-source zylos-standup voice agent, whose relay server implements exactly the function-call dispatch loop, semantic-VAD configuration, and compact-result discipline this article describes — not as a hypothetical, but as production code with real session logs behind it.

The headline findings: (1) OpenAI's Realtime API keeps the function-calling event contract essentially unchanged from its 2024 Chat Completions ancestor — tools in session.update, a function_call output item, function_call_output + response.create to resume — but wraps it in an audio-native turn-detection layer (semantic_vad) that has to be tuned around tool latency, not just speech pauses; (2) the documented latency budget for a voice agent to still feel responsive is roughly 300–800ms end-to-end, and OpenAI's own prompting guidance is explicit that a synchronous tool call inside that budget needs a spoken "preamble" ("I'll check that now") rather than silence or narrated reasoning; (3) tool results in a voice agent are not context for the model to reference silently — they are usually about to be spoken back, which means retrieval tools should return compact, pre-summarized structures, not raw search hits; and (4) competing platforms are starting to diverge architecturally on the hardest part of this problem — OpenAI and Google's Gemini Live API both block the turn on tool execution, while Amazon's Nova Sonic added native asynchronous tool calling specifically to let the agent keep talking while a slow retrieval call resolves in the background.

The Realtime API's Tool-Calling Loop

OpenAI's Realtime API is a persistent WebSocket (or WebRTC) session, not a request/response call. Tools are declared once, up front, in the session.update client event, as a flat array — each entry has type: "function", name, description, and a JSON-schema parameters object directly on the tool object, with no nested function wrapper the way the Chat Completions and Responses APIs use. tool_choice supports none / auto / required, same semantics as text-based function calling.

When the model, mid-conversation, decides a tool is warranted, the complete, authoritative call arrives as an output item on the response.done server event: { type: "function_call", name, call_id, arguments }, where arguments is a JSON-encoded string. There is also an incremental stream — response.function_call_arguments.delta followed by a terminal response.function_call_arguments.done — but because you generally need the full argument JSON before you can safely execute anything, response.done remains the reliable point to act on in most implementations; the deltas are mainly useful for showing "thinking" UI in non-voice clients, not for triggering execution early.

The client executes the function locally and reports back with two events: a conversation.item.create carrying { type: "function_call_output", call_id, output } (output is a string — typically JSON-stringified), immediately followed by a response.create to tell the model to resume generating — now with the tool result available as conversation context. This second event is easy to forget and is the single most common bug in Realtime tool integrations: without it, the model has the result sitting in state but no active response to speak it in, and the conversation appears to freeze.

A newer, architecturally distinct mechanism layered onto this in 2025 is MCP tool support: instead of the client executing the function, the Realtime API itself calls out to a remote MCP server or a built-in connector (Google Calendar, for example). The event sequence is different — mcp_list_tools.in_progress / .completed to discover the server's tool surface, response.mcp_call_arguments.delta/.done as the call is formed, an optional mcp_approval_request/mcp_approval_response gate for sensitive actions, and the result landing in response.output_item.done as an mcp_call item that becomes conversation context automatically, with no client-side round trip at all. The practical guidance from OpenAI here is to keep the tool surface narrow with allowed_tools, and to note that a remote MCP server does not automatically see the full conversation transcript — only whatever the model chose to put in the call arguments. This distinction matters for retrieval design, covered below: an MCP-backed knowledge search server is blind to context you haven't explicitly threaded through the call.

For a client-executed local tool (the pattern most production voice agents use for anything touching a private database), the loop is fully synchronous from the model's perspective: it cannot say another word until your function_call_output + response.create land. That synchronicity is the whole reason latency budgeting (below) matters so much more here than in a text chat.

A Production Example: The Voice Standup Agent's Tool Loop

the open-source zylos-standup component (src/lib/relay.js in the zylos-ai/zylos-standup repo) is a working instance of exactly this loop, and it's worth reading as a reference implementation rather than a toy example, because it encodes several hard-won constraints directly in code.

The agent, Luna, runs a voice daily-standup conversation and is given three tools declared in session.update:

  • submit_standup_summary — a terminal, structured-extraction tool called once the four standup topics (yesterday / today / blockers / meeting topics) have been covered. This is the "close the loop" tool pattern common to voice form-filling agents.
  • recall_member_history — an on-demand retrieval tool: "look at this person's last N daily reports." The model decides for itself when it needs this, per its instructions: "when you want to confirm what they said last time, or want to follow up on previous progress or blockers."
  • search_team_knowledge — a second retrieval tool over a keyword-indexed team knowledge base, invoked when the member mentions a project or term the model needs grounding on.

All three funnel through a single dispatcher, triggered off response.done:

case 'response.done': {
  const calls = (ev.response?.output || []).filter(i => i.type === 'function_call');
  for (const fc of calls) this.handleToolCall(fc, { client, upstream, member, reportDate, model, markSaved: () => { saved = true; } });
  break;
}

And the reply path is the canonical two-event sequence, wrapped in a small helper:

const respond = output => {
  safeSend(upstream, {
    type: 'conversation.item.create',
    item: { type: 'function_call_output', call_id: fc.call_id, output: typeof output === 'string' ? output : JSON.stringify(output) },
  });
  safeSend(upstream, { type: 'response.create' });
};

A few design decisions in this file are directly instructive for anyone building a similar agent:

A locked-down client event allowlist. The browser client can only send a fixed set of events upstream — input_audio_buffer.append/commit/clear, response.create, response.cancel, conversation.item.create, conversation.item.truncate. This means the browser cannot inject arbitrary session.update calls, forge tool results, or otherwise manipulate the session — tool execution and session configuration stay entirely server-side. For a voice agent whose tools touch a real database, this is a load-bearing security boundary, not an implementation detail.

Retrieval results are pre-shrunk before they ever reach the model. AgentContext.searchKnowledge truncates each knowledge-base hit to 600 characters before returning it as the tool output, with an explicit comment: "results are trimmed — this is the audio path." Likewise recallHistory returns only the structured fields (yesterday/today/blockers/topics arrays) rather than full transcripts. This is the compact-tool-result discipline discussed later in this article, applied concretely: nobody wants to sit through Luna reading back a 3,000-character knowledge base article.

Retrieval tools are instructed, not forced. The system instructions built by AgentContext.buildInstructions() explicitly tell the model when to reach for the two pull-based tools ("only call them when you actually need to — don't interrupt the flow of conversation") — an acknowledgment that a tool call is not free even when it's fast; it's a break in conversational rhythm that has to be earned.

Barge-in has a special termination path. When a user ends the call early (app.end client event), the relay doesn't just close the socket — it sends response.cancel followed immediately by a fresh response.create carrying override instructions: "the user wants to end — if you haven't submitted a summary yet, call submit_standup_summary right now, then say a brief goodbye." This is a clean example of steering a tool call to happen deterministically at a conversation boundary, rather than hoping the model gets there organically.

This is a small system — one relay file, three tools — but it exercises essentially the entire mechanic this article is about: declaration, dispatch, compact retrieval, and cancellation-aware wrap-up.

Turn-Taking and Barge-In: Semantic VAD Meets Tool Calls

Voice conversation requires the system to decide, continuously, whether the user has finished speaking. OpenAI's Realtime API exposes this as session.audio.input.turn_detection, with two modes. server_vad is volume/energy-based: it waits for silence_duration_ms of quiet (commonly 500ms–1000ms) after speech before concluding the turn is over. semantic_vad is a semantic classifier that scores, from the words themselves, the probability the user has actually finished their thought — it can wait through a mid-sentence pause and cut in faster when confident the utterance is complete. Its tuning knob is eagerness: low | medium | high | auto (auto = medium). low is documented as "let the user take their time to speak" — the standup agent above sets exactly this, { type: 'semantic_vad', eagerness: 'low' }, with a code comment explaining why: standup answers are often long, multi-clause narrations of yesterday's work, and an eager VAD would chop them into fragments and trigger premature responses.

Barge-in — the user interrupting the agent mid-response — is handled independently of tool state. The documented sequence is: VAD detects new user speech while the model is talking → client sends response.cancel to stop server-side generation → client sends conversation.item.truncate with the item id, content index, and audio_end_ms (how much of the assistant's audio the user actually heard) so the server's conversation history reflects only what was actually played, not what was generated-but-cut-off. A newer output_audio_buffer.clear event exists to flush any already-buffered-but-unplayed audio client-side, addressing edge cases where truncation alone left a fragment of stale audio queued.

What's notably not documented, as far as current public OpenAI materials go, is special-case behavior for turn detection or truncation while a function_call is in flight or while the model is mid-way through speaking a tool result. Turn detection appears to operate on the input audio stream independently of the model's internal generation state — from the API's point of view there's no distinction between "the model is reciting a fact it already knew" and "the model is reading a tool result it just got back." This has a real practical consequence worth designing around explicitly: if a user barges in exactly as a function_call_output you just sent is about to be consumed by the model's next response.create, you can end up racing a cancellation against a tool result. Because function_call_output is keyed to a call_id rather than a response_id, submitting it after a cancellation is generally still safe — it just updates conversation state — but firing an automatic follow-up response.create in that situation risks generating a response the user is no longer waiting for, or worse, a response that talks over the user's next utterance. The safe pattern (used implicitly by the standup relay's app.end handler) is to treat response.cancel as authoritative and gate any pending response.create behind a check for "is this still the active turn," rather than assuming the tool round trip you kicked off a moment ago is still wanted.

The relay code also has to defensively swallow a specific benign error: calling response.cancel when no response is actually active returns response_cancel_not_active, which the standup implementation explicitly filters out rather than surfacing as a user-facing error — a small but telling sign of how much of building these systems is handling the API's own race conditions gracefully rather than assuming a tidy state machine.

The Latency Budget: Why Silence Is the Enemy

The whole reason tool-calling design in voice agents is a distinct discipline from text agents is that the acceptable latency envelope for a natural-feeling voice turn is tight, and a synchronous tool call eats directly into it. Sourced figures vary by methodology and vendor, but they converge on a similar shape: human conversational turn-taking has an average gap of roughly 200ms; users start noticing a pause around 300–500ms; by 800ms it reads as sluggish; past ~1.5 seconds people start talking again or assume the line dropped. Deepgram's 2026 production guidance targets p50 latency under 250ms on a fully optimized stack and under 800ms at p95 even on a standard cloud stack (ASR + LLM + TTS as separate hops). Vapi's engineering breakdown assigns roughly 40–300ms to streaming ASR's first tokens, 100–400ms to LLM processing, and 50–250ms to warmed neural TTS, targeting sub-500ms as the product bar. The independent "OpenAI Realtime API: The Missing Manual" write-up targets ~800ms voice-to-voice as "a good target" for Realtime-API-based agents specifically, citing OpenAI's own reported ~500ms time-to-first-byte and budgeting the remainder for audio processing and endpointing.

None of those budgets have slack for a database query, a vector search, or an external API call layered in the middle. A tool round trip inside a live turn is: the model decides to call → arguments stream/complete → your server receives and executes → you build and send function_call_output → response.create → the model resumes generating → audio synthesis catches up to the new text. Even a fast, well-indexed lookup (tens of milliseconds of actual query time) adds a full extra network hop each direction plus model re-processing, and anything touching an external API, an LLM-based re-ranker, or a cold cache can easily blow past the entire 800ms budget on its own.

This is why LiveKit's guidance explicitly flags tool/function execution as a "potentially high" latency contributor, calls out that tools execute sequentially (so a turn invoking three tools compounds three round trips, not one), and recommends capping max_tool_steps and consolidating what would be multiple external calls into fewer, richer ones — advice that maps directly onto the standup agent's design of two purpose-built retrieval tools rather than a generic multi-step search API.

Filling the Silence: Preambles, Not Filler Words

Given that some tool calls simply cannot complete inside a "feels instant" window, the practical question becomes: what does the agent do with the dead air? OpenAI's own Realtime Prompting Guide is explicit and prescriptive on this point: use a spoken preamble before any tool call likely to take noticeable time — checking a record, looking up availability, verifying account state — so the user gets an acknowledgment instead of silence. The style guidance is specific: keep it to about one natural sentence, vary the phrasing turn to turn so it doesn't sound canned, describe the action, not the internal reasoning, and explicitly avoid meta-commentary like "Let me think...", "Hmm, one moment while I process that...", or "I am now going to call the lookup tool" — the user should never hear the word "function" or "tool." The preferred pattern reads naturally: "I'll check that order now," "Let me look up your appointment details," "I'll verify that before we make any changes." The standup agent applies this same principle at the instruction level, explicitly forbidding the model from narrating tool mechanics ("don't recite the tool or technical details") and telling it to weave retrieved facts into the follow-up question naturally instead.

A second, complementary technique described across the LiveKit/Pipecat ecosystem is intercepting the tool-call signal the instant it's emitted and immediately playing a short, pre-synthesized filler clip ("Let me pull that up...") from the audio pipeline while the backend resolves the call in parallel — masking the round trip with audio that costs zero additional model-generation latency, rather than asking the model to generate a spoken filler on the fly. This is a lower-level, pipeline-side variant of the same idea OpenAI's guide addresses at the prompting level.

On the data-shape side, OpenAI's cookbook on data-intensive Realtime apps adds a complementary set of recommendations that reduce the processing half of the latency, not just mask the waiting half: keep individual functions narrowly scoped so payloads stay small, filter and flatten data server-side before returning it (compact JSON or YAML beats a table dump), and pair large results with a short "hint" instructing the model how to summarize rather than reprocess the raw payload at length. Every one of these is really the same instinct applied at a different layer: minimize how much the model has to read before it can start talking again.

Retrieval Augmentation in a Live Voice Turn: Pre-Injection vs. Pull

There are two structurally different ways to get a user's history or a knowledge base into a live conversation, and they trade off against the same latency budget in opposite directions.

Pre-session context injection loads relevant history as text before or at the start of the session — via the instructions field in session.update, or by seeding conversation.item.create messages that replay prior turns before the live conversation begins. This avoids the tool round trip entirely: whatever's injected is just there, at zero incremental latency once the session starts. The cost is on the front end (session setup, not turn latency) and it's static — it reflects what you decided to load before the conversation started, not what turns out to be relevant three exchanges in. It's also bounded: Realtime sessions have real ceilings (independently reported around a 128k-token context and roughly a 15-minute session duration), so pre-loading someone's entire history isn't viable at scale, and irrelevant loaded context still costs processing time on every subsequent turn even if it's never referenced.

Mid-conversation tool-invoked retrieval is the pattern the standup agent uses for both of its pull tools: the model itself decides, based on what the person just said, that it needs recall_member_history or search_team_knowledge, and issues a normal function_call. This stays current and lean — nothing is fetched that isn't needed — but it incurs the full latency chain from the previous section on the turn where it happens, and it depends on the model correctly judging that a lookup is warranted, which is a soft, instruction-driven decision rather than a guarantee (hence the standup instructions' explicit "only when you actually need it, don't interrupt the flow" framing — over-eager retrieval is its own latency and awkwardness problem).

In practice, production voice agents don't pick one pattern exclusively — they layer them. The standup agent injects static, always-relevant context up front (team background, the specific member's role, a "probing guidance" block describing what to follow up on) via AgentContext.buildInstructions(), and reserves the tool-call path specifically for information that's expensive to pre-load in full (a member's multi-week report history) or that's genuinely conditional on what the conversation surfaces (a knowledge-base term the member happens to mention). That's a reasonable general heuristic: push what's always needed, pull what's conditionally needed — and design the pull tools to return the smallest structure that answers the specific question, not a general-purpose search API result set, because unlike a text agent's tool output, this result is either about to be read verbatim by a TTS voice or is expected to be woven into natural spoken language by the model — raw JSON dumps, long lists, or unbounded search hits are a liability either way.

It's worth noting OpenAI's MCP tool path complicates the "pull" side further: a remote MCP-backed knowledge server does not automatically see the conversation transcript, only whatever gets passed explicitly as call arguments — so a retrieval tool built as an MCP server needs the client-executed function layer (or careful argument construction) to supply enough conversational context for the search to be relevant, which is one more reason many voice-agent teams keep retrieval as a locally-executed function tool rather than delegating it to an MCP server, at least for anything that needs conversational grounding.

Amazon's Nova Sonic offers a genuinely different architecture worth flagging here rather than treating as a footnote: Nova 2 Sonic supports native asynchronous tool calling — the model can keep responding to new user input while a slow tool, including agentic RAG against Amazon Bedrock Knowledge Bases, resolves in the background, then surface the result once ready. That sidesteps the entire silence problem structurally instead of papering over it with preambles and filler audio, which is a meaningfully different bet than the synchronous, block-the-turn model OpenAI's Realtime API and Google's Gemini Live API both currently use.

Keeping Tool Results Compact: A Design Rule, Not an Optimization

It's worth stating plainly, because it's easy to under-weight coming from text-agent experience: in a speech-to-speech agent, a tool result isn't primarily context for the model to reason over silently — a meaningful fraction of it is often about to be read aloud, in whole or in paraphrase, by a synthesized voice, at speaking pace, to a human who cannot skim it. A 2,000-token JSON blob that a text agent would happily fold into its context window and mostly ignore becomes, in a voice agent, either an unacceptably long monologue or (more likely) a model that has to do real-time summarization work on the fly before it can speak, adding processing latency on top of the round-trip latency already discussed. The standup agent's knowledge-search tool hard-truncates each result to 600 characters before it ever reaches the model, and its history-recall tool returns only pre-structured fields rather than transcripts, for exactly this reason. The general design rule: retrieval tools for voice agents should do summarization and truncation server-side, in code, before the result reaches the model — don't rely on the model to compress a large payload gracefully under a tight latency budget, because that compression itself costs the time you're trying to save.

Practical Guidance for Building One

Pulling the above into a short checklist for anyone building a tool-calling voice agent on the Realtime API:

  • Declare narrowly. Fewer, purpose-built tools (as the standup agent does with exactly two retrieval tools) beat a general-purpose search API — each tool call is a full round trip inside a sub-second budget, and LiveKit's sequential-execution behavior means multiple tool calls in one turn compound.
  • Instruct the model on when, not just how, to call a tool. "Only call this when you actually need it" is as important a piece of the tool's prompt as its parameter schema — an unnecessary tool call is pure latency cost with no benefit.
  • Preamble slow calls. Follow OpenAI's own prompting guidance: one natural, varied sentence describing the action ("I'll check that now"), never meta-commentary about tools or reasoning, issued before the call, not after.
  • Truncate and pre-structure retrieval results in code, not in the prompt — assume anything you return may be spoken back close to verbatim.
  • Treat response.cancel as authoritative and gate any pending response.create (including one queued behind a tool result) on the turn still being live — don't assume a tool round trip you kicked off is still wanted by the time it resolves.
  • Tune turn detection to your content, not a platform default. Long, narrative answers (a standup report, a customer complaint) want semantic_vad at low eagerness; quick transactional exchanges can tolerate high or server_vad.
  • Layer push and pull. Inject what's always relevant at session start; reserve tool calls for what's genuinely conditional on the conversation, both to save latency and to keep the always-injected context from bloating with material that's rarely used.
  • Lock down which events the client audio surface can send upstream. The standup relay's allowlist pattern — audio + response-control events only, no raw session.update or forged tool results from the browser — keeps tool execution and session configuration authoritative on the server.

What's Still Open

Two things worth flagging honestly rather than glossing over: first, OpenAI's public documentation does not specify how turn detection and truncation are meant to interact with an in-flight function call or a response mid-way through reciting a tool result — behavior here appears to fall out of the general audio-stream-level VAD and truncation mechanics rather than being tool-aware by design, and teams building on this should test their specific barge-in-during-tool-call scenarios rather than assume documented behavior exists. Second, the industry is visibly not settled on synchronous versus asynchronous tool execution as the right default: OpenAI and Google's Gemini Live API both currently block the conversational turn on tool completion (Gemini's Live API notably requires the client to execute and return every tool call manually, with no automatic handling, and non-blocking function calls not yet generally supported as of the current docs), while Amazon's Nova Sonic bet on native async tool calling specifically to let the agent keep talking through a slow retrieval. Whether the synchronous model (with better preambles and faster retrieval) or the asynchronous model (with the added complexity of resuming a conversation the user has since moved past) wins out in practice is an open architectural question, not a settled one.

References