Getting a voice agent under 800ms, one hop at a time

The first voice agent we shipped had a median response time of about 1.9 seconds. On paper that seemed acceptable. On a real phone call it was unusable — callers assumed the line had dropped, said "hello?", and the agent, now hearing new speech, restarted its turn. Roughly one call in three collapsed into people talking over each other.

The fix wasn't one thing. It was measuring six hops separately and finding time in three of them.

What "latency" means here

The number that matters is not time-to-first-token or model TTFT or any single vendor metric. It's the interval between the caller finishing their sentence and the first audio of the reply reaching their ear. Everything in between is your budget, including the parts you don't control.

Rough thresholds, from conversational analysis research and from watching a lot of call recordings:

So the target is 800ms at p90, not at median. Median is a vanity metric here — the caller remembers the worst turn of the call, not the average one.

The six hops

Instrument each of these separately. If you only measure end to end, you will optimise the wrong one, and we did exactly that for a week.

Three of those rows account for nearly all the improvement. Take them in order of payoff.

1. Endpointing: the biggest slice, and it isn't compute

Endpointing is deciding that the caller has finished, rather than pausing. Almost every stack ships with a conservative silence threshold — commonly 700ms — because triggering early is worse than triggering late. Cut someone off mid-sentence and the conversation is ruined more thoroughly than by any delay.

But 700ms of silence is 700ms of your budget spent before a single byte of work has been done. It was, in our case, the single largest contributor and we hadn't even been counting it as latency.

The fix is not simply lowering the number. It's making the threshold depend on what was just said. A turn ending in a complete sentence with falling intonation can be closed quickly. A turn ending in "my postcode is..." plainly cannot.

# Adaptive endpointing. The silence threshold depends on whether the
# partial transcript looks finished, not on a fixed constant.

FAST_MS, DEFAULT_MS, PATIENT_MS = 220, 420, 900

TRAILING = re.compile(
    r"\b(um|uh|and|but|so|my|the|is|it'?s|number|address|postcode)\s*$",
    re.IGNORECASE,
)

def silence_threshold_ms(partial: str, state: TurnState) -> int:
    text = partial.strip()

    # Mid-thought: they are clearly about to continue. Wait.
    if TRAILING.search(text):
        return PATIENT_MS

    # We asked for a digit string; they may pause between groups.
    if state.expecting in ("phone", "postcode", "account_number"):
        return PATIENT_MS

    # Short, complete, and answers what we asked — close it fast.
    if state.expecting == "confirmation" and len(text.split()) <= 4:
        return FAST_MS

    return DEFAULT_MS

This took p90 endpointing from 700ms to 340ms while reducing interruptions, because the patient branch protects exactly the cases where the old fixed threshold was cutting people off.

Everyone tunes the model. Almost nobody tunes the silence detector, and it's frequently the largest number in the budget.

2. Speculative tool calls: start before you're asked

In a dispatch or lead-qualification agent, the expensive tool call is usually a lookup against a scheduling system or CRM — a few hundred milliseconds, in the middle of the turn, while the caller waits.

But by the time the caller is halfway through describing their problem, we usually already know their postcode and roughly what kind of job it is. That's enough to start the availability lookup while they're still talking.

async def on_partial_transcript(partial: str, session: CallSession) -> None:
    """Fires on every interim ASR result, mid-utterance."""
    hint = extract_slots(partial)          # cheap regex + gazetteer, no model

    if hint.postcode and not session.availability_task:
        # Fire and forget. If the turn ends up going somewhere else,
        # we throw the result away — a wasted read is cheap, and the
        # caller's silence is not.
        session.availability_task = asyncio.create_task(
            scheduling.find_slots(postcode=hint.postcode, job_type=hint.job_type)
        )

async def on_tool_call(name: str, args: dict, session: CallSession):
    if name == "find_slots" and session.availability_task:
        if args_match(session.prefetch_args, args):
            return await session.availability_task   # usually already done
    return await dispatch_tool(name, args)

Two rules make this safe. The prefetch must be a read with no side effects — never speculatively book anything. And the cached result is only used if the arguments the model eventually asks for match what was prefetched; otherwise it's discarded and the real call runs.

On calls where the prefetch hits, tool latency effectively disappears. It hits about 70% of the time.

3. Stream into TTS sentence by sentence

The naive pipeline waits for the complete model response, then sends it to text-to-speech, then plays the audio. That serialises two slow things.

Instead, buffer the token stream to the first sentence boundary, send that to TTS immediately, and keep going. The caller hears the first sentence while the second is still being generated.

SENTENCE_END = re.compile(r"(?<=[.!?])\s+")

async def stream_reply(token_stream, tts, audio_out):
    buffer = ""
    first_chunk = True

    async for token in token_stream:
        buffer += token

        # Flush the first sentence as soon as it's complete — that
        # is the number the caller actually experiences.
        parts = SENTENCE_END.split(buffer)
        while len(parts) > 1:
            sentence, buffer = parts[0], SENTENCE_END.split(buffer, 1)[1]
            await audio_out.send(await tts.synthesize(sentence, flush=first_chunk))
            first_chunk = False
            parts = SENTENCE_END.split(buffer)

    if buffer.strip():
        await audio_out.send(await tts.synthesize(buffer))

The whole "LLM full response" row disappears from the budget. On replies of two sentences or more this saved around 400ms.

One caveat worth knowing: some TTS voices sound subtly different when synthesising short fragments, because prosody is computed per request. If the seams are audible, most providers accept a continuation or context parameter to keep prosody consistent across chunks — check yours before shipping.

What we tried that didn't help

A smaller model. Dropping to a faster model saved about 120ms of first-token time and cost noticeably more in tool-calling reliability. Worth it in some domains; it wasn't in this one, where a wrong tool call costs a truck roll.

Colocating everything in one region. Real, but small — around 40ms. Do it if it's free; don't restructure your deployment for it.

Shortening the system prompt. Almost nothing with prompt caching enabled. Without caching it's worth checking, but cache first.

Measure it in production, per turn

Log every hop as a span on every turn, tagged with the call ID, and store it. Aggregate latency tells you nothing about the specific call where a customer hung up.

The metric we actually watch is not any of the individual hops — it's the share of turns over 1.2 seconds. That's the number that correlates with calls going wrong, and it's the one worth putting on a dashboard.

• Budget end-of-speech to first-audio, at p90, not median
• Measure endpointing as latency — it's usually your biggest slice
• Make the silence threshold adaptive, not constant
• Prefetch read-only tool calls from partial transcripts; never prefetch writes
• Stream into TTS at sentence boundaries, not on completion
• Alert on the percentage of turns over 1.2s