Aditya Dutt Pandey

Build · 2026

Published · 3 min read

Two speculative layers in a voice RAG system

The fast path answers from BM25-retrieved passages without a hosted LLM. An async recovery path checks uncertain answers, drafts alternatives in parallel and synthesizes across passages when needed.

I built the voice RAG system around two different kinds of speculation. One is about getting an answer out quickly. The other is about what to do when that answer might not be enough.

The latency target is under 200 ms from the transcribed question to an extractive answer. That is a target, not a guarantee. In production, the extractive answer step has measured around 25 ms, or tens of milliseconds, after speech-to-text when Render and Qdrant are in the same region. Speech-to-text is outside that measurement. The service runs on a 512 MB free-tier host, which is a separate memory constraint, not a latency promise.

First layer: answer from the passage

The hot path searches Qdrant with BM25, a sparse keyword-retrieval method. It is not dense vector search. BM25 is useful for the names, numbers and phrases that matter in questions about real documents.

The system extracts an answer from a retrieved passage instead of asking a hosted language model to generate one. The hot path makes zero hosted LLM calls. Current measured retrieval P95 is 13.67 ms, from the latest measured report. That number is retrieval latency, not end-to-end voice latency.

This is the first speculation: return the best extractive answer quickly, before spending time on a more involved check. The fast answer is grounded in a passage, but a single passage may not answer the whole question.

Second layer: check and recover asynchronously

An asynchronous recovery chain can validate the fast answer and replace it. Its work is outside the latency target and SLA. The groundedness check still incurs a linear LLM cost: each answer that goes through the check means another model call, even if the check decides to keep the original.

When the answer needs more than one passage, the recovery path uses a paper-style clustering approach to organize the relevant material. A small model drafts several candidate answers in parallel. A larger model verifies those drafts against the source material. The system ranks candidates using draft confidence multiplied by verifier support, then synthesizes a final answer from multiple passages.

I don't have prompt log-probabilities for the drafts, so this is not token-level self-consistency. Draft confidence is a ranking signal, not a calibrated probability. The verifier provides a separate support signal.

Because recovery can take longer, the fast answer may already be displayed when it finishes. Recovery can change that displayed answer if the passages support a better multi-passage response. It is an eventual correction, not a way to make every first answer correct.

Two paths, two costs

The immediate path is sparse retrieval plus extraction, with no hosted LLM and a measured answer step in the tens of milliseconds under the same-region setup. The recovery path spends LLM calls to check or build an answer across passages. It is intentionally not part of the under-200-ms SLA.

I described the original latency and memory constraints in Answering out loud in four Indian languages. The two paths explain what that fast answer does, and what happens when it is not enough.

By , CTO of tunyt

← All writing