Build · 2026
Published · 3 min read
Answering out loud in four Indian languages, in under 200 ms, on 512 MB
A voice assistant that answers questions in four Indic languages, grounded in real documents, inside a 200 ms budget, on a free server with 512 MB of memory. Why the fast path never generates, why retrieval uses two kinds of search, and what "speculative" means here.
Here's the brief I gave myself. Someone asks a question out loud in one of four Indian languages. They get an answer grounded in actual documents, not made up. The whole thing takes under 200 milliseconds. And it runs on a free-tier server with 512 MB of memory.
Every one of those constraints kills an obvious design. That's what made it fun.
Why 200 ms
In conversation, a pause is information. Ask someone a question and they take two seconds, and you assume they're unsure. A voice assistant that pauses feels broken before it has said anything. So the budget wasn't a vanity number. It was the difference between something that feels like talking and something that feels like waiting.
Decision 1: never generate on the fast path
The default way to build this is to retrieve some documents and ask a large language model to write an answer. It reads beautifully. It's also slow, and every generated word is a chance to make something up.
So the fast path doesn't generate at all. It's extractive: it finds the passage that answers the question and returns the answer straight from it. Pulling a span out of a document is fast, and it can't invent anything, because every word was already there.
Decision 2: two kinds of search, not one
Retrieval uses BM25 and Qdrant together. BM25 is old-school keyword search, and it's excellent at exact terms like names, numbers and specific phrases. Qdrant is a vector database, which searches by meaning, so it finds the right passage even when the question uses different words from the document.
Each one misses things the other catches. Using both costs a little more work and catches a lot more of the right passages. Retrieval takes about 25 ms, live, on that 512 MB box, which leaves most of the budget for everything else.
Decision 3: speculate first, check later
This is where the name comes from. The system answers fast on its best guess, the way a processor runs ahead on a guess about which way a branch will go.
Some questions don't fit the fast path. The answer is spread across several passages, or the best passage doesn't quite answer the question. So behind the fast answer runs an async recovery chain. It checks whether the answer is actually grounded in the source, and when one passage isn't enough, it stitches an answer together from several.
The common case gets the speed, and the hard case still gets the right answer. You don't make every question pay for the hardest one.
Decision 4: let 512 MB decide
Speech comes in through Sarvam's speech-to-text, which handles the Indic languages, and everything after it had to fit in 512 MB. That limit ruled out loading a big model next to the search indexes and hoping for the best.
Like the Raspberry Pis I ran Tixin on, the small box was a feature. Every component had to justify its memory, and the design is cleaner because nothing got in for free.
What I'd do again
I'd set the latency budget first and design backwards from it. If you start with the model and hope it's fast enough, it won't be. I'd extract before I generate, because a lot of questions already have their answer written down somewhere, and finding it is faster and more honest than rewriting it. And I'd keep the fast path fast and the slow path correct, because averaging them into one path leaves you with something that's neither.
The most useful thing an assistant can say quickly is something that was already true.