Case studies · Benchmark
Six Indian-language meetings: Sarvam's cloud models against our on-device pipeline
Saaras v4 averaged 5.2% word error across six meetings. The Whisper models we ship today averaged around 80%. Here is the full run, its cost, and its limits.
Why we ran it
Vlocalhost writes meeting notes on the user's own machine. In English that works today. Indian languages are where our on-device stack has to catch up, and we wanted to know by how much before claiming anything. Through the Sarvam Startup Program we compared Sarvam's models against exactly what we ship.
The setup
- Six short meetings, one each in Hindi, Hinglish (code-mixed), Indian English, Bengali, Tamil and Marathi: 134 seconds of audio in total. Each contains an owner, a task and a deadline.
- Speech-to-text: Sarvam's Saaras model (cloud; API model
saaras:v4, called on 6 Oct 2026) against faster-whisper tiny and small, int8, on a laptop CPU with no GPU. - Notes: sarvam-105b against Llama 3.2 3B and Qwen 2.5 1.5B running locally through Ollama. A note counts only if owner, task and deadline are all right.
Speech-to-text
Saaras stayed under 8% word error in every Indian language. Whisper did not misread so much as break down: tiny drifted from Hindi into romanised text, small wrote Bengali in Devanagari. English was the only parity point: Whisper small reached 4.6% against Saaras's 6.2%.
One setting mattered more than any model choice. On the same Hinglish audio, Saaras in its default transcribe mode scored
39.7% because it spelled English words in Devanagari; in codemix mode it scored 6.3%.
Meeting notes
| Model | Where | Action items kept | Decisions kept |
|---|---|---|---|
| sarvam-105b | cloud | 6 / 6 | 6 / 6 |
| Llama 3.2 3B | this laptop | 3 / 6 | 4 / 6 |
| Qwen 2.5 1.5B | this laptop | 1 / 6 | 4 / 6 |
The small local models wrote fluent English notes and lost the facts: deadlines moved from Friday to Monday, and one invented a festival. On Indian English all three kept both facts. Sarvam's tokenizer also read Tamil in 145 tokens where Llama 3.2 needed 508, 3.5× fewer.
Limits we publish with it
Six studio-clean synthetic clips flatter every system, and the audio was voiced by Sarvam's own Bulbul model, which may favour Saaras. Local notes were single runs. The next phase is roughly 40 hours of real, consented meeting audio.
Sources
- Sarvam AI · Sarvam
- faster-whisper · SYSTRAN, GitHub
- Llama 3.2 on Ollama · Ollama