Blog · Accuracy
When AI notes invent things: what the research says, and what we do about it
About 1% of Whisper transcripts contain whole invented phrases. A 1B model wrote unfaithful dialogue summaries 83% of the time in one study. Meeting notes need a check that does not depend on the model behaving.
Two places a note can go wrong
The transcript. In Careless Whisper, researchers found that “roughly 1% of audio transcriptions contained entire hallucinated phrases”, and that 38% of those included explicit harms. An AP investigation reported a researcher finding fabrications in 8 of 10 transcripts they examined, in a medical tool built on Whisper whose vendor deletes the original audio. Barański et al. showed that non-speech audio triggers recurring phrases, and proposed filtering them out. We saw one ourselves: “Thanks for watching.” appeared in a test transcript from microphone noise.
The summary. A language model then condenses the transcript, and small models are the ones that fit on a laptop. In Huang et al., Llama 3.2 1B produced unfaithful summaries in 64% of CNN/DM, 83% of SAMSum and 70% of XSum instances, as judged by GPT-4o span annotation. SAMSum is dialogue, the closest of the three to a meeting.
Larger models do better: on Vectara's leaderboard the best summarisers sit below 4%, and 4B-class open models around 6–7%. None is zero.
What happened to us
Our own notes once listed “Decided: then let's ship on Friday” for a phone-shop memo. The sentence came from an example in our prompt. The full write-up covers it; the lesson generalises.
A check that does not trust the model
- Cite everything. Every decision, task and value points at the line and second it came from. A wrong item becomes a one-click check.
- Ground in code, not in the prompt. Asking a model not to invent is a request. Removing any decision whose words are not in the transcript is a guarantee.
- Prefer extraction to generation. When a question is asked across meetings, assemble the answer from cited facts instead of letting a model write it.
- Keep the audio's owner in control. A transcript you can replay against your own recording is checkable; one whose audio was deleted is not.
Sources
- Careless Whisper: Speech-to-Text Hallucination Harms (Koenecke et al.) · arXiv, Feb 2024, rev. May 2024
- Researchers say an AI-powered transcription tool used in hospitals invents things no one ever said (AP, syndicated) · Associated Press, 27 Oct 2024
- Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio (Barański et al.) · arXiv / ICASSP 2025, Jan 2025
- Enhancing Faithfulness in Abstractive Summarization via Span-Level Fine-Tuning (Huang et al.) · arXiv, Oct 2025
- Hallucination Leaderboard · Vectara, updated 22 Sep 2026