Case studies · Accuracy test
Is this the right conversation? Teaching a small model to say no
Asked whether a lawyer's call was a shop-counter enquiry, a 3B model said yes and filed the case name as the product. Four ways of asking, 22 pairs each, on a laptop CPU.
The problem
VIBE mode fills a business's own record after every conversation. A shop that sets up a counter-enquiry format does not want a lawyer's call filed into it. So before filling anything, the app asks the local model whether the conversation is the kind the format is for.
While testing the new Records screen on 10 October, we tried the store-counter format on a recorded client call about a property dispute. The relevance question was part of the same prompt that fills the fields, and the model answered yes. The record came back with the case name, “Charmau versus Horizon Builders”, in the Product field. Every value was cited, so the mistake was easy to see. It was still a mistake.
What we tested
Two recorded conversations: a salesperson's note from a phone-store counter, and a lawyer's call with a client. Each was checked against the eleven formats Vlocalhost now ships as starting points. That is 22 pairs, with one right answer each: the store note belongs to the store-counter format, the call to client intake, and nothing else matches. Each question was put to Llama 3.2 1B and 3B through Ollama on a Windows laptop with no GPU.
What each change did
| Question | What happened |
|---|---|
| Yes / no, asked on its own before the fields | 1B got 20 of 22. 3B said yes too easily: 16 of 22, including the lawyer's call against the store format. |
| Yes / no, after naming the topic | Asking the model to describe the conversation first is a common fix. Here it broke the 1B model, which then said yes to almost everything: 2 of 22. |
| Multiple choice | “Which one of these is this conversation?”, with the format's description among the other kinds and “something else”. Both models got 20 of 22. Now the misses were mostly skips: the right conversation rejected. |
| Multiple choice, clearer descriptions | The skips were partly fair. The store recording was a staff note, not a customer at the counter, and the call was about an existing case, not a new matter. With the descriptions widened to say so, both models still got 20 of 22, and the store note was matched by both. |
What we shipped
- The multiple-choice question, asked first and on its own. When it says no, no fields are filled. That is cheaper as well as safer.
- No silent skips. A skipped conversation now appears in the Records screen with what it looked like (“a client and a lawyer discussing a case”) and a Fill it anyway button. A wrong skip costs one click instead of a missing record.
- Review before sending, per format. A business can hold every record for a person to check, and the model's original value is kept beside any edit.
What this does not show
- Two conversations is a small test. It shows which way each change moves the answers, not a rate you can expect.
- Neighbouring kinds still get confused: a store note and a support call, a property dispute and a site visit. That is why a person can always see and undo the decision.
- The Records screen and these changes are in the Pro early-access build, free during the preview, from 10 October.
Sources
- Llama 3.2 on Ollama · Ollama