Blog · On-device AI
Can a laptop run a meeting assistant? On-device AI in 2026, measured
Phone makers ship 3B models on-device, PC makers ship 40-TOPS NPUs, and 1B–3B open models were built for it. Here is what that buys on an ordinary laptop CPU.
The hardware and the models arrived
Apple describes its on-device foundation model as about 3 billion parameters, quantized to an average of 3.7 bits per weight. Microsoft defined Copilot+ PCs by “silicon capable of an incredible 40+ TOPS”. Meta released Llama 3.2 1B and 3B for “on-device agentic applications”, optimised for Arm and enabled for Qualcomm and MediaTek on day one.
What an ordinary laptop does today
We measured our early-access build on a Windows 11 laptop, CPU only, no GPU, no NPU acceleration:
Two honest readings. A meeting's notes arrive within a minute or two of the meeting ending, which is fine for most work. And a 20-second memo that also writes notes takes over two minutes on a CPU, which is why we transcribe while people speak and keep heavier steps out of the way.
The trade-offs that remain
- Small models need guard rails. 1B–3B models write fluent notes and still lose facts, which is why our notes are checked against the transcript (see hallucinations).
- Language coverage lags. On-device models are strongest in English; Indian languages need more (see the benchmark).
- Acceleration is uneven. NPUs and GPUs help a lot where the software supports them; many laptops in offices still run CPU-only.
Sources
- Introducing Apple's On-Device and Server Foundation Models · Apple Machine Learning Research, 10 Jun 2024
- Introducing Copilot+ PCs · Official Microsoft Blog, 20 May 2024
- Llama 3.2: Revolutionizing edge AI and vision with open, customizable models · Meta AI, 25 Sep 2024
- faster-whisper · SYSTRAN, GitHub