100B LLM on a CPU
Microsoft’s open BitNet framework claims it can run 100B‑parameter LLMs on a single CPU using 1.58‑bit weights, hitting about 5–7 tokens/sec and using ~82% less energy than alternatives while matching competitive accuracy after training on 4T tokens. That’s a concrete signal that huge models can move off the cloud and into edge machines like MacBooks or ARM devices. (x.com)
Microsoft Research published bitnet.cpp as an open-source inference framework on GitHub under the MIT license, with the repository showing roughly 34.5k stars and active commits. (github.com) A technical report describing BitNet b1.58 and the project's evaluation artifacts was posted to arXiv (arXiv:2504.12285). (arxiv.org) Microsoft has uploaded BitNet weights to Hugging Face as microsoft/bitnet-b1.58-2B-4T, representing a 2‑billion‑parameter release available under an open model card. (huggingface.co) Several outlets and analysts note that while the framework asserts the ability to operate much larger models, the largest model Microsoft has published to date is the 2B release, leaving a disparity between capability claims and publicly shipped model artifacts. (aihola.com) The codebase includes optimized ternary CPU kernels and published CPU benchmarks that report ARM performance uplifts (roughly 1.37×–5.07× over FP16 baselines) and explicitly lists future NPU support in its roadmap. (github.com) Community response generated rapid tutorials and third‑party walkthroughs showing local CPU runs on laptops and single‑board devices, and community reports indicate Microsoft pushed additional CPU optimizations in January 2026. (byteiota.com)
