Benchmark noise and tiny models

Benchmarks are getting noisier just as tiny artificial intelligence models get better. A new wave of 1.58-bit “ternary” models is pushing both trends into the same conversation. (huggingface.co) (openreview.net) A language model stores learned patterns as numbers called weights; ternary models squeeze those numbers down to three choices: -1, 0, or +1. Prism ML’s Ternary-Bonsai-4B says it packs a 4 billion-parameter Qwen3-based model into 1.05 GiB, down from 8.04 GB in half precision. (huggingface.co) Prism ML says that 4B model runs in an MLX-native 2-bit format on Apple Silicon, reaches 50 tokens per second on an iPhone 17 Pro Max, and averages 70.7 across six benchmark categories. The same model card says the weights are ternary across embeddings, attention projections, multilayer perceptron projections, and the language-model head. (huggingface.co) That claim lands into a research cycle that has been moving toward smaller models for phones and other constrained devices. An ACL 2025 paper reviewing more than 60 small language models said state-of-the-art small models can outperform 7B models on general tasks, though the authors also said in-context learning still lags and efficiency work remains unfinished. (aclanthology.org) The ternary idea is not limited to one release. An ICLR 2025 Spotlight paper on “Surprising Effectiveness of pretraining Ternary Language Model at Scale” reported that, above 1 billion parameters, ternary language models outperformed quantized and floating-point models for a given bit budget, and that a 3.9B ternary model matched a 3.9B floating-point model across its benchmarks. (openreview.net) Benchmarks, though, are only as clean as the test around them. Google’s benchmarking documentation lists CPU frequency scaling, address space layout randomization, and differences between cores as sources of variance that can change results from one run to the next. (github.com) Hardware reviewers run into the same problem from another angle. TechSpot’s CPU-testing guide says reviewers often use a powerful graphics card at 1080p to remove graphics bottlenecks, because changing the resolution can shift a test from measuring the processor to measuring the graphics card instead. (techspot.com) General-purpose benchmark suites also make tradeoffs. Geekbench 6 says it uses “real-world” workloads, machine-learning tests, and cross-platform comparisons, but any single score still compresses many different tasks into one number. (geekbench.com) That leaves tiny-model claims in a narrow lane: useful, but conditional. A 1.05 GiB model that runs on a phone and posts competitive scores can be a strong engineering result, while still needing task-by-task checks on accuracy, latency, and what the benchmark actually measured. (huggingface.co) (github.com) The immediate story is not that one benchmark settled the field. It is that compact models are improving fast enough that the test setup now matters almost as much as the score. (aclanthology.org) (openreview.net)