MLX Swift brings Apple Silicon inference
- SharpAI’s SwiftLM repository published new Apple Silicon builds this week for a native Swift inference server that serves MLX models through an OpenAI-compatible application programming interface on macOS and iPhone. - The project says it can stream experts from solid-state storage for mixture-of-experts models above 100 billion parameters and compress the key-value cache with TurboQuant to stretch memory on Apple hardware. - The release pushes MLX Swift beyond its research-only framing toward local serving tools developers can plug into existing clients. (github.com)
Machine learning inference is the step where a trained model answers a prompt, and SharpAI’s SwiftLM is trying to do that natively on Apple Silicon. (github.com) (swift.org) SwiftLM is an open-source server written in Swift that exposes a “strict OpenAI-compatible API,” so apps built for standard chat-completions style endpoints can point at a local Mac instead of a cloud model. (github.com) The repository’s latest public activity shows new commits on April 25 and a release build posted within the past week, along with downloadable macOS Apple Silicon binaries and a SwiftBuddy desktop app. (github.com 1) (github.com 2) To follow the claim, it helps to know the bottleneck. Large language models store a running memory of earlier tokens in a key-value cache, and that cache can eat up RAM during long generations. (github.com) TurboQuant is a compression method for that cache. A separate MLX implementation describes 3-times to 5-times memory compression with near-lossless quality, and SwiftLM says it uses TurboQuant-style key-value cache compression in its server. (github.com 1) (github.com 2) SwiftLM also advertises solid-state drive streaming for mixture-of-experts models above 100 billion parameters. In plain terms, that means keeping some model parts on storage and loading the needed experts on demand instead of forcing the whole model into memory at once. (github.com) That approach lines up with a broader push to make Apple hardware a serious local inference target. A January 2026 paper on vllm-mlx argued that Apple Silicon’s unified memory and bandwidth can support large local models, while existing tools still leave gaps in batching and multimodal support. (arxiv.org) There is an important caveat in the background material from Apple’s Swift team. When MLX Swift was introduced on February 20, 2024, Swift.org said MLX was intended for research rather than production deployment inside apps. (swift.org) SwiftLM is effectively a community answer to that gap: keep the Apple-native MLX stack, add a server wrapper, ship binaries, and mimic the application programming interface developers already use. (github.com) (swift.org) The immediate test is not whether Apple Silicon can run a demo model, but whether teams will trust a local Swift server to replace part of a cloud inference workflow. SwiftLM’s pitch is that the familiar interface stays the same while the hardware moves onto the desk. (github.com)
