New Swift SDK for On-Device Voice AI
A new modular Swift library, MLX-Audio-Swift SDK, has been released for building real-time, on-device voice agents on Apple platforms. The SDK is optimized for Apple Silicon and provides components for text-to-speech, speech-to-text, and voice activity detection. Developers can import only the specific modules they need for their applications.
- This SDK is built upon MLX, a machine learning framework developed by Apple's research division that provides APIs similar to NumPy and PyTorch. - It is optimized for the unified memory architecture of Apple Silicon, enabling efficient operations across the CPU and GPU without data duplication. - The SDK leverages the Apple Neural Engine (ANE), a dedicated AI accelerator first introduced in the A11 Bionic chip in 2017, to perform all computations on-device, ensuring that voice data is not sent to the cloud. - The modular architecture includes components for speech-to-speech models, speaker diarization, and audio codecs, in addition to text-to-speech and speech-to-text. - Pre-trained models compatible with the SDK are hosted on the MLX Community on Hugging Face, including models for text-to-speech (Qwen3-TTS, Soprano) and speech-to-text (Qwen3-ASR, Parakeet). - It requires target platforms of at least macOS 14 or iOS 17 and is installable via the Swift Package Manager. - The underlying MLX framework supports lazy computation, meaning arrays are only materialized when needed, which can optimize performance. - The developer behind the SDK is Prince Canuma, who also maintains a Python version of the MLX-Audio library.
