On-Device AI Benchmarking Framework Released
The new MobileAIBench framework benchmarks large language and vision-language models for local, on-device use. The framework responds to a growing demand for private, low-latency AI applications. The ability to run complex creative pipelines on local hardware is becoming a key differentiator for AI tools.
- The MobileAIBench framework was developed by Salesforce AI Research and is open-source, featuring a desktop library for evaluations and an iOS app for measuring on-device latency and hardware use. This allows developers to test how different levels of model compression, known as quantization, affect performance on real-world mobile hardware. - A key driver for on-device AI in creative fields is privacy and control over intellectual property; processing data locally prevents sensitive creative work and user data from being sent to external servers. This approach also ensures that creative workflows are not interrupted by poor internet connectivity and can reduce the latency that can occur with cloud-based services. - The debate around human-AI collaboration in creative work is shifting from AI as a simple tool to a co-creative partner, raising new questions about authorship and artistic agency. Frameworks are emerging to understand how to best structure this collaboration, focusing on AI augmenting human judgment rather than replacing it. - For builders creating AI tools, the development environment is evolving with AI-native IDEs and terminals. Tools like Cursor offer an AI-powered code editor built on VS Code, while Warp provides an AI-enhanced terminal experience. These tools are designed to integrate AI assistance directly into the coding workflow, from debugging to refactoring. - The ability to run models locally enables complex multi-tool creative pipelines, where practitioners can chain together different specialized AI tools for tasks like image generation, code assistance, and design. Platforms like Prompts.ai and Azure AI Foundry's Prompt Flow are emerging to help manage these multi-step AI workflows. - Advances in mobile hardware, particularly the inclusion of Neural Processing Units (NPUs), are making it feasible to run more complex generative AI models directly on devices. Benchmarking frameworks are crucial for developers to understand how different models will perform on a variety of chipsets from manufacturers like Apple, Qualcomm, and Samsung. - While MobileAIBench focuses on broad LLM and LMM tasks, other benchmarks are being developed specifically for creative applications. For example, the "Creative Tool Use Benchmark" evaluates an AI's ability to plan and execute multi-step creative workflows, and a separate benchmark by Springboards assesses the creative abilities of different models for advertising and marketing tasks. - The on-device approach is particularly relevant for multimodal creative tasks, which combine visual and text understanding. Open-source vision-language models like Idefics2 and Llama 3.2-Vision are becoming more efficient, making it possible to deploy them on local hardware for tasks such as analyzing images, understanding documents, and generating visual content.
