Edge AI Performance Benchmarked for Mobile Devices
Salesforce AI Research has published MobileAIBench, a new benchmark for evaluating large language and multimodal models on mobile devices. The findings show that modern edge hardware is capable of running sophisticated AI models at usable speeds. This technical validation supports the feasibility of developing on-device AI agents for consumer applications requiring privacy and low latency, such as real estate and fitness apps.
- The MobileAIBench framework is an open-source tool composed of a desktop evaluation library and an iOS app to measure on-device metrics like latency, CPU usage, and battery drain. Initial tests on an iPhone 14 revealed that while language models under 7 billion parameters are viable, more complex Large Multimodal Models (LMMs) had a time-to-first-token exceeding 60 seconds, indicating they may not yet be suitable for real-time mobile use. - While the benchmark was run on an iPhone 14, reviewers noted the importance of testing on AI-optimized hardware like the Snapdragon 8 Gen 3. For comparison, other research shows a 3-billion parameter model called Imp-3B can run on a Snapdragon 8 Gen 3 chip at a speed of about 13 tokens per second. - In real estate, agentic AI is already being deployed to automate multi-step workflows beyond simple chatbots. Applications include generating and reviewing lease agreements for legal compliance, processing tenant applications by verifying credit scores and IDs, and managing automated reminders for rent payments and renewals. - For fitness applications, companies like Tempo are leveraging AI with 3D motion capture systems to provide real-time form correction and personalized workout plans. This technology analyzes a user's movements during exercise to provide immediate feedback, functioning like a personal coach to optimize performance and prevent injury. - The challenge of inconsistent AI performance, which Salesforce researchers call "jagged intelligence," is a key hurdle for enterprise agents. To address this, Salesforce also developed CRMArena, a benchmark specifically for testing AI agents in realistic customer relationship management (CRM) scenarios to ensure reliability. - Venture capital investment is shifting from foundational AI infrastructure to vertical-specific AI applications targeting complex industries like real estate, finance, and healthcare. This signals a market maturation where investors are now backing companies solving tangible industry problems with applied AI. - According to Gartner, more than 50% of venture capital investments are predicted to go to AI-powered startups by 2026. Top VC firms like Sequoia Capital and Andreessen Horowitz are actively funding AI-native companies, with AI and machine learning deals capturing over 65% of U.S. venture capital in 2025. - A key insight from venture capitalists is that AI enables solo founders and small teams to build products and achieve outputs that previously required much larger organizations. This shift is fundamentally changing startup team structures and operational efficiency, making it more feasible for technical founders to launch companies with less initial capital.
