iPhone's 'Visual Intelligence' Aims to Kill Typing

This functionality is an evolution of features Apple has been building for years. It started with "Live Text" in iOS 15, which let users copy and translate text from photos, and was expanded in iOS 16 to work in paused videos. That same year, "Visual Look Up" was introduced to identify subjects like plants, landmarks, and pets within the Photos app. The latest iteration, branded "Visual Intelligence," is a core part of the "Apple Intelligence" suite and requires the on-device processing power of an iPhone 15 Pro or newer model. On the iPhone 16, a dedicated "Camera Control" button can activate the feature, allowing users to point their camera at an object to get information directly, without first taking a photo. Under the hood, Visual Intelligence integrates third-party AI to expand its capabilities. Pointing the camera and tapping "Ask" sends the query to ChatGPT, while tapping "Search" uses Google to find similar images. This allows the system to identify business hours from a sign, add an event from a flyer to a calendar, or look up a product seen in the real world. Apple's focus on visual data is backed by a string of strategic acquisitions. In 2023 alone, Apple acquired 32 AI startups. More recent deals include the acquisition of Q.AI for nearly $2 billion to analyze facial expressions and Prompt AI for its object detection technology. CEO Tim Cook has repeatedly signaled that visual intelligence is a key future platform for the company. The technology is seen as a foundational step for a new category of AI-driven wearable devices that can understand and interact with a user's surroundings. Industry analysts anticipate this technology will extend beyond the iPhone. Rumored future hardware includes camera-equipped AirPods and a wearable AI pendant or pin. These devices would use low-resolution cameras not for photography, but to provide contextual awareness for the AI, enabling more intuitive, real-world navigation and interaction.