Apple‘s recent updates through Apple Intelligence have significantly improved the camera and image features on iPhones, such as generating movies from your photos and offering generative image editing. One of the most exciting features for many users is natural language search, which lets you find photos simply by describing them in plain language.
However, a blog post by developer Jeff Johnson has revealed serious privacy concerns regarding this feature, designed to make it easier to search for landmarks and points of interest. Many iPhone users have already updated to iOS 18 without fully understanding that a feature called Enhanced Visual Search is now built into their devices.
How Enhanced Visual Search Works
This feature works by detecting a point of interest within a photo. A commonly cited example is the Eiffel Tower. On-device artificial intelligence (AI) creates a mathematical “fingerprint” of the specific area of the image, referred to by Apple as a “vector embedding.” This vector is then sent to Apple’s servers for further processing, with a privacy layer in place that includes fake queries to prevent the server from identifying the real ones. In return, Apple’s servers send back encrypted potential matches.
The Privacy Issue
While Apple claims that the process protects users’ privacy, Johnson argues that this behavior constitutes a privacy violation. He explains in his blog, “I never requested that my on-device experiences be ‘enriched’ by phoning home to Cupertino… If something happens entirely on my device, it’s private. But if my device sends data to the manufacturer, then it’s not entirely private.”
The Enhanced Visual Search feature debuted as part of iOS 18 and macOS 15 on September 16, 2024. However, Apple’s initial announcements about these releases mentioned only “natural language queries” and an “expanded understanding” of search, without clarifying that photo data would be sent to Apple’s servers by default.
On October 24th, Apple did provide a brief explanation about the feature in a technical document discussing machine learning and homomorphic encryption. This document mentioned Enhanced Visual Search and its role in identifying landmarks or points of interest. Additionally, Apple released a legal document detailing more privacy information. However, most users are unaware of these documents, and even fewer understand their implications.
A Silent Privacy Shift
Apple’s new feature might improve search functionality, but the lack of transparency regarding data transmission raises valid privacy concerns. Many users have unknowingly opted into sharing their photos with Apple’s servers as part of the Enhanced Visual Search experience. As this feature continues to roll out, it’s crucial for users to stay informed and consider the potential privacy trade-offs involved.