Bozzetto warns no on-device vector DB

Mobile AI teams want a simple stack: run an embedding model on the phone, search local vectors, and answer with a local model. That is the clean version of offline RAG. The appeal is obvious — lower latency, better privacy, and features that still work on a plane. But the stack is still rough around the edges, which is why Mirko Bozzetto’s complaint landed. The twist is that the problem is real, but the absolute version of it is already out of date. ObjectBox and Couchbase Lite both now ship embedded vector search aimed at mobile and edge use cases. ### What is the missing piece here? A vector database stores embeddings — the numeric fingerprints you generate from text, images, or other data so you can do semantic search. In a RAG app, that database is what lets the model pull the most relevant local notes, docs, or app data before it answers. Without that retrieval layer, “offline AI” usually means either a dumb keyword search or a model forced to answer from whatever you stuffed into its prompt window. (objectbox.io) ### So was Bozzetto wrong? Not exactly. He was directionally right about the developer experience. If you live in React Native or Flutter, there still is not a universally adopted, boring-default embedded vector layer the way there is for cloud stacks. But “there is no viable on-device vector DB” is too strong now. ObjectBox added on-device vector search for Dart and Flutter in May 2024, using HNSW indexing and explicitly framing it as a building block for on-device RAG. (objectbox.io) ### What exists today? ObjectBox is the clearest counterexample on the Flutter side. It says its Dart/Flutter release brought local vector indexing and nearest-neighbor search directly to the platform. On Android, the same company made the same pitch for Java and Kotlin — local vector search, private data, and on-device RAG. Couchbase Lite is pushing a broader embedded database story with vector search, SQL, sync, and offline-first mobile deployment. (objectbox.io) ### Why doesn’t that settle it? Because a vector index is only one layer of the stack. You still need embeddings on-device, which means shipping or calling a local model. You need storage that behaves well on iOS and Android. You need packaging that does not explode app size. And you need bindings that feel native inside Flutter or React Native instead of like a science project glued together with bridges and plugins. That is where teams still hit friction. ObjectBox itself notes that local embeddings may require extra research and tooling. (objectbox.io) ### Why is React Native still awkward? React Native has local-first databases, but vector-native support is less standardized. RxDB now shows an offline JavaScript vector database approach, and there are niche products trying to fill the gap, but there still is not one obvious default that everybody reaches for. In practice, many React Native teams still end up with hybrid designs — local cache, remote embeddings, cloud vector search, or both. (objectbox.io) ### What about “true offline AI”? That phrase hides several problems at once. Offline retrieval is one. Offline generation is another. A phone can store vectors and run nearest-neighbor search, but answering well also depends on whether the local model is small enough, fast enough, and good enough. Basically, the database problem is becoming solvable faster than the full product problem. (rxdb.info) ### Why does this matter now? Because mobile AI is shifting from demo features to product features. Once teams want personal knowledge search, private copilots, or always-available assistants, cloud-only designs start to look brittle. The good news is that the ecosystem is moving. The bad news is that it still feels fragmented. ### Bottom line? Bozzetto’s warning captures the pain, but not the whole map. On-device vector databases do exist now. What is still missing is the boring, cross-platform, one-command developer experience that would make offline mobile RAG feel normal instead of advanced. (objectbox.io)