On-device AI: Agents owning their data

- Alibaba's zvec is an open-source, in-process vector database designed to function like "the SQLite of vector databases" for on-device applications. It is built on Proxima, Alibaba's production-grade vector search engine, and can be embedded directly into an application as a library without needing a separate server. - On benchmarks, zvec has demonstrated the ability to deliver over 8,000 queries per second (QPS) on the VectorDBBench with the Cohere 10M dataset, which is more than double the performance of the previous leader, ZillizCloud. This level of performance is critical for real-time applications on resource-constrained devices. - For a portfolio project, an ML engineer could use an on-device vector database like zvec or ObjectBox to build a privacy-focused mobile application that performs Retrieval-Augmented Generation (RAG) entirely locally. This demonstrates skills in deploying models to edge devices, managing data pipelines, and working with modern AI tools, all of which are sought after by companies building features like Apple's FaceID or Tesla's Autopilot. - The concept of agents owning their data is a core tenet of Web3 and decentralized AI, aiming to give users sovereignty over their information by storing it across decentralized nodes instead of centralized servers. This architecture enhances security by removing single points of failure and allows for transparent, auditable AI agent behavior through technologies like blockchain. - In ML system design interviews, expect questions about handling the trade-offs between latency, accuracy, and scale, especially for on-device applications like a YouTube recommendation feed on mobile versus desktop. Interviewers will probe your ability to design end-to-end systems, including data ingestion, feature engineering, model deployment, and monitoring for issues like data drift. - On-device RAG is enabled by the increasing efficiency of "small" large language models such as Phi-3, Llama 3, and Google's Gemma 2, which are capable of running directly on user hardware. This local execution reduces latency, works offline, and addresses privacy concerns by preventing sensitive data from being sent to the cloud. - A key challenge in on-device AI is efficient memory management, as vector data can be memory-intensive. Solutions like ObjectBox employ techniques such as multi-layered caching and tight integration of its HNSW algorithm with the database persistence layer, avoiding the need to keep all vectors in memory. - The shift towards decentralized AI is creating an "AI Agent Economy" where autonomous agents can operate as economic participants with their own decentralized identifiers (DIDs). This allows them to prove their identity, own data, and exchange value programmatically without relying on centralized platforms for permissions and trust.