Google Cloud CISO Warns of New AI Threat Vectors
Google Cloud's Office of the CISO has released a report warning of emerging security risks as agentic AI systems become more widespread. The new threat vectors include model distillation attacks, adversarial experimentation, and risks to the AI supply chain. The report urges developers to prioritize secure model integration and robust monitoring.
- Model distillation, also known as model extraction, involves an attacker repeatedly querying a target AI model to gather enough input-output pairs to train their own "student" model that mimics the original's capabilities. This is considered intellectual property theft, and Google has observed campaigns targeting its Gemini model with over 100,000 prompts to extract its underlying logic. - Adversarial experimentation involves testing AI models with malicious or harmful inputs to find vulnerabilities. Attackers can use these techniques to cause models to bypass safety protocols, generate harmful content, or reveal sensitive information. - Risks to the AI supply chain include data poisoning, where training data is maliciously altered, and the use of compromised open-source models that may contain backdoors or other vulnerabilities. These threats can compromise the integrity of AI systems before they are even deployed. - The report is part of a larger initiative by Google's Threat Intelligence Group (GTIG) to track the adversarial misuse of AI. The group has observed threat actors from nations like China, Iran, and North Korea using AI to improve social engineering campaigns and automate reconnaissance. - Agentic AI systems introduce unique risks like "memory poisoning," where an attacker injects false information into an agent's memory to influence future decisions, and "tool misuse," where an agent with access to other applications is tricked into performing harmful actions. - The rise of autonomous AI agents creates an explosion of "Non-Human Identities" (NHIs), such as API keys and service accounts, which become targets for attackers to impersonate trusted agents and gain access to systems. - To counter these threats, Google is developing AI-driven defenses, including a system named "Big Sleep" created by Google DeepMind and Project Zero to identify vulnerabilities. The company also advocates for frameworks like the Secure AI Framework (SAIF) to establish security standards for building and deploying AI. - While attackers are increasingly using AI for malicious purposes, some experts, including Google Cloud's CISO Phil Venables, believe that in the long term, AI gives a structural advantage to defenders who can use it to analyze vast amounts of security data and automate defenses.
