Security budgets lift labeling need
A recent security-focused video argues attackers are using AI and highlights concrete defenses, signalling that security teams are budgeting for AI-related controls and tooling. That creates durable labeling demand for evaluations that test prompt injection resistance, safe tool use, escalation behavior, and policy adherence in agentic systems. Vendors that tie labeling services to measurable security outcomes will find clearer procurement paths. (youtube.com)
Security teams are starting to buy artificial intelligence defenses, and that is creating a steadier market for the labels used to test those systems. (youtube.com) In a recent interview on The Tech Trek, Impart Security chief executive Jonathan DiVincenzo said attackers are using large language models to adapt faster than web application firewalls, and he pointed to prompt injection and “invisible character” tricks as production threats. The video was posted about four months ago and lists runtime security, prompt injections, and hidden-input exploits among its main topics. (youtube.com) Those risks now sit inside mainstream security guidance. The Open Worldwide Application Security Project’s Top 10 for Large Language Model Applications 2025 puts prompt injection at No. 1 and lists “excessive agency,” or models taking actions they should not take, as a separate category. (owasp.org) A label in this market is not a sticker on content. It is a human or machine-generated judgment on whether a model resisted a malicious instruction, refused an unsafe tool call, escalated to a human reviewer, or followed a policy in a test case. (security.googleblog.com) Google DeepMind said on January 29, 2025 that it built an evaluation framework to automatically red-team indirect prompt injection, including a scenario where an agent reads an attacker-controlled email and is tested on whether it leaks private data. That kind of setup turns security behavior into pass-fail examples that must be labeled and measured over time. (security.googleblog.com) The National Institute of Standards and Technology has moved the same direction. Its Generative Artificial Intelligence Profile, released July 26, 2024 as part of the Artificial Intelligence Risk Management Framework, tells organizations to identify and manage risks unique to generative systems during design, use, and evaluation. (nist.gov) Microsoft has also formalized the work. Its Microsoft Learn documentation for the Artificial Intelligence Red Team includes guidance for building red teams for large language models, an open automation framework called Python Risk Identification Tool for red teaming generative systems, and lessons from testing 100 generative artificial intelligence products. (learn.microsoft.com) Budget signals are getting stronger outside the lab. Gartner said on March 31, 2025 that worldwide generative artificial intelligence spending would reach $644 billion in 2025, up 76.4% from 2024, while International Data Corporation said global security spending would rise 12.2% in 2025 as threats accelerated by generative artificial intelligence pushed buyers toward more controls. (gartner.com) (biztechreports.com) That spending pattern favors labeling vendors that can connect test data to security outcomes a buyer already tracks, such as prompt injection success rates, unauthorized tool execution, data leakage, and policy-violation rates. As more companies move from chatbots to agents that can read files, call tools, and take actions, the demand is shifting from generic quality labels to security-grade evaluation labels. (owasp.org) (security.googleblog.com) The near-term question is not whether companies will test these systems, but which suppliers can show that their labels change a security metric before procurement teams sign a contract. The more artificial intelligence security buying looks like ordinary cybersecurity buying, the easier that pitch becomes. (youtube.com) (nist.gov)
