Azure’s GUI‑driving agents

A language model usually works like a clerk behind a window: you hand it text, it hands back text. Microsoft is now offering a model in Azure that can look at a screen, decide where to click, type into boxes, and keep going through a multi-step task. (learn.microsoft.com) That sounds small until you remember how much business software still lives behind old graphical screens. A purchasing system from 2011, a claims portal with no application programming interface, or a desktop app with 14 buttons can all be used by a person with a mouse, and this model is meant to work the same way. (learn.microsoft.com) Microsoft put this into the Azure OpenAI Responses application programming interface, which is its newer stateful interface for agents. The same interface is meant to keep context across turns, call tools, search files, and now drive a user interface from one workflow. (learn.microsoft.com) (azure.microsoft.com) The basic loop is simple. Your code sends the model a screenshot, the model returns an action like click or type, your software performs that action in a browser or virtual machine, and then sends back a fresh screenshot so the model can choose the next step. (developers.openai.com) (learn.microsoft.com) OpenAI describes that setup as the model being the eyes and planner while your harness is the hands on the keyboard and mouse. That split matters because the model is not directly “inside” Windows or a website; it is steering an environment that your code controls. (developers.openai.com) Microsoft’s March 11, 2025 announcement pitched this for insurance claims, information technology service desks, supply chains, and healthcare record workflows. Those are all places where the hard part is often not reasoning alone but getting from one screen to the next without rewriting the whole software stack. (azure.microsoft.com) The reason enterprises care is that replacing legacy software is expensive and slow. If an agent can use the same buttons and forms that a human already uses, companies get a shortcut that does not require every old system to expose a clean application programming interface first. (learn.microsoft.com) (azure.microsoft.com) There are limits right away. Microsoft’s documentation says access to the model requires registration, and even customers who already have access to other limited-access models still have to apply for this one separately before they can deploy it. (learn.microsoft.com 1) (learn.microsoft.com 2) There are safety limits too. OpenAI’s own guidance says to run computer use in an isolated browser or virtual machine, keep a human in the loop for high-impact actions, and treat page content as untrusted input. (developers.openai.com) That warning is there because a screen can contain traps the same way an email can. A malicious webpage, a fake button, or text that tries to redirect the model can all show up inside the screenshot stream the model is using to decide what to do next. (developers.openai.com) The pricing and access story is also less friendly than a normal demo. Microsoft’s Azure OpenAI service generally needs an approved Azure subscription, and Microsoft support answers indicate free-trial or ordinary promotional paths often do not give straightforward access to Azure OpenAI in the first place. (learn.microsoft.com 1) (learn.microsoft.com 2) So the real news is not that a model can click a button. It is that Microsoft is trying to turn “use the software exactly like an employee would” into a supported cloud product, which is a much more practical bridge between language models and the huge pile of old enterprise software that still runs large companies. (azure.microsoft.com) (learn.microsoft.com)