Weekly feedback cycles
- Microsoft, OpenAI, LangChain, and Dust are all pushing the same production pattern: capture user corrections continuously, then feed them into evals, prompts, and model tuning. - OpenAI’s reinforcement fine-tuning now uses programmable graders to score candidate responses, while LangChain’s human-in-the-loop middleware can pause, edit, or reject agent actions. - The shift marks a move away from one-off prompt tweaks toward always-on “signals loops” that update agent behavior from live usage data. (azure.microsoft.com)
AI teams are moving from static prompts to recurring feedback loops that change agent behavior after deployment, not just before launch. (azure.microsoft.com) (developers.openai.com) In Microsoft’s October 21, 2025 framing, the “signals loop” means capturing user interactions in real time and feeding them back into model behavior and product changes. The company said that pattern is showing up in products including Dragon Copilot and GitHub Copilot. (azure.microsoft.com) OpenAI’s current tooling splits that loop into two layers. Its Evals stack lets teams test prompts and stored responses against graders, and its reinforcement fine-tuning lets developers train o4-mini with a numeric reward signal instead of a fixed answer key. (developers.openai.com 1) (developers.openai.com 2) That reward signal can target style, safety, or domain accuracy, and OpenAI says the platform cycles through prompts, samples several responses, scores them, and updates the model toward higher-scoring behavior. The company says reinforcement fine-tuning is currently supported only on o-series reasoning models and only for o4-mini. (developers.openai.com) A second layer sits above the model itself: the agent’s instructions, tools, and approval rules. Dust argued on January 15, 2026 that many enterprise teams are not really missing better base models; they are missing a system that turns votes, comments, and repeated corrections into updated agent configurations. (dust.tt) Dust said it sees thousands of pieces of agent feedback each month on its platform, including +1 and -1 votes, comments, and suggestions. Its description of the common cycle is weekly in practice: deploy, collect complaints, tweak instructions or data sources, redeploy, and repeat. (dust.tt) OpenAI is also formalizing prompt iteration as a product workflow. Its prompt optimizer is a dashboard chat interface that rewrites prompts to current best practices, and the company says it works best when paired with a dataset and evaluation set rather than a single ad hoc rewrite. (developers.openai.com) For higher-risk actions, teams are adding human checkpoints instead of letting agents run end to end. LangChain’s human-in-the-loop middleware can interrupt tool calls like writing files or executing SQL, then let a person approve, edit, or reject the action before execution resumes. (docs.langchain.com) That is a narrower claim than “self-adjusting” agents, but it shows where production systems are landing today: live feedback changes prompts, eval sets, and tool permissions faster than model vendors ship new releases. (docs.langchain.com) (developers.openai.com) (azure.microsoft.com) Anthropic’s safety work points in the same direction from the model side. Its next-generation Constitutional Classifiers report said the system cut jailbreak success from 86% to 4.4% in its tests by adding a separate classifier layer that blocks unsafe requests before they land. (anthropic.com) The through line is that agent quality is becoming an operations problem as much as a model problem. The teams that review outputs every week, update prompts with datasets, and gate risky actions with human approval are building agents that change between major model releases. (azure.microsoft.com) (developers.openai.com) (docs.langchain.com)
