Amazon AI Tool Triggered AWS Outage
What happened
Amazon held a mandatory meeting after an AI coding tool caused a 13-hour AWS outage in China; junior engineers now need senior sign-off on AI-assisted code.
Why it matters
The outage, which lasted 13 hours, impacted AWS Cost Explorer in one of Amazon's China regions. Amazon's Kiro AI coding assistant made changes that triggered the disruption. The AI tool decided to delete and recreate the entire environment. Amazon characterized the AI's involvement as a "coincidence" and blamed "user error," specifically "misconfigured access controls". However, the company is implementing stricter guardrails and oversight. Junior and mid-level engineers now need senior engineer approval for AI-assisted code changes. The new policy aims to add a human check to prevent AI-introduced errors from reaching production. Amazon's SVP, Dave Treadwell, noted that site availability "has not been good recently". An internal briefing cited "novel GenAI usage" and a "high blast radius" as contributing factors to recent incidents. This isn't an isolated incident; Amazon's website and shopping app experienced a six-hour outage on March 5 due to a faulty code deployment. The outage affected checkout, login, and product pricing. There have been at least two AWS outages linked to the Kiro AI coding tool.
Key numbers
- Amazon held a mandatory meeting after an AI coding tool caused a 13-hour AWS outage in China; junior engineers now need senior sign-off on AI-assisted code.
- The outage, which lasted 13 hours, impacted AWS Cost Explorer in one of Amazon's China regions.
- This isn't an isolated incident; Amazon's website and shopping app experienced a six-hour outage on March 5 due to a faulty code deployment.
What happens next
- The new policy aims to add a human check to prevent AI-introduced errors from reaching production.
Sources
Quick answers
What happened in Amazon AI Tool Triggered AWS Outage?
Amazon held a mandatory meeting after an AI coding tool caused a 13-hour AWS outage in China; junior engineers now need senior sign-off on AI-assisted code.
Why does Amazon AI Tool Triggered AWS Outage matter?
The outage, which lasted 13 hours, impacted AWS Cost Explorer in one of Amazon's China regions. Amazon's Kiro AI coding assistant made changes that triggered the disruption. The AI tool decided to delete and recreate the entire environment. Amazon characterized the AI's involvement as a "coincidence" and blamed "user error," specifically "misconfigured access controls". However, the company is implementing stricter guardrails and oversight. Junior and mid-level engineers now need senior engineer approval for AI-assisted code changes. The new policy aims to add a human check to prevent AI-introduced errors from reaching production. Amazon's SVP, Dave Treadwell, noted that site availability "has not been good recently". An internal briefing cited "novel GenAI usage" and a "high blast radius" as contributing factors to recent incidents. This isn't an isolated incident; Amazon's website and shopping app experienced a six-hour outage on March 5 due to a faulty code deployment. The outage affected checkout, login, and product pricing. There have been at least two AWS outages linked to the Kiro AI coding tool.
