Mock: diagnose 5% latency spike

A mock debugging thread starts with a narrow symptom: about 5% of application programming interface calls go slow only during peak traffic, which usually points to a tail problem, not a total outage. (aws.amazon.com) In distributed systems, “latency” is the wait time for a response, and the slowest slice often hides inside the tail rather than the average. Amazon’s Builders’ Library notes that many failures first appear as requests taking longer than usual, while clients keep holding memory, threads, and connections. (aws.amazon.com) That framing fits the simulation’s first checks: whether peak-hour traffic is being sent to a farther region, whether a database is spending longer waiting on locks, and whether retries are turning a small slowdown into a bigger burst. Cross-region transaction systems are especially exposed because each extra network round trip and each longer lock hold adds delay. (arxiv.org) The database piece matters in payments-style systems because one slow transaction can block another while it holds a lock, like a checkout lane that stays closed until one customer finishes. A recent paper on geo-distributed transaction middleware says performance drops when network latency rises and “lock contention span” gets longer across regions. (arxiv.org) The retry piece is where a 5% issue can widen. Amazon says retries help with partial and transient failures, but they can also increase load on a backend that is already near overload, especially if many clients retry together. (aws.amazon.com) Amazon Web Services’ Well-Architected guidance says teams should cap retries, add exponential backoff, and randomize them with jitter so clients do not create coordinated spikes. The same guidance warns against retrying at multiple layers of a stack, because compounded retries can create a retry storm. (docs.aws.amazon.com) Cold caches are the other peak-hour suspect. A cache is a fast copy of popular data kept close to the application, and when it is empty or expired, many requests fall through to the slower origin at once. (developers.cloudflare.com) Cloudflare described that pattern in December 2024 with a simple example: if an item is requested 10 times per second and the origin can only handle 1 request per second, an expired cache can send all 10 requests back to origin together. Cloudflare calls that a cache stampede and says request collapsing or cache locks can keep one refill request from becoming many. (blog.cloudflare.com) Google’s Site Reliability Engineering material puts those symptoms into the standard incident playbook: set service level objectives, monitor overload, and identify recovery paths before a partial slowdown becomes a customer-visible incident. The point is to measure the bad tail directly instead of letting averages hide it. (sre.google) So the practical checklist in the mock exercise is straightforward: compare latency by region, inspect lock waits and connection pools, trace retry volume after first failures, and check cache hit rates before and during the spike. A 5% latency jump is small enough to miss on a dashboard and large enough to signal the start of a larger overload cycle. (aws.amazon.com)