Circuit‑breaker and retry patterns
- Netflix’s Hystrix design, Amazon’s Builders’ Library, and Google Cloud retry guidance all describe the same resilience playbook: stop hammering sick services, wait with backoff, and probe carefully before reopening traffic. - The core mechanics are specific: circuit breakers move through closed, open, and half-open states, while retries use exponential backoff plus jitter so large fleets do not retry in lockstep. - The pattern spread because retries can multiply load across service stacks, and controlled fail-fast behavior limits cascade risk in distributed systems. (aws.amazon.com)
A circuit breaker in software is a traffic stoplight for failing services: let calls through when healthy, cut them off when errors spike, then test recovery with a few probes. (github.com) Netflix’s Hystrix documentation describes that flow with three states: closed for normal traffic, open to reject calls after failures, and half-open to allow limited test requests. (github.com) When the breaker is open, Hystrix sends callers to fallback logic instead of waiting on a dependency that is already timing out. Netflix said that keeps latency from piling up across threads and queues. (github.com) Retries solve a different problem. A retry says “try again later,” but only when the failure is likely to be temporary, such as a dropped connection or a brief overload. (cloud.google.com) Amazon’s Builders’ Library says retries can become dangerous in deep service chains because each layer may retry the next one. In a five-deep stack with three retries per layer, load on the database can multiply 243 times when the bottom layer starts failing. (aws.amazon.com) That is why Amazon recommends exponential backoff, which increases wait time after each failure, and jitter, which adds randomness so thousands of clients do not retry at the same instant. (aws.amazon.com) Google Cloud’s retry guidance makes the same point from the client side: retry only idempotent operations when possible, cap the number of attempts, and use exponential backoff to avoid creating more congestion. (cloud.google.com) The two patterns work together. A retry policy handles short-lived glitches, while a circuit breaker prevents a caller from spending more time and capacity on a dependency that is already failing hard. (aws.amazon.com) (github.com) Fallbacks are the last safety rail. Instead of returning fresh recommendations, a service might serve cached data, omit a nonessential widget, or return a clear error for an optional feature. (github.com) The practical rule across Netflix, Amazon, and Google is simple: fail fast, retry selectively, spread retries out, and never let one bad dependency drag an entire service fleet down with it. (github.com) (aws.amazon.com) (cloud.google.com)
