Circuit Breaking & Adaptive Load Shedding: Resilience4j & Google SRE Token Bucket Algorithms

In large-scale microservice platforms (Uber, Amazon, Google, Stripe), services interact via dependency DAGs.

When a downstream database or third-party payment gateway experiences latency spikes, upstream caller services continue flooding it with requests, queueing up worker threads and exhausting memory.

Left unmitigated, a localized microservice slowdown triggers a chain reaction that crashes the entire cloud infrastructure (Cascading System Failure).

To prevent cascading failures and protect system availability under extreme traffic surges, distributed architectures deploy Circuit Breakers and Adaptive Load Shedding.

By applying Resilience4j State Machines and Google SRE Adaptive Throttling, microservices automatically trip faulty downstream connections and shed excess load before CPU queues collapse.

This article details Circuit Breaker states, sliding-window error evaluation, probe health checks, and Google SRE client-side adaptive throttling.

Circuit Breaker State Machine & Adaptive Throttling Architecture

How Circuit Breaker state transitions protect downstream microservices and how Google SRE Adaptive Throttling dynamically drops excess requests:

Core Resilience Principles

1. Circuit Breaker Mechanics (Resilience4j / Envoy):
   • Closed State: Requests flow normally to the downstream service. The proxy maintains a sliding ring buffer tracking recent request results (success/failure).
   • Open State: If the failure rate exceeds a configurable threshold (e.g. $> 50%$ in a $100\text{-request}$ window), the circuit trips to OPEN. All incoming requests are rejected immediately (Short-Circuited) with a fallback response or HTTP 503, protecting downstream CPU from crashing.
   • Half-Open State: After a wait period (e.g. $10\text{ seconds}$), the circuit transitions to HALF-OPEN. A small sample of probe requests (e.g. $10$ requests) is allowed through. If probe calls succeed, the circuit resets to CLOSED; if any probe fails, it returns to OPEN.
2. Google SRE Adaptive Load Shedding (Algorithmic Throttling):
   • Fixed rate limits fail during sudden traffic spikes. Google SRE defined an Adaptive Client-Side Throttling probabilistic formula: $$P_{\text{rejection}} = \max\left(0, \frac{\text{requests} - K \cdot \text{accepts}}{\text{requests} + 1}\right)$$
   • Parameters:
     • $\text{requests}$: Total requests initiated by the client in the sliding window.
     • $\text{accepts}$: Number of requests accepted and successfully processed by the server.
     • $K$ ($\text{Multiplier}$, default $2.0$): Determines aggressiveness. As long as requests $\le K \cdot \text{accepts}$, no rejection occurs ($P = 0$).
   • As the server begins shedding or timing out requests, $\text{accepts}$ drops, causing $P_{\text{rejection}}$ to rise automatically and shed client traffic before overloading the backend.

Python Implementation: Circuit Breaker & Google SRE Adaptive Load Shedder

Here is a production-grade Python implementation of a Circuit Breaker State Machine and Google SRE Adaptive Load Shedding Engine:

import time
import random
from typing import List
from pydantic import BaseModel

class CircuitBreakerEngine:
    """
    Simulates Resilience4j Circuit Breaker State Machine (Closed, Open, Half-Open).
    """
    def __init__(self, failure_rate_threshold: float = 0.5, wait_duration_sec: float = 2.0, ring_buffer_size: int = 10):
        self.threshold = failure_rate_threshold
        self.wait_duration = wait_duration_sec
        self.buffer_size = ring_buffer_size
        
        self.state = "CLOSED"  # CLOSED, OPEN, HALF_OPEN
        self.sliding_window: List[bool] = []  # True = Success, False = Failure
        self.last_state_change_time = time.time()
        self.half_open_probe_count = 0

    def allow_request(self) -> bool:
        now = time.time()
        
        if self.state == "OPEN":
            if now - self.last_state_change_time >= self.wait_duration:
                self.state = "HALF_OPEN"
                self.half_open_probe_count = 0
                self.last_state_change_time = now
                print(" 🟡 [Circuit Breaker] Wait duration expired -> Transitioning to HALF-OPEN (Testing Probe Requests)")
                return True
            else:
                print(" 🔴 [Circuit Breaker SHORT-CIRCUIT] State: OPEN. Rejecting Request Immediately!")
                return False

        if self.state == "HALF_OPEN":
            if self.half_open_probe_count < 3:
                self.half_open_probe_count += 1
                return True
            return False

        return True  # CLOSED State

    def record_result(self, is_success: bool):
        if self.state == "CLOSED":
            self.sliding_window.append(is_success)
            if len(self.sliding_window) > self.buffer_size:
                self.sliding_window.pop(0)

            # Evaluate failure rate
            failures = self.sliding_window.count(False)
            rate = failures / len(self.sliding_window)
            
            if len(self.sliding_window) >= self.buffer_size and rate >= self.threshold:
                self.state = "OPEN"
                self.last_state_change_time = time.time()
                print(f" 🚨 [Circuit Breaker TRIPPED!] Failure Rate: {rate*100:.1f}% >= {self.threshold*100:.1f}% -> State: OPEN")

        elif self.state == "HALF_OPEN":
            if is_success:
                self.state = "CLOSED"
                self.sliding_window.clear()
                print(" 🟢 [Circuit Breaker RECOVERED] Probe Succeeded -> State: CLOSED")
            else:
                self.state = "OPEN"
                self.last_state_change_time = time.time()
                print(" 🔴 [Circuit Breaker Probe Failed] -> Reverting to State: OPEN")

class GoogleSREAdaptiveLoadShedder:
    """
    Simulates Google SRE Client-Side Adaptive Throttling Algorithm.
    """
    def __init__(self, K: float = 2.0):
        self.K = K
        self.requests = 0
        self.accepts = 0

    def should_drop_request((self) -> bool:
        self.requests += 1
        
        # Calculate Rejection Probability P = Max(0, (requests - K * accepts) / (requests + 1))
        num = self.requests - (self.K * self.accepts)
        den = self.requests + 1.0
        prob = max(0.0, num / den)

        if prob > 0 and random.random() < prob:
            print(f" ⚡ [Google SRE Load Shedding] Request DROPPED! (Rejection Prob: {prob*100:.1f}%)")
            return True

        return False

    def record_accept(self):
        self.accepts += 1

# Demonstration Execution
if __name__ == "__main__":
    cb = CircuitBreakerEngine(failure_rate_threshold=0.5, wait_duration_sec=1.0, ring_buffer_size=6)
    sre_shedder = GoogleSREAdaptiveLoadShedder(K=1.5)

    print("🚀 Demonstrating Circuit Breaker & Adaptive Load Shedding...")
    print("=" * 75)

    # 1. Circuit Breaker Simulation
    print("1. Simulating Downstream Service Outage & Circuit Tripping:")
    for i in range(8):
        if cb.allow_request():
            # Simulate 80% failure rate
            success = i < 2
            cb.record_result(is_success=success)
            print(f"   • Request #{i+1} Executed -> Result: {'Success' if success else 'FAILURE'}")

    # Wait for circuit breaker recovery timeout
    time.sleep(1.1)
    print("\n2. Testing Half-Open Probe Recovery:")
    if cb.allow_request():
        cb.record_result(is_success=True) # Probe succeeds!

    # 2. Google SRE Adaptive Load Shedding Simulation
    print("\n3. Simulating Google SRE Client-Side Adaptive Throttling:")
    for i in range(10):
        if not sre_shedder.should_drop_request():
            # Server accepts only 1 out of 3 requests
            if i % 3 == 0:
                sre_shedder.record_accept()
                print(f"   • Server ACCEPTED Request #{i+1}")
            else:
                print(f"   • Server TIMED OUT Request #{i+1}")

Resilience Gotchas & Best Practices

When configuring resilience mechanisms:

Real-World Enterprise Impact

Distributed systems adopting Resilience4j and Google SRE adaptive throttling (such as Netflix Hystrix, Envoy Proxy, and Google Cloud) report:

• 100% Elimination of Cascading Outages: Short-circuiting faulty downstream services protects upstream thread pools from queue exhaustion.
• Continuous System Availability Under $10\times$ Traffic Spikes: Adaptive load shedding gracefully drops un-processable requests while keeping backend CPU utilization stable.