Resilience Patterns with Resilience4j

#Introduction

In a distributed system, something is always failing somewhere — a slow database, a restarting pod, a dropped connection. Resilience means your service keeps working (perhaps in a degraded way) when its dependencies don't. Resilience4j is the standard library for this in Spring Boot.

#How Failures Cascade

The fix isn't one pattern but several working together:

<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot3</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

#1. Timeouts

Every remote call needs a timeout (see Inter-Service Communication). For asynchronous calls, Resilience4j's TimeLimiter adds one:

resilience4j:
  timelimiter:
    instances:
      pricing:
        timeout-duration: 2s
        cancel-running-future: true

#2. Retries

resilience4j:
  retry:
    instances:
      inventory:
        max-attempts: 3                       # 1 call + 2 retries
        wait-duration: 200ms
        enable-exponential-backoff: true
        exponential-backoff-multiplier: 2     # 200 ms, 400 ms…
        enable-randomized-wait: true          # jitter
        retry-exceptions:
          - java.io.IOException
          - org.springframework.web.client.ResourceAccessException
        ignore-exceptions:
          - com.shop.InvalidRequestException  # 4xx — retrying won't help

@Retry(name = "inventory")
public StockLevel stockFor(String sku) {
    return inventoryClient.stockFor(sku);
}

Rules for safe retries:

• Retry transient errors only (timeouts, connection resets, 503) — never validation errors (400) or "not found" (404).
• Use exponential back-off with jitter, so thousands of clients don't retry in lock-step (a retry storm).
• Only retry idempotent operations, or use idempotency keys.
• Keep the total retry time within the caller's timeout — retries multiply latency.
• Don't retry at every layer: if the gateway, service A and service B each retry 3 times, one failure becomes 27 calls.

#3. Circuit Breaker

A circuit breaker stops calling a dependency that is failing, fails fast for a while, then tests whether it has recovered. The state machine (CLOSED → OPEN → HALF_OPEN) is explained in detail in the Circuit Breaker pattern chapter.

resilience4j:
  circuitbreaker:
    instances:
      payment:
        sliding-window-type: COUNT_BASED
        sliding-window-size: 20
        minimum-number-of-calls: 10
        failure-rate-threshold: 50
        slow-call-duration-threshold: 2s
        slow-call-rate-threshold: 50
        wait-duration-in-open-state: 30s
        permitted-number-of-calls-in-half-open-state: 3

@CircuitBreaker(name = "payment", fallbackMethod = "paymentUnavailable")
public PaymentResult charge(PaymentRequest request) {
    return paymentClient.charge(request);
}

private PaymentResult paymentUnavailable(PaymentRequest request, Throwable ex) {
    return PaymentResult.pending(request.orderId());   // accept the order, charge later
}

#4. Bulkhead

Named after a ship's watertight compartments: if one floods, the ship stays afloat. A bulkhead limits how many concurrent calls go to each dependency, so a slow one can't consume every thread.

resilience4j:
  bulkhead:
    instances:
      recommendations:
        max-concurrent-calls: 10
        max-wait-duration: 0          # reject immediately when full

@Bulkhead(name = "recommendations", fallbackMethod = "noRecommendations")
public List<Product> recommendationsFor(String customerId) {
    return recommendationClient.forCustomer(customerId);
}

Resilience4j offers a semaphore bulkhead (above) and a ThreadPoolBulkhead that runs calls on a dedicated pool.

#5. Rate Limiter

Limit how often you call something (to respect a partner's quota) or how often an endpoint can be called:

resilience4j:
  ratelimiter:
    instances:
      sms-provider:
        limit-for-period: 50
        limit-refresh-period: 1s
        timeout-duration: 0

@RateLimiter(name = "sms-provider")
public void sendSms(String to, String text) { … }

#6. Fallbacks and Graceful Degradation

A fallback returns a sensible alternative when a call fails or a breaker is open:

Design what degraded looks like before the outage, so a failure in a non-critical feature never breaks a critical one like checkout.

#Combining Patterns

Resilience4j applies annotations in this order around the call (outermost first):

So each retry attempt is counted by the circuit breaker, and once the breaker opens, retries fail fast instead of hammering the dependency.

@Retry(name = "inventory")
@CircuitBreaker(name = "inventory", fallbackMethod = "stockUnknown")
@Bulkhead(name = "inventory")
public StockLevel stockFor(String sku) {
    return inventoryClient.stockFor(sku);
}

#Observability for Resilience

Resilience4j publishes Micrometer metrics (calls, failures, breaker state, retries, bulkhead usage) and Actuator endpoints (/actuator/circuitbreakers, health indicators). Alert when a circuit opens or a bulkhead is constantly full.

#Key Takeaways

• One slow dependency can take down a whole platform without protection.
• Combine timeouts, retries (with back-off and jitter), circuit breakers, bulkheads and rate limiters.
• Retry only transient failures and idempotent operations; don't retry at every layer.
• Design fallbacks so non-critical failures degrade gracefully.
• Resilience4j annotations rely on proxies; monitor breaker and bulkhead metrics.