Skip to content
Author: ytianle

Resilience Strategies⚓︎

In distributed systems, you may face network issues, hardware failures, or software bugs. resilience strategies are techniques and approaches used to design and implement systems that can withstand and recover from failures, ensuring high availability and reliability.

Types of Resilience Strategies⚓︎

We follow Polly's classification to categorize resilience strategies into two main types:

  • Reactive: These strategies focus on handling errors that the callbacks might throw or return.
  • Retry: Automatically retry a failed operation a certain number of times with a delay between attempts.
  • Circuit Breaker: When system struggles to handle requests, instead of waiting, it allows the system to fail fast and return an error immediately, preventing further strain on the system.
  • Fallback: Provides an alternative response or action when a primary operation fails.
  • Hedging: Send multiple requests to different endpoints and use the first successful response, improving latency and reliability.
  • Proactive: These strategies make proactive decisions to cancel or reject the execution of callbacks:
  • Timeout: Cancel the execution of a callback if it takes too long.
  • Rate Limiter: Limit the number of times a callback can be executed within a certain time frame.

Code Examples⚓︎

These examples use Polly v8.

  • For general-purpose resilience pipelines, using Polly directly is a good choice.
  • For ASP.NET Core HttpClient, prefer Microsoft.Extensions.Http.Resilience, which is built on top of Polly and integrates better with DI and named clients.
  • For Rate Limiter, you usually also need Polly.RateLimiting and System.Threading.RateLimiting.
  • Assume HttpClient httpClient and CancellationToken cancellationToken already exist in the surrounding code.
  • In the examples below, highlighted lines show the core resilience configuration.

Retry⚓︎

Use retry for transient failures such as temporary network glitches or short-lived 5xx responses.

using Polly;
using Polly.Retry;

var retryPipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
  .AddRetry(new RetryStrategyOptions<HttpResponseMessage>
  {
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
      .Handle<HttpRequestException>()
      .HandleResult(response => (int)response.StatusCode >= 500),
    MaxRetryAttempts = 3,
    Delay = TimeSpan.FromSeconds(1),
    BackoffType = DelayBackoffType.Exponential,
    UseJitter = true
  })
  .Build();

HttpResponseMessage response = await retryPipeline.ExecuteAsync(
  async ct => await httpClient.GetAsync("https://api.example.com/orders", ct),
  cancellationToken);

Circuit Breaker⚓︎

Use circuit breaker when a downstream dependency is already unhealthy and you want to fail fast instead of repeatedly hammering it.

using Polly;
using Polly.CircuitBreaker;

var circuitBreakerPipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
  .AddCircuitBreaker(new CircuitBreakerStrategyOptions<HttpResponseMessage>
  {
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
      .Handle<HttpRequestException>()
      .HandleResult(response => (int)response.StatusCode >= 500),
    FailureRatio = 0.5,
    SamplingDuration = TimeSpan.FromSeconds(10),
    MinimumThroughput = 8,
    BreakDuration = TimeSpan.FromSeconds(30)
  })
  .Build();

HttpResponseMessage response = await circuitBreakerPipeline.ExecuteAsync(
  async ct => await httpClient.GetAsync("https://inventory.example.com/stock", ct),
  cancellationToken);

Fallback⚓︎

Use fallback when the primary action fails and you can still return a safe substitute, such as cached data or a default response.

using System.Net;
using Polly;
using Polly.Fallback;

var fallbackPipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
  .AddFallback(new FallbackStrategyOptions<HttpResponseMessage>
  {
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
      .Handle<HttpRequestException>()
      .HandleResult(response => (int)response.StatusCode >= 500),
    FallbackAction = static args => Outcome.FromResultAsValueTask(
      new HttpResponseMessage(HttpStatusCode.OK)
      {
        Content = new StringContent("{\"source\":\"fallback-cache\"}")
      })
  })
  .Build();

HttpResponseMessage response = await fallbackPipeline.ExecuteAsync(
  async ct => await httpClient.GetAsync("https://pricing.example.com/catalog", ct),
  cancellationToken);

Hedging⚓︎

Use hedging when latency matters and sending another attempt to an alternate endpoint can reduce tail latency.

using Polly;
using Polly.Hedging;

var hedgingPipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
  .AddHedging(new HedgingStrategyOptions<HttpResponseMessage>
  {
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
      .Handle<HttpRequestException>()
      .HandleResult(response => (int)response.StatusCode >= 500),
    MaxHedgedAttempts = 2,
    Delay = TimeSpan.FromMilliseconds(200),
    ActionGenerator = args =>
    {
      var endpoints = new[]
      {
        "https://api-east.example.com/search",
        "https://api-west.example.com/search",
        "https://api-central.example.com/search"
      };

      var endpoint = endpoints[Math.Min(args.AttemptNumber, endpoints.Length - 1)];

      return async () =>
      {
        try
        {
          var response = await httpClient.GetAsync(endpoint, args.ActionContext.CancellationToken);
          return Outcome.FromResult(response);
        }
        catch (Exception ex)
        {
          return Outcome.FromException<HttpResponseMessage>(ex);
        }
      };
    }
  })
  .Build();

HttpResponseMessage response = await hedgingPipeline.ExecuteAsync(
  async ct => await httpClient.GetAsync("https://api-east.example.com/search", ct),
  cancellationToken);

Timeout⚓︎

Use timeout when an operation should not be allowed to wait forever.

using Polly;
using Polly.Timeout;

var timeoutPipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
  .AddTimeout(TimeSpan.FromSeconds(2))
  .Build();

HttpResponseMessage response = await timeoutPipeline.ExecuteAsync(
  async ct => await httpClient.GetAsync("https://reporting.example.com/summary", ct),
  cancellationToken);

Rate Limiter⚓︎

Use rate limiting to protect a shared dependency from excessive load.

using Polly;
using Polly.RateLimiting;
using System.Threading.RateLimiting;

var limiter = new SlidingWindowRateLimiter(new SlidingWindowRateLimiterOptions
{
  PermitLimit = 100,
  Window = TimeSpan.FromMinutes(1),
  SegmentsPerWindow = 4,
  QueueLimit = 0
});

var rateLimiterPipeline = new ResiliencePipelineBuilder()
  .AddRateLimiter(limiter)
  .Build();

await rateLimiterPipeline.ExecuteAsync(
  async ct => await SendMessageToDownstreamAsync(ct),
  cancellationToken);

Combined Example⚓︎

In real systems, you often compose several strategies together instead of using only one.

using Polly;
using Polly.Fallback;
using Polly.Retry;
using Polly.Timeout;

var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
  .AddFallback(new FallbackStrategyOptions<HttpResponseMessage>
  {
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
      .Handle<HttpRequestException>()
      .HandleResult(r => (int)r.StatusCode >= 500),
    FallbackAction = static args => Outcome.FromResultAsValueTask(
      new HttpResponseMessage(System.Net.HttpStatusCode.OK)
      {
        Content = new StringContent("fallback response")
      })
  })
  .AddRetry(new RetryStrategyOptions<HttpResponseMessage>
  {
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
      .Handle<HttpRequestException>()
      .HandleResult(r => (int)r.StatusCode >= 500),
    MaxRetryAttempts = 3,
    Delay = TimeSpan.FromMilliseconds(300)
  })
  .AddTimeout(TimeSpan.FromSeconds(2))
  .Build();

HttpResponseMessage response = await pipeline.ExecuteAsync(
  async ct => await httpClient.GetAsync("https://api.example.com/data", ct),
  cancellationToken);

Reference⚓︎