Notes: Retry Pattern in Go

In this blog we’ll cover retry pattern, what it is, when to use it and when to avoid it with an example.

What is the Retry Pattern?

The Retry Pattern is a strategy where we retry operations on failure. This is to make sure that business-critical operations don’t fail permanently due to transient errors.

Types of Retry Strategies

There are many patterns that are used as per the requirement.

1. Immediate Retry: In this strategy, we retry immediately on failure. This is helpful when we require low latency and failures are rare.

2. Fixed Delay Retry: In this strategy, we wait for a fixed amount of time between each retry e.g. trying every 2 seconds.

3. Exponential Backoff: In this strategy we increment retry time exponentially e.g. 2^n, which gives retry time 2s, 4s, 8s and so on. This pattern is commonly used with external APIs, distributed locks, and database contention. This also reduces pressure on failing system giving it time to recover and not bombard it with requests.

4. Exponential Backoff with Jitter: This adds a random time in exponential backoff. This is to avoid thundering herd problem in case a large scale system fails and retries start following the same pattern causing sudden load on the system frequently. Jitter adds randomness which helps in distribution of the load. This is the best and safest pattern to use in the most cases.

5. Limited Retry with Circuit Breaker: This is to avoid unnecessary retries on a system which has gone down without any signs of recovery. This is an important pattern to make sure that we are not blocking our clients unnecessarily for a resource which is down.

Real World Example

Let’s say you're calling a payment provider’s API to verify a transaction. You want to retry on transient failures like timeouts, but not on permanent failures like 4xx and some 5xx errors.

package main

import (
    "errors"
    "fmt"
    "math/rand"
    "net/http"
    "time"
)

// mock call to external API
func callExternalAPI() error {
    r := rand.Intn(10)

    if r < 3 {
        return errors.New("timeout")
    } else if r < 6 {
        return &httpError{code: 500}
    } else if r < 8 {
        return &httpError{code: 400}
    }

    return nil
}

type httpError struct {
    code int
}

func (e *httpError) Error() string {
    return fmt.Sprintf("HTTP %d error", e.code)
}

// return whether the request is retryable by verifying response code
// 4xx and 5xx will give permanent failures and exit retry
func isRetryable(err error) bool {
    if err == nil {
        return false
    }

    if e, ok := err.(*httpError); ok {
        return e.code < 400
    }

    return true // timeout, etc.
}

func retry(attempts int, sleep time.Duration, fn func() error) error {
    for i := 0; i < attempts; i++ {
        err := fn()
        if err == nil {
            return nil
        }

        if !isRetryable(err) {
            return fmt.Errorf("permanent error: %w", err)
        }

        // add jitter before retrying
        jitter := time.Duration(rand.Int63n(int64(sleep)))
        fmt.Printf("attempt %d failed: %v, retrying...\n", i+1, err)
        time.Sleep(sleep + jitter)

        sleep *= 2 // exponential backoff
    }

    return fmt.Errorf("all %d retry attempts failed", attempts)
}

func main() {
    rand.Seed(time.Now().UnixNano())

    err := retry(5, time.Second, callExternalAPI)

    if err != nil {
        fmt.Println("final Error:", err)
    } else {
        fmt.Println("call succeeded!")
    }
}

When NOT to Use Retry

• Permanent Failures (e.g. HTTP 4xx, some 5xx)

• Idempotent Operations

• Operations where duplication can occur

• Frequent timeout errors - prefer circuit breakers

• Non-business critical operations where users can manually retry

The Retry Pattern is a powerful resilience pattern when used wisely - especially for unreliable external systems. But used blindly, it can increase latency, load, and even cause data corruption.