Error Handling, Retries & Dead-Letter Topics

#Introduction

Sooner or later a consumer meets a record it can't process: the database is down, a downstream API times out, or the message itself is broken. How you handle that decides whether one bad record stops the whole partition, gets silently lost, or is parked safely for investigation.

#Transient vs Permanent Failures

Retry transient failures with back-off; don't retry permanent ones — send them to a dead-letter topic (DLT).

#The Poison-Pill Problem

Kafka delivers a partition's records in order. If one record always fails and the consumer keeps retrying it, nothing behind it in that partition is ever processed — the partition is blocked and lag grows forever.

The solution is always the same: bounded retries, then move the record aside (DLT) and continue.

#Spring Kafka's DefaultErrorHandler

When a @KafkaListener throws, the container's DefaultErrorHandler:

1. re-seeks the partition so the failed record is redelivered;
2. retries according to a back-off;
3. when retries are exhausted, calls a recoverer (by default it just logs) and moves on.

Configuring Back-Off and a Dead-Letter Topic

@Configuration
public class KafkaErrorConfig {

    @Bean
    public DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
        // After retries are exhausted, publish the record to <topic>-dlt, same partition
        DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template,
                (record, ex) -> new TopicPartition(record.topic() + "-dlt", record.partition()));

        // 1s, 2s, 4s, 8s … up to 5 attempts
        ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(4);
        backOff.setInitialInterval(1_000);
        backOff.setMultiplier(2.0);
        backOff.setMaxInterval(10_000);

        DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);

        // Permanent failures go straight to the DLT — no pointless retries
        handler.addNotRetryableExceptions(ValidationException.class, JsonProcessingException.class);
        return handler;
    }
}

Spring Boot automatically applies a single CommonErrorHandler bean to its auto-configured listener containers.

What Lands in the DLT

The DeadLetterPublishingRecoverer copies the original key and value and adds headers describing the failure:

#Blocking vs Non-Blocking Retries

The DefaultErrorHandler retries in place: while it waits and retries one record, the rest of that partition waits too. Fine for short delays (seconds) — but a 5-minute back-off would stall the partition and can exceed max.poll.interval.ms.

Non-blocking retries move the failed record to a separate retry topic with a delay, so the main partition keeps flowing:

@RetryableTopic

@Component
public class PaymentListener {

    @RetryableTopic(
            attempts = "4",                                            // 1 try + 3 retries
            backoff = @Backoff(delay = 1000, multiplier = 10.0, maxDelay = 60000),
            exclude = { ValidationException.class },                   // straight to DLT
            topicSuffixingStrategy = TopicSuffixingStrategy.SUFFIX_WITH_INDEX_VALUE)
    @KafkaListener(topics = "payments", groupId = "ledger-service")
    public void onPayment(PaymentEvent event) {
        ledgerService.record(event);
    }

    @DltHandler
    public void onDeadLetter(PaymentEvent event, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
        log.error("Payment {} reached the dead-letter topic {}", event.paymentId(), topic);
        alerting.notifyOnCall(event);
    }
}

Spring Kafka creates the retry and DLT topics, consumers for them, and the delay logic automatically.

#Deserialization Errors

If the value can't be deserialized, the exception happens inside poll() — before your listener runs — and a plain setup fails on the same record forever. Wrap the real deserializer in ErrorHandlingDeserializer:

spring:
  kafka:
    consumer:
      key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
        spring.json.trusted.packages: "com.shop.events"

The failure is turned into a DeserializationException that the error handler treats as not retryable and sends straight to the DLT.

#Handling the Dead-Letter Topic

A DLT is only useful if someone acts on it:

• Alert when records arrive in a DLT.
• Inspect the error headers to find the cause.
• Fix the bug or the data, then replay — a small tool or admin endpoint that reads the DLT and republishes records to the original topic.
• Give DLTs a long retention so nothing expires before it's investigated.

#Producer-Side Failures

#Key Takeaways

• Separate transient failures (retry with back-off) from permanent ones (straight to the DLT).
• Bounded retries + a dead-letter topic prevent poison pills from blocking partitions.
• DefaultErrorHandler + DeadLetterPublishingRecoverer = blocking retries that keep ordering.
• @RetryableTopic = non-blocking delayed retries, at the cost of ordering.
• Wrap deserializers in ErrorHandlingDeserializer.
• Monitor, investigate and replay dead-lettered records.