Kafka Consumers & Consumer Groups

#Introduction

A consumer reads records from Kafka by polling. Consumers that share a group.id form a consumer group and divide the topic's partitions between them. This chapter covers the consumer loop and how groups distribute work; committing offsets and rebalancing get their own chapter next.

#The Poll Loop

#A Plain Java Consumer

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "email-service");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(List.of("orders"));

    while (running) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> record : records) {
            System.out.printf("partition=%d offset=%d key=%s value=%s%n",
                    record.partition(), record.offset(), record.key(), record.value());
            sendEmail(record.value());
        }
    }
}

partition=0 offset=41 key=ORD-2 value={"status":"PLACED"}
partition=2 offset=17 key=ORD-1 value={"status":"PLACED"}
partition=2 offset=18 key=ORD-1 value={"status":"PAID"}

poll() does much more than fetch: it joins the group, sends heartbeats (on a background thread), triggers rebalances and, with auto-commit, commits offsets.

Runtime.getRuntime().addShutdownHook(new Thread(() -> {
    running = false;
    consumer.wakeup();
}));

#Consumer Groups

All consumers with the same group.id share the topic. A broker acting as the group coordinator tracks the members and their partition assignments.

Scaling a Group

Adding consumers increases throughput only up to the partition count. When consumers join, leave or crash, the group rebalances — partitions are redistributed (see the next chapter).

Multiple Groups

Each group has its own offsets and receives every record, independently.

#Partition Assignment Strategies

The assignor decides which consumer gets which partitions (partition.assignment.strategy):

The default is [RangeAssignor, CooperativeStickyAssignor]. With Kafka 4's new consumer group protocol (group.protocol=consumer), assignment moves to the broker and is incremental by default — covered in the next chapter.

#Where Does a New Group Start? auto.offset.reset

When a group has no committed offset for a partition (a brand-new group, or its offsets expired), auto.offset.reset decides where to start:

Once offsets are committed, the group always resumes from them.

#Key Consumer Settings

max.poll.interval.ms — the Classic Trap

Heartbeats run on a background thread, so a consumer can be "alive" yet stuck. If processing one poll() batch takes longer than max.poll.interval.ms, the consumer is kicked out of the group, its partitions are reassigned, and its commit fails — the same records are reprocessed by another consumer, again and again.

Fixes:

• reduce max.poll.records so each batch finishes sooner;
• make processing faster (batch database writes, avoid slow external calls);
• raise max.poll.interval.ms if processing is legitimately slow;
• move very slow work to another system (e.g. a job queue).

#Processing with Multiple Threads

The partition count limits parallelism per group. When processing is slow (calls to other services), options are:

1. More partitions and consumers — the simplest, keeps per-key ordering.
2. Spring Kafka concurrency — several consumer threads in one application instance (each with its own consumer).
3. Hand off to a worker pool — one consumer thread, many workers. Ordering per key and correct offset commits become your responsibility; libraries such as Confluent's Parallel Consumer do this safely.

#Key Takeaways

• Consumers poll; poll() also handles group membership and (by default) offset commits.
• A partition is assigned to exactly one consumer per group; parallelism ≤ partitions.
• Different groups consume independently with their own offsets.
• auto.offset.reset applies only when no committed offset exists.
• Keep each poll batch well within max.poll.interval.ms.
• KafkaConsumer is single-threaded; scale with partitions, consumers or careful worker pools.