Kafka in Production — Performance, Monitoring & Security

#Introduction

Getting Kafka working locally is easy; running it well in production means choosing the right partition counts, tuning clients for your workload, watching a handful of key metrics, and locking the cluster down. This chapter collects those practices.

#Choosing the Number of Partitions

Partitions are the unit of parallelism — and the hardest setting to change later.

partitions ≥ max(target throughput / throughput per producer partition,
                 target throughput / throughput per consumer)

Example: you need 60 MB/s; one consumer instance processes 5 MB/s → at least 12 partitions (rounding up to allow growth, perhaps 24).

#Tuning Producers

#Tuning Consumers

#Broker and Cluster Sizing

• Disk: daily volume × retention days × replication factor, plus ~30–40% headroom. Fast disks (NVMe/SSD); separate disks or volumes for logs if possible.
• Memory: Kafka relies on the OS page cache — give brokers plenty of RAM but a modest JVM heap (typically 4–8 GB); the rest serves cached reads.
• Network: often the real limit — replication multiplies traffic (RF = 3 means each write is copied twice more).
• Brokers: at least 3 for RF = 3; spread them across availability zones with broker.rack.
• Controllers: a dedicated quorum of 3 (or 5) KRaft controllers for production clusters.

#The Metrics That Matter

Kafka exposes metrics over JMX; collect them with Prometheus (JMX exporter, Kafka exporter for lag), Micrometer in Spring Boot, Burrow, or a managed platform's dashboards, and visualise them in Grafana.

Alert on Symptoms

• 🚨 Consumer lag growing for more than N minutes on a critical group.
• 🚨 Under-replicated or offline partitions > 0.
• 🚨 Disk usage above 80%.
• 🚨 Records arriving in a dead-letter topic.
• 🚨 No messages produced to a topic that should always have traffic.

#Security

By default, a Kafka cluster accepts unencrypted, unauthenticated connections — fine on a laptop, unacceptable in production.

# Allow the order service to write to orders, and the email service to read it as its group
./kafka-acls.sh --bootstrap-server broker:9093 --command-config admin.properties \
  --add --allow-principal User:order-service --operation Write --topic orders

./kafka-acls.sh --bootstrap-server broker:9093 --command-config admin.properties \
  --add --allow-principal User:email-service --operation Read --topic orders --group email-service

# Spring Boot client over SASL_SSL with SCRAM
spring:
  kafka:
    bootstrap-servers: broker1:9093,broker2:9093
    properties:
      security.protocol: SASL_SSL
      sasl.mechanism: SCRAM-SHA-512
      sasl.jaas.config: >
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="${KAFKA_USERNAME}" password="${KAFKA_PASSWORD}";

#Managed Kafka

Running Kafka yourself means handling upgrades, broker failures, rebalancing partitions across brokers, monitoring and security. Managed services — Confluent Cloud, Amazon MSK, Aiven, Azure Event Hubs (Kafka API), Redpanda Cloud — take much of that on. Many teams choose them unless Kafka operations are a core competency.

#Production Checklist

• Replication factor 3, min.insync.replicas=2, unclean leader election off.
• Producers: acks=all, idempotence on, sensible linger.ms/compression, failures logged and alerted.
• Consumers: idempotent processing, bounded retries, dead-letter topics, ErrorHandlingDeserializer.
• Partition counts planned with headroom; meaningful keys.
• Retention sized for the longest consumer outage; disks with headroom.
• Topic auto-creation disabled; topics defined as code.
• Schemas with a registry and compatibility checks.
• TLS + authentication + ACLs; secrets outside code.
• Monitoring: lag, under-replicated partitions, disk, latency; alerts on DLTs.
• Graceful shutdown of producers (close()) and consumers.

#Key Takeaways

• Plan partition counts from consumer throughput, with headroom.
• Tune producers with batching and compression; tune consumers with fetch sizes and batch processing.
• Watch consumer lag, under-replicated/offline partitions, disk, latency and DLT arrivals.
• Secure clusters with TLS, SASL or mTLS, and ACLs.
• Managed Kafka services remove much of the operational burden.