Introduction to Apache Kafka

#Introduction

Apache Kafka is a distributed event-streaming platform. Applications publish streams of events to Kafka, Kafka stores them durably and in order, and other applications subscribe to and process those events — in real time or later.

It was created at LinkedIn in 2010, open-sourced in 2011, and is now an Apache project used by most large technology companies.

#What Is an Event?

An event (also called a record or message) is a fact that something happened:

{
  "eventType": "OrderPlaced",
  "orderId": "ORD-1001",
  "customerId": "C-42",
  "amount": 2499.00,
  "timestamp": "2026-10-15T10:15:30Z"
}

Events are immutable — you don't change "order placed"; you publish a new event such as "order cancelled".

#The Problem Kafka Solves

Without a messaging system, services call each other directly:

This creates problems:

• Tight coupling — the order service must know every consumer and change whenever one is added.
• Cascading failures — if the email service is down, placing an order fails or slows down.
• Load spikes — a sale-day burst hits every downstream service at once.
• No replay — once a call is made, there's no history to reprocess.

With Kafka, the producer publishes once and each consumer reads independently:

#Kafka's Three Capabilities

#Why Kafka Is Fast and Scalable

• Append-only log — writes go to the end of a file, which disks and the OS page cache handle extremely efficiently.
• Partitioning — a topic is split into partitions spread across brokers, so reads and writes scale horizontally.
• Batching and compression — producers send records in compressed batches.
• Zero-copy reads — brokers send data from the page cache to the network without copying it through the application.
• Pull-based consumers — consumers fetch at their own speed; the broker doesn't track per-message delivery state.

A modest cluster handles hundreds of thousands to millions of events per second.

#Messaging Models

Point-to-Point (Queue)

Each message is consumed by one consumer and then removed — like a job queue.

Publish–Subscribe

Each message is delivered to every subscriber.

Kafka Does Both

Kafka uses consumer groups:

• consumers in the same group share the work — each event goes to one consumer in the group (queue-like);
• different groups each receive every event independently (pub/sub-like).

And unlike a traditional queue, reading a message doesn't delete it — it stays until its retention period ends, so new consumers can start from the beginning.

#Common Use Cases

#Kafka vs Other Messaging Systems

#Limitations

• Operational complexity — a production cluster needs capacity planning, monitoring and upgrades (managed services reduce this).
• Ordering only within a partition, not across a whole topic.
• Not a database — no ad-hoc queries; you read sequentially by offset.
• Not ideal for per-message routing, priorities or delays — a traditional broker suits those better.
• Large messages (several MB) hurt performance; store the payload elsewhere and send a reference.

#Key Takeaways

• Kafka is a distributed event-streaming platform: publish, store and process events.
• It decouples services — producers write to topics without knowing who reads them.
• Messages are retained and replayable, not deleted on consumption.
• Consumer groups give both queue and pub/sub semantics.
• It's fast thanks to append-only logs, partitioning, batching, compression and zero-copy.
• Ordering is guaranteed only within a partition.