Kafka 101

Imagine a food-delivery application. You place an order using a mobile app.

After that, several things need to happen — almost all at once:

1. The restaurant must receive the order.
2. The payment service must process the payment.
3. The delivery service must find a delivery partner.
4. The inventory service must update item availability.
5. The notification service must send you an update.
6. The analytics system must record the transaction.
7. The fraud-detection system may need to inspect it.

One action — Order Placed — creates work for many different systems.

The challenge isn’t only sending the data. The system must also handle:

• Services becoming temporarily unavailable
• Sudden traffic spikes
• Duplicate events
• New services being added later
• Events processed at different speeds
• Data needing to be replayed
• Failures without losing important information

That is the problem Apache Kafka helps solve.

Kafka didn’t appear in a world with no messaging or integration tools. Applications already communicated using several approaches.

Order Service
    ├── calls Payment Service
    ├── calls Inventory Service
    ├── calls Notification Service
    ├── calls Delivery Service
    └── calls Analytics Service

Order Service ──┐
Payment Service ├── Shared Database
Inventory       ┤
Reporting       ┘

Every 10 seconds:
SELECT * FROM orders WHERE processed = false;

Producer → Queue → Consumer

Kafka was originally developed at LinkedIn, which needed to move large volumes of activity data between systems — page views, searches, ads, and user interactions. LinkedIn described Kafka as a persistent, efficient, distributed messaging system when it open-sourced the project in 2011.

The goal was a system that could act as a central event backbone. Instead of every application directly integrating with every destination, applications could publish events to Kafka — and other systems could consume those events independently.

Application A ──→ Application B
Application A ──→ Application C
Application A ──→ Application D
Application A ──→ Database E
Application A ──→ Analytics F

Application A ──→ Kafka ──→ Application B
                        ├──→ Application C
                        ├──→ Application D
                        ├──→ Database E
                        └──→ Analytics F

The producer doesn’t need to know every consumer. Consumers don’t need to run at the same speed. A new consumer can be added without changing the original producer. This creates loose coupling between systems.

Kafka as a digital event highway

Imagine a highway used by delivery vehicles.

• Producers place packages onto the highway.
• Kafka organizes and carries those packages.
• Topics represent different routes.
• Partitions are parallel lanes.
• Brokers are distribution centres maintaining those routes.
• Consumers collect the packages they are interested in.
• Offsets identify how far each consumer has travelled.
• Consumer groups divide the collection work between multiple workers.
• Replication keeps backup copies in case one distribution centre fails.
• Retention determines how long packages remain available.

The analogy isn’t technically perfect, but it builds a useful mental model:

Functional perspective

From a business perspective, Kafka sits between event-producing systems and event-consuming systems. It doesn’t usually create the business event itself — it transports, stores, and distributes the event so interested applications can respond.

BUSINESS EVENTS
────────────────────────────
Order placed
Payment completed
User registered
Inventory updated
Shipment dispatched

             │
             ▼

      APACHE KAFKA
       Event Backbone

             │
             ▼

BUSINESS REACTIONS
────────────────────────────
Send notification
Update analytics
Reserve inventory
Start delivery
Check for fraud
Update search index
Store in data warehouse

Functional example

Suppose the Order Service publishes OrderCreated. Different systems react independently:

OrderCreated
    ├── Payment Service starts payment processing
    ├── Inventory Service reserves products
    ├── Notification Service sends confirmation
    ├── Analytics Service records a sale
    ├── Recommendation Service updates preferences
    └── Fraud Service evaluates the transaction

The Order Service doesn’t need to call all six services directly. It publishes one event; Kafka makes that event available to the interested consumers.

The complete flow, in eight steps:

1. Step 1: Something happens

   A business action occurs and the application creates an event.

   Customer places an order
   
   { "type": "ORDER_CREATED", "orderId": "ORD-10045" }

2. Step 2: A producer publishes the event

   The producer chooses a topic and may provide a key.

   Order Service → Kafka
   
   Topic: orders
   Key: customer-501

3. Step 3: Kafka selects a partition

   A topic can be divided into multiple partitions. The event key helps determine which partition receives the event — events with the same key are generally routed to the same partition, so related events stay ordered. Kafka guarantees ordering within a topic-partition, not across every partition in the topic.

   orders
       ├── Partition 0
       ├── Partition 1
       ├── Partition 2
       └── Partition 3

4. Step 4: A broker stores the event

   A broker is a Kafka server; a cluster is a group of brokers. Partitions are distributed across brokers, and the event is appended to the end of its partition log.

   Kafka Cluster              Broker 1 → orders-0
       ├── Broker 1          Broker 2 → orders-1
       ├── Broker 2          Broker 3 → orders-2
       └── Broker 3

5. Step 5: Kafka assigns an offset

   Every event inside a partition receives a position called an offset, meaningful within that specific partition. The combination Topic + Partition + Offset identifies the event's exact position.

   Partition 0
   
   Offset 0 → Order A created
   Offset 1 → Order A paid
   Offset 2 → Order B created
   Offset 3 → Order A shipped

6. Step 6: Consumers read the event

   A consumer subscribes to one or more topics and requests events at its own pace. Kafka tracks a consumer group's committed position so processing can resume from the right offset after a restart.

7. Step 7: Consumer groups share the work

   Within a normal consumer group, each partition is assigned to one consumer at a time — so two consumers in the same group never process the same partition simultaneously. A second, independent consumer group can read the same topic and maintain its own progress.

   orders topic
       ├── Order Processing Group
       ├── Analytics Group
       └── Notification Group

8. Step 8: Events remain available

   Reading an event doesn't delete it. Kafka retains events per the topic's retention configuration — a restarted consumer continues from its previous offset, a new consumer group can read from the start, and a consumer can deliberately replay earlier events.

Kafka stores data, but that doesn’t make it a general replacement for a relational database. Kafka and a database solve different primary problems:

A common architecture uses both

Application
    │
    ├── saves current business state → Database
    │
    └── publishes what happened → Kafka

The database may store the current order state (Order 10045 = SHIPPED) while Kafka holds the history: OrderCreated → PaymentCompleted → InventoryReserved → DeliveryAssigned → OrderShipped. The database answers “what is the current state?”; Kafka can answer “what events happened, and in what order within the relevant partition?”

The full definitions live in the glossary — here’s the compact version you’ll use constantly:

Event / Message

A unit of data sent through Kafka, such as 'order placed' or 'payment completed'.

Producer

The application or service that sends events to Kafka.

Consumer

The application or service that reads events from Kafka.

Broker

A Kafka server that stores events and serves clients.

Cluster

A group of brokers working together for scale and reliability.

Topic

A named stream of events, like orders, payments, or notifications.

Partition

A topic is split into partitions so Kafka can scale and process data in parallel.

Offset

The position of a message inside a partition. Consumers track offsets to know what they have read.

Consumer group

A set of consumers that share the work of reading a topic.

Key

A value used to decide which partition an event goes to; the same key usually keeps order together.

Replication

Copies of partition data stored on multiple brokers for fault tolerance.

Retention

How long Kafka keeps events so they can be read or replayed later.

Example topics: orders, payments, shipments, notifications, clicks, logs.

A frequent beginner misunderstanding is:

Partition 0: A → B → C
Partition 1: X → Y → Z

Kafka preserves A before B before C, and X before Y before Z — but it doesn’t provide one automatic total order across both partitions.

How to preserve ordering for one entity

Use a stable key: Key = orderId. All events for the same order can then be directed to the same partition, preserving the sequence OrderCreated → PaymentCompleted → OrderShipped.

order-101 → P1
order-101 → P1
order-101 → P1

Assume a topic has four partitions: P0 P1 P2 P3

Consumer 1 → P0, P1, P2, P3

Consumer 1 → P0, P1
Consumer 2 → P2, P3

Consumer 1 → P0
Consumer 2 → P1
Consumer 3 → P2
Consumer 4 → P3

Consumer 1 → P0
Consumer 2 → P1
Consumer 3 → P2
Consumer 4 → P3
Consumer 5 → Idle
Consumer 6 → Idle

Adding consumers beyond the available partitions doesn’t increase traditional consumer-group parallelism for that topic.

orders topic
    ├── Group A: Inventory Service
    ├── Group B: Notification Service
    └── Group C: Analytics Service

Each group independently consumes the topic — Kafka doesn’t make the three groups compete for one shared copy.

Suppose a consumer reads an event and then crashes. Kafka doesn’t depend only on the consumer’s memory — the consumer group can continue from its last committed offset. Depending on when the offset was committed, the event may be processed again. This is why Kafka applications should often be designed for idempotency.

Idempotency in simple terms

Processing the same event twice shouldn’t produce an incorrect result. Instead of blindly adding a payment twice, the application can check a unique transaction ID — “has transaction TX-9001 already been processed?” If yes, the duplicate is safely ignored.

Kafka provides delivery and transactional capabilities, but application-level correctness still depends on how the producer, consumer, database, retries, and offset commits are designed.

Older Kafka tutorials frequently mention Apache ZooKeeper. Modern Apache Kafka uses KRaft, Kafka’s built-in metadata quorum. Kafka 4.0 became the first major Kafka release operating entirely without ZooKeeper, and ZooKeeper mode was removed from Kafka 4.x.

For a beginner lesson, the simplified architecture is:

Kafka Cluster
    ├── Brokers store and serve event data
    └── KRaft controllers manage cluster metadata

You may still encounter ZooKeeper in older production systems and older learning material, but new Kafka 4.x explanations should start with KRaft.

Kafka can be a strong choice when a system needs:

• Event-driven microservices

  Services communicate by publishing and reacting to business events (OrderCreated, PaymentCompleted, ShipmentDispatched).

• Real-time data pipelines

  Data continuously moves between applications, databases, search engines, analytics platforms, data warehouses and data lakes.

• High-volume event processing

  The system needs to handle large and continuous streams of data.

• Independent consumers

  Multiple applications need to process the same event for different purposes.

• Replayability

  Past events may need to be processed again — rebuilding a search index, recalculating analytics, recovering after a consumer failure, testing a new service against retained data.

• Application decoupling

  Producers shouldn't require direct knowledge of every downstream consumer.

• Audit-style event history

  A durable sequence of business events is valuable on its own.

Kafka is widely used for high-performance data pipelines, event-driven applications, streaming analytics, and data integration.

Kafka adds operational and architectural complexity. It may be unnecessary when:

• A simple API call solves the requirement.
• Only one small application uses the data.
• The event volume is low.
• Replay is unnecessary.
• Asynchronous processing provides no benefit.
• A normal job queue is sufficient.
• The team cannot operate or monitor a distributed system.
• Immediate synchronous confirmation is mandatory.
• The real requirement is only database storage and querying.

Example

For a small application:

Contact Form → Save to Database → Send Email

Introducing a Kafka cluster may be excessive here — a direct service call or lightweight queue may be easier to build, test, operate, and maintain.

A traditional message queue commonly focuses on distributing jobs or messages to workers. Kafka focuses on maintaining distributed event logs that can be retained, replayed, and consumed by independent groups. Neither approach is universally better — use the architecture that matches the requirement.

Kafka versus RabbitMQ, at a beginner level

This is only a starting comparison. Actual selection depends on durability, ordering, throughput, routing, operational experience, latency, and processing requirements.

Scenario: online order processing

The Order Service produces:

{
  "eventId": "evt-501",
  "eventType": "ORDER_CREATED",
  "orderId": "ORD-10045",
  "customerId": "CUS-20",
  "amount": 1499.00
}

Topic: orders
Key: ORD-10045
→ Partition: 2, Offset: 845

Consumer groups react independently:

inventory-service-group     → reserves inventory
notification-service-group  → sends order confirmation
analytics-service-group     → updates sales analytics
fraud-service-group         → checks suspicious activity

Later, payment completes:

{
  "eventId": "evt-502",
  "eventType": "PAYMENT_COMPLETED",
  "orderId": "ORD-10045",
  "paymentId": "PAY-9009"
}

The same orderId is used as the key, so Kafka routes the related event to the same partition — supporting ordered processing for that order.

Apache Kafka provides a durable and scalable way to move events between systems. Remember this basic flow:

Producer
    ↓
Topic
    ↓
Partition
    ↓
Broker
    ↓
Consumer Group
    ↓
Consumer

And remember these five ideas:

1. A producer publishes an event.
2. The event is written to a topic partition.
3. A broker stores that partition.
4. Consumers read events and track progress using offsets.
5. Different consumer groups can process the same event independently.