Skip to content
Kafka Is Fun

Kafka 101 Setup Guide

Installation, configuration, producers, consumers, and clusters

This guide walks through a practical Kafka setup from download to a running single-node or multi-node cluster — starting with the easiest local setup, then moving toward partitioning, producer strategies, consumers, and clustered deployment. Kafka 4.x uses KRaft mode, which removes ZooKeeper and relies on a Raft-based metadata quorum instead.

New to Kafka concepts first? Read Kafka 101 for the “why Kafka exists” primer before working through the setup below.

Start the setupRead Kafka 101 first

What you will set up

By the end of this guide, you will know:

  • Where to download Kafka and what to install
  • How to format storage and start a broker in KRaft mode
  • The server.properties settings that matter for a local setup
  • How to create topics and produce/consume messages from the terminal
  • How Kafka chooses partitions — sticky, key-based, round robin, custom
  • How consumer groups divide up partitions
  • How to stand up a 3-broker cluster with replication
  • The mistakes that trip up most beginners
Jump to a section

1. Where to download Kafka

The official Apache Kafka downloads page publishes the current binary and source releases, and the quickstart guide walks through the latest release workflow. Kafka can be run either from the downloaded files directly or through Docker images.

What to download

  • Binary release — if the goal is to run Kafka locally.
  • Source release — if the goal is to build Kafka from source.
  • Docker image — if the goal is a container-based setup.

For a local installation, download the binary distribution (kafka_2.13-<version>.tgz) from the official downloads page.

2. Prerequisites

The Kafka quickstart requires Java 17 or newer for current releases. Before starting, verify Java is installed and available in the terminal:

bash
java -version

3. Folder structure

macOS / Linux

If Kafka is extracted into /opt/kafka or /usr/local/kafka, the folder structure usually looks like this:

kafka_2.13-4.x.x/
kafka_2.13-4.x.x/
├── bin/
├── config/
├── libs/
├── licenses/
├── site-docs/
└── tars/
  • Kafka root: /usr/local/kafka or /opt/kafka
  • Config files: /usr/local/kafka/config
  • Binaries: /usr/local/kafka/bin
  • Logs: configured by log.dirs in server.properties

Windows

On Windows, the folder structure is the same after extraction, but scripts live under bin\windows\ and use .bat files instead of .sh.

  • Kafka root: C:\kafka\kafka_2.13-4.x.x
  • Config files: C:\kafka\kafka_2.13-4.x.x\config
  • Binaries: C:\kafka\kafka_2.13-4.x.x\bin\windows
  • Logs: configured by log.dirs

4. Single-node Kafka setup

Kafka 4.x uses KRaft mode, which removes ZooKeeper and relies on a Raft-based metadata quorum instead. The quickstart uses this KRaft-based flow for standalone mode.

Step 1: Extract Kafka

bash
tar -xzf kafka_2.13-4.x.x.tgz
cd kafka_2.13-4.x.x

Step 2: Generate a cluster UUID

KRaft requires formatting the log directory with a cluster UUID before starting the server.

bash
bin/kafka-storage.sh random-uuid

Step 3: Format the storage directory

bash
bin/kafka-storage.sh format --standalone -t <CLUSTER_ID> -c config/server.properties

Step 4: Start Kafka

bash
bin/kafka-server-start.sh config/server.properties

5. Windows setup

Windows uses .bat scripts inside bin\windows\ instead of the shell scripts used on macOS/Linux. The same three-step KRaft flow applies:

powershell
# Generate cluster UUID
.\bin\windows\kafka-storage.bat random-uuid

# Format storage
.\bin\windows\kafka-storage.bat format --standalone -t <CLUSTER_ID> -c .\config\server.properties

# Start Kafka
.\bin\windows\kafka-server-start.bat .\config\server.properties

Running Kafka inside WSL2 instead of native Windows avoids the .bat scripts entirely, but brings its own set of gotchas worth knowing before you spend an hour debugging a connection that was never going to work:

  • Clients on Windows can’t always reach a broker in WSL via localhost. If localhost:9092 from a Windows-side client hangs or refuses to connect, try the IPv6 loopback address explicitly ([::1]:9092) — WSL2’s networking sometimes resolves localhost differently than you’d expect across the Windows/Linux boundary.
  • IntelliJ on Windows can’t see a JDK installed inside WSL. If you install Java only inside the WSL distro, a Windows-side IDE won’t find it as a valid SDK. Install a JDK on the Windows side too (or run the IDE inside WSL) rather than assuming one JDK install covers both environments.
  • A broker started inside WSL still needs correct listener config to be reachable. Uncomment and adjust the listeners line in server.properties if the defaults don’t bind to an address your Windows-side tools can actually reach.

6. Important server configuration

The main configuration file is config/server.properties. Common settings for a local single-broker setup look like this:

config/server.propertiesproperties
node.id=1
process.roles=broker,controller
listeners=PLAINTEXT://:9092,CONTROLLER://:19092
advertised.listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:19092
controller.listener.names=CONTROLLER
controller.quorum.bootstrap.servers=localhost:19092
inter.broker.listener.name=PLAINTEXT
log.dirs=/tmp/kafka-logs
Common server.properties settings, explained
PropertyPurpose
node.idUnique node identifier in KRaft mode.
process.rolesTells Kafka whether the process is a broker, controller, or both.
listenersNetwork sockets the server binds to.
advertised.listenersHost/port sent back to clients in metadata.
controller.quorum.bootstrap.serversController bootstrap addresses used by brokers to join the quorum.
log.dirsLocal storage path for logs and segments.

7. Creating topics

Create a topic

bash
bin/kafka-topics.sh --create \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 1

The quickstart demonstrates creating topics with kafka-topics.sh right after the server starts.

Describe a topic

bash
bin/kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092

List topics

bash
bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Delete a topic

bash
bin/kafka-topics.sh --delete --topic orders --bootstrap-server localhost:9092

8. Producer basics

Start a console producer

bash
bin/kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092

The official console producer guide uses the same pattern for sending records from the terminal.

Produce keyed messages

bash
bin/kafka-console-producer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --property parse.key=true \
  --property key.separator=:
example inputtext
customer-101:Order Created
customer-101:Order Paid
customer-202:Order Created

This is the preferred approach when ordering matters for an entity, because all messages with the same key route to the same partition.

Producer configs to know
ConfigWhy it matters
acks=allWaits for leader and ISR acknowledgments.
enable.idempotence=truePrevents duplicate writes on retries.
retriesControls retry attempts.
linger.msImproves batching efficiency.
batch.sizeControls batch size.
compression.typeReduces network and storage cost.

9. Message distribution strategies

Kafka can choose partitions in different ways depending on whether a message has a key and whether a custom partitioner is configured.

1. Sticky partitioner

This is the default for keyless messages. Kafka keeps sending to the same partition temporarily to build efficient batches, then switches when the batch fills or the linger timeout expires.

2. Key-based hashing

When a key exists, Kafka uses a deterministic hash route:

partition = hash(key) % number_of_partitions

3. Round robin partitioner

To force sequential distribution, use the built-in round robin partitioner:

partitioner.class=org.apache.kafka.clients.producer.RoundRobinPartitioner

4. Custom partitioner

A custom partitioner.class can implement routing based on tenant, region, record type, or any application-specific logic.

10. Consumer basics

Start a consumer

bash
bin/kafka-console-consumer.sh --topic orders --bootstrap-server localhost:9092 --from-beginning

Consume with a group id

bash
bin/kafka-console-consumer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --group order-group \
  --from-beginning

Print topic, partition, and key

bash
bin/kafka-console-consumer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --group order-group \
  --from-beginning \
  --property print.topic=true \
  --property print.partition=true \
  --property print.key=true

Consume multiple topics with one group

The Kafka console consumer supports regex-based multi-topic subscription with --include. Use this when you want a single consumer group to read from bothorders and payments.

bash
bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --include "orders|payments" \
  --group my-group \
  --from-beginning \
  --property print.topic=true \
  --property print.partition=true

11. Consumer configs to know

ConfigWhy it matters
group.idDefines the consumer group.
auto.offset.resetDetermines where to start when no offset exists.
enable.auto.commit=falseLets the application control offset commits.
max.poll.recordsLimits batch size per poll.
partition.assignment.strategyControls partition assignment and rebalancing.

12. Consumer group behavior

A consumer group can subscribe to multiple topics, and Kafka distributes all partitions of all subscribed topics across the consumers in the group.

  • A single consumer can read multiple partitions.
  • A single consumer can even receive partitions from multiple topics.
  • A partition can never be processed by two consumers in the same group at the same time.

13. Ordering and partitioning

Kafka guarantees ordering only inside one partition. If you need per-customer or per-order ordering, use a stable key and keep all events for that entity on the same partition.

Practical topic design

Use casePartition strategy
Audit logsKeyless + sticky partitioner
OrdersKey = orderId
PaymentsKey = paymentId or customerId
Multi-tenant workflowsKey = tenantId + business id

14. Increasing complexity: multi-broker cluster

A single broker is fine for learning, but production uses multiple brokers for availability and replication. A simple 3-broker cluster can be created by running three Kafka processes, each with its own server.properties file and port configuration.

Core idea

  • Each partition has one leader and one or more followers.
  • Leaders handle reads/writes.
  • Followers replicate data.
  • If the leader dies, Kafka elects a new leader from the in-sync replicas.

Typical production settings

properties
replication.factor=3
min.insync.replicas=2
auto.create.topics.enable=false
default.replication.factor=3

15. How to spin up a 3-broker cluster

Example cluster layout

BrokerClient portController portLog directory
Broker 1909219092broker1/
Broker 2909319093broker2/
Broker 3909419094broker3/

Broker 1 properties

properties
node.id=1
process.roles=broker,controller
listeners=PLAINTEXT://:9092,CONTROLLER://:19092
advertised.listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:19092
controller.quorum.bootstrap.servers=localhost:19092,localhost:19093,localhost:19094
log.dirs=/tmp/kafka-logs-1

Broker 2 properties

properties
node.id=2
process.roles=broker,controller
listeners=PLAINTEXT://:9093,CONTROLLER://:19093
advertised.listeners=PLAINTEXT://localhost:9093,CONTROLLER://localhost:19093
controller.quorum.bootstrap.servers=localhost:19092,localhost:19093,localhost:19094
log.dirs=/tmp/kafka-logs-2

Broker 3 properties

properties
node.id=3
process.roles=broker,controller
listeners=PLAINTEXT://:9094,CONTROLLER://:19094
advertised.listeners=PLAINTEXT://localhost:9094,CONTROLLER://localhost:19094
controller.quorum.bootstrap.servers=localhost:19092,localhost:19093,localhost:19094
log.dirs=/tmp/kafka-logs-3

Start each broker

Run each broker in a separate terminal:

bash
bin/kafka-server-start.sh config/server1.properties
bin/kafka-server-start.sh config/server2.properties
bin/kafka-server-start.sh config/server3.properties

16. Topic partitioning strategy in a cluster

A simple way to start is to create topics with partition counts that match expected parallelism.

Example
TopicPartitionsReason
orders12Good for scaling consumers later
payments3 or 6Depends on expected load
audit-events1 or 3Ordering may matter more than throughput

17. Common mistakes

  1. 1. Using one partition and expecting many consumers to be active

    If a topic has only one partition, only one consumer in the group can actively process it. Additional consumers will sit idle.

  2. 2. Using random keys

    If every message gets a random UUID key, ordering by entity is broken because each message may route to a different partition.

  3. 3. Enabling auto topic creation in production

    auto.create.topics.enable=true can create accidental topics because of typos such as order vs orders.

  4. 4. Over-partitioning

    Too many partitions create operational overhead and make rebalances slower.

  5. 5. Mixing ordered and unordered workloads

    Keep business-critical ordered topics separate from high-volume log topics.

  6. 6. Forgetting group.id

    Without a consumer group, consumption behavior becomes harder to manage and offset tracking is not coordinated.

  7. 7. Using round robin for business keys

    partitioner.class=org.apache.kafka.clients.producer.RoundRobinPartitioner is great for even load, but not when same-entity ordering matters.

  8. 8. Wrong advertised listener values

    If advertised.listeners points to the wrong host or port, clients can connect initially but fail on metadata refresh.

18. Command cheat sheet

Topic commands

bash
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
bin/kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic orders --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Producer commands

bash
bin/kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092
bin/kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092 --property parse.key=true --property key.separator=:

Consumer commands

bash
bin/kafka-console-consumer.sh --topic orders --bootstrap-server localhost:9092 --from-beginning
bin/kafka-console-consumer.sh --topic orders --bootstrap-server localhost:9092 --group order-group --from-beginning
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --include "orders|payments" --group my-group --from-beginning

Cluster commands

bash
bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-group

19. Final mental model

Think of Kafka in layers:

  1. Download and install Kafka
  2. Start one server
  3. Create a topic
  4. Send and receive messages
  5. Add keys for ordering
  6. Add partitions for parallelism
  7. Add brokers for fault tolerance
  8. Tune consumer groups and rebalancing
  9. Use cluster-safe configs in production