Kafka Setup on Windows OS | Basic Installation, Setup, Verification, Cluster Setup, Storage

In this article, we will go through very simple & basic installation of Kafka on windows machine.

Basic Kafka Setup

Apache ZooKeeper

Kafka needs Zookeeper installed & running first. So lets start with installing Zookeeper.

  • Download – Download zookeeper from  https://zookeeper.apache.org/releases.html. It will be GZIP file like “apache-zookeeper-3.5.7-bin.tar.gz“. Feel free to download latest version available.
  • Unzip – Ungzip & then untar downloaded zookeeper at some directory like C:\...\apache-zookeeper-3.5.7-bin. You can use any unzip utility like 7zip to ungzip & untar.
  • Config – Zookeeper needs a config file at location C:\...\apache-zookeeper-3.5.7-bin\conf. So just copy sample file that came with installation from conf/zoo_sample.cfg  as conf/zoo.cfg .
    • If config file is missing then you might see error
  • Data directory – Create “data” folder under zookeeper & change config zoo.cfg to dataDir=C:\...\apache-zookeeper-3.5.7-bin\data
  • Start – then start zookeeper as shown below

Apache Kafka

Now that Zookeeper is installed & running, we can download & install Kafka.

  • Download – Download Kafka from https://kafka.apache.org/quickstart#quickstart_download i.e. “kafka_2.12-2.4.0.tgz
  • Unzip – Unzip downloaded Kafka at location like C:\...\kafka_2.12-2.4.0\
  • Config – Kafka needs config/server.properties which comes with installation with default configurations. We will use the same.
  • Start – Start Kafka as shown below. Note that scripts for windows are located in kafka_2.12-2.4.0\bin\windows

Verify Setup using Windows Command Prompt

Create simple topic using command

Verify created topic by listing topics

Publish/produce message from command prompt

Use this command to publish data to specific topic. Kafka internally store it in file storage directory. Refer Kafka Logs section for more details.

Consume message from command prompt

Use –from-beginning so that you can read all messages from given topic. Provide topic name using –topic.

Kafka Cluster Setup

Setup additional Kafka broker

We can use same installation with different config file to run multiple kafka broker servers on windows. Having multiple brokers as a cluster makes Kafka scalable & highly available. Generally cluster is setup across multiple machines but for simplicity we will setup on same machine.

Keep earlier kafka server running & follow below steps for setting up another server.

  • Config – Copy config/server.properties & paste with different name config/server-1.properties. In this file, modify/add below properties. Basically we are creating separate kafka broker with different id & log directory. We will run this kafka broker on different port.

  • Start – Now start server with this new config file.

Verify cluster

In separate command prompt you can verify that multiple brokers are started.

Create replicated topic

Create topic with replication & partition factor as 2.

Publish record to replicated topic

Consume record from first Kafka broker (Port: 9092)

Consume record from second Kafka broker (Port: 9093)

As you can see that the produced/published message was replicated to both Kafka brokers & was available in both the brokers.

Verify topics & their replications

Use below command to check topics, their partitions, replications etc. You can see in below output that topic ‘HelloCluster’ has 2 replicas. THe earlier created topic ‘Hello’ has a single copy available.

Kafka Logs / Data Storage

In server.properties file under C:\...\kafka_2.12-2.4.0\config , there is property log.dirs=/tmp/kafka-logs . This property indicates where kafka logs i.e. kafka data will be stored. This directory is not for server log statements but it is actual data that is published to topic.

In windows, default kafka log directory or data storage directory is C:\tmp\kafka-logs . Under this directory you can find sub-directories with the names of topics created. Below is files from the topic “Hello”.

Further Reading

Kafka using Java | Basic code examples of Features | Producer, Consumer, Offset, Partition, Replication

Leave a Reply

Your email address will not be published. Required fields are marked *