Installing Kafka on Windows
Kafka application development, debugging and the unit testing process is complicated. You might be set off for developing a Kafka producer, a consumer or a Kafka Streams application. In all cases, you need to install and configure at least four tools on your local machine.
- JDK 1.8
- Single node Kafka cluster
- A build tool such as Maven 3
- An IDE such as IntelliJ IDEA
Item 1, 3 and 4 are everyday activities for an experienced Java developer. However,
installing a single node Kafka cluster on your local machine is a specific activity
for the Kafka developer.
Apache Kafka quick start is very well documented to start on Linux machine. So, your life is easy when you have a Linux or a Mac laptop with an IDE such as IntelliJ IDEA. However, a windows machine is a typical case for a lot of people. In this post, we will cover JDK and a single node Kafka installation on Windows 10 operating system.
Installing JDK 1.8
Download Oracle JDK 1.8 latest update from Oracle tech network website. You may have to accept the license agreement and download an appropriate installer for your system type.
Once you download the appropriate JDK 1.8 installer. Execute
the Installer and follow the on-screen instructions. The installation process
should be relatively straightforward.
Installation wizard allows you to select the JDK installation location. You can
keep the default value for the JDK installation location.
However, remember the place because that is going to be your Java Home.
Once the JDK installation is complete, you must configure your PATH and JAVA_HOME environment variables. You can set up your environment variables using the following steps.
- Start system settings in the control panel. You can directly start it by pressing Win + Pause/Break key.
- Click on Advanced System Settings.
- In the advanced tab, click the environment variable button.
- In the user variables section, add new JAVA_HOME environment variable as your Java installation directory. In a typical case, your Java installation directory should be something like C:\Program Files\Java\jdk1.8.0_191. However, if you have installed JDK as per the steps defined above, you should have noted down your JDK location as advised earlier.
- Now you can add a new PATH environment variable and specify the path as %JAVA_HOME%\bin
The final step is to test your JDK installation. Start windows command prompt and test JDK using below command.
The output should be something similar to below.
java version "1.8.0_191" Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
Configure and Test Kafka
Installing single node Apache Kafka cluster on Windows 10 is as straightforward as doing it on Linux. You can follow the steps defined below to run and test Kafka on Windows 10 operating system.
Configure Kafka for Windows 10
- Download the Kafka binaries. This post is based on Kafka 2.0.0, and hence we assume that you are downloading 2.0.0 version for Scala 2.12.
- Un-compress the kafka_2.12-2.0.0.tgz file. You might want to use 7-zip, and you may have to use it twice to extract the files correctly.
- To make it work, we would be changing the Zookeeper data directory location.
Open kafka_2.12-2.0.0\config\zookeeper.properties file and change the Zookeeper data directory location config to a valid windows directory location. An example value is given below.
data Dir = C:\zookeeper_data
We also need to make some changes in the Kafka configurations.
Open kafka_2.12-2.0.0\config\server.properties and change/add following configuration properties.
log.dirs=C:\kafka_logs offsets.topic.num.partitions = 1 offsets.topic.replication.factor = 1 min.insync.replicas=1 default.replication.factor = 1
The Zookeeper and the Kafka data directories must already exist. So, make sure that
the C:\zookeeper_data and C:\kafka_logs is there.
You might have already learned all the above in the earlier section. The only difference is in topic default values. We are setting topic defaults to one, and that makes sense because we will be running a single node Kafka.
Finally, add Kafka bin\windows directory to the PATH environment variable. This directory contains a bunch of Kafka tools for the windows platform. We will be using some of those in the next section.
Starting and Testing Kafka
Kafka needs Zookeeper, so we will start Zookeeper on your Windows 10 machine using below command.
Start a windows command prompt and execute the above command. It should start the
zookeeper server. Minimize the command window and let the zookeeper running in that
Start a new command window and start Kafka Broker using below command.
The above command should start the single node Kafka Broker. Minimize the window
and let the broker running in that window.
Now, you can test the Zookeeper and Kafka. Execute below command to test the Zookeeper and Kafka broker registration to the Zookeeper server.
kafka_2.12-2.0.0\bin\windows\zookeeper-shell.bat localhost:2181 ls /brokers/ids
You can create a new Kafka topic using below command.
kafka_2.12-2.0.0\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
You can list the available topics using below command.
kafka_2.12-2.0.0\bin\windows\kafka-topics.bat --list --zookeeper localhost:2181
You can start a console producer and send some messages using below command.
kafka_2.12-2.0.0\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
You can start a console consumer and check the messages that you sent using below command.
kafka_2.12-2.0.0\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
In this post, we learned Installing a single-node Kafka broker on your local machine running Windows 10 operating system. This installation will help you to execute your Kafka application code locally and help you debug your application from the IDE. In the next section, we will learn to configure and use an IDE for the Kafka development.