Apache Kafka Foundation Course - Installing Kafka in Google Cloud

Welcome back to Apache Kafka Tutorial. This video is an update for Apache Kafka 1.x. Many of my viewers don't have access to a Linux machine. Some of them had a Linux machine, but they struggle to download and install JDK and Kafka. This video will help them to quickly setup Apache Kafka in a Google Cloud VM. Google cloud VMs are quite cheap, and if you are a first-time user, they offer one-year free access to various Cloud services. This video will help you to get quick access to latest Kafka VM in Google Cloud. I recommend all my students and followers to gain access to GCP. I have a video tutorial as well to set up your free GCP account . You can take advantage of free GCP service for your learning efforts.
Great. Let's start.

Installing Kafka in Google Cloud

I am assuming you already have access to Google Cloud Account. You can follow these steps to set up a single node Kafka VM in Google Cloud.

  1. Login to your GCP account.
  2. Go to GCP products and services menu.
  3. Click Cloud Launcher
  4. Search for Kafka. You will see multiple options. For a single node setup, I prefer to use the Google VM Image. You can also try the single node Bitnami image. There is a multi-node Bitnami Image as well. However, they designed it for production usage with larger VM configurations. For your learning purposes, the Google image is good enough.
  5. Select the Kafka VM Image.
  6. You will notice the Kafka version, Operating system, and other packages.
  7. Your Kafka usage is free, but you have a cost associated with the VM for CPU, Memory and the disk space. But don't worry about it. Google is charging you on a per hour basis. You also have a free credit for a year.
  8. Click the launch button. You can review and change some configurations, but the default settings are fair enough.
  9. Click the Deploy button at the bottom of the page. That's it. Just wait for few minutes and GCP will launch your single node Kafka VM.
  10. You can SSH to the VM from your deployment page, or you can go to your homepage, then visit the compute engine page, and SSH to your VM.
  11. Once you finish your work, you can select the VM and stop it. Your billing will stop. You can come back next day, pick the VM and start it again.
  12. You Kafka VM comes preconfigured. All services are up and running. You can start using it right away.

Kafka Quick Start

Let's do some simple things. Go to Kafka documentation quick start page. You don't need to perform step 1 and step 2. We already completed the setup, and the services are already running. You can copy the command to create a Kafka topic and execute it. Our Kafka VM comes with appropriate PATH settings. So, we just copy the command. There is no need to specify the path.

Next command is to list the available topics.

You can open two SSH windows. Start a console consumer in one window and a console producer in another window. Let's do that. Start the producer.

Start a consumer.

Now, you can type some messages in the producer window. You will see them arriving in the consumer window. Press Control+C to terminate the consumer and similarly the producer.
If you no longer need the Kafka VM, navigate to deployment manager and delete it. Removing your VM directly, doesn't clean up everything. So, instead of just deleting your VM from compute instance page, make sure to delete it from your deployments page.
Great. That's it for this session.
Thank you for watching Learning Journal. Keep learning and Keep growing.


You will also like: