How to create Avro messages using Kafka Producer

We want to rewrite the Json Serializer and Kafka Producer example and use out of the box Avro Serializer offered by Confluent community edition. Basically, we want to do the following.

  1. Use avro-maven-plugin to autogenerate StockData class from Avro schema definition.
  2. Read data from a CSV and load it into a StockData object.
  3. Send StockData objects to Kafka using Confluent Avro Serializer and Confluent Schema Registry.

This example uses Confluent Schema Registry and Avro serializer that comes with Confluent community edition of Apache Kafka. Hence, you must use Confluent Kafka community distribution.
This example will create JSON Serializer to help you understand the details of implementing a custom serializer.
This example is an excerpt from the Book Kafka Streams – Real-time Stream Processing
For a detailed explanation of the example and much more, you can get access to the Book using below link.

Creating Avro Message Schema

The example is mostly the same as we used for JSON serializer. We have taken the same example and modified it to work with Confluent Avro serializer and schema registry. The first step is to define Avro schema and generate Avro compatible Java object to represent your stock data. The code below shows the schema definition.

You can access fully function project in our GitHub folder.

You can get more details about Avro schema definition syntax at the official Avro documentation. You can generate Avro compatible Java objects from Avro schema using avro-maven-plugin. The example on GitHub repository comes with a maven pom.xml file that contains all the required dependencies.

Reading Avro Messages

The second step is to load your CSV data to your Java object. Code listing 8.6 shows the method to do the same.

Serializing Avro Messages for Kafka

Once you have the list of objects, all you need is to configure the serializer and send the objects to the dispatcher who will intern send it to Kafka. The code below shows the code snippet to do the same.

Read More

Author : Prashant Pandey -

You will also like: