We want to rewrite the Json
Serializer and Kafka
example and use out of the box Avro Serializer offered by Confluent community
edition. Basically, we want to do the following.
Use avro-maven-plugin to autogenerate StockData class from Avro schema
Read data from a CSV and load it into a StockData object.
Send StockData objects to Kafka using Confluent Avro Serializer and Confluent
This example uses Confluent Schema Registry and Avro serializer that comes with
Confluent community edition of Apache Kafka. Hence, you must use Confluent Kafka
This example will create JSON Serializer to help you understand the details of
implementing a custom serializer.
This example is an excerpt from the Book Kafka Streams –
Real-time Stream Processing For a detailed explanation of the example and much more, you can get access to
the Book using below link.
Creating Avro Message Schema
The example is mostly the same as we used for JSON serializer. We have taken the
same example and modified it to work with Confluent Avro serializer and schema
The first step is to define Avro schema and generate Avro compatible Java object to
represent your stock data. The code below shows the schema definition.
You can get more details about Avro schema definition syntax at the official Avro
documentation. You can generate Avro compatible Java objects from Avro schema using
avro-maven-plugin. The example on GitHub repository comes with a maven pom.xml file
that contains all the required dependencies.
Reading Avro Messages
The second step is to load your CSV data to your Java object. Code listing 8.6 shows
the method to do the same.
Serializing Avro Messages for Kafka
Once you have the list of objects, all you need is to configure the serializer and
send the objects to the dispatcher who will intern send it to Kafka. The code below shows
the code snippet to do the same.