How to create a JSON serializer for Kafka Producer

We want to extend the Kafka Multithreaded Producer example, and instead of dealing with a simple line of text, we want to represent each line as a Java Object. While sending Java Object to Kafka, you must serialize it to byte[]. Hence, we want to create a JSON Serializer using jackson-databind for serializing Java Objects to byte[].
This example will create JSON Serializer to help you understand the details of implementing a custom serializer.
This example is an excerpt from the Book Kafka Streams – Real-time Stream Processing
For a detailed explanation of the example and much more, you can get access to the Book using below link.

Creating JSON Producer for Kafka

We will be using com.fasterxml.jackson.databind library for implementing a JSON serializer. However, you are free to use any other JSON library such as Google’s Gson or something else of your choice. The code below shows a JSON serializer implementation.

You can access fully function project in our GitHub folder.

Using Kafka JSON Serializer

The example data file contains a CSV record. We wanted to read the CSV and convert it into a Java Object. The code below shows a simple function that reads a CSV file and creates a list of StockData object.

Once you have a list of objects, you are ready to use your JsonSerializer. To use your custom json serializer, you must set VALUE_SERIALIZER_CLASS_CONFIG as shown in the code below.

The example project is using jsonschema2pojo-maven-plugin to generate the file using a JSON schema automatically. Rest of the code is straightforward and mostly inherited from our Producer Threads example.

Read More

Author : Prashant Pandey -

You will also like:

Scala named arguments

Learn about named arguments and default values in Scala functions with examples.

Learning Journal

First Class Functions

Function is a first-class citizen in functional programming. What does it mean?

Learning Journal

Pure Function benefits

Pure Functions are used heavily in functional programming. Learn Why?

Learning Journal

Free virtual machines

Get upto six free VMs in Google Cloud and learn Bigdata.

Learning Journal

Referential Transparency

Referential Transparency is an easy method to verify the purity of a function.

Learning Journal