How to create a custom partitioner for Kafka Producer

We want to create a simple producer example that does the following things.

1. Take the topic name and an integer N as a program argument.
2. Sends N number of simple text messages to the given topic.
3. The message key should start from 1 and increment by one for each new message.
4. Create a custom partitioner that sends all messages with even key to the first partition and odd key to the second partition.

This example will help you understand the process of implementing custom partitioner. To execute this example, you must create a topic with just two partitions. Once your data is partitioned into two groups, you can leverage Kafka consumer APIs to ensure that all odd messages are processed by one consumer and the even messages are processed by the other consumer. The idea is simple, but the same notion is leveraged heavily by the Kafka Streams API for aggregation and other features.
This example will create JSON Serializer to help you understand the details of implementing a custom serializer.
This example is an excerpt from the Book Kafka Streams – Real-time Stream Processing
For a detailed explanation of the example and much more, you can get access to the Book using below link.

Creating Kafka Message Partitioner

The code below shows the odd/even partitioner implementation.

You can access fully function project in our GitHub folder.

Using Kafka Custom Partitioner

Using your custom partitioner is straightforward. All you need to do is to configure the PARTITIONER_CLASS_CONFIG value as shown in the code below.

Read More