How to create a Multithreaded Producer in Java

Assume you have multiple data files. We want to create an application that takes a list of data files as an argument and does following.

  1. Instantiate a Kafka Producer
  2. Create one independent thread to process each data file. For example, if you are supplying three data files to the application, it must create three threads, one for each data file.
  3. Start independent threads with one data file for each thread and share the same Kafka Producer Instance among all threads.
  4. Each thread is responsible for reading records from one file and sending them to Kafka producer in parallel.
  5. Wait for all threads to complete processing and finally, close the application.

This example is an excerpt from the Book Kafka Streams – Real-time Stream Processing
For a detailed explanation of the example and much more, you can get access to the Book using below link.

Creating Multithreaded Kafka Producer

The solution to the given problem can be implemented in two parts.

  1. Create a reusable message dispatcher that can send messages to Kafka.
  2. Create a main application that starts multiple threads using the dispatcher.

Creating a Kafka Message Dispatcher

Code Listing below shows the code snippet for the Dispatcher Class.

Creating Kafka Producer

The next part of the puzzle is to implement a main() method to create threads and start them. The main() method is implemented in the DispatcherDemo class as shown in the Code Listing below.

You can access fully function project in our GitHub folder.

Read More

Author : Prashant Pandey -

You will also like:

Scala named arguments

Learn about named arguments and default values in Scala functions with examples.

Learning Journal

First Class Functions

Function is a first-class citizen in functional programming. What does it mean?

Learning Journal

Pure Function benefits

Pure Functions are used heavily in functional programming. Learn Why?

Learning Journal

Free virtual machines

Get upto six free VMs in Google Cloud and learn Bigdata.

Learning Journal

Referential Transparency

Referential Transparency is an easy method to verify the purity of a function.

Learning Journal