Simple Way to Implement Kafka with Java – Source Code on GitHub

Implement Kafka with Java:

Apache Kafka is the buzz word today. Everyone talks about it writes about it. So I have also decided to dive into it and understand it. I will try to put some basic understanding of Apache Kafka and then we will go through a running example.

Is Kafka similar to traditional Message Broker? Well answer to this can be Yes and No. “Yes” because it gives you the similar functionality of traditional brokers. “NO” because it offers you much more functionality than traditional brokers.

Apache Kafka – Concepts

When we talk about Kafka we need to have few things clear.


Kafka is always run as a cluster. Cluster is nothing but one instance of the Kafka server running on any machine. You can have such many clusters or instances of Kafka running on the same or different machines. You can specify the protocol and port on which Kafka runs in the respective properties file.


Kafka treats topics as categories or feed name to which messages are published. The core concept here is similar to a traditional broker. On top of that,

  1. the Kafka topics are always multi-subscriber.
  2. topics can have single or multiple partitions which store messages with unique offset numbers
  3. Kafka topics retain all the published messages whether or not they have been consumed. The records are freed based on the configurable retention period.

Implement Kafka with Java

(Image Courtesy :


ZooKeeper is a centralized service for maintaining and providing distributed synchronization and providing group services. As Kafka is distributed as a clustered framework, it highly depends on Zookeeper to keep its clusters in sync.

Now let’s see how we can actually get some hands-on Kafka

Download and Installation

You can download the Kafka distribution from this link. Once downloaded, untar the archive to the folder of your choice. The distribution has scripts for both Linux and Windows environments. We are using Windows scripts here and default ports and directories.  As mentioned earlier Kafka uses Zookeeper, so you have to first start the Zookeeper server.

Start Zookeeper

After Zookeeper is started its time to start Kafka. For demo purposes, we will use a single cluster.

Start Kafka

For our testing, we will create a topic named “test”. The command for same is:

We are ready to connect to this newly created Kafka topic and publish and consume some messages. I am using the Simple Spring Boot project with Kafka dependencies included. Below are few major dependencies from my pom.xml

In this example, I have written simple Consumer and Producer classes. These classes are implementing the Runnable interface and they are producing or consuming from the topic that they receive as a command-line parameter.

Read here on How to pass command line parameters with Spring Boot

The Producer

The Consumer

In the main application, I have injected Simple Task Executor. This executor will start either producer or consumer based on the parameter we send.

The Main Application

As I am using threaded example, I have added log file generation using log back. So when we run our consumer, the log file will get appended with the message read from the topic.

Start Java Consumer

This will run the jar and create myapplication.log file

After this our consumer thread is started and it will log the message to the log file when it receives.

Send Message using Producer

The above command sends a message to our test topic. And now if you check your log file you will see the message in it. This is logged by our consumer

To confirm same, let’s try and run official test script that is distributed with Apache Kafka


I found Apache Kafka simple to implement. I will try my hands on some more aspects of Apache Kafka and share it with readers. Meanwhile, you can download the source code from our repository.

Please feel free to comment or ask questions.

Download from Git

Add a Comment

Your email address will not be published. Required fields are marked *