Simple and effective way to implement Apache Kafka with Java

Apache Kafka is the buzz word today. Every one talks about it, writes about it. So I have also decided to dive in it and understand it. I will try to put some basic understanding about Apache Kafka and then we will go through a running example.

Is Kafka similar to traditional Message Broker? Well answer to this can be Yes and No. “Yes” because it gives you the similar functionality of traditional brokers. “NO” because it offers you much more functionality than traditional brokers.

Apache Kafka – Concepts

When we talk about Kafka we need to have few things clear.

Cluster:

Kafka is always run as cluster. Cluster is nothing but one instance of Kafka server running on any machine. You can have such many clusters or instances of kafka running on same or different machines. You can specify the protocol and port on which kafka runs in the respective properties file.

Topics:

Kafka treats topics as categories or feed name to which messages are published. The core concept here is similar to traditional broker. On top of that

  1. the kafka topics are always multi-subscriber.
  2. topics can have single or multiple partition which store messages with unique offset numbers
  3. Kafka topics retain the all the published messages whether or not they have been consumed. The records are freed based on configurable retention period.

Apache Kafka Topic

(Image Courtesy : kafka.apache.org)

 Zookeeper:

ZooKeeper is a centralized service for maintaining and providing distributed synchronization, and providing group services. As kafka is distributed as clustered framework, it highly depends on Zookeepr to keep its clusters in sync.

Now lets see how we can actually get some hands on Kafka

Download and Installation

You can download the Kafka distribution from this link. Once downloaded, untar the archive to the folder of your choice. The distribution has scripts for both linux and windows environment. We are using Windows scripts here and default ports and directories.  As mentioned earlier Kafka uses Zookeeper, so you have to first start the Zookeeper server.

Start Zookeeper

After Zookeeper is started its time to start Kafka. For demo purpose we will use single cluster.

Start Kafka

For our testing we will create a topic named “test”. Command for same is

We are ready to connect to this newly created kafka topic and publish and consume some messages. I am using Simple Spring Boot project with Kafka dependencies included. Below are few major dependencies from my pom.xml

In this example I have written simple Conusmer and Producer classes. These classes are implementing Runnable interface and they are producing or consuming from topic that they receive as command line parameter.

Read here on How to pass command line parameters with Spring Boot

The Producer

The Consumer

In the main application, I have injected Simple Task Executor. This executor will start either producer or consumer based on the parameter we send.

The Main Application

As I am using threaded example, I have added log file generation using logback. So when we run our consumer, the log file will get appended with the message read from topic.

Start Java Consumer

This will run the jar and create myapplication.log file

After this our consumer thread is started and it will log the message to the log file when it receives.

Send Message using Producer

Above command sends message to out test topic. And now if you check your log file you will see the message in it. This is logged by our consumer

To confirm same, lets try and run official test script that is distributed with Apache Kafka

Summary

I found Apache Kafka simple to implement. I will try my hands on some more aspects of Apache Kafka and share it with readers. Mean while you can download the source code from our repository.

Please feel free to comment or ask questions.

Download from Git

Add a Comment

Your email address will not be published. Required fields are marked *