How to build Message Driven Application using Kafka and Spring Boot

In this article, we will create a simple Message Driven Application using Apache Kafka and Spring Boot. We have already seen how we connect to Kafka using plain java clients. In this tutorial, we will see Spring Boot Kafka capability and how it makes your life easier.

Softwares used:

For the demonstration purpose, we will use the web starter project which has support for Spring Boot and Thymeleaf.

Read How quickly you can create your web project with super configuration.

Maven Dependency

Spring has very strong support for Kafka. We will add that dependency to our pom file. Below are the main notable dependencies that our application is going use.

To simulate real-life scenario for our message driven application, we will assume that messages will be published to Kafka topic from external system or source. We will use the command line utility that ships with Kafka.

Consumer Configuration

We need to configure ConsumerFactory and a KafkaListenerContainerFactory to consume messages off Kafka topic. After that, we can use @KafkaListener annotation on any method to turn that to our message receiver.

Note: We also need @EnableKafka annotation in the configuration class. This will enable Spring to detect all our methods that are annotated with @KafkaListener.

The properties for consumer factory can be found in out application properties file.

Kafka Listener Config

Below is our code for the listener.  The method listen is annotated with @KafkaListener. This method will be invoked when a message is posted on topic name opencodez.

Okay, now you must have observed that I have some code related to controller and SseEmitter. Whats it doing there?

Well, if we simply wanted to show the message on console then it would not have been necessary. But we are doing an end to end application, where I want to see the message sent to Kafka topic to be available on frontend UI.

Server-Sent Events

For our example, we want to push messages to frontend as soon as we get it on our topic. So we need to establish a one-way communication from server to the client. Server-sent events are just perfect for that and Spring has inbuilt support for that in the form of SseEmitter.

Let us define our message controller, which will be mapped to one of the URLs in our application. As soon as the user hits that page, an instance of SseEmitter is created and sent back to the client to get data from.

In our controller, we are creating an emitter and adding that to list. When pushing message to client latest emitter from the list is pulled and used.

Also to keep our application safe in memory, we have added some callbacks that remove the emitter from the list once its timed out or the request is marked as complete. (Refer the KafkaListener code as shown earlier)

Server-Sent Events: Client View

We have seen how data is pushed from Server to the client. Now let’s see how its received at the client’s end. The client has to define and configure EventSource. As this may not be supported in all the browser we are using a Jquery-SSE Plugin.

This plugin tries to use the native EventSource object if it supported by the browser. If there is no native support the request is made by ajax requests (polling). You do not need to change the server side nor the client side.

Once we include the required javascript file, you can use above code to bind it to the server.

Running the Application

We are all set now and ready to run our application. Once application is and running visit the page at url:http://localhost:8080/messages

To send messages to our topic, we are using inbuilt producer script from Kafka. Check the screen from running application.

Conclusion

Spring makes it very easy to integrate Kafka with the web application. We have seen how we can develop a Message Driven Application with the help of Spring Boot and Apache Kafka.

Complete source code for this article can be downloaded from our GitHub. Before trying out the code, please make sure that Kafka server is running and the topics are created manually.

Download Code

Add a Comment

Your email address will not be published. Required fields are marked *