Posts

Making of Message Gateway with Kafka - Part 3

Image
Balancing service provider connections across application nodes One of the  challenges with cluster-ready applications is balancing specific processes throughout the cluster nodes. By process I mean part of the application logic that runs in separate thread.  For example, some process can read data from database and export it in PDF files, another example is schedule job that does something periodically (e.g. generating some reports). 
In many cases such processes must run only on one node at the same time, but if that node fails, the process should be restarted on another node. Running more then one instance of  same process (for example, each node runs one instance) will cause troubles in many cases. For example , if our process must only process data in certain state, and after processing it state should be updated, having two processes operating on same data could lead to serious data inconsistencies.
Our case Our application has to send messages to several different service prov…

Making of Message Gateway with Kafka - Part 2

Image
Assembling multi-part messages using KafkaStreams API What and why?  In previous post we gave general overview of the product we are building - message gateway. In this post we are going to describe how we process message parts and assemble whole messages.

One issue with  SMS is that messages are limited to 160 characters. The way SMS protocol overrides that limitation is by using so called "Long SMS messages". Basically, multiple SMS messages are sent as parts of final message which is displayed to end user only after all parts arrive.

SMPP protocol supports this feature, so we need to support it in our product as well. Since we are not limited to SMPP  protocol only, we made our own abstraction of multi-part messages. It boils down to this: every message is multi-part and can have one or more parts. Messages with one part are logically not multi-part, but from technical perspective it's easier to treat all messages in same way, so we don't make structural differen…

Making of Message Gateway with Kafka - Part 1

Image
The beginning 
Here at Inovativni trendovi we recently started new project that I 'm very excited about. The reason for my excitement is that we are using Apache Kafka on it. We also decided to write it in Kotlin, so this is going to be superfun! Spring-boot based project written in Kotlin, using Apache Kafka. Who can ask for more!

Few words about the product we are building...
It is SMS message gateway that receives messages from customers , routes and sends them to appropriate SMS service provider which delivers them to end user. It receives delivery reports from service suppliers and sends them back to customer that sent original message. Messages can also be sent in other direction - from end user to customer.
The product has to provide high availability and able to handle high load, in other words horizontal scaling  must be supported. Primary protocol used between customers and our product as well as between our product and SMS service providers is SMPP.



This is actually se…

Kafka demo at JavaCro 2016 conference

Image
In my previous post (published almost a year ago), I tried to summarize basic concepts of  distributed messaging system called Kafka, and to provide very basic tutorial for people who are new to this product.
At the end of that post I promised to give another "how to" once 0.9 version is released, version that introduces new consumers API. Version 0.9 is released six months ago, so I am little late with my promise, but better late then never :)

Few days ago JavaCro 2016 conference was held in Rovinj, Croatia, where I presented basic Kafka concepts to (mostly) people of croatian Java community and demonstrated simple spring Boot application using Kafka 0.9.  Since, I already explained basic Kafka concepts in previous post, this one will be focused on only on new consumer API and how to use it.



Starting Kafka
So lets start with setup.
Download Kafka platform 2.0.1 from http://packages.confluent.io/archive/2.0/confluent-2.0.1-2.11.7.zip and unzip it to location of your choice. …

Apache Kafka - First Contact

Image
Finally I caught some time to dig into Kafka, a high-throughput distributed messaging system, as authors define it.

Essentially, Kafka is publish-subscribe messaging system, it can be used for several use case scenarios but in this post we will focus on tradinional message broker use case.
Kafka stores every message on disk. Yet, it is very fast. If you are interested how is this achieved, read it later :)
It is distributed by design - it is run as a cluster comprised of one or more servers each of which is called a broker.


Important concepts - Kafka stores messages in categories called topics.
- Topics consists of one or more "partitions"
- Each partition is an ordered, immutable sequence of messages that is continually appended to
- Processes that publish messages to Kafka are called producers
- Processes that subscribe to topics and fetch messages are called consumers
- Messages are not deleted after delivery, but after configured period of time.

Partitions are replicated …