Making of Message Gateway with Kafka - Part 1

The beginning 

Here at Inovativni trendovi we recently started new project that I 'm very excited about. The reason for my excitement is that we are using Apache Kafka on it. We also decided to write it in Kotlin, so this is going to be superfun! Spring-boot based project written in Kotlin, using Apache Kafka. Who can ask for more!

Few words about the product we are building...
It is SMS message gateway that receives messages from customers , routes and sends them to appropriate SMS service provider which delivers them to end user. It receives delivery reports from service suppliers and sends them back to customer that sent original message. Messages can also be sent in other direction - from end user to customer.
The product has to provide high availability and able to handle high load, in other words horizontal scaling  must be supported. Primary protocol used between customers and our product as well as between our product and SMS service providers is SMPP.

This is actually second version of our product. First one was also Spring based, it's using Apache Cassandra for storing messages, Akka is used for processing and PostgreSQL DB for storing configuration.

Main problem we have with current version is that making it horizontally  scalable  required lot of "manual" workflow orchestration.  For example, after message is received, it's written to Cassandra, and than passed to Akka actor that handles routing and sends it to another actor that handles  sending it to specific SMS Service provider. If the node crashes before actor managed to process the messages in it's mailbox, that must be detected and those messages must be loaded to another node or reloaded on same node after it's restarted.

With Apache Kafka, we can add or remove application nodes without worrying about migrating data when one of the processors goes down. Kafka client API, more specifically it's consumer groups (logical consumers) abstraction, handles this automatically, we only need to make sure that consumers doing same job have same group id. If one node goes down , another will automatically pick up where the first one left off.

All that complex logic we have in current version handling node crashes or restarts becomes unnecessary.

With KafkaStreams API processing messages requires much less code than we have in current version and everything is cluster ready out-of-the-box.

Kafka also provides persistence and replication so we also don't need Cassandra anymore.

Kotlin has many advantages over Java generally, but there is one specific feature that could be very useful on this project: Kotlin coroutines. Since we need some kind of throttling while sending messages to SMS services providers, we need to make pauses between sends. With Java this is hard to achieve without blocking a thread, but in Kotlin this is very easy by using coroutines delay function. Calling delay on coroutine doesn't block the thread but it's returning it to the thread pool.

In next post  we will talk about more specific stuff: Reading messages from customers through single or multiple connections (on same or different application nodes) and aggregating multi-part message parts in clustered environment.