Making of Message Gateway with Kafka - Part 1

The beginning 


Here at Inovativni trendovi we recently started new project that I 'm very excited about. The reason for my excitement is that we are using Apache Kafka on it. We also decided to write it in Kotlin, so this is going to be superfun! Spring-boot based project written in Kotlin, using Apache Kafka. Who can ask for more!

Few words about the product we are building...
It is SMS message gateway that receives messages from customers , routes and sends them to appropriate SMS service provider which delivers them to end user. It receives delivery reports from service suppliers and sends them back to customer that sent original message. Messages can also be sent in other direction - from end user to customer.
The product has to provide high availability and able to handle high load, in other words horizontal scaling  must be supported. Primary protocol used between customers and our product as well as between our product and SMS service providers is SMPP.



This is actually second version of our product. First one was also Spring based, it's using Apache Cassandra for storing messages, Akka is used for processing and PostgreSQL DB for storing configuration.

Main problem we have with current version is that making it horizontally  scalable  required lot of "manual" workflow orchestration.  For example, after message is received, it's written to Cassandra, and than passed to Akka actor that handles routing and sends it to another actor that handles  sending it to specific SMS Service provider. If the node crashes before actor managed to process the messages in it's mailbox, that must be detected and those messages must be loaded to another node or reloaded on same node after it's restarted.

With Apache Kafka, we can add or remove application nodes without worrying about migrating data when one of the processors goes down. Kafka client API, more specifically it's consumer groups (logical consumers) abstraction, handles this automatically, we only need to make sure that consumers doing same job have same group id. If one node goes down , another will automatically pick up where the first one left off.

All that complex logic we have in current version handling node crashes or restarts becomes unnecessary.

With KafkaStreams API processing messages requires much less code than we have in current version and everything is cluster ready out-of-the-box.

Kafka also provides persistence and replication so we also don't need Cassandra anymore.

Kotlin has many advantages over Java generally, but there is one specific feature that could be very useful on this project: Kotlin coroutines. Since we need some kind of throttling while sending messages to SMS services providers, we need to make pauses between sends. With Java this is hard to achieve without blocking a thread, but in Kotlin this is very easy by using coroutines delay function. Calling delay on coroutine doesn't block the thread but it's returning it to the thread pool.

In next post  we will talk about more specific stuff: Reading messages from customers through single or multiple connections (on same or different application nodes) and aggregating multi-part message parts in clustered environment.




Comments

  1. The Message Bus SMS Gateway connects to the Kafka server using the Kafka transport. The Kafka transport runs on the following:

    1) zookeeper-3.4.8.jar
    2) kafka-clients-0.10.0.1.jar

    ReplyDelete
  2. Yeah!! SMPP SMS Gateway providers make sure that the bulk messages being sent through an SMPP server are actually received by the intended recipients. There are delivery reports to prove the delivery of messages. Also, the messages remain on standby mode if the devices of the recipients are not used and get delivered once the users switch on their systems. Contact us: +91- 9981122255 or Taniya@MsgClub.co.in

    ReplyDelete
  3. Very nice post here thanks for it .I always like and such a super contents of these post. SMSMessenger provide the one of the best Promotional bulk SMS Service to there customers. For the further details you can visit on Promotional Bulk SMS in Chennai

    ReplyDelete

Post a Comment

Popular posts from this blog

Making of Message Gateway with Kafka - Part 3

Making of Message Gateway with Kafka - Part 2