Making of Message Gateway with Kafka - Part 1

The beginning 


Here at Inovativni trendovi we recently started new project that I 'm very excited about. The reason for my excitement is that we are using Apache Kafka on it. We also decided to write it in Kotlin, so this is going to be superfun! Spring-boot based project written in Kotlin, using Apache Kafka. Who can ask for more!

Few words about the product we are building...
It is SMS message gateway that receives messages from customers , routes and sends them to appropriate SMS service provider which delivers them to end user. It receives delivery reports from service suppliers and sends them back to customer that sent original message. Messages can also be sent in other direction - from end user to customer.
The product has to provide high availability and able to handle high load, in other words horizontal scaling  must be supported. Primary protocol used between customers and our product as well as between our product and SMS service providers is SMPP.



This is actually second version of our product. First one was also Spring based, it's using Apache Cassandra for storing messages, Akka is used for processing and PostgreSQL DB for storing configuration.

Main problem we have with current version is that making it horizontally  scalable  required lot of "manual" workflow orchestration.  For example, after message is received, it's written to Cassandra, and than passed to Akka actor that handles routing and sends it to another actor that handles  sending it to specific SMS Service provider. If the node crashes before actor managed to process the messages in it's mailbox, that must be detected and those messages must be loaded to another node or reloaded on same node after it's restarted.

With Apache Kafka, we can add or remove application nodes without worrying about migrating data when one of the processors goes down. Kafka client API, more specifically it's consumer groups (logical consumers) abstraction, handles this automatically, we only need to make sure that consumers doing same job have same group id. If one node goes down , another will automatically pick up where the first one left off.

All that complex logic we have in current version handling node crashes or restarts becomes unnecessary.

With KafkaStreams API processing messages requires much less code than we have in current version and everything is cluster ready out-of-the-box.

Kafka also provides persistence and replication so we also don't need Cassandra anymore.

Kotlin has many advantages over Java generally, but there is one specific feature that could be very useful on this project: Kotlin coroutines. Since we need some kind of throttling while sending messages to SMS services providers, we need to make pauses between sends. With Java this is hard to achieve without blocking a thread, but in Kotlin this is very easy by using coroutines delay function. Calling delay on coroutine doesn't block the thread but it's returning it to the thread pool.

In next post  we will talk about more specific stuff: Reading messages from customers through single or multiple connections (on same or different application nodes) and aggregating multi-part message parts in clustered environment.




Comments

  1. The Message Bus SMS Gateway connects to the Kafka server using the Kafka transport. The Kafka transport runs on the following:

    1) zookeeper-3.4.8.jar
    2) kafka-clients-0.10.0.1.jar

    ReplyDelete
  2. Yeah!! SMPP SMS Gateway providers make sure that the bulk messages being sent through an SMPP server are actually received by the intended recipients. There are delivery reports to prove the delivery of messages. Also, the messages remain on standby mode if the devices of the recipients are not used and get delivered once the users switch on their systems. Contact us: +91- 9981122255 or Taniya@MsgClub.co.in

    ReplyDelete
  3. Very nice post here thanks for it .I always like and such a super contents of these post. SMSMessenger provide the one of the best Promotional bulk SMS Service to there customers. For the further details you can visit on Promotional Bulk SMS in Chennai

    ReplyDelete
  4. Hey Guys ! Get rid of Microsoft office issues.

    As we know Microsoft Office is a big software and sometimes users encounter some problems while trying to set up an office and we have already uploaded various articles on how to fix those errors and easily install office on one’s Pc/Mac or any other device which is supported by Microsoft.


    office.com/setup



    office.com/setup

    ReplyDelete
  5. The gateway of Kafka which is shown in the blog is easy & the procedure of how it starts & ends are also much very informative.

    office.com/setup

    ReplyDelete
  6. I should thank you for posting this blog because the subject is trendy today and everybody needs to find out about it. The article is fascinating and it might be valuable for the individuals.Bulk SMS Marketing Features

    ReplyDelete

  7. sms bulk is a term
    that allows the user or company to send out text messages to a
    group of people at the same time. ... This is why bulk SMS is
    also known as SMS marketing as it refers to marketing
    via text messages rather than notifications or alerts

    ReplyDelete
  8. promotional sms is used to send offers, discounts or promotions to new and existing customers.

    ReplyDelete

Post a Comment

Popular posts from this blog

Making of Message Gateway with Kafka - Part 3

Kafka demo at JavaCro 2016 conference