Making of Message Gateway with Kafka - Part 1
The beginning
Few words about the product we are building...
It is SMS message gateway that receives messages from customers , routes and sends them to appropriate SMS service provider which delivers them to end user. It receives delivery reports from service suppliers and sends them back to customer that sent original message. Messages can also be sent in other direction - from end user to customer.
The product has to provide high availability and able to handle high load, in other words horizontal scaling must be supported. Primary protocol used between customers and our product as well as between our product and SMS service providers is SMPP.
This is actually second version of our product. First one was also Spring based, it's using Apache Cassandra for storing messages, Akka is used for processing and PostgreSQL DB for storing configuration.
Main problem we have with current version is that making it horizontally scalable required lot of "manual" workflow orchestration. For example, after message is received, it's written to Cassandra, and than passed to Akka actor that handles routing and sends it to another actor that handles sending it to specific SMS Service provider. If the node crashes before actor managed to process the messages in it's mailbox, that must be detected and those messages must be loaded to another node or reloaded on same node after it's restarted.
With Apache Kafka, we can add or remove application nodes without worrying about migrating data when one of the processors goes down. Kafka client API, more specifically it's consumer groups (logical consumers) abstraction, handles this automatically, we only need to make sure that consumers doing same job have same group id. If one node goes down , another will automatically pick up where the first one left off.
All that complex logic we have in current version handling node crashes or restarts becomes unnecessary.
With KafkaStreams API processing messages requires much less code than we have in current version and everything is cluster ready out-of-the-box.
Kafka also provides persistence and replication so we also don't need Cassandra anymore.
Kotlin has many advantages over Java generally, but there is one specific feature that could be very useful on this project: Kotlin coroutines. Since we need some kind of throttling while sending messages to SMS services providers, we need to make pauses between sends. With Java this is hard to achieve without blocking a thread, but in Kotlin this is very easy by using coroutines delay function. Calling delay on coroutine doesn't block the thread but it's returning it to the thread pool.
In next post we will talk about more specific stuff: Reading messages from customers through single or multiple connections (on same or different application nodes) and aggregating multi-part message parts in clustered environment.
The Message Bus SMS Gateway connects to the Kafka server using the Kafka transport. The Kafka transport runs on the following:
ReplyDelete1) zookeeper-3.4.8.jar
2) kafka-clients-0.10.0.1.jar
Yeah!! SMPP SMS Gateway providers make sure that the bulk messages being sent through an SMPP server are actually received by the intended recipients. There are delivery reports to prove the delivery of messages. Also, the messages remain on standby mode if the devices of the recipients are not used and get delivered once the users switch on their systems. Contact us: +91- 9981122255 or Taniya@MsgClub.co.in
ReplyDeleteVery nice post here thanks for it .I always like and such a super contents of these post. SMSMessenger provide the one of the best Promotional bulk SMS Service to there customers. For the further details you can visit on Promotional Bulk SMS in Chennai
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteThe gateway of Kafka which is shown in the blog is easy & the procedure of how it starts & ends are also much very informative.
ReplyDeleteoffice.com/setup
I should thank you for posting this blog because the subject is trendy today and everybody needs to find out about it. The article is fascinating and it might be valuable for the individuals.Bulk SMS Marketing Features
ReplyDeletepromotional sms is used to send offers, discounts or promotions to new and existing customers.
ReplyDeleteI am an I.T Consultant living in the U.S. I am very much involved in software issues in computers and Mac and providing a suitable solution to customers.
ReplyDeleteoffice.com/setup
office.com/setup
office.com/setup
office.com/setup
office.com/setup
canada immigration consultants in dubai
ReplyDeleteAngstromPod provides best automation marketing platform, can help to automates multi channel marketing campaigns. Automated marketing campaign to fit the better customer engagement & increase sales in Florida & Colorado.
ReplyDeleteSMS Marketing Services Florida
Best Automation Marketing Platform Florida
SMS Marketing Services Florida - Best & Fast Bulk SMS Service Provider In Colorado. For marketing purposes, Bulk SMS plays an Important Role
ReplyDeleteBest Automation Marketing Platform Florida
Email Marketing Platform Colorado
Best Postcard Mailing Services Colorado