Inovatrend blog

In this post we are  taking a deeper look into KafkaStreams API's "through" operation behavior and  potential problems if it is not used carefully. Some time ago I was writing a streaming application using KafkaStreams API and it supposed to do something like this: 1. Read message from "input topic" 2. Apply first transformation 3. Output transformed message to some "side topic" 4. Apply second transformation 5. Output message to "final topic" KStream.through() operation seems logical choice for 3. step - outputting message to side topic and continue to process the message. But if you haven't read this method's javadoc carefully (like I wasn't), you might be surprised with it's behavior. I expected that "through" method will only produce the message to side-topic and continue with following operations on the message. But instead it outputs the message to specified topic and returns new KStream buil

Using aggregate and join operations to handle message receipt confirmations It' been a while since last blog post in this series, but now we continue our journey. After sending SMS messages to service providers, next thing we have to do is to keep track of it's delivery status. There are two asynchronous responses that we should receive from our SMS service providers (aka "suppliers") after we send them a message. 1. Supplier sends us so called "submit response" which confirms that SMS message is received and accepted for further processing on their side. 2. If we requested it in our send request, supplier sends us delivery report for  SMS message. This is report about status of delivery to mobile user. In this post we deal with first one ... Before we go further, note that we support so-called  "long sms" - messages that are sent as several SMPP messages that represent parts of the whole message. In order to support that, we have

Balancing service provider connections across application nodes One of the  challenges with cluster-ready applications is balancing specific processes throughout the cluster nodes. By process I mean part of the application logic that runs in separate thread.  For example, some process can read data from database and export it in PDF files, another example is schedule job that does something periodically (e.g. generating some reports).  In many cases such processes must run only on one node at the same time, but if that node fails, the process should be restarted on another node. Running more then one instance of  same process (for example, each node runs one instance) will cause troubles in many cases. For example , if our process must only process data in certain state, and after processing it state should be updated, having two processes operating on same data could lead to serious data inconsistencies. Our case Our application has to send messages to several different

Assembling multi-part messages using KafkaStreams API What and why?  In previous post we gave general overview of the product we are building - message gateway. In this post we are going to describe how we process message parts and assemble whole messages. One issue with  SMS is that messages are limited to 160 characters. The way SMS protocol overrides that limitation is by using so called "Long SMS messages". Basically, multiple SMS messages are sent as parts of final message which is displayed to end user only after all parts arrive. SMPP protocol supports this feature, so we need to support it in our product as well. Since we are not limited to SMPP  protocol only, we made our own abstraction of multi-part messages. It boils down to this: every message is multi-part and can have one or more parts. Messages with one part are logically not multi-part, but from technical perspective it's easier to treat all messages in same way, so we don't make structural di

The beginning  Here at Inovativni trendovi  we recently started new project that I 'm very excited about. The reason for my excitement is that we are using Apache Kafka on it. We also decided to write it in Kotlin , so this is going to be superfun! Spring-boot based project written in Kotlin, using Apache Kafka. Who can ask for more! Few words about the product we are building... It is SMS message gateway that receives messages from customers , routes and sends them to appropriate SMS service provider which delivers them to end user. It receives delivery reports from service suppliers and sends them back to customer that sent original message. Messages can also be sent in other direction - from end user to customer. The product has to provide high availability and able to handle high load, in other words horizontal scaling  must be supported. Primary protocol used between customers and our product as well as between our product and SMS service providers is SMPP . Thi