KafkaStreams API "through" operation behavior

By -


In this post we are  taking a deeper look into KafkaStreams API's "through" operation behavior and  potential problems if it is not used carefully.

Some time ago I was writing a streaming application using KafkaStreams API and it supposed to do something like this:

1. Read message from "input topic"
2. Apply first transformation
3. Output transformed message to some "side topic"
4. Apply second transformation
5. Output message to "final topic"

KStream.through() operation seems logical choice for 3. step - outputting message to side topic and continue to process the message.

But if you haven't read this method's javadoc carefully (like I wasn't), you might be surprised with it's behavior.

I expected that "through" method will only produce the message to side-topic and continue with following operations on the message.
But instead it outputs the message to specified topic and returns new KStream built from that topic.

The "problem" with this implementation is that if "someone else" (another streaming app, or another producer) writes messages to this "side topic", you will start getting output from "through" without ever "calling" it from your streaming application.

Bottom line, when using "through" operation you must be completely aware of how is target topic used in other places in your application and even in some external applications if they have access to this topic.

I wrote small app that demonstrates this behavior. You can find it at our github

For our project this behavior was "show stopper", and we switched to using KafkaProducer for writing  message to side topic.

Comments

  1. UnknownJuly 19, 2018 at 9:40 AM

    You can still accomplish a "side output topic"

    KStream stream = builder.stream("input-topic"):
    stream.to("side-output-topic"); // former "through()"
    stream.filter(); // filter() just an example

    Re-using `stream` twice will "broadcast" (without actual copying) each record into both downstream operators: in the example to() and filter().

    ReplyDelete
    Replies
    1. Igor BuzatovićJuly 23, 2018 at 2:01 PM

      Thanks Matthias, this is definitely more elegant then using KafkaProducer.

      Delete

Post a Comment

Popular posts from this blog

Making of Message Gateway with Kafka - Part 3

By -
Image
Balancing service provider connections across application nodes One of the  challenges with cluster-ready applications is balancing specific processes throughout the cluster nodes. By process I mean part of the application logic that runs in separate thread.  For example, some process can read data from database and export it in PDF files, another example is schedule job that does something periodically (e.g. generating some reports).  In many cases such processes must run only on one node at the same time, but if that node fails, the process should be restarted on another node. Running more then one instance of  same process (for example, each node runs one instance) will cause troubles in many cases. For example , if our process must only process data in certain state, and after processing it state should be updated, having two processes operating on same data could lead to serious data inconsistencies. Our case Our application has to send messages to several different
Read more

Making of Message Gateway with Kafka - Part 1

By -
Image
The beginning  Here at Inovativni trendovi  we recently started new project that I 'm very excited about. The reason for my excitement is that we are using Apache Kafka on it. We also decided to write it in Kotlin , so this is going to be superfun! Spring-boot based project written in Kotlin, using Apache Kafka. Who can ask for more! Few words about the product we are building... It is SMS message gateway that receives messages from customers , routes and sends them to appropriate SMS service provider which delivers them to end user. It receives delivery reports from service suppliers and sends them back to customer that sent original message. Messages can also be sent in other direction - from end user to customer. The product has to provide high availability and able to handle high load, in other words horizontal scaling  must be supported. Primary protocol used between customers and our product as well as between our product and SMS service providers is SMPP . Thi
Read more

Kafka demo at JavaCro 2016 conference

By -
Image
In my  previous post  (published almost a year ago), I tried to summarize basic concepts of  distributed messaging system called Kafka, and to provide very basic tutorial for people who are new to this product. At the end of that post I promised to give another "how to" once 0.9 version is released, version that introduces new consumers API. Version 0.9 is released six months ago, so I am little late with my promise, but better late then never :) Few days ago  JavaCro 2016  conference was held in Rovinj, Croatia, where I presented basic Kafka concepts to (mostly) people of croatian Java community and demonstrated simple spring Boot application using Kafka 0.9.  Since, I already explained basic Kafka concepts in previous post, this one will be focused on only on new consumer API and how to use it. Starting Kafka So lets start with setup. Download Kafka platform 2.0.1 from  http://packages.confluent.io/archive/2.0/confluent-2.0.1-2.11.7.zip  and unzip it to locati
Read more