Posts

Showing posts from July, 2018

KafkaStreams API "through" operation behavior

Image
In this post we are  taking a deeper look into KafkaStreams API's "through" operation behavior and  potential problems if it is not used carefully.

Some time ago I was writing a streaming application using KafkaStreams API and it supposed to do something like this:

1. Read message from "input topic"
2. Apply first transformation
3. Output transformed message to some "side topic"
4. Apply second transformation
5. Output message to "final topic"

KStream.through() operation seems logical choice for 3. step - outputting message to side topic and continue to process the message.

But if you haven't read this method's javadoc carefully (like I wasn't), you might be surprised with it's behavior.

I expected that "through" method will only produce the message to side-topic and continue with following operations on the message.
But instead it outputs the message to specified topic and returns new KStream built from that topic