# Kafka Streams in Action: Real-time apps and microservices with the Kafka Streams API

An episode of [CyberSecurity Summary](https://invisiblenetwork.ai/podcast/1816777677), 22 Oct 2026, 22 min.

Audio: https://dts.podtrac.com/redirect.mp3/api.spreaker.com/download/episode/75148317/the_distributed_architecture_of_kafka_streams.mp3

A Java library designed for building real-time, event-driven applications and microservices. The text explores the evolution of data processing, moving from traditional batch processing to modern stream processing where data is handled immediately as it arrives. Readers are introduced to the core architecture of Apache Kafka, including fundamental components like producers, consumers, topics, and partitions. The documentation uses a retail example from a fictional company, ZMart, to illustrate how to construct a topology of processing nodes for tasks such as data masking and rewards calculations. Furthermore, the material covers advanced topics including stateful operations, the KTable API, and interactive queries for monitoring data flows. Overall, the text serves as a technical guide for developers to master high-throughput, fault-tolerant stream processing within the Kafka ecosystem. You can listen and download our episodes for free on more than 10 different platforms: https://linktr.ee/cyber_security_summary Get the Book now from Amazon: https://www.amazon.com/Kafka-Streams-Action-William-Bejeck/dp/1617294470?&linkCode=ll2&tag=cvthunderx-20&linkId=f6976ad8c63c87b3a55170f6e7026b97&language=en_US&gaOptInStatus=true&ref_=as_li_ss_tl Discover our free courses in tech and cybersecurity, Start learning today: https://linktr.ee/cybercode_academy

## Transcript

[0:00] Speaker 1: Imagine for just a second that you are buying a
[0:02] Speaker 1: tube of toothpaste.
[0:04] Speaker 2: Just, you know, a totally standard Tuesday afternoon errand.
[0:07] Speaker 1: Exactly. You are at a massive retail chain. Let's call
[0:11] Speaker 1: it Zmart. You are at the register. You tap your
[0:15] Speaker 1: debit card on the reader and you hear that little beep.
[0:19] Speaker 2: And to you, that beep is just a millisecond.
[0:22] Speaker 1: Right.
[0:22] Speaker 2: It's nothing. Nothing.
[0:23] Speaker 1: But in that exact same millisecond, there is this incredibly
[0:26] Speaker 1: complex chain reaction firing off across a distributed network.
[0:30] Speaker 2: Oh, absolutely.
[0:31] Speaker 1: I mean your credit card number is instantly masked for privacy.
[0:34] Speaker 1: Your loyalty points are calculated and written to your profile.
[0:38] Speaker 1: Regional purchase trends are updated to reflect a spike in
[0:41] Speaker 1: toothpaste sales. In your specific ZIP code.
[0:45] Speaker 2: And, well, the entire transaction is securely archived for auditing, too. Yeah,
[0:48] Speaker 2: all of that. It really is this massive invisible ballet
[0:51] Speaker 2: of data choreography. We take it completely for granted because,
[0:54] Speaker 2: you know, we're so seamlessly on the front end. But
[0:57] Speaker 2: getting a system to process millions of those concurrent transactions
[1:01] Speaker 2: in real time without just buckling under the sheer I.O. load,
[1:05] Speaker 2: that required completely rethinking everything. how we design distributed systems.
[1:09] Speaker 1: Which is exactly why you are listening to this custom-tailored
[1:12] Speaker 1: deep dive today. When you hear that beep, you are
[1:16] Speaker 1: kicking off a massive chain reaction.
[1:18] Speaker 2: You really are.
[1:19] Speaker 1: And to understand how a system handles millions of those
[1:22] Speaker 1: reactions a second without melting down... We're pulling apart the
[1:26] Speaker 1: architecture laid out in William P. Bedrick Jr. 's book,
[1:29] Speaker 1: Kafka Streams in Action.
[1:30] Speaker 2: It's a fantastic resource for this.
[1:32] Speaker 1: It really is. OK, let's unpack this, because before we
[1:34] Speaker 1: can appreciate how Kafka Streams processes infinite data in milliseconds today,
[1:39] Speaker 1: we have to look back at the original architectural bottleneck
[1:43] Speaker 1: that forced this evolution in the first place.
[1:45] Speaker 2: Yeah, going back to around 1998 really sets the right
[1:48] Speaker 2: context here.
[1:48] Speaker 1: The dawn of the big data era.
[1:50] Speaker 2: Exactly. Google had just been founded. And they introduced the
[1:54] Speaker 2: PageRank algorithm. Their engineering challenge was totally unprecedented. I mean,
[1:59] Speaker 2: they didn't just need to store the internet.
[2:01] Speaker 1: Right, which is hard enough. Right.
[2:03] Speaker 2: But they needed to evaluate the importance of a website
[2:06] Speaker 2: by computing the number and quality of inbound links across
[2:09] Speaker 2: the entire web graph.
[2:11] Speaker 1: And traditional relational databases, you know, running on massive, expensive,
[2:15] Speaker 1: monolithic servers, they simply couldn't handle that compute load.
[2:19] Speaker 2: Not even close. Yeah. The disk seeks alone would take months.
[2:22] Speaker 2: So they had to invent a completely new processing paradigm,
[2:26] Speaker 2: which eventually became MapReduce.
[2:27] Speaker 1: And MapReduce borrowed heavily from functional programming, right?
[2:30] Speaker 2: It did. Instead of moving massive amounts of data to
[2:33] Speaker 2: the processing code, which just saturates your network bandwidth immediately,
[2:39] Speaker 2: MapReduce moved the code to the data.
[2:41] Speaker 1: Oh, wow.
[2:42] Speaker 2: Yeah, it utilized a massive scatter-gather operation. You distribute chunks
[2:46] Speaker 2: of the web index across thousands of cheap commodity machines.
[2:49] Speaker 1: Okay, so I want to use an analogy here for you.
[2:51] Speaker 1: It's like having a literal mountain of pennies to count, right?
[2:54] Speaker 2: Sure, yeah.
[2:55] Speaker 1: Instead of doing it yourself, you divide the pennies among
[2:57] Speaker 1: a thousand friends. That's distributing the load.
[3:00] Speaker 2: Exactly. And then you have them count their individual piles.
[3:03] Speaker 1: Right, which is the map function running locally. Yes.
[3:06] Speaker 2: And then they just add all their final numbers together
[3:09] Speaker 2: into one big total.
[3:10] Speaker 1: Which is the reduce function. It's essentially parallelizing the workload
[3:14] Speaker 1: across a cluster.
[3:15] Speaker 2: That's spot on. And a byproduct of relying on thousands
[3:19] Speaker 2: of cheap commodity servers was that, well, hardware failure became
[3:24] Speaker 2: a mathematical certainty.
[3:25] Speaker 1: Right. Something is going to break. Always.
[3:27] Speaker 2: So instead of buying million-dollar storage area networks with custom
[3:30] Speaker 2: hardware redundancy, they baked fault tolerance directly into the software layer.
[3:35] Speaker 1: By replicating the data across multiple physical nodes.
[3:39] Speaker 2: Exactly. That distributed compute model, which became open-sourced as Hadoop,
[3:44] Speaker 2: it really solved the big data storage and processing problem.
[3:47] Speaker 2: But it introduced a new constraint.
[3:48] Speaker 1: Time.
[3:49] Speaker 2: Time, exactly. MapReduce is fundamentally a batch processing engine. It
[3:53] Speaker 2: requires data to be bounded.
[3:54] Speaker 1: Meaning you collect logs for a day, load them into
[3:57] Speaker 1: the distributed file system, run a massive MapReduce job overnight,
[4:00] Speaker 1: and then you look at the results the next morning.
[4:02] Speaker 2: Right, which works beautifully for, like, building a web search
[4:06] Speaker 2: index or running a monthly payroll.
[4:08] Speaker 1: But if Zmart wants to detect a fraudulent credit card
[4:11] Speaker 1: swipe or dynamically adjust pricing based on real-time inventory levels...
[4:16] Speaker 1: A 12-hour batch delay is totally useless.
[4:20] Speaker 2: It's worse than useless. It's a liability.
[4:22] Speaker 1: Yeah.
[4:22] Speaker 2: The business requirement shifted from historical analysis to instantaneous reaction.
[4:28] Speaker 1: And that operational shift required abandoning batches entirely, right? In
[4:31] Speaker 1: favor of stream processing. Yeah.
[4:33] Speaker 2: In stream processing, you are dealing with an infinite, unbounded
[4:36] Speaker 2: flow of data. The computation happens continuously as the events arrive.
[4:41] Speaker 1: So there is no waiting for a batch window to close. None.
[4:44] Speaker 2: And crucially, there is no micro-batching either, where you group
[4:48] Speaker 2: events into tiny one-second intervals. It is true, per event processing.
[4:52] Speaker 1: You see this in high-frequency trading algorithms where, you know,
[4:55] Speaker 1: a millisecond delay costs millions.
[4:57] Speaker 2: Or in network security monitoring, where intrusion signatures need immediate isolation.
[5:02] Speaker 2: Or even tracking runners via shoe chips in a marathon.
[5:04] Speaker 1: Oh, yeah. You cross a checkpoint, your family gets a
[5:07] Speaker 1: text instantly. You don't want to batch text an hour
[5:09] Speaker 1: later saying you finished.
[5:11] Speaker 2: Exactly. But obviously... Stream processing isn't a silver bullet. If
[5:15] Speaker 2: a university is analyzing the five-year efficacy of a new
[5:18] Speaker 2: math curriculum or an economist is running complex historical regressions.
[5:23] Speaker 1: They don't need real-time streams.
[5:24] Speaker 2: No, they need the deep historical aggregation that gotch processing
[5:28] Speaker 2: excels at. Stream processing is strictly for the now. It's
[5:32] Speaker 2: for extracting immediate value from data in motion before it
[5:35] Speaker 2: comes to rest in a database.
[5:37] Speaker 1: Okay, but I have a pushback here. If the data
[5:39] Speaker 1: is literally infinite and never stops, like every transaction across
[5:43] Speaker 1: thousands of Zmart stores worldwide flowing in continuously, there has
[5:47] Speaker 1: to be a physical limit. If the system is trying
[5:50] Speaker 1: to calculate loyalty points and run fraud checks on an
[5:52] Speaker 1: endless firehose, I assume the downstream applications eventually get overwhelmed.
[5:57] Speaker 1: How do you prevent the system from hitting back pressure
[5:59] Speaker 1: where it just throws its hands up, blocks the incoming data?
[6:02] Speaker 1: And crashes the pipeline.
[6:04] Speaker 2: What's fascinating here is how Kafka Streams circumvents that back
[6:07] Speaker 2: pressure bottleneck using graph theory, specifically by modeling the application
[6:12] Speaker 2: as a directed acyclic graph, or DAGE. A DAGE, okay.
[6:16] Speaker 2: Think of a DAG as a processing topology. The source
[6:20] Speaker 2: node at the very top is the raw transaction feed.
[6:23] Speaker 2: From there, the data flows downward through a series of
[6:26] Speaker 2: specialized processing nodes.
[6:28] Speaker 1: So in the Zmart scenario, node 1 applies a masking
[6:31] Speaker 1: function to the credit card field. Right. Then node 2
[6:34] Speaker 1: filters the record to extract just the ZIP code and
[6:38] Speaker 1: item for regional analytics.
[6:39] Speaker 2: Exactly. And node 3 aggregates the purchase amount into the
[6:43] Speaker 2: customer's loyalty profile.
[6:44] Speaker 1: And finally, node 4, the sync node, writes the transformed
[6:48] Speaker 1: record out to a new destination.
[6:50] Speaker 2: You've got it. The reason this pipeline doesn't choke and
[6:52] Speaker 2: require backpressure blocking is due to its depth-first execution model.
[6:57] Speaker 1: Depth first. What does that mean in practice?
[6:59] Speaker 2: It means when your single toothpaste transaction hits the source node,
[7:03] Speaker 2: the Kafka Streams library doesn't queue it up and wait
[7:05] Speaker 2: for a batch. It pushes that exact record through the
[7:08] Speaker 2: entire depth of the graph.
[7:10] Speaker 1: Oh, wow. Yeah.
[7:11] Speaker 2: It masks it, filters it, aggregates the rewards, and writes
[7:15] Speaker 2: it to the sink, completely finishing the lifecycle of that
[7:17] Speaker 2: one event before the thread reaches back up to grab
[7:20] Speaker 2: the next customer's transaction from the source.
[7:22] Speaker 1: Wait, really? So because it's processing one record fully through
[7:26] Speaker 1: the chain... There are no intermediate buffers filling up between
[7:29] Speaker 1: node 1 and node 2.
[7:31] Speaker 2: Exactly the point. The memory footprint remains incredibly small and predictable.
[7:35] Speaker 2: It processes at the speed of the downstream graph.
[7:38] Speaker 1: That's incredibly efficient.
[7:39] Speaker 2: It is. But for a DAG to process that smoothly
[7:42] Speaker 2: without crashing, it can't hold the entirety of the infinite
[7:46] Speaker 2: stream in memory. It needs a profoundly fast, highly scalable
[7:50] Speaker 2: intermediary pipe to ingest the raw fire hose and serve
[7:53] Speaker 2: it up to the DAG.
[7:54] Speaker 1: Which is where we drop a layer deeper into the architecture.
[7:57] Speaker 1: Kafka Streams is just the processing library, right? It rides
[8:01] Speaker 1: on top of Apache Kafka, which is the actual distributed
[8:04] Speaker 1: message broker handling the data storage and transit.
[8:07] Speaker 2: Precisely. Kafka acts as a stateless, decoupled data hub. The
[8:11] Speaker 2: ZMark cash registers act as producers, blindly firing JSON payloads
[8:15] Speaker 2: into Kafka.
[8:16] Speaker 1: And the loyalty applications act as consumers, independently pulling records out.
[8:21] Speaker 1: Neither side knows about the other's existence or processing speed.
[8:24] Speaker 2: And the underlying storage mechanism making that decoupling possible is
[8:28] Speaker 2: elegantly primitive. It is an append-only, totally time-ordered distributed log.
[8:35] Speaker 1: That sounds so simple compared to a massive database. It is.
[8:39] Speaker 2: Traditional relational databases use complex B-trees and random disk I-O
[8:43] Speaker 2: to update records in place, which is computationally expensive. Kafka
[8:48] Speaker 2: simply takes the incoming byte array and appends it to
[8:50] Speaker 2: the very end of a file.
[8:52] Speaker 1: Sequential disk I.O., because the disk head just writes forward
[8:55] Speaker 1: and never has to seek back and forth to overwrite data.
[8:58] Speaker 2: Exactly. Appending to a log on a cheap hard drive
[9:00] Speaker 2: can actually be faster than random writes to expensive RAM.
[9:04] Speaker 2: It achieves incredible write throughput.
[9:05] Speaker 1: That makes a ton of sense.
[9:07] Speaker 2: But even a fast log on a single server will
[9:09] Speaker 2: eventually bottleneck on network bandwidth. So to achieve true horizontal scale,
[9:14] Speaker 2: Kafka uses partitions.
[9:15] Speaker 1: Okay, partitions.
[9:16] Speaker 2: Yeah, it takes a logical topic like ZMAR transactions, and
[9:19] Speaker 2: shatters it into mobile physical logs distributed across numerous broker
[9:23] Speaker 2: machines in the cluster.
[9:24] Speaker 1: So if you have millions of transactions pouring in and
[9:27] Speaker 1: you split them across, say, 50 partitions, how does the
[9:31] Speaker 1: system maintain any sense of logical order?
[9:34] Speaker 2: That's the million dollar question. Right.
[9:36] Speaker 1: Because if I make a purchase and then I return
[9:38] Speaker 1: the item five minutes later, those events have to be
[9:41] Speaker 1: processed sequentially.
[9:43] Speaker 2: This is where routing keys are critical. Data in Kafka
[9:46] Speaker 2: is typically sent as key value pairs. If the key
[9:49] Speaker 2: is null, Kafka optimizes for load balancing and distributes the
[9:53] Speaker 2: records round robin across all partitions.
[9:55] Speaker 1: Just spreading it evenly. Right.
[9:58] Speaker 2: But if you assign a key like a customer ID
[10:00] Speaker 2: Kafka guarantees order, it calculates the hash code of that key,
[10:04] Speaker 2: applies a modulo operation against the total number of partitions,
[10:07] Speaker 2: and routes it.
[10:08] Speaker 1: Okay, because the math is deterministic, every single transaction with
[10:12] Speaker 1: customer ID 12345 will always write to partition 7.
[10:16] Speaker 2: Yes.
[10:16] Speaker 1: And the consumer reading partition 7 will process all of
[10:19] Speaker 1: Jane Doe's actions in the exact chronological order they occurred.
[10:23] Speaker 1: But here's where it gets really interesting, based on a
[10:26] Speaker 1: specific edge case in the text. The book discusses custom partitioners.
[10:31] Speaker 1: Let's say Zmart's architects decide to use a composite key
[10:34] Speaker 1: made up of the customer ID and the transaction timestamp.
[10:37] Speaker 2: Which breaks the deterministic routing entirely.
[10:40] Speaker 1: Because the timestamp is always changing. Right.
[10:43] Speaker 2: If you run a hash function on a key that
[10:44] Speaker 2: includes a constantly changing timestamp, the output is unique every
[10:49] Speaker 2: single time. So Jane's Monday purchase goes to partition 2,
[10:53] Speaker 2: and her Tuesday return goes to partition 40.
[10:56] Speaker 1: And your sequential loyalty point calculation completely falls apart.
[10:59] Speaker 2: Exactly.
[10:59] Speaker 1: So developers can write a custom partitioner class. The custom
[11:03] Speaker 1: code intercepts the composite key, strips out the timestamp just
[11:07] Speaker 1: for the routing calculation, and hashes only the customer ID. Yep.
[11:11] Speaker 2: The full composite key is still saved in the log,
[11:13] Speaker 2: but the custom routing math ensures Jane's entire purchase history
[11:17] Speaker 2: lines up sequentially on a single partition.
[11:20] Speaker 1: That is so clever.
[11:21] Speaker 2: It really showcases the flexibility of the API. However, architecting
[11:25] Speaker 2: a cluster requires balancing those partitions carefully.
[11:29] Speaker 1: Yeah.
[11:29] Speaker 2: I mean, you might assume that creating 10,000 partitions for
[11:32] Speaker 2: a topic yields maximum parallel throughput.
[11:35] Speaker 1: That would be my first thought, yeah.
[11:37] Speaker 2: But every partition requires open file handles on the underlying
[11:40] Speaker 2: operating system. Furthermore, if you have 10,000 partitions but only
[11:44] Speaker 2: 50 consumer application instances running, each consumer has to manage
[11:49] Speaker 2: the thread switching for 200 partitions.
[11:51] Speaker 1: Oh, wow. Which creates significant consumer lag and overhead during
[11:56] Speaker 1: cluster rebalances.
[11:57] Speaker 2: It's a very delicate tuning exercise. It's art and science.
[12:00] Speaker 2: But scaling out to multiple brokers introduces the same hardware
[12:03] Speaker 2: realities that MapReduce faced. Distributed servers will inevitably catch fire,
[12:08] Speaker 2: lose network connectivity, or just suffer kernel panics.
[12:11] Speaker 1: And if Jane Doe's data is strictly routed to partition
[12:14] Speaker 1: 7 on broker A and broker A dies, her transaction
[12:17] Speaker 1: stream just stops.
[12:18] Speaker 2: Right, unless you have a backup. Kafka guarantees fault tolerance
[12:21] Speaker 2: through replication, and it relies on an external coordination service
[12:25] Speaker 2: called Apache Zookeeper to manage the chaos.
[12:27] Speaker 1: Zookeeper.
[12:28] Speaker 2: Zookeeper's job is cluster state management. When a Kafka cluster
[12:32] Speaker 2: boots up, Zookeeper elects one specific Kafka broker to act
[12:36] Speaker 2: as the active controller.
[12:37] Speaker 1: And the controller oversees the replica assignments, right? For every
[12:41] Speaker 1: topic partition, there is one designated leader broker and multiple
[12:45] Speaker 1: follower brokers.
[12:46] Speaker 2: Exactly. Producers only write to the leader. The followers passively
[12:50] Speaker 2: consume those messages from the leader exactly like a regular
[12:54] Speaker 2: consumer application would.
[12:55] Speaker 1: Oh, I see.
[12:56] Speaker 2: That's what creates an in-sync replica or ISR list.
[12:59] Speaker 1: So if the leader broker undergoes a catastrophic failure, the
[13:02] Speaker 1: zookeeper controller detects the dropped connection. Yep.
[13:06] Speaker 2: It immediately consults the ISR list to find a follower
[13:08] Speaker 2: broker that has acknowledged the exact same offset as the
[13:12] Speaker 2: dead leader. It promotes that follower to be the new
[13:14] Speaker 2: leader and seamlessly redirects all producer and consumer traffic.
[13:18] Speaker 1: And the applications barely register a latency blip.
[13:20] Speaker 2: Usually they don't even notice.
[13:22] Speaker 1: Okay, but I have to ask. If Zookeeper is the
[13:24] Speaker 1: omniscient brain managing all these leader elections and tracking the
[13:28] Speaker 1: ISR lists, doesn't that make Zookeeper a massive single point
[13:31] Speaker 1: of failure? If Zookeeper crashes... The Kafka cluster loses its
[13:36] Speaker 1: controller and the whole ecosystem collapses.
[13:39] Speaker 2: That is a great observation. And a single Zookeeper instance
[13:42] Speaker 2: would be a catastrophic vulnerability. That's why Zookeeper itself operates
[13:47] Speaker 2: as a distributed cluster, known as an ensemble.
[13:49] Speaker 1: Oh, okay.
[13:50] Speaker 2: It utilizes a quorum-based consensus algorithm. If you have a
[13:54] Speaker 2: five-node Zookeeper ensemble, as long as a strict majority, so
[13:58] Speaker 2: three nodes are alive and agree on the state of
[14:00] Speaker 2: the cluster, they can continue coordinating Kafka.
[14:03] Speaker 1: So it's voting, basically.
[14:05] Speaker 2: Exactly. This prevents split-brain scenarios where a network partition might
[14:09] Speaker 2: cause two different brokers to think they are the leader.
[14:11] Speaker 2: By strictly isolating the control plane logic within ZooKeeper from
[14:15] Speaker 2: the heavy data plane I.O. handled by the Kafka brokers,
[14:18] Speaker 2: the architecture remains highly resilient.
[14:21] Speaker 1: That separation of concerns makes total architectural sense, but there
[14:24] Speaker 1: is a physical constraint we haven't solved yet. Storage limits.
[14:28] Speaker 2: Right, the hard drives.
[14:29] Speaker 1: Yeah. An append-only log on a healthy cluster is still
[14:33] Speaker 1: appending infinite data. Basic physics dictates the hard drives will
[14:36] Speaker 1: eventually fill up. If we can't do in-place updates to
[14:39] Speaker 1: overwrite old data, the system eventually dies of digital hoarding.
[14:43] Speaker 1: I assume Kafka has a garbage collection strategy.
[14:46] Speaker 2: It does. It utilizes two distinct retention strategies, depending on
[14:49] Speaker 2: the nature of the data. Log deletion and log compaction.
[14:53] Speaker 1: Okay, let's start with deletion.
[14:54] Speaker 2: Log deletion is straightforward. Kafka doesn't store a partition as
[14:58] Speaker 2: one massive, endless file. It rolls the log into smaller
[15:02] Speaker 2: file segments based on timestamps, or file size limits. A
[15:06] Speaker 2: common retention policy might dictate that once a segment is
[15:09] Speaker 2: older than seven days, Kafka just deletes the entire file
[15:12] Speaker 2: segment from disk.
[15:13] Speaker 1: Which makes sense for ephemeral event streams. If you are
[15:16] Speaker 1: tracking telemetry data from a delivery fleet, you probably don't
[15:20] Speaker 1: need sub-second GPS coordinates from a truck three years ago.
[15:23] Speaker 1: You just delete the old segments and reclaim the disk space.
[15:26] Speaker 2: Exactly. But log compaction is fundamentally different.
[15:29] Speaker 1: Right. And here's an anthology for you. Log deletion is
[15:32] Speaker 1: like throwing away last year's wall calendar. But log compaction
[15:36] Speaker 1: is like having an address book. When your friend moves,
[15:39] Speaker 1: you don't keep their old addresses. You just erase and
[15:42] Speaker 1: update their single entry.
[15:43] Speaker 2: That is a perfect way to put it. Compaction is
[15:46] Speaker 2: designed for keyed data that represents a continuously evolving state
[15:51] Speaker 2: rather than isolated events.
[15:52] Speaker 1: Like a stock ticker.
[15:53] Speaker 2: Exactly. If the key is AAPL for Apple stock, the
[15:57] Speaker 2: stream might receive 10,000 price updates in a single day.
[16:01] Speaker 1: But if a consumer application boots up, it doesn't want
[16:04] Speaker 1: to process 10,000 historical price ticks. It only cares about
[16:07] Speaker 1: the absolute latest valuation. Right.
[16:10] Speaker 2: Compaction clears out the noise. Instead of deleting data based
[16:13] Speaker 2: on arbitrary age, a background process called a log cleaner
[16:17] Speaker 2: thread sweeps through the segments. It looks at the keys,
[16:20] Speaker 2: and it guarantees that it will retain at least the
[16:22] Speaker 2: most recent message for every specific key while aggressively deleting
[16:26] Speaker 2: older duplicate versions.
[16:28] Speaker 1: And to handle actual removals, developers can send what's called
[16:31] Speaker 1: a tombstone marker, right? Which is just sending a null
[16:33] Speaker 1: value for a key.
[16:34] Speaker 2: Yes. That acts as a flag for the log cleaner
[16:37] Speaker 2: to completely erase that key from the partition during its
[16:40] Speaker 2: next sweep. And if we connect this to the bigger picture,
[16:44] Speaker 2: log compaction is the secret to maintaining state in real-time applications.
[16:49] Speaker 2: How so? Well, if Zmart's Kafka Streams application is aggregating
[16:53] Speaker 2: a real-time count of total sales by region... It stores
[16:57] Speaker 2: that aggregation in a local embedded database, typically RocksDB.
[17:02] Speaker 1: But if the server running that application crashes, the local
[17:05] Speaker 1: RocksDB state is lost.
[17:07] Speaker 2: Precisely. To recover, Kafka Streams uses a compacted internal topic
[17:11] Speaker 2: as a highly durable changelog for that local state store.
[17:14] Speaker 1: Oh, I see.
[17:15] Speaker 2: When the application restarts on a new server, it doesn't
[17:17] Speaker 2: have to recalculate the sales totals by replaying millions of
[17:20] Speaker 2: raw transactions from the beginning of time.
[17:22] Speaker 1: Because it simply replays the heavily compacted changelog topic, instantly
[17:26] Speaker 1: restoring the most recent values into its local RocksDB, and
[17:29] Speaker 1: resumes processing.
[17:30] Speaker 2: Exactly. It's an automatic memory-safe state mechanism.
[17:34] Speaker 1: It is brilliant. So we've mapped out the distributed infrastructure.
[17:37] Speaker 1: We have time-ordered logs, deterministic partitioning, zookeeper quorums, and compacted
[17:42] Speaker 1: state stores. The final piece is how a software engineer
[17:46] Speaker 1: actually writes the code that rides on top of all
[17:48] Speaker 1: this plumbing.
[17:49] Speaker 2: Yeah, the author walks through a Hello World implementation using
[17:52] Speaker 2: the Kafka Screams DSL, or domain-specific language. It centers around
[17:57] Speaker 2: a core abstraction called the KStream object, which represents the
[18:00] Speaker 2: continuous record stream.
[18:02] Speaker 1: And the application they build is called the Yelling app, right?
[18:04] Speaker 2: Yes, the Yelling app. Its sole function is to ingest
[18:07] Speaker 2: a stream of text, convert it to uppercase, and output
[18:11] Speaker 2: the transformed stream.
[18:12] Speaker 1: The code structure for this is incredibly clean. Step one
[18:16] Speaker 1: is configuration assigning an application ID and pointing the app
[18:19] Speaker 1: to the Kafka broker addresses. Step two requires defining the SERDs.
[18:24] Speaker 2: Ah, SERDs. Serializers and deserializers.
[18:26] Speaker 1: Right.
[18:27] Speaker 2: Kafka brokers, as we discussed, are incredibly fast because they
[18:30] Speaker 2: are intentionally dumb. They don't know if the log contains JSON, Avro,
[18:35] Speaker 2: or plain text. They only deal in raw byte arrays.
[18:38] Speaker 1: Which allows Kafka to use zero-copy optimizations. transferring data directly
[18:43] Speaker 1: from the disk buffer to the network socket without the
[18:46] Speaker 1: CPU having to decode it.
[18:48] Speaker 2: Exactly. But the Java application obviously needs to read the text.
[18:51] Speaker 2: So the developer provides a string serde, which acts as
[18:54] Speaker 2: the translation layer, deserializing the byte arrays back into Java
[18:58] Speaker 2: string objects as they enter the DAG and serializing them
[19:01] Speaker 2: back to bytes when they exit.
[19:02] Speaker 1: Once the serdes are configured, you build the actual topology.
[19:06] Speaker 1: You instantiate a source node pointing to an input topic.
[19:09] Speaker 1: Then you apply a stateless transformation node. The author uses
[19:13] Speaker 1: a simple Java 8 lambda function, which guarantees immutability. It
[19:19] Speaker 1: doesn't modify the original record. It creates a transformed copy
[19:23] Speaker 1: and passes it to the final sync node, which writes
[19:25] Speaker 1: it to an output topic. What stands out looking at
[19:28] Speaker 1: the syntax is that it utilizes a fluent interface.
[19:30] Speaker 2: Yes, a design pattern popularized by Martin Fowler and Eric Evans.
[19:35] Speaker 1: Right. Instead of writing disconnected procedural scripts with mutable variables,
[19:40] Speaker 1: the developer chains the methods together in a single fluid sequence.
[19:43] Speaker 1: You literally write builder.stream.mapvalues.2.
[19:48] Speaker 2: The code reads, exactly like the logical day topology it represents.
[19:53] Speaker 2: And the elegance of the API is that whether you
[19:55] Speaker 2: are uppercasing text in a sandbox or performing highly complex
[20:00] Speaker 2: stateful joins of real-time inventory feeds against customer loyalty streams
[20:04] Speaker 2: at Zmart, you are using the exact same composable DSL.
[20:08] Speaker 1: The complexity is abstracted away into composable blocks.
[20:11] Speaker 2: Allowing engineers to focus purely on the business logic rather
[20:14] Speaker 2: than distributed system fault tolerance.
[20:17] Speaker 1: Which brings our architectural journey full circle. We started with
[20:20] Speaker 1: MapReduce and the realization that while batch processing could aggregate
[20:23] Speaker 1: massive historical datasets, it couldn't react in real time.
[20:26] Speaker 2: Yeah, we had to evolve.
[20:27] Speaker 1: We explored how Apache Kafka solved the real-time ingestion bottleneck
[20:31] Speaker 1: by relying on append-only logs, distributed partitions, and ZooKeeper's quorum consensus.
[20:36] Speaker 1: We uncovered how log compaction allows applications to maintain state
[20:40] Speaker 1: without drowning in infinite data.
[20:41] Speaker 2: All crucial pieces.
[20:42] Speaker 1: And finally, we saw how the Kafka Streams DSL allows
[20:45] Speaker 1: developers to compose complex DAG topologies with just a few
[20:49] Speaker 1: lines of fluent code. So what does this all mean
[20:52] Speaker 1: for you?
[20:53] Speaker 2: Well, it means that the next time you interact with
[20:55] Speaker 2: a modern digital system, whether it's an immediate fraud alert
[20:58] Speaker 2: on your phone, dynamic surge pricing on a rideshare app,
[21:02] Speaker 2: or an instant inventory update, you understand the plumbing.
[21:06] Speaker 1: You are generating an event that is being routed, partitioned,
[21:09] Speaker 1: and processed continuously through a DAG. You aren't just querying
[21:13] Speaker 1: a database. You are participating in an infinite, unbounded stream.
[21:17] Speaker 2: And that requires a paradigm shift for anyone architecting systems today.
[21:21] Speaker 2: You know, the foreword of the book emphasizes moving away
[21:24] Speaker 2: from traditional command thinking.
[21:26] Speaker 1: Command thinking. Yeah.
[21:27] Speaker 2: In a command-driven world, systems are passive. You send a
[21:30] Speaker 2: request to a database and you wait for a response.
[21:33] Speaker 2: The shift to event thinking means recognizing that a business
[21:36] Speaker 2: is not a static database of what has happened. An
[21:39] Speaker 2: enterprise is a living, continuous stream of real-time events.
[21:42] Speaker 1: When you stop trying to capture static snapshots and instead
[21:45] Speaker 1: design systems to react to continuous state changes, the architectural
[21:49] Speaker 1: possibilities completely open up. It's an entirely different way of
[21:52] Speaker 1: modeling reality. Are we just static databases of memories, or
[21:56] Speaker 1: are we continuous streams of real-time events constantly rolling and
[21:59] Speaker 1: compacting our own logs? Go ponder that.

Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them.