CHAPTER 05

Event-Driven Architectures, Message Queues & Log Streaming

Decoupling microservices using asynchronous event streams enables buffer control, backpressure management, event replayability, and horizontal consumer scaling.

1. Message Queues (RabbitMQ) vs Distributed Logs (Apache Kafka)

Message Queue (e.g. RabbitMQ / AMQP)

Smart broker, dumb consumer. Pushes messages to workers via exchanges (Direct, Fanout, Topic). Messages are deleted from the queue immediately once acknowledged by a consumer.

Best for: Complex routing, transient task processing, RPC work queues.

Distributed Commit Log (e.g. Apache Kafka)

Dumb broker, smart consumer. Partitioned append-only log on disk. Messages are retained based on time/size policies regardless of consumption. Multiple consumer groups track their own offsets independently.

Best for: Event streaming, replayable logs, high-throughput metrics, CQRS.

2. Apache Kafka Architecture & Partitioning

Kafka topics are divided into Partitions. Each partition is an ordered, immutable sequence of records. Messages within a partition are assigned a sequential ID called an Offset.

Ordering Guarantee: Kafka guarantees total message order ONLY within a single partition, NOT across partitions.

3. Zero-Copy Kernel Optimization (`sendfile()`)

Traditional data transfer copies data 4 times between OS kernel space and user application space. Kafka avoids user-space copy overhead by using the Linux sendfile() system call:

Disk \(\longrightarrow\) OS Page Cache \(\xrightarrow[\text{Direct DMA Transfer}]{\text{sendfile()}}\) NIC Network Buffer

Bypassing user-space memory copies reduces CPU cycles dramatically and allows Kafka to saturate network interface cards (NICs) at hardware line rate.

4. Exactly-Once Semantics (EOS)

Achieving EOS requires combining three features:

Idempotent Producer: Assigns unique Producer ID (PID) and sequence numbers to every batch. Duplicate sequence numbers sent due to network retries are dropped by the broker.
Transactional Coordinator: Coordinates atomic writes across multiple topic partitions in a single "Consume-Transform-Produce" cycle.
Read Committed Isolation: Downstream consumers only read messages that belong to successfully committed transactions.

5. The Transactional Outbox Pattern & CQRS

Updating a database and publishing an event to Kafka in separate operations risks consistency failure if the service crashes between the database commit and event dispatch.

Transactional Outbox Solution:

1. Service saves entity AND inserts an event record into an outbox table inside the SAME ACID database transaction.