First local event flow
Create a topic with 3 partitions, send 10 keyed messages, and verify only per-partition ordering (not global ordering). Capture one screenshot or log snippet.
Data / Knowledge library
Apache Kafka (kafka)
a distributed event streaming platform
Why learn Apache Kafka (kafka)?
Kafka lets teams move high-volume event data between systems reliably and in near real time. It decouples producers from consumers, supports replay, and scales through partitioned logs.
Where it came from
Kafka was created at LinkedIn around 2010 for activity/event stream transport. It was open-sourced in 2011 and became an Apache top-level project in 2012. Kafka Connect and Kafka Streams broadened Kafka beyond messaging into integration and stream processing. Current architecture direction centers on KRaft to simplify cluster operations.
- Kafka was created at LinkedIn around 2010 for activity/event stream transport.
- It was open-sourced in 2011 and became an Apache top-level project in 2012.
- Kafka Connect and Kafka Streams broadened Kafka beyond messaging into integration and stream processing.
- Current architecture direction centers on KRaft to simplify cluster operations.
Build your understanding
Work through these exercises in order. Try the task before opening a reference.
- Create a topic with 3 partitions, send 10 keyed messages, and verify only per-partition ordering (not global ordering). Capture one screenshot or log snippet.
- Run two consumers in the same group against a multi-partition topic. Trigger a rebalance by stopping one consumer, then explain partition reassignment in 2-3 lines.
- Configure a producer with idempotence and strong durability. Explain why acks=all plus retries improves reliability and what trade-off it introduces.
- Sketch a pipeline: DB change events -> Kafka -> stream processor -> analytics store. Name topic keys, retention policy, and failure-recovery strategy.
- Simulate consumer failure after processing records but before commit. Restore processing with minimal duplicates; document your commit strategy.
More ways to practice
Consumer-group scaling check
Run two consumers in the same group against a multi-partition topic. Trigger a rebalance by stopping one consumer, then explain partition reassignment in 2-3 lines.
Safer producer settings
Configure a producer with idempotence and strong durability. Explain why acks=all plus retries improves reliability and what trade-off it introduces.
Mini CDC pipeline design
Sketch a pipeline: DB change events -> Kafka -> stream processor -> analytics store. Name topic keys, retention policy, and failure-recovery strategy.
Offset recovery drill
Simulate consumer failure after processing records but before commit. Restore processing with minimal duplicates; document your commit strategy.
Documentation and sources
- Apache Kafka Documentation (kafka.apache.org)
- Apache Kafka Quickstart (kafka.apache.org)
- Kafka Improvement Proposals (KIPs) (cwiki.apache.org)
- Kafka APIs Overview (kafka.apache.org)
Catalog research notes
- KRaft mode (Raft-based metadata quorum) became the strategic path, reducing ZooKeeper dependency.
- Recent releases improved operational stability, balancing, and observability for large clusters.
- Ecosystem usage grew around Kafka Connect, Streams, and managed Kafka services for real-time data products.
Connect the ideas
Alternatives
Explore next
Real World Example
Real-time Logistics Event Synchronization
Global logistics companies managing complex supply chains.
The business problem
Legacy systems often suffer from data silos where shipping status updates are trapped in isolated databases, causing significant delays in customer notification and inventory reconciliation.
How this topic solves it
Kafka acts as a distributed event streaming platform, decoupling data producers from consumers by storing events in durable topics, allowing multiple downstream services to consume the same status updates in real-time.
- Initialize the Kafka cluster environment using storage formatting tools.
- Define a dedicated shipping-updates topic to serve as the central event log.
- Deploy a source connector via Kafka Connect to ingest status changes from legacy databases.
- Implement a Kafka Streams application to filter and aggregate shipment milestones.
- Expose the processed event stream to customer-facing dashboards and inventory systems.
Data flows from legacy databases through Kafka into multiple consuming services.
Source of truth
Ingestion layer
Durable event log
Processing logic
Consumer
- Legacy DB → reads → Kafka Connect
- Kafka Connect → produces → Kafka Topic
- Kafka Topic → consumes → Kafka Streams
- Kafka Streams → updates → Dashboard
Expected business outcome
Improved operational visibility and reduced latency in cross-departmental data synchronization.
Limitations and checks
The primary trade-off is the operational complexity of managing cluster state and partition distribution, which must be monitored by checking the consumer lag and broker health metrics.