Receives raw telemetry
Cloud / Knowledge library
Data Sharding
Partitions records across nodes by a routing rule
Why learn Data Sharding?
It scales storage and write throughput beyond one machine
The problem it solves
It divides a dataset that no single node should own
Where it came from
Data Sharding is a distributed systems concept that partitions records across nodes by a routing rule. Its guarantees and tradeoffs depend on the concrete implementation and operating context.
Build your understanding
Work through these exercises in order. Try the task before opening a reference.
- Define Data Sharding and distinguish it from a neighboring concept.
- Choose a realistic task where Data Sharding is appropriate.
- Identify a failure mode caused by misusing Data Sharding.
- Interpret a small design or result involving Data Sharding.
- Explain one tradeoff or limitation of Data Sharding.
Documentation and sources
- Designing Data-Intensive Applications (dataintensive.net)
Connect the ideas
Explore next
Real World Example
Global Logistics Route Optimization Sharding
A global logistics provider managing real-time tracking data for millions of shipping containers across multiple continents.
The business problem
The central database node experiences severe write contention and storage exhaustion because all incoming telemetry data from global sensors is directed to a single primary instance, leading to latency spikes during peak shipping hours.
How this topic solves it
Data sharding distributes the load by partitioning records across multiple nodes based on a routing rule, specifically using the container's region ID as a partition key. This ensures that telemetry data is distributed evenly across a cluster, allowing the system to scale horizontally by adding nodes as the number of containers increases.
- Define a partition key based on the geographic region ID of the shipping container.
- Configure a routing layer that maps specific region ID ranges to designated storage nodes.
- Initialize the cluster nodes to host their respective partitions of the global telemetry dataset.
- Update the ingestion service to include the region ID in the routing metadata for every incoming record.
- Monitor the distribution of data across nodes to ensure no single partition becomes a hotspot.
Explains how incoming telemetry is routed to specific nodes based on region.
Applies partition rule
Stores regional data
Stores regional data
- Ingestion Service → forwards data → Routing Layer
- Routing Layer → routes by region ID → Node: North America
- Routing Layer → routes by region ID → Node: EMEA
Expected business outcome
The system achieves balanced resource utilization across the cluster, eliminating single-node bottlenecks and allowing for predictable growth as the fleet size expands.
Limitations and checks
A primary limitation is the difficulty of performing cross-shard queries. If a logistics manager needs to generate a report spanning multiple regions, the system must perform a scatter-gather operation, which increases latency. To check for this, monitor the frequency of cross-shard join operations and query execution times.