Data partitioning

01Divide ownership, not just machine count

An analytics service keeps 2 TB of customer events. One database is running out of useful storage and write capacity. The team considers giving different machines different parts of the data.

Distinguish two splits:

• Horizontal partitioning divides records. For example, one group of customers’ events belongs to one destination and another group belongs elsewhere.
• Vertical partitioning separates fields or groups, such as keeping large event payloads apart from frequently read metadata.

A shard is an owned subset of records assigned to a database server or server group. State the physical layout: several shards may share a machine, and one shard may have copies on several machines. Merely dividing a table within one server does not add the processing capacity of several servers.

Work the first capacity estimate:

1. Dividing 2 TB among four shards gives an average of 500 GB each.
2. Check actual customer sizes. Equal customer counts need not produce equal stored bytes.
3. Check actual request rates. If customer K produces half the traffic and all of K belongs to one shard, that shard may overload while others are idle.

Copying and splitting are separate choices. Four shards with three copies of each produce twelve shard copies. Each holds only its assigned subset, rather than the entire 2 TB. The physical machine count depends on where those copies are placed.

02Choose a placement rule from the queries

Most queries ask for one customer’s events from the last day. Choose placement by following both that read and the incoming writes.

For the large customer K, divide each day’s events into three known groups:

1. Calculate a repeatable hash from each event’s identifier.
2. Use it to choose group 0, 1, or 2. Repeating the same event returns to the same group.
3. Place the groups on different destinations. A directory might record that K uses shards 2, 5, and 7.
4. For a day-wide read, ask all three known groups for their matching records.
5. Merge those records in the requested order and construct the next page consistently with that order.

One request causing several storage requests is called fan-out. Here it spreads K’s writes, but adds read work and more destinations that must respond. Ordinary customers can remain in one group. Check the real event distribution; hashing a name is not proof of equal work.

03Move data while requests continue

Suppose K outgrows its current destination. Existing records must move while new events still arrive. A completed snapshot does not contain writes made afterward.

A supported migration procedure needs these stages:

1. Save which records are moving and the old and new destinations, so the move can recover after a restart.
2. Copy the existing records to the new destination.
3. Keep an ordered change log at the old destination and apply its writes to the new copy while requests continue.
4. For the final handoff in this example, briefly pause new writes for K and let its in-flight writes finish. Record the last accepted change-log position and verify that the new destination has applied every change through that position.
5. Save the new destination in the routing directory and make the old destination forward stale-route requests there instead of accepting local writes. Resume K’s writes at the new destination.
6. Keep the saved ownership and move progress so a restart resumes the same destination rather than reopening both writers. Remove the old copy only after the new route and retained copies are verified.

The pause-and-forward sequence is one illustrative contract, not a complete migration implementation. Use a store that supports that handoff and crash recovery. Other stores may use ownership versions or another coordination protocol; changing a routing map alone does not transfer records or later writes.

Rules spanning shards also need coordination. A money transfer cannot be treated as complete after debiting one account if the credit fails elsewhere. Consider the permitted alternatives:

• Keep both changes within one supported local transaction when the placement permits it.
• Use a transaction protocol that coordinates the independently failing stores.
• If the product allows it, show a pending transfer and use a recovery workflow to complete or compensate it. That is a different visible guarantee.

Measure bytes, write rate, waiting work, and response-time distributions per shard. A fleet average can hide K’s bottleneck. Include the routing key in ordinary requests, and plan late events, retention, and deletion across every relevant group.