Distributed_systems on Vivek's Field Notes

Scaling Databases with Sharding

Sat, 28 Feb 2026 20:16:56 +0530

Introduction to Sharding

Sharding is the process of scaling a database by spreading data across multiple servers, or shards. It is the go-to solution for large organizations managing data at a petabyte scale. Industry leaders like Uber, Shopify, Slack, and OpenAI all leverage sharding to manage their massive datasets.

In a typical small-scale application, one or more app servers connect to a single, monolithic database. This server stores all persistent data, from user accounts to application state. However, as data grows, this single point of failure and bottleneck must be addressed.

Sharded Architecture

In a sharded setup, we divide the total data into portions, each hosted on a separate database server.

Initially, your application code might try to manage these shards directly—keeping track of which row lives where and maintaining multiple open connections. While manageable with two shards, this approach becomes a maintenance nightmare when dealing with hundreds.

The Proxy Layer

A more robust solution is to use an intermediary proxy. Application servers connect only to this proxy, which then routes queries to the correct shard.

However, proxies introduce their own challenges:

Throughput Limits: If a proxy reaches its capacity, queries are queued, adding latency.
Scalability: To handle high volumes, you must deploy multiple proxy servers to prevent them from becoming the bottleneck.

Sharding Strategies

The sharding strategy—the rules determining data placement—is critical for performance and balance. This usually involves a shard key: the column(s) used to route data.

1. Range Sharding

Data is routed based on predefined ranges of values. For example, IDs 1-25 might go to Shard A, 26-50 to Shard B, and so on.

[!WARNING] Naive range-based sharding with monotonically increasing IDs often leads to “Hot Shards”. If you insert IDs 1 to 25 sequentially, only the first shard is active while others remain idle.

2. Hash Sharding

The proxy generates a cryptographic hash of the shard key for each row. Each shard is then responsible for a specific range of hashes.

Best Practice: Choose a key with high cardinality (e.g., user_id).
Avoid: Columns like name, where popular values can still create hotspots despite hashing.
Optimization: Hashing fixed-size integers (user_id) is generally faster than hashing variable-width strings.

3. Lookup Sharding

A separate mapping table tracks exactly which data belongs on which shard. This offers maximum flexibility but requires an additional lookup for every query.

Real-World Case Study: PostgreSQL and ChatGPT

While sharding solves many scale problems, specific database architectures like PostgreSQL’s MVCC (Multiversion Concurrency Control) introduce unique write penalties that companies like OpenAI have had to navigate.

The “Copy-on-Write” Penalty

In Postgres, updates are not performed “in-place.” Updating even one byte results in Write Amplification, where the entire row is copied to create a new version. This strains I/O and leads to Read Amplification, as queries must scan through “dead” versions (old rows) to find live ones.

The “Bloat” Problem

Old row versions (Dead Tuples) don’t disappear instantly, leading to table bloat and increased autovacuum overhead. If writes outpace reclamation, performance collapses. Every update also requires updating all indexes to point to the new physical row location, adding CPU stress.

Strategies from the OpenAI Engineering Team

To ensure services like ChatGPT and their API remain responsive during massive write spikes, several strategies are employed:

Minimizing Primary Load: Read traffic is offloaded to replicas whenever possible. Queries that must remain on the primary (e.g., those part of write transactions) are strictly optimized for efficiency.
Selective Migration: Shardable, write-heavy workloads are migrated to systems like Azure CosmosDB.
Application-Level Optimizations: Redundant writes are eliminated, and “lazy writes” are introduced to smooth out traffic spikes.
Rate Limiting: Strict limits are enforced during background tasks, such as backfilling table fields, to prevent excessive write pressure.

Optimization & Best Practices

Query Optimization

Avoid “OLTP anti-patterns” that can degrade services:

Simplify Joins: A query joining 12 tables (as seen in some historical ChatGPT SEVs) can crash a service during a spike. Move complex join logic to the application layer.
ORM Awareness: Object-Relational Mapping tools can generate inefficient SQL; always review the output.
Timeout Management: Configure idle_in_transaction_session_timeout to prevent idle queries from blocking critical processes like autovacuum.

Cross-Shard Penalties

Queries spanning multiple shards add excessive network and CPU overhead. Aim for single-shard queries whenever possible. Additionally, avoid shard keys that change frequently, as moving rows between shards to maintain strategy integrity is expensive.

Infrastructure & Latency

Adding a proxy introduces a network hop, typically adding ~1ms of latency.

Server Proximity: If proxies and shards are in the same data center, this latency is negligible.
Proven Success: Slack uses Vitess to manage massive sharded clusters with an average query latency of just 2ms.

High Availability

Replicas aren’t just for reads; they are your safety net. If a primary fails, traffic can be instantly failed over to a replica, preventing hours of downtime.

Storage and Retrival

Wed, 25 Feb 2026 23:32:36 +0530

In particular, there is a big difference between storage engines that are optimized for transactional workloads and those that are optimized for analytics.

An index is an additional structure that is derived from the primary data. Many data‐ bases allow you to add and remove indexes, and this doesn’t affect the contents of the database; it only affects the performance of queries. Maintaining additional structures incurs overhead, especially on writes. For writes, it’s hard to beat the performance of simply appending to a file, because that’s the simplest possible write operation. Any kind of index usually slows down writes, because the index also needs to be updated every time data is written.

Hash Index

Let’s say our data storage consists only of appending to a file, as in the preceding example. Then the simplest possible indexing strategy is this: keep an in-memory hash map where every key is mapped to a byte offset in the data file—the location at which the value can be found, as illustrated in Figure 3-1. Whenever you append a new key-value pair to the file, you also update the hash map to reflect the offset of the data you just wrote (this works both for inserting new keys and for updating existing keys). When you want to look up a value, use the hash map to find the offset in the data file, seek to that location, and read the value. This may sound simplistic, but it is a viable approach. In fact, this is essentially what Bitcask (the default storage engine in Riak) does [3]. Bitcask offers high-performance reads and writes, subject to the requirement that all the keys fit in the available RAM, since the hash map is kept completely in memory. The values can use more space than there is available memory, since they can be loaded from disk with just one disk seek. If that part of the data file is already in the filesystem cache, a read doesn’t require any disk I/O at all. A storage engine like Bitcask is well suited to situations where the value for each key is updated frequently. For example, the key might be the URL of a cat video, and the value might be the number of times it has been played (incremented every time someone hits the play button). In this kind of workload, there are a lot of writes, but there are not too many distinct keys—you have a large number of writes per key, but it’s feasible to keep all keys in memory. Moreover, since compaction often makes segments much smaller (assuming that a key is overwritten several times on average within one segment), we can also merge several segments together at the same time as performing the compaction, as shown in Figure 3-3. Segments are never modified after they have been written, so the merged segment is written to a new file. The merging and compaction of frozen seg‐ ments can be done in a background thread, and while it is going on, we can still con‐ tinue to serve read and write requests as normal, using the old segment files. After the merging process is complete, we switch read requests to using the new merged seg‐ ment instead of the old segments—and then the old segment files can simply be deleted.

Each segment now has its own in-memory hash table, mapping keys to file offsets. In order to find the value for a key, we first check the most recent segment’s hash map; if the key is not present we check the second-most-recent segment, and so on. The merging process keeps the number of segments small, so lookups don’t need to check many hash maps. Lots of detail goes into making this simple idea work in practice. Briefly, some of the issues that are important in a real implementation are:

The hash table must fit in memory, so if you have a very large number of keys, you’re out of luck. In principle, you could maintain a hash map on disk, but unfortunately it is difficult to make an on-disk hash map perform well. It requires a lot of random access I/O, it is expensive to grow when it becomes full, and hash collisions require fiddly logic [5]. • Range queries are not efficient. For example, you cannot easily scan over all keys between kitty00000 and kitty99999—you’d have to look up each key individu‐ ally in the hash maps.

Docker & kubernetes

Fri, 14 Nov 2025 12:38:31 +0530

Docker

Open Source

Mapper Reducer

Wed, 30 Jul 2025 22:28:21 +0530

Map Reduce

Problem faced by google:

Large Data like crawled pages over WWW. They need to do some analysis over this data. It’s really not possible to store all this data in one system and to analyse this data serially wil take a lots of time. So they created MapReduce Issue:

parallelize Computation
distribute the data
handle failure cases
load balancing

Programming Model

For example in large database we are trying to calculate number of occurrence of each words Map: receive a document

map(String key, String value):
	// key: document name
	// value: document contents
	for each word w in value:
		EmitIntermediate(w, "1");

Reduce: will receive intermediate values

reduce(String key, Iterator values):
	// key: a word
	// values: a list of counts
	int result = 0;
	for each v in values:
		result += ParseInt(v);
	Emit(AsString(result));

Mapper produce (the,1) (map,1) (function,1) Reducer receive (“the”,{1,1,1,1,2,3,}) Reducer produce (“the”,9)

map (k1,v1) → list(k2,v2)

reduce (k2,list(v2)) → list(v2)

Implementation

We Split the data into M split. Input split is processed in parallel by different machine.
We split intermediate value into R split using partitioning function specified by user. ex (hash(value)%R). Steps
The MapReduce library in the user program first splits the input files into M pieces of typically 16 megabytes to 64 megabytes (MB) per piece (controllable by the user via an optional parameter). It then starts up many copies of the program on a cluster of machines.
One of the copies of the program is special – the master. The rest are workers that are assigned work by the master. There are M map tasks and R reduce tasks to assign. The master picks idle workers and assigns each one a map task or a reduce task.
A worker who is assigned a map task reads the contents of the corresponding input split. It parses key/value pairs out of the input data and passes each pair to the user-defined Map function. The intermediate key/value pairs produced by the Map function are buffered in memory
Periodically, the buffered pairs are written to local disk, partitioned into R regions by the partitioning function. The locations of these buffered pairs on the local disk are passed back to the master, who is responsible for forwarding these locations to the reduce workers.
When a reduce worker is notified by the master about these locations, it uses remote procedure calls to read the buffered data from the local disks of themap workers. When a reduce worker has read all intermediate data, it sorts it by the intermediate keysso that all occurrences of the same key are grouped together. The sorting is needed because typically many different keys map to the same reduce task. If the amount of intermediate data is too large to fit in memory, an external sort is used
The reduce worker iterates over the sorted intermediate data and for each unique intermediate key encountered, it passes the key and the corresponding set of intermediate values to the user’s Reduce function. The output of the Reduce function is appended to a final output file for this reduce partition.
When all map tasks and reduce tasks have been completed, the master wakes up the user program. At this point, the MapReduce call in the user program returns back to the user code.

for each completed map task,the master stores the locations and sizes of the R intermediate file regions produced by the map task. Updates to this location and size information are received as map tasks are completed. The information is pushed incrementally to workers that have in-progress reduce tasks.

Fault Tolerance

Master pings every machine time to time if they failed to response in time then master make the task assigned to that system as failure and reschedule that task to some other worker. Completed map task is reschedule on failure cuz intermediate data is stored in local disk but completed reduce task in not reschedule on failure cuz output is stored in global disk.

It is easy to make the master write periodic checkpoints of the master data structures described above. If the master task dies, a new copy can be started from the last checkpointed state. However, given that there is only a single master, its failure is unlikely; therefore our current implementation aborts the MapReduce computation if the master fails. Clients can check for this condition and retry the MapReduce operation if they desire. Reduce produce 1 output file Mapper produce R output file When a map task completes, the worker sends a message to the master and includes the names of the R temporary files in the message. When a reduce task completes, the reduce worker atomically renames its temporary output file to the final output file. If the same reduce task is executed on multiple machines, multiple rename calls will be executed for the same final output file.

We split mapper phase into M split and reduce phase in R split ideally (M&R ) should be much larger than number of worker.

Having each worker perform many different tasks improves dynamic load balancing
speeds up recovery when a worker fails we tend to choose M so that each individual task is roughly 16 MB to 64 MB of input data (so that the locality optimization described above is most effective)

One of the common causes that lengthens the total time taken for a MapReduce operation is a “straggler”: a machine that takes an unusually long time to complete one of the last few map or reduce tasks in the computation. When MapReduce operation is close to completion master assign in progress task to backup machines.he task is marked as completed whenever either the primary or the backup execution completes.