Tips on Vivek's Field Notes

Scaling Databases with Sharding

Sat, 28 Feb 2026 20:16:56 +0530

Introduction to Sharding

Sharding is the process of scaling a database by spreading data across multiple servers, or shards. It is the go-to solution for large organizations managing data at a petabyte scale. Industry leaders like Uber, Shopify, Slack, and OpenAI all leverage sharding to manage their massive datasets.

In a typical small-scale application, one or more app servers connect to a single, monolithic database. This server stores all persistent data, from user accounts to application state. However, as data grows, this single point of failure and bottleneck must be addressed.

Sharded Architecture

In a sharded setup, we divide the total data into portions, each hosted on a separate database server.

Initially, your application code might try to manage these shards directly—keeping track of which row lives where and maintaining multiple open connections. While manageable with two shards, this approach becomes a maintenance nightmare when dealing with hundreds.

The Proxy Layer

A more robust solution is to use an intermediary proxy. Application servers connect only to this proxy, which then routes queries to the correct shard.

However, proxies introduce their own challenges:

Throughput Limits: If a proxy reaches its capacity, queries are queued, adding latency.
Scalability: To handle high volumes, you must deploy multiple proxy servers to prevent them from becoming the bottleneck.

Sharding Strategies

The sharding strategy—the rules determining data placement—is critical for performance and balance. This usually involves a shard key: the column(s) used to route data.

1. Range Sharding

Data is routed based on predefined ranges of values. For example, IDs 1-25 might go to Shard A, 26-50 to Shard B, and so on.

[!WARNING] Naive range-based sharding with monotonically increasing IDs often leads to “Hot Shards”. If you insert IDs 1 to 25 sequentially, only the first shard is active while others remain idle.

2. Hash Sharding

The proxy generates a cryptographic hash of the shard key for each row. Each shard is then responsible for a specific range of hashes.

Best Practice: Choose a key with high cardinality (e.g., user_id).
Avoid: Columns like name, where popular values can still create hotspots despite hashing.
Optimization: Hashing fixed-size integers (user_id) is generally faster than hashing variable-width strings.

3. Lookup Sharding

A separate mapping table tracks exactly which data belongs on which shard. This offers maximum flexibility but requires an additional lookup for every query.

Real-World Case Study: PostgreSQL and ChatGPT

While sharding solves many scale problems, specific database architectures like PostgreSQL’s MVCC (Multiversion Concurrency Control) introduce unique write penalties that companies like OpenAI have had to navigate.

The “Copy-on-Write” Penalty

In Postgres, updates are not performed “in-place.” Updating even one byte results in Write Amplification, where the entire row is copied to create a new version. This strains I/O and leads to Read Amplification, as queries must scan through “dead” versions (old rows) to find live ones.

The “Bloat” Problem

Old row versions (Dead Tuples) don’t disappear instantly, leading to table bloat and increased autovacuum overhead. If writes outpace reclamation, performance collapses. Every update also requires updating all indexes to point to the new physical row location, adding CPU stress.

Strategies from the OpenAI Engineering Team

To ensure services like ChatGPT and their API remain responsive during massive write spikes, several strategies are employed:

Minimizing Primary Load: Read traffic is offloaded to replicas whenever possible. Queries that must remain on the primary (e.g., those part of write transactions) are strictly optimized for efficiency.
Selective Migration: Shardable, write-heavy workloads are migrated to systems like Azure CosmosDB.
Application-Level Optimizations: Redundant writes are eliminated, and “lazy writes” are introduced to smooth out traffic spikes.
Rate Limiting: Strict limits are enforced during background tasks, such as backfilling table fields, to prevent excessive write pressure.

Optimization & Best Practices

Query Optimization

Avoid “OLTP anti-patterns” that can degrade services:

Simplify Joins: A query joining 12 tables (as seen in some historical ChatGPT SEVs) can crash a service during a spike. Move complex join logic to the application layer.
ORM Awareness: Object-Relational Mapping tools can generate inefficient SQL; always review the output.
Timeout Management: Configure idle_in_transaction_session_timeout to prevent idle queries from blocking critical processes like autovacuum.

Cross-Shard Penalties

Queries spanning multiple shards add excessive network and CPU overhead. Aim for single-shard queries whenever possible. Additionally, avoid shard keys that change frequently, as moving rows between shards to maintain strategy integrity is expensive.

Infrastructure & Latency

Adding a proxy introduces a network hop, typically adding ~1ms of latency.

Server Proximity: If proxies and shards are in the same data center, this latency is negligible.
Proven Success: Slack uses Vitess to manage massive sharded clusters with an average query latency of just 2ms.

High Availability

Replicas aren’t just for reads; they are your safety net. If a primary fails, traffic can be instantly failed over to a replica, preventing hours of downtime.

Storage and Retrival

Wed, 25 Feb 2026 23:32:36 +0530

In particular, there is a big difference between storage engines that are optimized for transactional workloads and those that are optimized for analytics.

An index is an additional structure that is derived from the primary data. Many data‐ bases allow you to add and remove indexes, and this doesn’t affect the contents of the database; it only affects the performance of queries. Maintaining additional structures incurs overhead, especially on writes. For writes, it’s hard to beat the performance of simply appending to a file, because that’s the simplest possible write operation. Any kind of index usually slows down writes, because the index also needs to be updated every time data is written.

Hash Index

Let’s say our data storage consists only of appending to a file, as in the preceding example. Then the simplest possible indexing strategy is this: keep an in-memory hash map where every key is mapped to a byte offset in the data file—the location at which the value can be found, as illustrated in Figure 3-1. Whenever you append a new key-value pair to the file, you also update the hash map to reflect the offset of the data you just wrote (this works both for inserting new keys and for updating existing keys). When you want to look up a value, use the hash map to find the offset in the data file, seek to that location, and read the value. This may sound simplistic, but it is a viable approach. In fact, this is essentially what Bitcask (the default storage engine in Riak) does [3]. Bitcask offers high-performance reads and writes, subject to the requirement that all the keys fit in the available RAM, since the hash map is kept completely in memory. The values can use more space than there is available memory, since they can be loaded from disk with just one disk seek. If that part of the data file is already in the filesystem cache, a read doesn’t require any disk I/O at all. A storage engine like Bitcask is well suited to situations where the value for each key is updated frequently. For example, the key might be the URL of a cat video, and the value might be the number of times it has been played (incremented every time someone hits the play button). In this kind of workload, there are a lot of writes, but there are not too many distinct keys—you have a large number of writes per key, but it’s feasible to keep all keys in memory. Moreover, since compaction often makes segments much smaller (assuming that a key is overwritten several times on average within one segment), we can also merge several segments together at the same time as performing the compaction, as shown in Figure 3-3. Segments are never modified after they have been written, so the merged segment is written to a new file. The merging and compaction of frozen seg‐ ments can be done in a background thread, and while it is going on, we can still con‐ tinue to serve read and write requests as normal, using the old segment files. After the merging process is complete, we switch read requests to using the new merged seg‐ ment instead of the old segments—and then the old segment files can simply be deleted.

Each segment now has its own in-memory hash table, mapping keys to file offsets. In order to find the value for a key, we first check the most recent segment’s hash map; if the key is not present we check the second-most-recent segment, and so on. The merging process keeps the number of segments small, so lookups don’t need to check many hash maps. Lots of detail goes into making this simple idea work in practice. Briefly, some of the issues that are important in a real implementation are:

The hash table must fit in memory, so if you have a very large number of keys, you’re out of luck. In principle, you could maintain a hash map on disk, but unfortunately it is difficult to make an on-disk hash map perform well. It requires a lot of random access I/O, it is expensive to grow when it becomes full, and hash collisions require fiddly logic [5]. • Range queries are not efficient. For example, you cannot easily scan over all keys between kitty00000 and kitty99999—you’d have to look up each key individu‐ ally in the hash maps.

Ai agents Notes

Tue, 10 Feb 2026 21:24:16 +0530

Worlflow

Prompt chaining decomposes a task into a sequence of steps, where each LLM call processes the output of the previous one. You can add programmatic checks on any intermediate steps to ensure that the process is still on track.

When to use this workflow: This workflow is ideal for situations where the task can be easily and cleanly decomposed into fixed subtasks. The main goal is to trade off latency for higher accuracy, by making each LLM call an easier task.

Prompt chaining decomposes a task into a sequence of steps, where each LLM call processes the output of the previous one. You can add programmatic checks (see “gate” in the diagram below) on any intermediate steps to ensure that the process is still on track.

Routing classifies an input and directs it to a specialized followup task. This workflow allows for separation of concerns, and building more specialized prompts. Without this workflow, optimizing for one kind of input can hurt performance on other inputs.

When to use this workflow: Routing works well for complex tasks where there are distinct categories that are better handled separately, and where classification can be handled accurately, either by an LLM or a more traditional classification model/algorithm.

LLMs can sometimes work simultaneously on a task and have their outputs aggregated programmatically. This workflow, parallelization, manifests in two key variations:

Sectioning: Breaking a task into independent subtasks run in parallel.
Voting: Running the same task multiple times to get diverse outputs.

When to use this workflow: Parallelization is effective when the divided subtasks can be parallelized for speed, or when multiple perspectives or attempts are needed for higher confidence results. For complex tasks with multiple considerations, LLMs generally perform better when each consideration is handled by a separate LLM call, allowing focused attention on each specific aspect.

In the orchestrator-workers workflow, a central LLM dynamically breaks down tasks, delegates them to worker LLMs, and synthesizes their results.When to use this workflow: This workflow is well-suited for complex tasks where you can’t predict the subtasks needed (in coding, for example, the number of files that need to be changed and the nature of the change in each file likely depend on the task). Whereas it’s topographically similar, the key difference from parallelization is its flexibility—subtasks aren’t pre-defined, but determined by the orchestrator based on the specific input.

In the evaluator-optimizer workflow, one LLM call generates a response while another provides evaluation and feedback in a loop. When to use this workflow: This workflow is particularly effective when we have clear evaluation criteria, and when iterative refinement provides measurable value. The two signs of good fit are, first, that LLM responses can be demonstrably improved when a human articulates their feedback; and second, that the LLM can provide such feedback. This is analogous to the iterative writing process a human writer might go through when producing a polished document.

Text to SQL in Pinterest

The user asks an analytical question, choosing the tables to be used.

The relevant table schemas are retrieved from the table metadata store.
The question, selected SQL dialect, and table schemas are compiled into a Text-to-SQL prompt.
The prompt is fed into the LLM.
A streaming response is generated and displayed to the user.

The table schema acquired from the metadata store includes: Table name Table description Columns Column name Column type Column description

Low-Cardinality Columns

Certain analytical queries, such as “how many active users are on the ‘web’ platform”, may generate SQL queries that do not conform to the database’s actual values if generated naively. For example, the where clause in the response might bewhere platform=’web’ as opposed to the correct where platform=’WEB’. To address such issues, unique values of low-cardinality columns which would frequently be used for this kind of filtering are processed and incorporated into the table schema, so that the LLM can make use of this information to generate precise SQL queries.

Context Window Limit

Extremely large table schemas might exceed the typical context window limit. To address this problem, we employed a few techniques:

Reduced version of the table schema: This includes only crucial elements such as the table name, column name, and type. Column pruning: Columns are tagged in the metadata store, and we exclude certain ones from the table schema based on their tags.

you are a {dialect} expert.

Please help to generate a {dialect} query to answer the question. Your response should ONLY be based on the given context and follow the response guidelines and format instructions.

===Tables
{table_schemas}

===Original Query
{original_query}

===Response Guidelines
1. If the provided context is sufficient, please generate a valid query without any explanations for the question. The query should start with a comment containing the question being asked.
2. If the provided context is insufficient, please explain why it can't be generated.
3. Please use the most relevant table(s).
5. Please format the query before responding.
6. Please always respond with a valid well-formed JSON object with the following format

===Response Format
{{
 "query": "A generated SQL query when context is sufficient.",
 "explanation": "An explanation of failing to generate the query."
}}

===Question
{question}

spider dataset : https://arxiv.org/pdf/2204.00498

An offline job is employed to generate a vector index of tables’ summaries and historical queries against them. If the user does not specify any tables, their question is transformed into embeddings, and a similarity search is conducted against the vector index to infer the top N suitable tables. The top N tables, along with the table schema and analytical question, are compiled into a prompt for LLM to select the top K most relevant tables. The top K tables are returned to the user for validation or alteration. The standard Text-to-SQL process is resumed with the user-confirmed tables.

Offline Vector Index Creation

Table Summarization There is an ongoing table standardization effort at Pinterest to add tiering for the tables. We index only top-tier tables, promoting the use of these higher-quality datasets. The table summarization generation process involves the following steps:

Retrieve the table schema from the table metadata store. Gather the most recent sample queries utilizing the table. Based on the context window, incorporate as many sample queries as possible into the table summarization prompt, along with the table schema. Forward the prompt to the LLM to create the summary. Generate and store embeddings in the vector store. The table summary includes description of the table, the data it contains, as well as potential use scenarios. Here is the current prompt we are using for table summarization:

prompt_template = """
You are a data analyst that can help summarize SQL tables.

Summarize below table by the given context.

===Table Schema
{table_schema}

===Sample Queries
{sample_queries}

===Response guideline
 - You shall write the summary based only on provided information.
 - Note that above sampled queries are only small sample of queries and thus not all possible use of tables are represented, and only some columns in the table are used.
 - Do not use any adjective to describe the table. For example, the importance of the table, its comprehensiveness or if it is crucial, or who may be using it. For example, you can say that a table contains certain types of data, but you cannot say that the table contains a 'wealth' of data, or that it is 'comprehensive'.
 - Do not mention about the sampled query. Only talk objectively about the type of data the table contains and its possible utilities.
 - Please also include some potential usecases of the table, e.g. what kind of questions can be answered by the table, what kind of analysis can be done by the table, etc.
"""

Query Summarization Besides their role in table summarization, sample queries associated with each table are also summarized individually, including details such as the query’s purpose and utilized tables. Here is the prompt we are using:

prompt_template = """
You are a helpful assistant that can help document SQL queries.

Please document below SQL query by the given table schemas.

===SQL Query
{query}

===Table Schemas
{table_schemas}

===Response Guidelines
Please provide the following list of descriptions for the query:
-The selected columns and their description
-The input tables of the query and the join pattern
-Query's detailed transformation logic in plain english, and why these transformation are necessary
-The type of filters performed by the query, and why these filters are necessary
-Write very detailed purposes and motives of the query in detail
-Write possible business and functional purposes of the query
"""

NLP Table Search When a user asks an analytical question, we convert it into embeddings using the same embedding model. Then we conduct a search against both table and query vector indices. We’re using OpenSearch as the vector store and using its built in similarity search ability.

Considering that multiple tables can be associated with a query, a single table could appear multiple times in the similarity search results. Currently, we utilize a simplified strategy to aggregate and score them. Table summaries carry more weight than query summaries, a scoring strategy that could be adjusted in the future.

Other than being used in the Text-to-SQL, this NLP-based table search is also used in the general table search in Querybook.

RAG

Docker & kubernetes

Fri, 14 Nov 2025 12:38:31 +0530

Docker

Open Source

Go Lang

Sun, 10 Aug 2025 15:24:40 +0530

Interface

An interface type in Go is kind of like a definition. It defines and describes the exact methods that some other type must have.

type Stringer interface {
 String() string
}

We say that something satisfies this interface (or implements this interface) if it has a method with the exact signature String() string.

type Book struct {
 Title string
 Author string
}

func (b Book) String() string {
 return fmt.Sprintf("Book: %s - %s", b.Title, b.Author)
}

Wherever you see declaration in Go (such as a variable, function parameter or struct field) which has an interface type, you can use an object of any type so long as it satisfies the interface.

func WriteLog(s fmt.Stringer) {
 log.Print(s.String())
}

Because this WriteLog() function uses the fmt.Stringer interface type in its parameter declaration, we can pass in any object that satisfies the fmt.Stringer interface.

package main

import (
 "fmt"
 "strconv"
 "log"
)

// Declare a Book type which satisfies the fmt.Stringer interface.
type Book struct {
 Title string
 Author string
}

func (b Book) String() string {
 return fmt.Sprintf("Book: %s - %s", b.Title, b.Author)
}

// Declare a Count type which satisfies the fmt.Stringer interface.
type Count int

func (c Count) String() string {
 return strconv.Itoa(int(c))
}

// Declare a WriteLog() function which takes any object that satisfies
// the fmt.Stringer interface as a parameter.
func WriteLog(s fmt.Stringer) {
 log.Print(s.String())
}

func main() {
 // Initialize a Count object and pass it to WriteLog().
 book := Book{"Alice in Wonderland", "Lewis Carrol"}
 WriteLog(book)

 // Initialize a Count object and pass it to WriteLog().
 count := Count(3)
 WriteLog(count)
}

output:

2009/11/10 23:00:00 Book: Alice in Wonderland - Lewis Carrol
2009/11/10 23:00:00 3

Advantage

To help reduce duplication or boilerplate code.
To make it easier to use mocks instead of real objects in unit tests.
As an architectural tool, to help enforce decoupling between parts of your codebase.

the empty interface type interface{} is kind of like a wildcard. Wherever you see it in a declaration (such as a variable, function parameter or struct field) you can use an object of any type.

package main

import "fmt"


func main() {
 person := make(map[string]interface{}, 0)

 person["name"] = "Alice"
 person["age"] = 21
 person["height"] = 167.64

 fmt.Printf("%+v", person)
}

Error Handling

The error type is an interface type. An error variable represents any value that can describe itself as a string. Here is the interface’s declaration:

type error interface {
 Error() string
}

Hugo Shortcuts

Thu, 22 May 2025 15:35:24 +0530

Shortcuts

to create a page

hugo new my-new-page.md

to create a blog

hugo new blog/my-new-post.md

run locally

hugo server

This shortcuts will be used by me in future.