DDIA Chapter 6 — Replication (Part 1)

Chapter 6 of Designing Data-Intensive Applications introduces replication, a core concept in data-intensive systems. Instead of storing data on just one machine, replication keeps copies of the same data on several machines.

At first glance, this seems expensive. Keeping multiple copies takes more storage, and buying or renting additional machines adds cost. So why replicate data? What benefits does it bring? What makes replication difficult in practice, and what tradeoffs do different approaches involve? These are the questions the chapter explores.

Why Replicate Data?

Replication comes at a cost, but it also offers several benefits. The book highlights three main ones.

First, replication can bring data closer to users, making reads faster. Suppose a user is in Taiwan, but the database is in the United States. Every time the user browses a page, the request has to cross the Pacific to reach the data center, and the response has to travel back. That distance adds latency.

Deploying read replicas closer to users can reduce this latency. If a local replica can handle some reads, users in Taiwan do not need to reach a US data center for every request. In a single-leader architecture, however, writes usually still go to the leader. Geographic replicas primarily improve latency for reads that can be served by those replicas.

Second, replication improves availability. If all the data lives on one machine, a failure on that machine can take the service down. With multiple replicas, another machine can take over when one fails, keeping the system available. Replication therefore reduces the impact of a single point of failure.

Third, replication increases read throughput. Many systems handle far more reads than writes: people browse products more often than they place orders, and read social media posts more often than they publish them. If one machine handles every read, traffic will eventually hit that machine's capacity limit. Multiple replicas let the system distribute requests across machines, increasing overall read throughput.

Single-Leader Replication

Replication raises an unavoidable question: how do we ensure that every replica eventually receives all the data? Whenever the database accepts a write, its replicas need to be updated too. Otherwise, their data will diverge. Single-leader replication is one of the most common ways to address this problem.

As the name suggests, one machine accepts writes, and the others replicate its changes. There are two roles: the leader, also called the primary database or primary node, and the followers, also called replicas or secondary nodes. All client writes go to the leader, which forwards the changes to its replicas. Replicas do not accept direct client writes. Whether they serve reads depends on the database's capabilities and the system's configuration.

!image.png

Having one primary database and multiple replicas gives writes a clear entry point and makes the setup relatively straightforward to maintain. Many databases, including PostgreSQL, MySQL, and MongoDB, use single-leader replication as their default replication approach.

Synchronous vs. Asynchronous Replication

Single-leader replication is a relatively simple architecture, but it still presents several practical challenges.

The first is deciding what counts as a successful write. Imagine a user updating their profile picture on a social media site. The write reaches the primary database, which saves the new data and sends the change to its replicas. Should the primary immediately report success, or should it wait for the replicas to acknowledge the update?

The diagram below shows several possible timelines. One option is for the primary to return ok as soon as it completes the change, with all replicas updating asynchronously. Another is to return ok once one replica, such as Replica 1, has completed the update. This combines synchronous and asynchronous replication: Replica 1 is synchronous, while the others are asynchronous. A third option is to wait for every replica to finish updating before returning ok, making all replication synchronous.

!image.png

What tradeoffs would you expect from these three approaches?

Synchronous replication reduces the risk of losing acknowledged writes if the leader fails. Depending on the configuration, the system may guarantee that at least one replica, a specified number of replicas, or all synchronous replicas have received and durably stored the change. This offers stronger protection for systems where data safety matters, such as financial or inventory systems. The obvious downside is that users wait longer for ok. If a replica has a problem, the primary may wait even longer for its acknowledgment. That blocking can hurt the user experience.

Asynchronous replication takes the opposite approach. The primary can return ok to the client quickly, improving responsiveness. But because it does not ensure that replicas have successfully applied the update, there is a greater risk of data loss. This approach is common where low latency matters and some data loss is acceptable, such as event analytics logs or browsing history tracking. Losing a small slice of data in these systems may not cause a serious incident.

A mix of synchronous and asynchronous replication offers a compromise. For example, the primary can return ok to the user as soon as one replica acknowledges the write. This ensures that at least one other node has saved the change without waiting for every replica, keeping latency lower.

In practice, the choice depends on the system's requirements. A mixed setup might wait until a majority of replicas have updated—for example, three out of five—before returning ok. There is no universal answer. The decision depends on how much the system prioritizes data safety relative to low latency.

The Challenge of Adding a Replica

Choosing between synchronous and asynchronous replication is not the only challenge. Adding a new replica also takes care.

You cannot simply start another machine and copy the primary database's data onto it, because most primary databases are constantly changing. Clients keep writing new data. It is like taking a book with ten chapters to a copy shop, only to discover that two more chapters have been added by the time the copying finishes. Your copy is already incomplete.

An obvious solution is to lock the primary database and temporarily stop accepting writes, but that reduces availability. Many systems still take this approach when creating an initial replica because it makes consistency easier to achieve and verify. The tradeoff is that the service cannot accept writes normally during that period.

To add a replica without downtime, the book describes an approach based on a snapshot of the primary database. First, copy the snapshot to the new replica, giving it the primary's state at a particular point in time. Then identify all changes made after that point—using a log sequence number in PostgreSQL, or a binlog position or GTID in MySQL—and apply any changes not already included in the snapshot. This lets the replica catch up with the primary.

The Cost of Replicas

As we discussed at the beginning, multiple replicas increase operating costs. In a modern cloud architecture, renting high-performance disks or memory can be expensive. If much of the database's data is rarely accessed, keeping it on expensive disks and in memory across multiple replicas is like renting an office in a prime location just to store cardboard boxes. It is hard to justify on cost alone.

This has led to architectures that incorporate object storage services such as AWS S3 or Google Cloud Storage. These services can be much cheaper than cloud databases while offering strong durability and cross-region replication capabilities. S3, for example, makes it straightforward to automate replication across regions.

Object storage may be inexpensive and convenient, but its access latency is much higher than that of a disk. In practice, systems using object storage often adopt a tiered architecture: frequently accessed data stays on disk or in memory, while less frequently accessed data lives in object storage.

A more recent idea is a diskless architecture. Here, local disks are not the durable storage layer. Data that needs to persist lives in object storage, while local disks and memory serve only as caches. This makes the primary database node close to stateless. If it fails, the durable data remains in object storage. There is still a tradeoff, though: recovering from a primary failure requires fetching data from object storage, which adds access latency.

The second half will explore how to handle replication lag. When replicas have not caught up with the leader, users may read stale data. How should applications respond? That follow-up is available exclusively to E+ members.

Support ExplainThis

If you found this content helpful, please consider supporting our work with a one-time donation of whatever amount feels right to you through this Buy Me a Coffee page.

Creating in-depth technical content takes significant time. Your support helps us continue producing high-quality educational content accessible to everyone.