DDIA Chapter 6 — Replication (Part 2)

In DDIA Chapter 6 — Replication (Part 1), we discussed replication in data systems and its most basic architecture: single-leader replication. In the second half of Chapter 6, the authors explore two other approaches: multi-leader replication and leaderless replication.

Multi-Leader Replication

One of the main drawbacks of single-leader replication is that every write must go through the same leader. If that node fails or becomes unreachable, the database cannot accept new writes. Imagine that a replica in Asia is working perfectly, but the primary database in the Americas is unreachable. The database right in front of you is healthy, yet you still cannot write to it.

This is one reason multi-leader replication exists. If relying on a single primary database means one node can block writes, perhaps adding more nodes that can accept writes will solve the problem.

In simple terms, single-leader replication looks like this:

Multi-leader replication, by comparison, looks like this:

Any node that accepts writes acts as a primary database. When a primary receives a write, it replicates the change to the other primaries. In that sense, each primary also acts as a replica of the others.

In a typical deployment spanning multiple regions, each region has one primary database and several read-only replicas. This keeps the overall architecture simpler. The regional primaries replicate changes between regions, and each primary then passes incoming changes to its local replicas.

Benefits of Multi-Leader Replication

Multi-leader replication offers several advantages over a single-leader setup. The first is better performance. With one leader, writes from other regions must travel farther over the network. A write from Taiwan to a primary database in the United States, for example, incurs latency because of the physical distance. If Asia has its own primary, the request can go to that nearby database instead, reducing latency.

The second advantage is greater tolerance of failures. If the region hosting the primary fails in a single-leader setup, the system must perform a cross-region failover, promoting a replica elsewhere to become the primary. That process can be complicated. In a multi-leader setup, primaries in other regions can keep accepting writes when one region fails. The system therefore may avoid the write outage that a single-leader architecture would experience. This resilience is particularly valuable when high availability matters.

The third advantage is greater tolerance of network failures between regions. A single-leader architecture depends heavily on cross-region connectivity: users elsewhere need to reach the primary to write data. Longer network paths can also create more opportunities for problems. Multi-leader replication still involves cross-region communication, but it mainly happens between regional primaries. Replicas communicate with their local primary within the same region, making that replication less sensitive to cross-region network problems.

Challenges of Multi-Leader Replication

These benefits come with unavoidable challenges. The biggest is consistency.

The book gives an example involving bank transactions. Suppose an account has a balance of 100, with primary databases in regions A and B. Because replication takes time, both primaries currently see a balance of 100. If each region receives a request to withdraw 80, primary A may accept it because 80 is less than 100. Primary B reaches the same conclusion.

Each withdrawal looks valid on its own, but together they amount to 100 - 80 - 80, leaving an invalid balance of -60. That may not even be the final stored balance if the system uses last-write-wins conflict resolution. The key problem is that the system accepted both withdrawals of 80, even though it should not have.

Avoiding this conflict requires coordination between regions when processing writes. That communication adds latency. This is a fundamental limitation of distributed systems, and choosing multi-leader replication means confronting it. When making an engineering decision, we need to ask what matters most for the system at hand: low latency or consistency across regions?

Multi-leader replication is therefore a less suitable choice when an application must enforce global consistency. Examples include preventing negative bank balances, ensuring inventory never drops below zero, and making sure a ticket is sold to only one person. In these scenarios, the cost of inconsistency will often outweigh the benefits of multi-leader replication.

How Do Multiple Leaders Replicate Data?

After discussing the benefits and challenges, the book explains how data moves between leaders. It uses the term replication topology to describe how changes propagate between nodes. Common multi-leader topologies include all-to-all, circular, and star, with all-to-all being the most general approach.

As the diagram shows, in an all-to-all topology, each primary sends its writes directly to every other primary. This offers strong fault tolerance: nodes have direct paths between them, so a failure on one connection does not block communication between the other nodes.

The downside is more connections and greater complexity. Each node has more replication partners to manage. Ordering is another challenge. Primaries are often in different regions, so network transmission times vary. A client might insert a record on primary A and then update it on primary B. If B has not yet received the original insert, the update can arrive before the insert, reversing their causal order.

In a circular topology, each node forwards changes only to the next node in the ring. This is much simpler than all-to-all: each node communicates only with its predecessor and successor. But if a node fails, the ring breaks until the topology is reconfigured. Nodes farther along the ring stop receiving earlier changes, making this approach relatively fragile.

A star topology has a central node. Changes from any node first go to the center, which then forwards them to the other nodes. Every node except the center has a simpler job, managing communication with just the central node. But if the center fails and the system does not complete failover immediately, replication across the system stalls. The center is a single point of failure.

These topology choices illustrate a familiar tradeoff in distributed systems. A simpler approach, such as a circular topology, costs less to manage but lacks alternative paths when failures occur. A more fault-tolerant approach brings greater management complexity.

The second half of this article will introduce leaderless replication. That content is available exclusively to E+ members.

Support ExplainThis

If you found this content helpful, please consider supporting our work with a one-time donation of whatever amount feels right to you through this Buy Me a Coffee page.

Creating in-depth technical content takes significant time. Your support helps us continue producing high-quality educational content accessible to everyone.