Reducing Latency Through Colocation

In its first two chapters, Latency gives an overview of what latency is and how to measure it. From this chapter onward, the book shifts its focus to latency optimization, beginning with an approach called colocation.

What Is Colocation, and How Does It Reduce Latency?

Imagine a web application whose client is in Asia, server is in the Americas, and database is in Europe. When a user performs an action, the client's request might travel 10,000 kilometers to reach the server. The server's request to the database then travels another 5,000 kilometers in each direction before the response makes the 10,000-kilometer journey back to the client.

Data takes time to travel. Even over fiber-optic networks, those extra distances add at least 150 milliseconds of latency—and that estimate does not account for submarine cables following indirect routes rather than straight lines. Moving both the server and database to Asia, or even into the same region as the client, could bring the network latency between them below 10 milliseconds. Deploying the application itself closer to users could reduce the network latency of the entire request even further.

In other words, simply moving machines physically closer together can substantially reduce latency. Placing components near one another in this way is called colocation. It works because reducing the distance between nodes reduces the communication latency of a distributed system.

Colocation is not limited to separate machines. The same principle can reduce latency within a single computer. The author uses the CPU as an example: even inside one machine, the distance between data and the CPU affects access latency. If data is in a CPU cache, the processor can retrieve it in under 10 nanoseconds. If it is not cached, the CPU must access DRAM, which takes roughly 100 nanoseconds—a difference of up to an order of magnitude.

Edge Computing as an Example

In practice, edge computing is probably the first concept that comes to mind when discussing colocation. It moves computation and data closer to end users to reduce latency. By shortening the distance between nodes, edge computing lowers the latency introduced by communication.

This communication latency can be divided into two broad categories: geographical latency and last-mile latency.

Geographical latency comes from the physical distance between two nodes. The earlier example of sending a request from Asia to the Americas and returning data from the Americas to Asia illustrates this round-trip cost. Placing the server and database together reduces it, but moving both closer to end users can reduce it further. Deploying server-side resources nearer to clients is another form of colocation.

A content delivery network (CDN) is the most widely used way to do this in practice. CDNs operate servers in data centers around the world. Developers can place frequently requested scripts, images, and videos on these CDN nodes so that the content is physically closer to clients, reducing the time needed to transfer it. We covered this topic in more detail in our earlier article, Essential System Design Concepts — CDN (Content Delivery Network)?, which is worth revisiting.

Last-mile latency is the delay between the nearest backbone network and the user's device. Based on distance alone, this portion should theoretically take less than one millisecond. In reality, various factors can push last-mile latency into the tens of milliseconds, so it cannot be ignored.

Intranode Latency

The previous section focused on internode latency, where communication between separate nodes in a distributed system is the primary source of delay. Intranode latency, by contrast, covers delays within a single node. These delays arise from both the software stack and hardware architecture, and colocation can reduce them as well.

When two machines communicate over a network, their data passes through multiple layers of abstraction. Each layer uses protocols that define how data is sent and received. TCP/IP and HTTP, both familiar to frontend and backend engineers, operate at different layers of this stack.

After a client sends an HTTP request, the application-layer data is passed to TCP and encapsulated in an IP packet. If the underlying network uses Ethernet, that packet is then encapsulated in an Ethernet frame. Each layer may also offer a choice of protocols. At the transport layer, for example, an application can use UDP instead of TCP.

Compared with TCP, UDP can be better suited to low-latency networks inside a data center. Its lower latency comes with a tradeoff: UDP does not guarantee delivery. If a router drops a network packet containing a UDP datagram, the network stack will not retransmit it. On a comparatively unreliable public network, the application layer must be able to tolerate data loss; otherwise, UDP may be a poor choice.

TCP, by contrast, provides reliable, ordered delivery and retransmits lost data. That reliability adds overhead, however, so TCP may not be ideal for applications that are highly sensitive to latency. Several industry solutions, such as Aeron, therefore build reliability on top of UDP, aiming to preserve UDP's low latency while approaching TCP's reliability.

Some of these layers remain involved even when separate processes communicate on the same machine. A server and database running locally still need an interprocess communication mechanism if they belong to different processes. For example, a TCP connection to a local PostgreSQL instance still passes through the operating system's TCP/IP network stack, adding a small amount of latency.

Support ExplainThis

If you found this content valuable, please consider supporting our work with a one-time donation of whatever amount feels right to you through this Buy Me a Coffee page.

Creating in-depth technical content takes significant time. Your support helps us continue producing high-quality educational content accessible to everyone.