What Is Latency, and Why Does It Matter?

When designing systems, low latency is an important non-functional requirement for many products. A reader recently asked whether we could write a guided reading of the book Latency. After reading through it, I found it insightful: the book offers useful mental models and practical methods for building low-latency systems. We will cover it chapter by chapter in future member articles.

In this article, we will start with what latency actually means, why it matters to software engineers, and how latency often needs to be traded off against other technical concerns.

What Is Latency?

In everyday conversation, people often describe a system as "fast" or "laggy." In system design, however, latency is the more precise term. That is why engineers are more likely to talk about "low latency" than simply saying something is "fast."

Latency is the amount of time between something happening and the moment we can observe its result.

For example, when you use a light switch at home, the time between pressing the switch and the bulb turning on is the latency of turning on the light. With a traditional light switch, the latency is so low that it is almost impossible to notice. But with a smart bulb, after tapping the button in a mobile app, you may need to wait a short moment before the light turns on. That noticeable feeling of being "one beat late" is a sign of higher latency.

Traditional lights and smart lights differ in latency because a smart light has to go through more layers after the user presses the button. The command may first travel over a wireless network to a control hub, then reach the light through Bluetooth or another communication protocol. That extra transmission time makes the latency higher than a traditional light switch.

Modern software systems work the same way. When a user enters a URL in the browser and presses Enter, it takes some time before the page appears on screen. That time is latency. The delay exists because many things happen after the user presses Enter: the browser performs DNS lookup to resolve the domain into an IP address, establishes a connection, completes a TLS handshake if needed, sends an HTTP request, receives the response, then renders the HTML, CSS, and JavaScript.

Each step adds to the total latency. So when a user feels that a website is slow, the cause may not be a single issue. It may be the accumulated result of DNS resolution, the network, the server, the database, frontend rendering, and more. When developers inspect latency, they should not look only at one piece of code. They need to expand the full path and examine each segment to identify what is actually causing excessive latency.

Why Low Latency Matters

Now that we have a basic understanding of latency, let's discuss why low latency matters. In both real-world system design and technical interviews, "how would you reduce latency?" is a common question. Why is latency so important in both contexts? The book approaches this from several angles.

User Experience

First, latency directly affects user experience. If a website takes 10 seconds to show content, most people will likely leave. Amazon and Google have both studied this topic and found that reducing latency can improve customer purchase behavior and user engagement. For example, Amazon found that every 100 milliseconds of latency could have roughly a 1% impact on sales. From a conversion-rate perspective, lowering system latency can produce meaningful business impact.

It is worth noting, though, that human perception of latency is not linear. Once latency crosses a certain threshold, user experience can suddenly feel much worse. In practical terms, changes within 100 milliseconds often feel instantaneous to humans, so the difference between 100 ms and 60 ms may not be noticeable. But once latency crosses that threshold, users begin to feel it. Akamai has reported that 53% of mobile site visitors leave pages that take longer than 3 seconds to load.

Because latency is tied to experience, a system can sometimes provide a better user experience even when total latency stays the same. Large language models are a good example. LLM responses often take some time to complete, so a common industry practice is streaming: as soon as a new chunk of output is available, it is sent to the client. Instead of waiting 30 seconds and then seeing the full response all at once, users can see the first token around the first second, then watch the response continue to arrive over the next 1 to 30 seconds. This makes users much more willing to stay in the application.

Real-Time Requirements and Efficiency

Beyond user experience, some systems and applications have extremely strict real-time requirements. In those cases, reducing latency as much as possible becomes a core design goal. For example, if the sensors in a pacemaker or a self-driving car have high latency, the consequences can become life-threatening.

The author also points out that optimizing latency is, in many cases, the same as making a program more efficient. For decades, Moore's law meant that software could often get lower latency without changing, simply because hardware kept improving and CPUs gained more parallel execution units. But as transistors became smaller and power density increased, CPU frequency growth began to stall. In that context, software teams can no longer assume that the next generation of hardware will automatically make everything faster. They need to actively reduce unnecessary computation, network calls, and data transformations so the software runs more efficiently and with lower latency.

How to Describe Latency

After understanding why latency matters, the next question is how to describe it. In general, we describe latency using units of time. For example, the latency of accessing data from an SSD is about 100 microseconds, while a network round trip from New York to London is about 60 milliseconds.

The book lists several common latency numbers, which are also useful for engineering estimates. If this idea is unfamiliar, it is worth reviewing the article Numbers and Estimation Every Engineer Should Know.

There are a few important ideas behind these numbers. First, latency has a hard limit. Light travels about 300,000 kilometers per second, which is the theoretical physical lower bound. In the real world, optical fiber is not a perfect vacuum, so light signals travel more slowly than that theoretical limit. Once you recognize this limit, you can start thinking about optimization under physical constraints. If there is only so much distance a signal can travel per second, then shortening the distance between machines can reduce latency.

Second, these latency values are measured across powers of ten, so the difference between "fast" and "slow" may be tens or thousands of times. But when people look at the table above, those numbers may still feel abstract. Below is Grace Hopper's classic MIT lecture. In the talk, Hopper uses physical wire to show how far an electrical signal can travel in one nanosecond, about 30 centimeters, and compares that with the distance it can travel in one microsecond.

If you cannot feel the difference between a nanosecond and a microsecond, it is easy to underestimate latency optimization. Hopper's demonstration makes the low-latency world feel concrete: time is extremely sensitive at this scale. Moving machines closer together or removing unnecessary detours in data transmission can produce a meaningful improvement.

Tradeoffs Between Latency and Other Metrics

Latency matters, but it is not the only important system metric. In the final part of this chapter, the author discusses how latency trades off against other metrics.

The Tradeoff Between Latency and Throughput

Throughput is another important performance metric. It describes how much work a system successfully completes per unit of time: how many requests per second, how many messages per minute, or how many tasks per hour.

In simple terms, latency asks, "How long does one thing take to finish?" Throughput asks, "How much work actually gets completed per second, minute, or hour?" A familiar real-world analogy is shipping goods from Taiwan to the United States. Ocean freight has high throughput and can carry large items, but it has poor latency because delivery takes weeks. Air freight has lower throughput because a plane carries less than a cargo ship, but it has better latency because delivery can happen in days.

Sometimes latency and throughput need to be traded off. The author uses laundry as an analogy for CPU pipelining. Suppose washing clothes takes 30 minutes, drying clothes takes 60 minutes, and drying can only start after washing finishes. There are several possible designs.

The first design is to dry each batch immediately after washing it. As shown below, the latency for one batch of laundry is 90 minutes, and the throughput is 1/90 batch per minute. However, if we use an approach similar to CPU pipelining, the washer starts the second batch as soon as the first batch is done, and the dryer starts the next batch as soon as it becomes available. That looks like the second version in the diagram. In this case, throughput improves to 1/60 batch per minute, but latency becomes worse: the second batch takes 120 minutes to complete, because it starts washing at minute 30 and finishes at minute 150.

This is the tradeoff between latency and throughput. One approach finishes a single task faster; the other completes more work overall. In engineering practice, neither is inherently better. What matters is which approach fits the situation.

Support ExplainThis

If you found this content valuable, please consider supporting our work with a one-time donation of whatever amount feels right to you through this Buy Me a Coffee page.

Creating in-depth technical content takes significant time. Your support helps us continue producing high-quality educational content accessible to everyone.