DDIA Ch1 Reading Guide — Tradeoffs in Data-Intensive System Design

"Designing Data-Intensive Applications" is a comprehensive guide written by Martin Kleppmann, a distributed systems researcher at Cambridge University, published in 2017. It has become one of the essential reading materials for anyone approaching system design. With the second edition released in 2026, we will be providing chapter-by-chapter reading guides throughout 2026's E+ backend content, covering key takeaways and lessons from each chapter.

Both the first and second editions of "Designing Data-Intensive Applications" approach the topic from a broader perspective, examining what aspects developers should focus on when building data-intensive application systems.

The first edition explores three angles: reliability, scalability, and maintainability. However, the second edition shifts focus to ask: "what tradeoffs should backend leaders in teams make when designing real-world systems?" Since the second edition integrates the concepts of reliability, scalability, and maintainability into Chapter 2, we'll focus on the second edition's content in this guide. A deeper exploration of these three important non-functional requirements will come in our next reading guide.

Returning to the book's title, the author emphasizes that data-intensive applications should be distinguished from computation-intensive applications. For computation-intensive systems, the core concern is accelerating computation speed, so the focus centers on parallelizing work and optimizing algorithms.

For data-intensive applications, the emphasis shifts toward data handling, covering questions like:

• How do we store data? (Which database should we choose from the many options available)
• How do we accelerate data retrieval? (Through caching and similar techniques)
• How do we make searching and filtering faster? (Through indexing strategies)
• How do we process data in real-time? (Through streaming approaches)
• How do we process accumulated data periodically? (Through batch processing)

Each of these questions requires thoughtful judgment and careful selection in practice. Subsequent chapters explore these topics in depth. Chapter 1 begins by examining several key tradeoffs from a high-level perspective.

Is Your System Transactional (Operational) or Analytical?

When a data system handles modest data volumes, there's often no pressing need to separate concerns. A SQL database's flexibility allows it to handle both transactional and analytical tasks adequately. However, as data volume grows, such as when an e-commerce platform manages hundreds of thousands of products or a social media platform serves millions of users, treating everything in a single system typically creates problems.

A common challenge emerges because the system serves not only backend engineers but also data analysts and data scientists. When the system contains millions or tens of millions of records, an analyst's query can cause system performance to degrade significantly. Either you ask the analyst to pause their work, which halts business intelligence work, or the degradation impacts end users. This creates an untenable resource contention.

This fundamental conflict is why modern large-scale, data-intensive systems split concerns into separate systems handling each task type. These generally fall into two categories: Online Transaction Processing (OLTP) for transactional work, and Online Analytical Processing (OLAP) for analytical work.

OLTP systems typically comprise various backend services that execute reads and modifications in response to user operations. OLTP systems process small datasets per operation—updating a single record, for instance. Because they respond to user actions, the data processed is always current.

OLAP systems serve data scientists and analysts. Since analysis doesn't require modifying data, OLAP systems focus on querying. These queries span large datasets—for example, calculating annual revenue for a merchant requires aggregating an entire year's transaction records. Common OLAP systems include BigQuery, Redshift, and Snowflake, with open-source options like ClickHouse.

Data Warehouses and Data Lakes

Given these analytical requirements, data-intensive systems typically introduce a separate OLAP system alongside the OLTP system. This context gave rise to the data warehouse concept, which became the dominant OLAP solution.

A data warehouse allows data scientists and analysts to query freely without impacting OLTP operations. In a system with multiple OLTP databases—such as a Sales DB handling transactions, an Inventory DB managing stock, and a Geo DB planning delivery routes—an ETL (Extract-Transform-Load) process can consolidate data into a warehouse. Data professionals query the warehouse directly rather than individual operational databases, preserving system performance.

However, data warehouses involve tradeoffs. The ETL process transforms data upfront—cleaning and standardizing formats—before loading. This approach, which predetermines data structure before storage, optimizes queries with efficient SQL, but reduces flexibility.

As analytical requirements grow more complex, data scientists often need formats outside traditional relational databases. To accommodate diverse analytical needs, the data lake concept emerged. Data lakes follow an ELT (Extract-Load-Transform) pattern: data loads after extraction without enforcing specific structure. Transformation happens later, tailored to each use case. This enables systems to serve varied analytical requirements dynamically.