DDIA Chapter 5 Guide — Encoding and Evolution, Part 2

In DDIA Chapter 5 — Encoding and Evolution, Part 1, we covered the first half of the chapter: encoding formats such as JSON, XML, Protocol Buffers, and Avro. As discussed there, we need encoding and decoding because once data leaves the current process's memory, it must be converted into a form that another program can read. Whether data is sent to another server, written to a database, or placed on a message queue, every destination needs to understand the data format.

It is not enough for different systems to agree on a format once. When systems evolve, new and old versions must continue to work together. That means data format evolution needs to consider both forward compatibility, where old code can handle new data, and backward compatibility, where new code can read old data.

After comparing encoding formats, the chapter turns to the places where data actually flows: databases, services, workflow engines, and asynchronous messaging. It also discusses how data moves through event-driven architectures.

Databases Need Compatibility

Many people think of a database as a place where data is simply stored. Another useful way to think about it is as a data handoff point. One version of a service writes data at one point in time, and another version may read it later. The reader might be newer code, which is why compatibility matters for databases.

If the database format is not backward compatible, a newer version of the application may fail to read data written by an older version. After deployment, old records could become unreadable. From a software engineering perspective, that is a serious incident.

Modern data systems also often have multiple services reading from and writing to the same database. These service instances may be upgraded gradually: one instance first, then another, until the old version is gone. During that window, upgraded instances may write data in the new format while old instances are still running and may encounter that new data. Forward compatibility is what keeps the old instances from breaking.

For example, suppose an order system originally has three states: pending, paid, and cancelled. Later, refund support adds a new state: refunded. Once the new service starts writing refunded, old services that do not know that state may fail if they assume only the original three values exist. A forward-compatible design prevents the system from crashing just because it sees an unfamiliar value.

Databases also naturally contain records written at different times. Some records may have been written today; others may have been written years ago. Those records may follow different versions of the schema, so the application needs to handle them gracefully.

One way to deal with this is data migration: rewrite old data into the new format. But at large scale, full migrations can be expensive. For simple changes, using defaults such as null may be easier. More complex changes may still require application-level migration logic to ensure data can be interpreted correctly.

Client-Server Communication

Beyond databases, modern systems also need communication between clients and servers, and between servers themselves. In all of these interactions, the two sides need to agree on a data format.

In web development, the browser is the client and the web server is the server. When a user opens a page, the browser sends GET requests to fetch HTML, CSS, JavaScript, images, and other static assets so it can render the page and make it interactive.

In a microservices architecture, one server can be the client of another server. For example, an order service may call a payment service. From the payment service's perspective, the order service is the client. The same pattern applies when a backend system calls a third-party payment API.

In web systems, clients usually call servers through APIs. Among the many API styles, REST is one of the most common. REST is often used with HTTP: URLs represent resources, HTTP methods such as GET and POST express operations, and standard HTTP features handle topics such as caching and content negotiation.

Even with REST, clients still need to know many details: which HTTP method each endpoint uses, what the request should contain, and what the response looks like. This is where interface definition languages, or IDLs, help. You can think of an IDL as an API specification. For REST/HTTP APIs, OpenAPI is common. For RPC-style APIs, gRPC commonly uses Protocol Buffers .proto files to describe service interfaces.

Here is the OpenAPI example from the book. It defines an API called Ping, Pong. A local server at http://localhost:8080 exposes a /ping endpoint, which can be called with GET. On success, it returns a JSON object containing a message.

openapi: 3.0.0
info:
  title: Ping, Pong
  version: 1.0.0
servers:
  - url: http://localhost:8080
paths:
  /ping:
    get:
      summary: Given a ping, returns a pong message
      responses:
        "200":
          description: A pong
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
                    example: Pong!

With a format like this, teams can communicate API contracts much more clearly. Many frameworks can also generate type definitions and validation code from an IDL, so developers can update clients automatically rather than hand-maintaining every detail.

The Problem with RPC

Any discussion of client-server communication eventually runs into remote procedure calls, or RPC, a model that has existed since the 1970s. The idea is to make a call to a remote procedure look like a local function call. This can make distributed systems feel simple in code, but the book warns that network calls are still fundamentally different from local function calls.

RPC can make it too easy to forget that the request is crossing a network. Once a network is involved, many things can fail: connectivity can break, latency can spike, data can arrive in unexpected formats, and versions can drift. Many of these failures are outside your direct control, so code must explicitly defend against them.

For example, because network failures are possible, code often needs retry logic. But once retries exist, duplicate execution becomes a concern. We discuss this in more detail in API Design — How to Design Stable and Predictable APIs.

From the author's perspective, treating remote services as if they were local objects is not very useful, because they are fundamentally different. That does not mean RPC should be avoided. In practice, tools like gRPC can be very useful. The key is to remember that the call still goes over the network, and the code must explicitly handle timeouts, retries, and version compatibility.

Support ExplainThis

If you found this content valuable, please consider supporting our work with a one-time donation of whatever amount feels right to you through this Buy Me a Coffee page.

Creating in-depth technical content takes significant time. Your support helps us continue producing high-quality educational content accessible to everyone.