How to Diagnose Problems and Fix Errors During Incidents?

In software engineering, you will inevitably run into all kinds of problems. When that happens, you need to investigate effectively and fix the error properly. Solving problems is at the core of a software engineer's job, so improving your ability to solve problems can be a meaningful boost to your engineering career.

Many people think some engineers are naturally better at debugging, or that debugging depends heavily on experience. In reality, regardless of talent or experience level, you can speed up problem solving by using a more structured approach. In this article, we will look at the effective troubleshooting method introduced in Google's SRE book.

The Four Steps of Problem Solving

Google's SRE book breaks problem solving into two core requirements: a method for troubleshooting and an understanding of the system itself. Because everyone works with different systems in daily work, we will not focus on system knowledge here. Instead, we will go deeper on the troubleshooting method.

You can think of troubleshooting in software engineering as a hypothetico-deductive process. It is similar to how scientists run experiments: observe a phenomenon, propose a hypothesis, test that hypothesis in different ways, and keep ruling out incorrect hypotheses until the scope narrows and the hypothesis is verified.

For example, suppose a website suddenly becomes slow. Without a hypothesis-driven approach, you might start guessing randomly: "Could Redis be broken?" or "Could yesterday's merged PR have caused this?" You might get lucky and guess correctly, but this approach is usually inefficient.

A better approach is to first identify where the failure is happening. For example: "The monitoring dashboard for XX API shows a large number of errors. The error type is related to timeouts, and P95 latency has indeed increased sharply." Once you have located a slow API, you should still avoid guessing that the cache or database is the issue. Instead, inspect further: "If this were caused by the database, database latency should also increase. But the database latency dashboard has no significant change, and CPU usage has not noticeably changed either, so the problem is likely elsewhere."

Google's SRE book describes the troubleshooting process in the following flow. Let's go through it step by step.

Triage and Examination

When a problem occurs, triage is the first thing to do. Triage means judging how severe the problem is, how large the impact is, and what incident level should be used to respond. In practice, many kinds of problems can occur, but not every one deserves the highest level of response. If every small issue triggers a major effort, the team will quickly burn through its resources. As a note, our article How should software engineers do monitoring well? discusses different priority levels.

During triage, the focus is not to find the bug. The focus is to understand the impact of the event: how many users are affected, whether only part of a feature is broken or the entire service is down, whether the problem is isolated or expanding, and so on.

After the initial triage, do not rush into root cause analysis. At this stage, the most important thing is to keep the system running as well as possible. Google's SRE book specifically emphasizes that you should "make the system work as well as it can under the circumstances."

For example, you might shift traffic away from a broken cluster to another cluster that is still working. If the problem can spread, you may even need to decisively shed part of the traffic and stop serving some requests. That may sound counterintuitive, but if shedding some traffic prevents the entire system from collapsing, it can be the more reasonable decision.

The book uses the phrase "stop the bleeding." This is similar to emergency medicine: if a patient is bleeding heavily, doctors do not slowly study exactly why the patient was injured. They stop the bleeding first to make sure the problem does not get worse. The same idea applies to system failures. End users do not care whether the engineering team can perfectly identify the root cause in the moment. They care whether the product works.

After the initial triage and mitigation, the next step is examination. Ideally, the system should make problems visible to the engineering team. To do that, you need solid monitoring and observability. We previously discussed these two topics in How should software engineers do monitoring well? and What is Observability? Why do we need it?, so those are worth revisiting.

Diagnosing Where the Problem Is

After examining the problem, the next step is to use the evidence you have seen to determine where the problem is most likely to be. To diagnose effectively, the book recommends "simplify and reduce": break the problem into smaller parts.

Ideally, every component in a system should have a clearly defined interface. The interface defines what input it expects and what output it returns, and those should be verifiable. So the most direct way to determine whether a component is failing is to test the relationship between its inputs and outputs.

More specifically, you can think of this as following a data flow. During examination, ask:

• Is the input received at this step correct?
• Is the output produced by this step correct?
• At which step does the data start to look wrong?
• Which component is still fine before it, but broken after it?

Take "users cannot find products through search" as an example. Instead of searching blindly through a large system, you can diagnose the data flow across the whole path. Start from the frontend request and check whether the request sent by the frontend is correct, whether the API receives the same data the frontend sent, and whether the API gets results when it calls the search service behind it. Following the path this way helps you identify where the problem is.

The book notes that in larger systems, tracing linearly like this can take a long time. That is why it recommends dividing and conquering: split the system into parts. In practice, the work can be split into two tracks. One person traces the frontend-to-backend path, while another traces the backend-to-other-microservices path. This can make it much more efficient to identify the issue.

During this process, you may find several possible problem areas. Treat these as hypotheses. Once you have hypotheses, the next step is to solve the problem based on them.

Testing and Solving the Problem

When solving the problem, you also need to observe whether your hypothesis is being validated. Google's SRE book mentions two ways to validate a hypothesis. One is to compare reality against theory. For example, if you suspect the database is causing an API to slow down, you should see related metric changes, such as movement in database CPU usage. If those signals are absent, the problem is probably not there.

Another approach is to actively change the system and observe the response. For example, if your hypothesis is that a new version caused the issue, you can roll back and observe whether the system returns to a stable state. Or, if your hypothesis is that a feature flag change caused the issue, you can try turning off the feature flag and then observe whether the system stabilizes.

This is similar to how doctors diagnose patients. A symptom may have several possible underlying causes, and doctors may not be able to pinpoint the answer immediately. Instead, they form hypotheses based on symptoms, run different tests to collect more data, or apply a treatment and observe how the patient responds.

In practice, problems often have both a surface-level issue and a deeper issue. During an actual incident, stopping the damage should take priority. Solving the surface-level problem first, without immediately digging all the way into root cause, is often the more important move. For example, suppose a payment system issue prevents users from checking out, and the observability dashboard shows that a new deployment caused the error rate to rise. The right response is to roll back immediately and make system recovery the main goal, rather than digging into the root cause while users continue to be affected.

When testing hypotheses, the book highlights a particularly important idea: test mutually exclusive alternatives. In other words, a good test should clearly tell you, "If the result is A, it supports this hypothesis more; if the result is B, it supports another hypothesis more."

For example, if the search API returns a 502, your hypotheses might be that the frontend-facing BFF layer is broken, or that the search microservice behind the BFF is broken. If you can send a request directly to the microservice layer and the result is broken there too, you can clearly identify where the issue is. Once the problem is confirmed, you can take the appropriate action.