2.15 Debugging and troubleshooting

Overview and motivation

Debugging is the disciplined work of finding out why a system does something it should not, and troubleshooting is the same skill turned on a running production system under time pressure. Both are the scientific method applied to defects: you observe a surprising behaviour, form a hypothesis about its cause, design an experiment that would confirm or refute it, and let the evidence, not your hunch, tell you what to change. Done this way, debugging is a learnable, teachable engineering skill. Done as folklore, it becomes superstition: changing random lines, restarting servers, and hoping.

For a large team, the difference is expensive. A single hard defect can pull in engineers across several services, consume on-call hours, and stall a release. When each person debugs by instinct, that effort does not compound, because nobody can reproduce or explain what anyone else tried. When the team shares a method (reproduce first, isolate by search, capture the bug in a failing test, then fix), the same effort turns into a repeatable process and a growing regression suite. Debugging connects tightly to testing strategy (chapter 2.4), to software quality (chapter 2.11), and to the construction habits (chapter 2.9) that make code diagnosable in the first place.

In enterprise and government settings, the stakes rise. Enterprise defects cross service and team boundaries, so the person seeing the symptom is rarely the person who owns the cause. Government systems add constraints most engineers never meet: air-gapped or restricted environments where you cannot attach a debugger to production, reproducible builds that must be diagnosed from artefacts, and audit trails that must record what you changed and why. In all three, the goal is the same: replace guessing with evidence.

Key principles

Reproduce before you theorise. A bug you cannot trigger on demand is a rumour, not a defect.
Debugging is hypothesis testing. State what you believe, then design the cheapest experiment that could prove you wrong.
Read the error and the stack trace first. The system usually tells you where it broke before you change a line.
Search the problem space, do not scan it. Halve the suspect region each step instead of reading top to bottom.
Reduce to the minimum. Strip the case down until only the essential trigger remains.
One change at a time. Shotgun edits destroy the evidence that would have told you which change mattered.
Capture the bug in a failing test before you fix it. The fix is only proven when that test goes green and stays green.
Find the root cause, not the nearest symptom. A patch that hides the symptom leaves the defect to return.

Recommendations

Reproduce the defect reliably before changing anything

Your first job is a reliable reproduction: a set of steps or an automated case that triggers the bug on demand. Without it you cannot tell a real fix from a coincidence, because the symptom may come and go for reasons you never controlled. Pin down the inputs, the environment, the versions, and the timing. If the bug is intermittent, hunt for the hidden variable that makes it appear (a specific data record, a clock boundary, a concurrent request) until the reproduction is dependable. A reliable reproduction is the single most valuable artefact in debugging, because everything after it becomes measurable.

Read the error, the logs, and the stack trace before you touch code

Before you form a single theory, read what the system already told you. The stack trace (the record of the call chain at the moment of failure) usually names the file, line, and sequence that failed. The exception message, the log lines around it, and the values in scope narrow the search before you have changed anything. Engineers waste hours theorising about causes the traceback ruled out on line one. Treat the error output as the first witness, read it carefully and completely, and only then decide what to investigate.

Isolate by binary search of the problem space

Do not scan the code top to bottom. Search it. Use binary search: find a point where the state is still good and a point where it is already bad, then check the midpoint, and repeat, halving the suspect region each time. This turns a thousand-line search into ten questions. When the regression appeared over a range of commits, apply the same idea to history with bisection: git bisect walks the commit range, and you mark each revision good or bad until it names the exact change that introduced the defect. Automate the good-or-bad test and bisection runs itself.

Reduce to a minimal reproducible example

Once you can trigger the bug, shrink it. A minimal reproducible example is the smallest input and code path that still fails: remove data, features, and steps until any further removal makes the bug disappear. Reduction is not busywork; each element you eliminate is a cause you have ruled out, so the minimal case often points straight at the defect. When the input is large or structured, automate the shrinking with delta debugging, an algorithm that systematically removes chunks of a failing input to find the minimal failing subset. A small, self-contained reproduction is also the best possible bug report to hand another team.

Instrument with logs, then use an interactive debugger

Match the tool to the bug. Logging and targeted instrumentation are best when you need to see behaviour over time, across processes, or in an environment you cannot pause. An interactive debugger, which lets you set breakpoints, step line by line, and inspect live state, is best when you can run the code locally and need to watch a single execution closely. Add instrumentation as a deliberate experiment tied to a hypothesis, not as scattered print statements, and remove it or promote it to permanent structured logging once the bug is solved. In production, lean on observability-driven debugging: high-cardinality events and distributed tracing (chapter 9.2) let you follow one request across many services, which is often the only way to debug a distributed system you cannot attach a debugger to.

Write a failing test that captures the bug before you fix it

Before you write the fix, write a test that fails because of the bug. This does three things at once: it proves you actually understand the cause, it defines exactly what “fixed” means, and it becomes a permanent guard. Then make the fix and watch the test go green. That test now joins your suite as a regression testing guard, so the same defect cannot return unnoticed. This practice ties debugging directly to your testing strategy (chapter 2.4): every hard bug you solve leaves the suite stronger than it found it, and a flaky test gets the same treatment (reproduce the nondeterminism, then guard against it) rather than a retry annotation.

Find the root cause, and keep the analysis blameless

Fixing the symptom is not fixing the bug. Trace the failure back to its true origin, asking why at each layer until you reach a cause you can remove rather than mask. For defects that reached production, run a blameless root-cause analysis as part of incident management (chapter 9.3): focus on the system and process conditions that let the bug ship and survive, never on the individual who wrote the line. Blame drives information underground, and debugging runs on information. The output is both a fix and a change to how the class of defect gets caught earlier next time.

Trade-offs: pros and cons

Approach	Pros	Cons
Logging and instrumentation	Works in production and distributed systems; captures behaviour over time	Noise, cost, and log sprawl; may perturb timing bugs
Interactive debugger	Precise, live state inspection; fast for local bugs	Useless in restricted or air-gapped production; can hide concurrency bugs
Reproduce-first discipline	Turns guessing into measurement; enables a failing test	Slow up front; some bugs are genuinely hard to trigger
Binary search and bisection	Fast isolation, even in unfamiliar code	Needs a reliable good-or-bad test; hard when bugs interact
Delta-debugging reduction	Shrinks huge inputs automatically to the trigger	Setup cost; assumes the failure is deterministic
Fix the symptom now	Restores service fast under pressure	Leaves the root cause to return; accrues debt

The central tension is speed versus certainty. Under a production incident you may need to stop the bleeding first (a rollback or a symptom patch) to restore service, and that is legitimate. The mistake is stopping there. Resolve the tension by separating the two jobs: mitigate fast to protect users, then reproduce, find the root cause, and add the regression guard before you consider the defect closed. A symptom fix without a follow-up is a bug you have agreed to meet again.

Questions to discuss with your team

When someone hits a hard bug, what is the first thing they do, and is it reproduce or guess? The honest answer reveals whether your team has a shared method or a room full of private folklore. Ask people to narrate their last difficult defect out loud: did they get a reliable reproduction first, or did they start changing code and restarting things? A team that reproduces first can hand a bug between people, because the reproduction travels; a team that guesses cannot, because every attempt is unrepeatable. This matters more as the team grows, since the person who sees a symptom is increasingly not the person who can fix it. If the default is guessing, agree on reproduce-first as a norm and make a clean reproduction the price of admission to a bug ticket.
Do the bugs we fix come back, and would we know if they did? A defect that returns is a defect whose root cause was never removed and whose fix was never guarded by a test. Pull your last quarter of incidents and reopened tickets and count how many were repeats or close cousins of earlier bugs. Each repeat is evidence that the team patched a symptom, skipped the failing test, or stopped the root-cause analysis too early. The fix is a rule: no bug is closed until a test that fails on the old behaviour passes on the new one and joins the suite. Bring one recent recurring bug and ask what guard would have caught it, because that guard is what you were missing.
Can we debug our production systems at all, given how we are allowed to touch them? In enterprise and especially government environments, you often cannot attach a debugger, cannot reproduce with real data, and cannot change a running system without an audit trail. If your only debugging technique is a local interactive debugger, you are blind exactly where the hardest bugs live. Ask what evidence a production failure actually leaves behind: structured logs, distributed traces (chapter 9.2), core dumps, or reproducible build artefacts. Decide now what you must capture by default so that a future incident is diagnosable, because you cannot add instrumentation to a failure that already happened. In regulated settings, confirm that the same trail also satisfies your audit obligations.
When a production incident forces us to stop the bleeding fast, how do we make sure the root cause still gets found afterwards? Under an incident, a rollback or a symptom patch is the right first move to protect users, but the danger is that the ticket closes the moment service returns and the underlying defect is never diagnosed. For a large team this is where debt accumulates invisibly, because the same class of failure resurfaces on a different service and a different on-call engineer months later. Bring your last few severity-one incidents and check each one: did a reproduction, a root-cause analysis, and a regression guard follow the mitigation, or did the story end at “service restored”? Agree an explicit rule that a mitigated incident stays open until the root cause is understood and guarded, and name who owns that follow-through. In enterprise and government settings, tie this to your incident-management process (chapter 9.3) so the post-incident review is a required, auditable step rather than a courtesy that slips when the next fire starts.
How much of a failure can we actually reconstruct after the fact, and who decided what we capture by default? You cannot attach instrumentation to a failure that has already happened, so the diagnosability of any incident is fixed in advance by the logs, traces, metrics, and dumps you chose to emit. The competing consideration is cost and noise: high-cardinality events and full tracing are not free, and over-logging buries the signal while inflating storage and, in regulated contexts, your data-retention exposure. Bring a real recent incident and ask what evidence it left behind, then work backwards to what you wish you had captured and what it would cost to keep. Decide deliberately which signals are on by default versus sampled or opt-in, and record that decision so it is a policy, not an accident. For an enterprise or government system, add who is accountable for that observability budget and whether the captured trail also satisfies audit, privacy, and data-residency obligations.
Do we treat debugging as a taught, measurable skill, or do new engineers absorb it by osmosis? Debugging is learnable, yet most teams never teach it explicitly, so juniors inherit whatever folklore sits nearest them, and the reproduce-first method spreads unevenly or not at all. The tension is that deliberate teaching (pairing on hard bugs, writing up post-incident findings, tracking metrics) costs senior time that always feels needed elsewhere. Bring two numbers to the discussion: your repeat-defect rate and your time-to-diagnose, because if you cannot measure them you cannot tell whether your method is improving or decaying. Consider whether onboarding includes a real debugging exercise and whether root-cause findings actually feed earlier detection. In a large or public organisation, a documented, measured debugging practice also becomes evidence of engineering rigour that auditors, regulators, and oversight bodies increasingly expect to see.

Sector lens

Startup. With a handful of engineers and no slack, your goal is to make bugs cheap to reproduce and impossible to forget, not to build heavy process. Lean on git bisect, a fast local reproduction, and one failing test per fixed bug, because that habit costs minutes and stops you re-paying for the same defect while you are trying to ship. Skip formal postmortems, but never skip the regression test: it is the one artefact small enough to always afford and valuable enough to always keep.

Small business. You likely have no dedicated reliability or observability specialist and a tight tooling budget, so favour what your stack already gives you: readable stack traces, structured logs, and the tracing built into the frameworks and hosted services you have bought. When you evaluate a new platform, weigh how diagnosable it makes failures, because a cheap tool that hides what went wrong costs you far more in guessing time than the licence saved. Reproduce-first and one-change-at-a-time are free disciplines that pay off fastest when nobody has hours to spare.

Enterprise. Your hard bugs cross service and team boundaries, so the person who sees the symptom rarely owns the cause, and a shared method matters more than any individual’s skill. Standardise reproduce-first, binary-search isolation, a failing test before the fix, and blameless postmortems across teams, and invest in distributed tracing (chapter 9.2) so one request can be followed across services. Manage debugging as a measured capability: track repeat-defect rate and time-to-diagnose, and feed root-cause findings back into earlier detection so the same class of failure does not tour your service map.

Government. Procurement rules, restricted environments, and public accountability shape how you may debug at all. You often cannot attach a debugger to production or copy citizen data to a laptop, so design for diagnosis from what is allowed: reproducible builds, synthetic records in an isolated enclave, and structured logs and traces captured by default. Record every diagnostic step and every change in the audit trail, and require that vendors expose enough telemetry and build reproducibility for you to investigate failures independently rather than depend on the supplier’s word.

Examples

Startup. A four-engineer team keeps seeing checkout fail for a fraction of users, but never in testing. Instead of guessing, one engineer captures a reliable reproduction by replaying the exact failing request payload, then reads the stack trace they had been ignoring, which points at a date-parsing call. A quick git bisect across the week’s commits names the change that switched a date library. They write a failing test with the offending timestamp, fix the parser, watch the test go green, and keep it in the suite. The whole investigation takes an afternoon because they reproduced before they theorised, and the bug never returns.

Enterprise. A payments platform sees intermittent timeouts that no single team can explain, because the symptom shows up in checkout but the cause lives three services away. On-call engineers use distributed tracing (chapter 9.2) to follow one failing request across service boundaries and find a downstream call that occasionally deadlocks under concurrent load, a classic race condition where the result depends on unlucky timing between threads. They reproduce it with a load test, capture it in a failing integration test, fix the locking, and run a blameless postmortem (chapter 9.3) that adds a tracing span and an alert so the next occurrence is caught in minutes, not days.

Government. A benefits agency runs its case system in an air-gapped environment where engineers cannot attach a debugger to production and cannot copy citizen data to their laptops. A calculation defect surfaces in reconciliation. The team debugs from what the environment does allow: structured logs, a reproducible build they can stand up in an isolated test enclave, and synthetic records that recreate the failing case. Every diagnostic step is recorded in the audit trail, the fix ships with a failing-then-passing test as evidence, and the root-cause analysis feeds a new pre-release check. Because the reproduction used synthetic data, no citizen record ever left the boundary.

Business case: motivations, ROI, and TCO

The return on disciplined debugging is measured in engineer hours not spent guessing and in defects that do not recur. An undiagnosed intermittent bug can consume days of senior time and repeated on-call escalations; a reproduce-first method turns that into a bounded, delegable task, and the failing-test habit stops the same defect from billing you again next quarter. Across a large organisation, the compounding effect of never re-paying for the same bug is substantial, and it directly improves the change-failure rate and mean time to recovery that leadership already tracks.

The total cost of ownership is mostly training and tooling, and it is modest. You need shared conventions (reproduce first, one change at a time, a failing test before the fix), debuggers and tracing already common in the toolchain, and the observability investment described in chapter 9.2. The larger, hidden cost is the alternative: a culture of superstition where engineers apply shotgun changes, symptoms are patched and return, and on-call load grows without bound. Reducing on-call toil alone often justifies the investment, and the case to leadership is simplest stated as fewer repeat incidents and faster recovery for a one-time cost in habits and instrumentation.

Anti-patterns and pitfalls

Shotgun debugging: changing many things at once, so even a fix teaches you nothing about the cause.
Fixing without reproducing: declaring victory on a bug you were never able to trigger on demand.
Ignoring the error output: theorising about causes the stack trace already ruled out.
Symptom patching: silencing the symptom while the root cause survives to return.
Print-statement sprawl: scattered debug output left in the code, adding noise instead of a tied-to-hypothesis experiment.
Skipping the regression test: fixing the bug but leaving no guard, so it can quietly come back.
Retrying flaky tests: hiding nondeterminism with retries instead of debugging the underlying race condition or heisenbug, a bug that changes or vanishes the moment you try to observe it.
Blame-driven postmortems: punishing the author, which drives underground the information debugging depends on.

Maturity model

Level 1, Initiate: Debugging is individual folklore and reaction. Engineers guess, apply shotgun changes, and restart things. Bugs are fixed at the symptom, reproductions are rare, and the same defects recur. Production is barely diagnosable, and nobody can hand a bug to anyone else because no attempt is repeatable.
Level 2, Develop: Some engineers reproduce reliably, read stack traces, and use debuggers, but the practice is inconsistent and varies from person to person and team to team. Logging exists but is noisy and unstructured. Fixes sometimes ship with a failing test, often not, and root-cause analysis happens only when someone insists.
Level 3, Standardise: Reproduce-first, binary-search isolation, one change at a time, and a failing test before the fix are documented team norms enforced across the organisation. Bisection and delta-debugging reduction are common practice. Production has structured logging and tracing (chapter 9.2), and blameless postmortems (chapter 9.3) are the standard response for every escaped defect.
Level 4, Manage: The debugging practice is measured and controlled against baselines. Repeat-defect rate, time-to-diagnose, reopened-ticket count, and the share of fixes that shipped with a regression test are tracked per team and reviewed on a cadence. Reproduction and root-cause completion are treated as gates rather than good intentions, and trends against the baseline drive where you invest in tooling, training, and observability.
Level 5, Orchestrate: Debugging is a taught skill integrated with quality (chapter 2.11) and incident management (chapter 9.3), and the whole loop adapts continuously. Observability is designed in so most production bugs are diagnosable without a debugger, every solved bug strengthens the regression suite, and root-cause findings feed earlier detection so classes of defect are prevented rather than re-diagnosed. The organisation rebalances effort as its systems and failure modes evolve, and the repeat-defect rate keeps falling.

Ideas for discussion

What fraction of your recent bugs were reproduced reliably before anyone changed code, and what does that fraction say about your method?
When a regression appears, does your team reach for bisection, or read code by hand until someone spots it?
How diagnosable is your production system today, and what would you give to have captured about a failure that already happened?
Do your fixes consistently ship with a failing-then-passing test, and if not, where does that discipline break down?
How do you handle flaky tests: debug the nondeterminism, or paper over it with retries?
Is debugging taught deliberately to new engineers, or are they left to absorb folklore by osmosis?

Key takeaways

Debugging is hypothesis testing: reproduce reliably, read the error and stack trace, then isolate by binary search and bisection rather than scanning.
Reduce the failure to a minimal reproducible example, using delta debugging for large inputs, because each element removed is a cause ruled out.
Match the tool to the bug: instrumentation and tracing for production and distributed systems (chapter 9.2), interactive debuggers for local investigation.
Write a failing test that captures the bug before you fix it, so the fix is proven and the defect is guarded against for good (chapter 2.4).
Find and remove the root cause, run blameless postmortems (chapter 9.3), and treat debugging as a learnable skill, not folklore.
Change one thing at a time; shotgun changes and symptom patches destroy evidence and invite the bug back.

References and further reading

David J. Agans, Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems
Andreas Zeller, Why Programs Fail: A Guide to Systematic Debugging
Andreas Zeller and Ralf Hildebrandt, “Simplifying and Isolating Failure-Inducing Input” (the delta debugging algorithm)
Brian W. Kernighan and Rob Pike, The Practice of Programming (chapter on debugging)
Andrew Hunt and David Thomas, The Pragmatic Programmer (the chapters on debugging and assertions)
Steve McConnell, Code Complete: A Practical Handbook of Software Construction (the debugging chapter)
John Regehr, “Reducers Are Fuzzers” and related writing on test-case reduction
Charity Majors, Liz Fong-Jones, and George Miranda, Observability Engineering (debugging production with high-cardinality telemetry and tracing)
Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy, eds., Site Reliability Engineering (blameless postmortems and production debugging)