A framework for debugging the hard way

Software Engineering  · 

Debugging is a skill that has some peculiar aspects. After some years working as a software engineer, you usually realize it’s one of the hardest and most important skills to master. But at the same time, it receives little attention in Computer Science curriculums and is often viewed as accessory. Maybe that’s because people tend to focus on things that are indeed accessory, like debugging tools, while the core aspect, in my opinion, is that good debugging is an application of good problem-solving skills. And that is not at all accessory. On the contrary: being able to understand a problem and tackle it in a structured, effective way is what makes engineers good.

AI, as with most things, is making this even more complicated. Before it, starting a debugging effort meant taking a deep breath and putting your mind in a state of deep investigation. Now, it usually means sending a prompt to the LLM telling it there’s an error and asking it to fix it. It’s hard to argue against the fact that this is simply easier and usually more efficient. In my experience, AI is pretty good at debugging. However, this practice may be turning us into even lazier problem solvers.

I’m super excited about AI and a heavy user of it. But I’ve been trying to be very conscious of the places within my craft where resorting to it by default may be taking the joy out of my work (which, for me, comes mostly from learning). We may need to choose less efficiency sometimes if we want to keep the habit of learning by doing hard stuff.

That’s why I’ve recently started contributing to open source again. There’s been a lot of discussion in the field about the relevance of contributing to OSS after AI. In many projects, what we’re seeing is a race to spend tokens in pursuit of higher numbers in people’s GitHub history, without much real thought behind it. Maintainers have a hard time trying to distinguish between people who genuinely want to contribute and those who simply delegate the work to AI and hence often deliver poor PRs. Some projects are even blocking AI submissions.

So I’ve been trying to contribute to OSS by intentionally avoiding this pattern. I do use a lot of AI in these debugging sessions, but never to generate code directly. I’ve been using it as a teacher and a pair debugger, and it’s certainly an amazing one. I believe there’s never been a better time to try to get good at problem solving, as long as you avoid the temptation of delegating the thinking to AI and use it to help you understand instead of execute.

I’ll try to dissect some of my contributions in this new phase of interaction with open-source projects in the post-AI era.

For that, first I’ll lay down a generic framework, which I try to frame as a scientific method and that I usually try to follow when debugging. Then, while going through each actual concrete debugging session, I’ll reflect on where I adhered to that framework and where I diverged from it (and, possibly, adjust the framework so it mirrors reality more closely).

My goal is to reach a problem-solving structure that draws from what I understand as good problem solving and that is validated by real debugging efforts.

The Framework

Here’s the conceptual framework:

1. Verify the bug and expected behavior

First of all, we need to make sure there is indeed a bug.

A bug is a flaw in the system, which may be observable through symptoms or not. To be able to characterize it as a flaw, we need to know what the expected behavior is. What should the system be doing that it is not doing?

2. Reproduce the bug

Once the bug’s existence is confirmed, we should start by reproducing it. This reproduction must be deterministic (meaning, same inputs must always produce the same output) and it should be as isolated and cheap as possible.

For example, if you run some code to reproduce the bug, you should try to make it as small and fast as possible. Making it small leaves variables that don’t interfere out, which reduces the amount of complexity you need to deal with. Making it fast speeds up the feedback loop, which sets the speed of everything else coming after.

Before trusting any reproduction, confirm you’re running the code you think you are: the right version, build, branch and environment. A repro against the wrong code can hide the bug or make a correct fix look like it does nothing.

Before experimenting, define an oracle: an automatic check that tells you exactly whether a result is correct. Build the expected answer independently of the code under test, and compare everything that matters, not a proxy.

3. Localise

Once you’re able to reproduce the bug, you want to find its root cause. This phase has one goal: narrow down where the wrong behaviour comes from (for example, one function or one line).

Although we’re looking for a scientific method to do that, the truth is there’s some guessing in this step (which gets more precise with experience). Given what you observed so far, you need a hypothesis so you have somewhere to start.

The hypothesis is the first step in a loop that in each iteration is supposed to narrow the search space, as in a binary search.

Hypothesis -> Prediction -> Experiment -> Update

The loop is how you reason. Black box vs white box is what kind of experiment you run in the “experiment” step, and to some extent where hypotheses come from.

Hypothesis  ──►  Prediction  ──►  Experiment  ──►  Update
     ▲                                │              │
     │                   black box: vary inputs      │
     │                   white box: read / trace     │
     └──── choose the next hypothesis and the  ◄─────┘
           cheapest experiment that can test it

Each turn of the loop picks one mode. The switch decision lives in the Update step. After each result, ask: what’s the cheapest experiment that would most reduce my uncertainty now? Sometimes that’s another input variation, sometimes it’s reading or printing something inside.

The Stop Condition

Exit the loop when you can point to the defect and your explanation accounts for all observations.

Counterfactual: removing the cause should remove the bug. This is what proves the story.

Depth rule: explain enough to fix the cause and predict where else it bites. Stop there.

4. Fix

This is usually the easy part. Fix the root cause.

5. Verify

Add a test that failed before and passes after the fix. Also, run the entire test suite to guarantee no regressions.

6. Generalise the Cause

A bug is usually the visible symptom of a mistaken assumption, and that assumption often appears in more than one place. After fixing the defect, say the assumption out loud, for example: “integer inputs stay exact through this code”. Then search everywhere it could also be false: other code paths that share the logic, other input types or values that reach the code, and similar functions elsewhere. Maybe write a test for each case you find.

The Issue

Now let’s see the first issue and try to correlate the steps I took to the framework phases:

The issue reported that pd.interval_range silently produced wrong results for large integer endpoints. The reproduction asked for four consecutive width-1 intervals starting at 2**53: start=2**53, end=2**53 + 4, freq=1. It got back three intervals, all (9007199254740992, 9007199254740992]. Each one has the same left and right bound, so all of them are empty. Nothing raised an error or a warning, and the result even reported an integer dtype (interval[int64]), so a user had no sign that anything had gone wrong.

1. Verify the bug and expected behavior

Well, the author of the issue was the one that observed its existence. The expected behavior is also explained in the issue and documented in the pandas documentation.

2. Reproduce the bug

The issue included a suggested reproduction:

import pandas as pd

start = 2**53

result = pd.interval_range(
    start=start,
    end=start + 4,
    freq=1,
)

print("Pandas version:", pd.__version__)
print("Number of intervals:", len(result))
print("Generated intervals:", result.tolist())
print("Empty intervals:", result.is_empty.tolist())

expected = [
    (start + i, start + i + 1)
    for i in range(4)
]

print("Expected interval boundaries:", expected)

Three things to note:

  1. Making sure I was running the code I thought I was. Reproducing against my build was the hardest part of this phase: setting up the environment, facing a missing pandas._libs.pandas_parser error, then checking pandas.__file__ pointed at my clone. A repro on the wrong build is worse than none.

  2. The repro’s check was done by eye. The snippet printed expected next to the actual result, and I compared them visually. That’s fine for one run. But once I automated it, the check got weaker (I checked length and emptiness only), and that weak check misled the next hour of experiments. You need a strong oracle.

  3. Minimising. The snippet was small, but not minimal. n=1 turned out to be enough, and n=4 actually hid the fact that the boundary depends on end, not start. Shrinking the input revealed something about the bug. That’s why minimising is part of reproduction and not just tidying up.

3. Localise

Context: the bug was that pd.interval_range returned wrong intervals for large integer endpoints. The root cause turned out to be one line in a roughly 15-line branch. A float “nudge” (end + freq * 0.1) pushed an integer computation into float64, which loses precision for large values.

What worked: precise, falsifiable predictions. The key turn was predicting that if the failure depended on end rather than start, then with 1 interval it would start at 2**50 − 1, and with 100 intervals at 2**50 − 100. Both predictions held, which turned a vague pattern into a precise rule. Adding asserts to the binary search also paid off: it exposed that failures weren’t monotonic, instead of silently returning a wrong threshold.

What didn’t work:

Takeaway: start with an explicit hypothesis, however rough. At each update, honestly ask whether reading or tracing the code is now the cheapest next experiment. For a small code path, it usually is.

4. Fix

Context: once localised, the cause was clear. With all-integer inputs, interval_range built the interval boundaries with np.arange(start, end + freq * 0.1, freq). The float term forced float64 arithmetic, which drops the last boundary from about freq · 2**50 and rounds the boundaries above 2**53. The periods code path had the same problem through np.linspace.

What was done: when start, end and freq are all integers that fit the target dtype, a small helper builds the boundaries with exact integer arithmetic. Otherwise the code falls back to the original float path unchanged. NumPy’s integer arange computes its length with a float division, so the helper counts the boundaries with Python integers.

5. Verify

What was done:

What it missed: every test used Python integers, so NumPy integer inputs were never exercised. A review bot later found an overflow that only NumPy scalars trigger.

6. Generalise the Cause

What was done: the mistaken assumption could be stated as “integer inputs stay exact through this code”. Checking where else it held led to the second code path (periods, via np.linspace), which had the same problem and was fixed in the same change. Writing the fix also surfaced a related instance one level down: NumPy’s own integer arange computes its length with a float division.

What it missed: the assumption was checked against code paths and values, but not against input types. With NumPy integer endpoints, computing end - start could overflow int64 before the integer path was even reached, so the result fell back to the float path and was rounded. A review bot found it after the PR was opened, and it was fixed in a follow-up commit.