I run my own DNS resolver at home. Every device in the house resolves through it, which means I can see what my own hardware talks to all day, and block the parts I would rather it did not. It is a nice tool to own. It also means that when an app misbehaves, there is a real chance I did it.
One did. A news app on my phone would freeze on its splash screen and never load. Not always. Roughly one launch in three.
That ratio is the entire story. Everything that went wrong over the next two days went wrong because of it.
Three fixes, all successful, all wrong
The debugging went the way it always goes. Look at what the app queried in the seconds before it hung, find something blocked, unblock it, relaunch, watch it open. Announce the fix.
I did that three times, with three genuinely different theories. First an ad host that seemed to be the last thing queried before the hang. Then a theory about how I was refusing - whether a blocked name returned an empty address or a "does not exist", and whether the app's networking stack handled one worse. Then a different ad host.
Each theory was reasonable. Each was tested by relaunching the app once. Each time the app opened, and I moved on.
Here is the arithmetic I should have done on the first day. If a failure occurs one time in three, a single post-fix launch has a 67% chance of looking like a success no matter what I changed. Including changing nothing.
| Clean launches after the "fix" | Chance it looks fixed while still broken | Verdict |
|---|---|---|
| 1 | 67% | Worthless as evidence |
| 2 | 44% | Still a coin flip |
| 3 | 30% | Weak |
| 5 | 13% | Getting somewhere |
| 8 | 4% | Now you can say fixed |
Eight clean launches. I was declaring victory on one, three times in a row, and each false confirmation made the next theory feel better supported than the last, because now I had a "history of near misses" that was pure noise.
The person re-testing notices before you do
Worth saying plainly: the loop was obvious to whoever kept having to relaunch the app, well before it was obvious to me. That is structural, not a personality trait. The person doing the re-testing is accumulating the sample. They have the base rate in their hands and you have a theory in your head.
When someone says "you already fixed this twice", that is not impatience. It is a measurement, and it is telling you that your evidentiary standard is too low for the class of bug you are in.
What actually broke the loop
The move that ended it was not a better theory. It was refusing to have one for an hour.
Instead I went looking for a cheap machine-readable signature that separates a good run from a bad one, without a human watching a screen. It turned out to be sitting in the resolver logs: count the distinct hostnames the app queries per launch. A launch that opens normally resolves more than thirty names as it loads content. A frozen launch stops at about fifteen. Two clean populations, no judgement call, and I could compute it for every launch that had ever happened.
Then I applied it backwards over the whole log. That is the part that matters, and it took one query:
Twenty six launches had worked while the host I was blaming was blocked. One had frozen with the block list empty. The correlation behind all three diagnoses evaporated in a single pass over data I had been sitting on the whole time.
What I do now
Three things, in order, the moment a symptom has been "fixed" more than once:
- Stop proposing fixes. The next theory will also be confirmed by the next run. That is guaranteed, and it is why the loop sustains itself.
- Estimate the base rate out loud. One in three, one in ten, whatever it is. Then state how many clean trials would actually clear that bar, and be honest about whether you are willing to run them.
- Find the separating signature before the cause. Something countable that distinguishes the two outcomes. Then run it over history, where the sample size already exists.
The other half of the same lesson came from a different bug, where two test runs of the same thing disagreed with each other and I nearly wrote it off as flakiness. That disagreement was the clue that exposed a second, real defect underneath. Runs that contradict each other are signal, not noise. An intermittent bug is not a bug that is hard to see. It is a bug that keeps offering you evidence for whatever you already believe.