Postmortem: Two Bugs in the Safety Path

I spent today fixing two bugs that lived entirely inside the bot's defensive code. Not the logic that decides what to do — the logic that's supposed to stop the bot when something looks wrong. One of them paused the whole service over a phantom problem. The other was a "safe" fallback value that survived two audit passes with holes in it big enough to drive a NaN through.

Both are closed now. But the pattern they share is worth writing down, because I keep relearning it: guard paths are code too, and code that runs rarely gets tested rarely, reviewed gently, and trusted by default. That combination is exactly how a safety net becomes a tripwire.

The false pause

The bot maintains a hard invariant: every record in the external state it reconciles against must be explainable by its own ledger. If reconciliation ever finds a record it can't account for, that record is flagged as unowned, and an unowned flag pauses the service. This is deliberate paranoia. An unexplained record means either the ledger is corrupt or something outside the bot has been writing state, and both of those are stop-everything events.

Today that tripwire fired on nothing.

The proximate cause was a filter. One of the maintenance routines only cares about a subset of the records, so it asks the reconciliation layer for records matching a particular scope. The bug: the scope filter was applied before the ownership check instead of after. Records belonging to a different scope weren't excluded from the comparison — they were compared and found missing. From the filtered view's perspective, perfectly well-known records looked like orphans. Unowned. Pause.

Nothing was actually wrong. The ledger was consistent, every record was accounted for, and the service stopped anyway because a query scoped for one purpose leaked into a check designed for another. The fix was conceptually one sentence: a scope filter narrows what you look at, never what counts as known. Ownership is a property of the whole ledger, and the ownership check must always run against the full set, no matter how the caller scoped the request.

The uncomfortable part is the failure direction. I built the pause to fail safe, and in one sense it did — the service stopped rather than acted. But false-positive alarms have a real cost that compounds: every phantom alarm trains the operator (me) to treat the alarm as noise. A safety system that cries wolf is quietly disabling itself.

Three rounds to trust a fallback

The second bug was smaller and, honestly, more embarrassing, because I audited it twice and blessed it twice before catching the real problem.

One of the bot's summary reports normalizes its figures against an external numeric reference value — a denominator, nothing more. When the fresh value isn't available, the code falls back to a cached copy. Reasonable. "Safe," even. That word is doing a lot of unearned work.

Audit round one flagged that the fallback was being used unconditionally in one path — no check that the fallback was even populated. I fixed it: qualify the fallback, only use it when it exists. Done, I thought.

Audit round two, rated high severity, pointed out that "exists" and "is a number you can do math with" are different claims. A NaN sails through most naive checks because every comparison against NaN is false — including the ones you wrote to reject bad values. A NaN reference value wouldn't crash the summary; it would silently poison every downstream ratio into more NaN, and NaN has a talent for looking like "missing data" rather than "bad data." I added an explicit NaN guard and, more importantly, a test that pins the ordering of the checks, because a guard that runs after the value has already been used is decoration.

Audit round three closed the last gap: NaN isn't the only pathological float. Infinity is finite-check-failing but NaN-check-passing; zero and negative values are arithmetically fine and semantically absurd for a denominator. The final contract is the one I should have written on day one: the fallback is accepted only if it is finite and strictly positive. Everything else is treated as "we do not have a value," and the summary says so instead of computing confidently on garbage.

Three rounds. For a fallback whose entire job was to be the boring, safe branch.

Guards are an attack surface

Here's the thesis both bugs point at: defensive code has all the properties that make code dangerous. It runs on the rare path — and the rarest guard paths are also the least exercised. It's psychologically pre-approved — "that's the safety check" — so review skims it. And its failures are asymmetric in sneaky ways: a guard that's too eager stops you on nothing, and a guard that's too lax launders bad data into every calculation downstream while wearing a badge that says validated.

If I threat-modeled my own codebase, the guard paths are where I'd attack. So that's how they need to be reviewed: adversarially, with the same energy as the happy path. Feed the fallback NaN, infinity, zero, negatives — not just "missing." Ask of every filter what invariant it might be scoping out of existence. Test the order guards run in, not just that they exist somewhere in the function.

What changed concretely: the ownership check is pinned to the full ledger regardless of caller filters, the reference-value fallback has a finite-positive contract with ordering tests, and my audit checklist now has a standing item that reads, roughly, "attack the safety code first." The incident is closed, and the tests now pin the intended behavior.

These two guard paths are stronger now than they were this morning. Mostly because I finally stopped assuming they were.