Nothing Found and Never Looked Are Not the Same Answer
There's a bug class I now recognize on sight, and I only learned to see it after finding it six times in six unrelated places in the same codebase. It has no standard name that I know of, so I've been calling it vacuous success: a result that looks like a verdict but has lost track of where it came from.
The precise defect is a loss of provenance. Three very different situations —
- I evaluated the inputs and they're fine
- I never evaluated anything
- My input was missing or unreadable
— all get encoded into the same value, and that value is the reassuring one. The failure mode is not that the check is wrong. It's that the check may be absent, and its absence is indistinguishable from a pass.
The shape of it
Almost every safety check I write returns something shaped like "here are the problems I found." Zero problems means all clear. That's a perfectly natural contract, and it's also a trap, because there is a second way to end up with zero problems: never look.
If the scan can't open its input, if the evaluator gets a malformed record and bails, if the loader finds no data and defaults to an empty collection — the check returns the same value it returns on a genuinely healthy system. Downstream, something logs SUCCESS. An operator (me) reads the log and moves on. The check has become a placebo, and the more reliably it "passes," the more trust it accumulates.
What makes this class nastier than an ordinary bug is that it can strengthen over time. A check that crashes tends to get fixed quickly. A check that silently degrades into a no-op can get promoted to load-bearing infrastructure, precisely because it never causes trouble.
Six places it was hiding
Over a couple of weeks I found and fixed the same defect in six subsystems that share no code, no author intent, and no obvious lineage. In each one, ask the same question: given the output, can you tell which of the three situations produced it?
A file scanner. It searched artifacts for a specific class of corruption. When it couldn't actually read what it was pointed at, it reported no corruption found. A scan that did not run reported the same thing as a clean scan.
A calendar check. A pre-trade guard consults an events calendar to decide whether a name is off-limits for the session. When the calendar lookup failed — the upstream unavailable, the response unusable — it reported a clean scan. It had performed no scan at all. This is the version that scared me most, because "no events found" and "I couldn't check for events" have opposite risk implications, and the code had collapsed them into one return value.
A periodic evaluator. A recurring job that walks open positions and applies a time-based protective rule. Under a specific input condition it walked nothing, then announced success. Its output could not distinguish "every position was evaluated and none needed action" from "no position was evaluated."
An alert deduplicator. Suppression logic that hides a repeat page when the underlying safety work has already run. It was keying off narrative — a message that described the work — rather than a durable record proving the work happened. The provenance loss here is one step removed: the suppression decision inherits its evidence from prose, so "the work ran" and "something mentioned the work" become the same signal. Prose about safety is not safety.
An operational gate. A precondition check that's supposed to block a step when it can't verify things are sound. Its failures were silent: when the gate itself failed to execute, the pipeline read that as consent and carried on. A gate that fails open on its own error is a very expensive comment.
A universe loader. The screener composes its candidate set from a mandatory baseline plus an optional overlay file. When the optional file was missing, the load path produced an empty result that then replaced the mandatory baseline rather than leaving it intact. Here the collapsed states are "the overlay is legitimately empty" and "the overlay is missing entirely" — the loader treated an absent input as an authoritative empty answer, and that answer silently destroyed present required input.
Six subsystems, one defect. That ratio is what convinced me it isn't six mistakes; it's one habit of mind expressed six times.
The common cause
The habit is treating "no findings" as the natural default value of a check. It's such a comfortable default: empty list, false, None, zero. Every language hands you one for free. Exception handlers reach for it instinctively — except: return [] feels defensive, and it is, but it's defending the caller from an error message, not defending the system from the underlying condition.
There's a second contributor: checks that return only a verdict, with no room to say anything else. A boolean has exactly two states and needs at least three — passed, failed, and didn't run. When the type can't express the third state, the third state gets encoded as one of the other two, and in all six of these cases it was encoded as the safe-looking one.
The fix: make output carry proof of work
In every one of the six cases the remedy had the same structure, which is the part I find genuinely useful.
A check's result must carry evidence that it executed. Not a verdict, a verdict plus a witness: how many items were examined, which source was consulted, when the underlying data was produced.
Zero examined items is inconclusive when the contract expected more than that. Not always — an empty input set can be a perfectly good, fully verified answer, and several of these subsystems have legitimate quiet days. The distinction is whether the system can independently establish that the emptiness is real. If the contract requires at least one input, or if the code can't prove the empty set came from a successful read rather than a failed one, then zero examined items is not a pass; it's an inconclusive, and inconclusive is a distinct outcome callers must handle. That proof usually comes cheaply — a flag saying the source was reached, a count of records the source claimed to have — and once you have it, an honest empty result and a silent no-op stop looking alike.
Inability to check is an error, not a clean result. If the input can't be read, the upstream can't be reached, or the record can't be parsed, the check fails loudly and the caller decides what to do about it. Failing loudly is not the same as crashing the system — several of these now degrade to a restricted mode rather than proceeding as if verified. But something has to observe the difference.
Suppression requires a durable artifact. Deduplication, caching, and "already handled" logic must key off a record the work itself wrote, never off a message describing it.
Absent optional input can't destroy present required input. Composition of a mandatory base with an optional overlay needs the missing-overlay path to be explicitly a no-op, not an empty merge.
There's a coda I like, though it's a cousin of this bug class rather than another instance of it. After all this, I found two tests for a freshness check that had been failing every single weekend, because the thing they asserted about only happens on weekdays. Those tests weren't lying — a visible weekend failure is the opposite of a silent pass. But the effect on me was similar: a check whose result I'd learned to discount on some days is a check I've partly stopped reading, and the next real failure has somewhere to hide. Honest checks need tests that are meaningful whenever they run. Same underlying question — can you trust what this output is telling you? — one level up.