A Green Test Is a Claim, Not a Proof

I treat every output of my trading bot with suspicion. If the bot says a position is closed, I want proof from the broker. If a watchdog says the system is healthy, I want to know what it actually checked. A "no problems found" from a scan that never ran isn't a clean result. It's missing data.

For a long time I didn't apply that same suspicion to my own test suite. A green checkmark felt like the end of the discussion. Then, over a few weeks, I found three different ways my tests had been passing for the wrong reason. Together they changed how I think about coverage.

The regression test that couldn't fail

It started with an ordinary bug. The part of the bot that decides whether the market is open couldn't tell when a day was an exchange holiday. So on holidays it saw a "missing" session and sent me alerts about a problem that didn't exist. The fix was simple: teach the gate about the holiday calendar.

As usual, I added regression tests with the fix, and they had two different jobs. The holiday test reproduced the original bug: it should fail against the old code and pass against the fix. The other test covered the opposite case: an ordinary trading day must not be treated as a holiday. That one wasn't reproducing the bug. It was guarding against overcorrection, because the easiest way to break a fix like this is to go too far, so that every day starts to look like a day off. A test like that is supposed to pass both before and after the fix. What it has to do is fail if an ordinary day ever gets classified as a holiday.

Both tests passed, and I moved on. Then a follow-up review found that the "ordinary day" test never reached the holiday check it was named for. The path it exercised went around the classification entirely. So if the classifier had been broken to call every day a holiday, which is exactly the overcorrection the test existed to catch, the test would still have been green.

That's the most dangerous kind of test. A missing test is at least a gap you might notice, even if missing behaviors can stay invisible in a coverage report. A test that can't fail looks like protection, shows up in the coverage report, and gets cited in a review as evidence. It takes up the space where a real guarantee should be.

The fix to the test was small. The lesson took more work to absorb: a test hasn't proven anything until you've seen it fail against a violation of the guarantee it claims to protect. For a test that reproduces the original defect, that violation is the original broken code, so it has to go red against the old code and green against the fix. For a supporting test, like the ordinary-day guard, the original code isn't the relevant failure. I deliberately introduce the violation it's meant to catch (here, making an ordinary day look like a holiday) and confirm it turns red. If it doesn't, the test is decoration.

The suite that broke on weekends

The second problem was the reverse. Tests that should have passed were failing, and only on certain days.

Several test harnesses for the bot's execution drills and reconciliation logic quietly depended on the real calendar. Part of the code they exercised asks "is the exchange in session right now?" The tests never answered that question themselves, so the real clock did. On a weekday the answer happened to suit the test. On a weekend it didn't, and a suite that had been green on Friday went red on Saturday without a single code change.

A test that depends on the wall clock is testing two things at once: your code, and what day it is. Only one of those is under review. The fix was to pin the exchange session inside the harness, so every run sees the same fixed session no matter when it runs. After the first harness was fixed, I checked the others and found the same hidden dependency in the reconciler tests.

The same pattern came back later, more subtly. A test for a guard around certain paid tools passed or failed depending on the exact moment it ran, because it read "now" in one place and compared it against a value computed a little later. The fix was to pin the whole test to one fixed instant in UTC. The time-sensitive tests I've reviewed now take their time as an explicit input, and that's the standard for any new one: it declares its time the same way it declares any other fixture.

I've since found a related problem: tests reading the environment they happen to run in. One test was picking up the real log directory from the environment rather than a temporary one. It worked on my machine, which is exactly the kind of test you shouldn't trust.

The test that didn't prove its name

The third problem is the quietest. One test was named for a specific guarantee: when a certain cleanup path runs, the most severe class of exceptions (the kind that means "stop now", not "something went wrong") must pass through untouched rather than being swallowed. The test ran that code path and it passed. But it never actually checked that the severe exception survived.

The name made a promise the body didn't keep. Anyone reading the test list, including me, would believe that guarantee was covered. I rewrote the test so it raises the exact exception class in question and asserts that the same object comes out the other side. It's a small change, but now the name and the assertion say the same thing.

An audit of another module turned up the same issue. Its tests checked the shape of the code, not its behavior. They were replaced with behavioral tests, one per guarantee.

Coverage as specified work

After enough of these, I stopped treating coverage as a percentage to push up. Now it's work that gets specified before it's written and audited after.

The process looks like a small feature release:

1. Spec first. For one area at a time (broker readiness recovery, connection handling, how order identity is stored, atomic state writes, close-path failures), I write down which behaviors and failure modes must be covered and what "covered" means for each one.
2. An independent review of the spec. A second reviewer, in my case a different AI model acting as an adversary, looks for vague spots in the spec before any tests exist.
3. Write the tests against the spec, not against whatever the code happens to do right now.
4. Record the verified result, including anything the work turned up.

The last step matters because coverage work turns things up. In one area, writing the specified tests exposed a real bug: after a lifecycle had already been resolved, the code still accepted stale observations that should have been ignored. In another, trying to cover a retry fallback showed that it was dead code that could never run. So it was deleted rather than tested. In a third, the spec showed that date and time boundaries had to be tested against the same database engine used in production. A lightweight stand-in doesn't handle time zones the same way, so a test passing against it wouldn't establish how the production engine behaves at those boundaries.

Aiming at a number alone wouldn't have ensured any of those discoveries. A coverage percentage can't tell you whether a test is able to fail, whether it depends on the day of the week, or whether its assertions match its name.

What I believe now

A green test is output, like anything else the system produces, and it gets the same scrutiny. Before I trust one, I want to know three things: have I seen it go red against a real violation of what it guards, does it control its own clock and environment, and does its assertion prove what its name promises?

It's more work per test. But I'd much rather have fewer tests that can each fail for a real reason than a large suite that's green whatever happens.