Manufacturing failures: the escalation drill harness

The problem with rare paths

The close-escalation system exists for the moments when a routine close order stops being routine: the order hangs, the broker cancels it, a restart loses track of it, something else fills first. Each of those branches is code that runs almost never. These branches occur too infrequently and unpredictably to provide a dependable validation cycle — and when one finally does fire, it fires exactly once, under real conditions, with no second take.

That's a poor testing regime. Unit tests cover the branches individually, but they run against mocks that behave the way I believe the broker behaves. Code review checks the logic against the spec, but the spec encodes the same beliefs. The failure mode I was actually worried about — the system confidently doing the wrong thing because reality disagrees with my model of it — is largely invisible to both. Waiting for production to exercise the path means the first real test happens with real money on the line, and a sample size of one.

So I inverted it: instead of waiting for failures, manufacture them.

The drill provisions its own fixture

The first design problem was surprisingly not the failures — it was the subject. A close-escalation drill needs something to escalate against, and my first instinct was to borrow whatever state the non-production environment happened to hold. That coupled the drill to that environment's schedule and inventory, which meant drills could only run when it cooperated. After going in circles on sequencing, I realized the constraint was self-inflicted: the escalation logic doesn't care how old its subject is or why it exists. The drill could provision its own minimal, self-contained fixture, run the entire escalation lifecycle against it, and tear it down afterward.

That decision made the harness self-contained: set up a fixture, put the system into the state the drill is exercising, force the failure, measure the outcome, tear down. No waiting, no borrowing, repeatable on demand.

The most useful tool in the kit is the one that injects controlled external-state inconsistencies: it makes the outside world diverge from what the system believes about it, without going through the system's own bookkeeping. From the system's perspective, something it believes is alive has silently changed state underneath it — which is the class of scenario the escalation system exists to handle, and one that my mocks, in this project, never reproduced. The drill manufactures the desynchronization on purpose, then watches whether the system notices and reconciles.

Reality disagreed immediately

The drill earned its keep before the first full run finished. Just building it surfaced friction the escalation code had been silently wrong about: real integration responses differed from the assumptions baked into the mocks, so inputs had to be normalized before they were usable, and the drill had to select its target resources unambiguously rather than taking the first plausible-looking match. Every one of those was a fact about the real integration that the mocks had quietly papered over.

Then the drills themselves started finding defects in production code. Four, in the first week — and, notably, not all of the same kind:

A reconciliation didn't survive a restart (persistence gap). When the system detected that an externally terminated order was its own and healed its record, that conclusion lived only in memory. Restart the process and it was forgotten. The fix makes the reconciled state durable, so a conclusion the system has already reached is never re-derived from scratch. In hindsight, a unit test could have caught this; nobody had thought to write one.

A replacement order had no patience of its own (timeout omission). When escalation replaced a stuck close with a more aggressive one, the replacement inherited no independent deadline — it could hang exactly the way its predecessor had. The invariant is now explicit: every replacement receives an independently bounded, nonzero response window. Also plausibly unit-testable; also never tested, because the design never stated the invariant that every outstanding order carries its own deadline.

Attribution trusted unstable keys (integration assumption). Order events were being matched to positions using identifiers that can collide or drift across sessions. Attribution now binds on a single stable identity for each order, which also let me delete an earlier, blunter safeguard that had been discarding legitimate events. This one genuinely required the real integration to expose: the mocks handed out identifiers that were stable by construction.

One category of order event never reached its handler (event-routing defect). The system subscribed to several kinds of lifecycle events and routed each to the escalation handler — except one, which was consumed upstream and never forwarded. The consequence was that a particular escalation branch could never fire, no matter what the broker did, and every test of that branch had been exercising a handler that would never be called in the real wiring. The invariant it violated is simple to state: every lifecycle event the system subscribes to must have a path to the component that acts on it. The fix connected the missing route, and I proved it by building a dedicated drill fixture whose lifecycle exercises exactly that event, running it end to end, and watching the handler finally receive it.

All four were fixed, re-drilled, and verified on the non-production environment within the same week. That loop — force the state, observe the defect, fix, force it again, watch it pass — is something waiting for production would not have given me.

What the drill actually bought

The honest summary: the existing code review and the existing test suite missed all four. But the reasons differ, and only one of the four strictly needed the real integration to expose. The attribution defect was a mismatch between my model of the broker and the broker itself — something mocks built from my model could not, by construction, reveal. The other three were application and wiring gaps: a missing persistence step, a missing timeout, and an event route that was never connected. Better unit tests could plausibly have caught the first two if anyone had thought to state the invariants first; the routing gap hid because every test exercised the handler in isolation rather than the wiring that was supposed to reach it. The drill caught all four for the same reason: it exercised the real path end to end and measured the outcome, rather than checking each piece against what I already believed.

The drill also changed what "done" means for this subsystem. A rare path used to be done when the code merged and the tests passed. Now it's done when the drill has forced that path against the real integration on a non-production environment and the measured outcome matched the spec. It's a higher bar, and after this week, I don't trust the lower one on its own anymore.

This is an engineering journal, not trading advice. Nothing here is a recommendation to trade anything.