When deployment removed a health check

Two failures, one of them the subject here

Mid-August I found the broker gateway dead. It had stopped hours earlier, and none of the checks I had in place had said anything.

Why the gateway stopped is its own question; this post is about the second failure, the one where nothing noticed for hours in a system that had a check written specifically to notice.

The check that would have caught the dead gateway was a scheduled job. It had been in place for months. It was not in place anymore.

Why nothing noticed

The deploy installs the scheduled jobs as one of its steps. The installer wrote one set of jobs and never applied the second set, the one that included the gateway check. The file defining that second set existed in the repo. It was correct. The deploy just never installed it. So each deploy since that step was written had been replacing the host's schedule with a version that lacked the gateway check.

I want to be precise about the shape of this bug, because I first got it wrong. My first incident note said the gateway check's definition was missing. It was not missing. It was present, reviewed, and unused. The defect was in the installer, not the artifact. That correction mattered, because "write the missing file" would have fixed nothing.

There was a second, smaller version of the same problem next to it. A separate check matched running processes against a pattern, and the pattern had drifted out of sync with the actual process list. The stale pattern made that check ineffective.

So: the primary check was uninstalled by the deploy, and the secondary check had quietly stopped meaning anything. One check was absent; the other was ineffective. Neither raised an alert.

The fix inside the installer, and why it is not enough alone

The obvious fix is "make the deploy install the right schedule", and I did that the same day. That fix is necessary. The installer was wrong and it had to be repaired.

But a repaired installer has the same weakness as the thing it fixes. It is one more step in the same script, and the next refactor of that script can drop it just as silently as the last one did. The installer fix closes this bug. It does not close the class.

The class is this: nothing in the system could tell the difference between a check that found no problem and a check that never ran. I had already learned this once for a nightly integrity scan and written it up. Here it was again, one layer up.

Watching the checks

What grew out of the postmortem was a small supervision pattern with three pieces. I describe them here as design, not as a map of what runs where. Each one answers a question the old setup could not.

A heartbeat registry. Every supervised job writes a timestamped heartbeat under its own name at the end of a successful run. That timing is the important part. A heartbeat is not "I was invoked". It is "I ran to completion without error". A job that starts and hangs, or starts and crashes, writes nothing, and its last heartbeat goes stale. A separate evaluator reads the registry and alerts when any expected heartbeat is stale or absent.

"Expected" is doing work in that sentence. The evaluator should not learn which jobs exist by watching what has written before, because a job that never ran would never appear. The set of supervised jobs is declared alongside their schedule, and the evaluator checks that declared set. A job that was never installed shows up as a job with no heartbeat, which is what it is.

Two details came out of the design review with the second model I use as a critic. First, a heartbeat from the future should be treated as a failure, not a success. A clock skew or a bad write should not buy a job an alibi. Second, every path by which the evaluator can disable itself has to be loud. A monitor that quietly turns itself off is the original bug wearing a new hat.

What the heartbeat does not prove: that the job's check was correct. A job can complete successfully while looking for the wrong thing, which is exactly what the pattern-drift check had been doing. The registry catches jobs that stop running. It does not catch jobs that run and are wrong.

A schedule drift alarm. The installed schedule is compared against the version the repo says should be installed, and a mismatch alerts. This is the check that could have detected the schedule mismatch on the first deploy after the regression, instead of on the day the gateway happened to die, provided the alarm itself remained installed, executed successfully, and could deliver an alert. It fails closed: an unreadable schedule is drift, not "nothing to report".

The drift alarm is itself a check that can be uninstalled, and heartbeat monitoring only notices its absence while the heartbeat evaluator is still running. That is the general limitation of this design. A stale heartbeat is only an alert if something is still there to read it and a path still exists to deliver the alert. Keeping the evaluator small makes it easier to inspect and test, but simplicity is not the same as observability. At some point the recursion has to stop, and the honest thing is to know where it stops.

A startup path that verifies its dependency. The third piece is a pattern for the thing that consumes the gateway rather than for the checks around it: a start routine that verifies its dependency is reachable before declaring success, with a bounded wait rather than an open-ended retry. Without that, a start against a dead dependency can appear to succeed while leaving the application unable to function, in exactly the way this incident had. This is not a monitor. It is a refusal to create a new silent failure on top of an existing one.

Exercised before promotion

None of this was promoted on the strength of tests alone. Each piece was exercised in a pre-production environment against the failure it was written for. The two alerting pieces had to alert: a heartbeat was deliberately allowed to go stale, and the installed schedule was deliberately edited by hand. The startup guard had a different pass condition: a start was attempted with the dependency stopped, and the start had to refuse.

Those exercises exposed edge cases the existing tests had not represented, including one path where the drift comparison could read an incomplete input and report clean. Those were fixed before promotion.

The exercises also produced a small, satisfying confirmation. The schedule wipe showed up there too. A scheduled job I believed I had turned off was still running, because the change that turned it off lived in the schedule the deploy kept overwriting. The bug had been demonstrating itself in front of me for weeks.

What I take from it

Monitoring has to be monitored, and the monitor of the monitors should be simple enough to inspect and test, with the understanding that simplicity alone does not make its failures visible.

"Healthy" is a claim about a specific layer. Each time I have been burned by a green light, it was because the light was measuring something adjacent to what I cared about. Now, when I add a health check, I write down what would still be broken while it reports green. The answer is never "nothing".

And I no longer treat a fix inside the failed component as the whole fix. The installer had to be repaired, and it was. But the fix that makes the repair hold is the one that asks, separately, whether the deploy left the host in the state the repo says it should be in. The two are complementary. One closes the bug. The other can make some installation regressions visible, for as long as the verifying check and its alert path are themselves still working.