Earning the Right to Act on Production
There is a particular kind of automation I have learned to be slow about: the kind that can take production down on purpose.
Most of the monitoring I write is inert by construction. It looks, it decides something is wrong, it tells me, and then it stops. It can be wrong — it can miss the thing entirely, it can tell me everything is fine when it isn't, and enough noise from it will train me to ignore it. Those are real costs. What it mostly cannot do is reach into a running system and interrupt it. Automation that acts can, and that difference in immediate authority is what the rest of this post is about.
Recently I worked through both sides of it: what a piece of acting automation has to establish before I grant it authority, and what to do with one that was already acting without having established it.
The failure that started it
The bot depends on an authenticated session that has to be renewed periodically. Occasionally that renewal doesn't complete cleanly, and the system lands in a state where every liveness signal is green and the session behind them is not usable. The process is alive. The dependency answers. The work still can't happen — and you only discover that at the moment you try.
That happened more than once before I stopped treating each occurrence as a one-off. The manual fix was always the same shape: notice, reset the session, carry on. It is exactly the kind of thing that should be automated, and exactly the kind of thing that is dangerous to automate, because the remediation disrupts a component everything else depends on.
So I wrote a watchdog for it — and then spent considerably more effort on the constraints than on the detection.
Scope evidence to the session, not the process
The first version had a bug worth naming, because it is generic rather than specific to my setup.
The watchdog decided a session was healthy by looking for evidence that an authentication had succeeded. That evidence lived in the long-lived process's own history. Which meant: once a login had ever succeeded during that process's lifetime, the watchdog could keep finding the record and keep concluding things were fine — long after the session the record referred to was gone.
A stale success was being read as current evidence.
The fix is a principle more than a patch: health evidence has to be keyed to the lifetime of the thing it claims to describe. Evidence produced by a previous session is not evidence about this one, and a check that can't tell the two apart isn't a health check. It's the monitoring equivalent of a cache key. If the key is wrong, what you're holding isn't proof, it's a souvenir.
Bound what an intervention can reach
The second constraint was blast radius. When the watchdog decides something is wrong and intervenes, what exactly is it allowed to touch?
The naive answer is "whatever it takes." The answer I wanted was narrower, and stating it requires keeping three things apart. Take a sanitized version of the structure:
- Trigger — the specific condition observed, attached to a specific identified unit of work. Not "something looks wrong," but "this operation is in this state."
- Authorized target — the one unit of work named by that trigger. Not its siblings, not the subsystem, not the process.
- Action — the single intervention permitted against that target, and nothing more.
Two things are worth separating here, because the post has already put both in play. Interrupting one identified unit of work is a narrow intervention: it stops that operation and leaves everything around it running. Recovering a shared session is a much broader one, because the session is a dependency of everything, and anything in flight against it is affected whether or not it was part of the diagnosis. The sanitized structure above illustrates only the narrow case — it is the shape I use when a single operation is the thing in the wrong state, not a description of what a session reset does. Session-level recovery is the wide-radius action, and it is precisely why the rest of this post is careful about who gets to trigger it.
The scope guarantee I was aiming for, in the narrow case: an intervention derives its authority entirely from the condition that produced it, and cannot outlive that condition. If the triggering condition clears, the authority to act on it is gone — the automation does not get to keep holding the brakes on the strength of something that stopped being true.
That guarantee has a hole, and I'd rather name it than pretend otherwise. The guarantee is anchored to the last check of the condition, and nothing later. Between that check and the intervention actually landing, the world can change: the check has already passed, the action is already in flight, and what gets stopped may be a unit of work that is no longer in the state that justified stopping it. Anything that changes after the last check can invalidate the justification for an action already on its way. The only real close is to make revalidation and intervention atomic — to recheck and act as one indivisible step — and that would have meant making the intervention path substantially more complex than the hazard justified. So I wrote it into the accepted-risk register instead: a ratified list of hazards I've decided to live with, which my own auditing process reads and doesn't re-file as a new finding each time. A known accepted risk is a different thing from an unknown one, and the register is what keeps that distinction from eroding.
Observe before granting authority
The watchdog ran first in observe mode: full detection, full logging of what it would have done, no authority to act.
That period earns something specific and not more. It produces evidence about how the detector behaved in the cases that actually occurred while it was watching: whether it fired on real conditions and stayed quiet the rest of the time, against traffic no test fixture reproduces — a rare interaction between a dependency, a network, and a clock. That's genuinely hard to get any other way, and it is still evidence about observed cases only.
What it cannot show is just as specific. Passive observation only sees the failures that happen to occur while you're watching, and the failure in question is rare enough that a quiet window is weak evidence about the failures that didn't show up. It says nothing at all about what happens when the intervention executes: whether the recovery works, whether it's safe when the diagnosis is wrong, what state the system is in on the far side.
The complement is a controlled recovery exercise — inducing the failure deliberately and watching the intervention run — and it has the same shape of limitation. It produces evidence about the action's behaviour in the scenarios you thought to construct, which is not the same as evidence that the action is safe in general. Neither body of evidence settles the other's question, and neither one is complete on its own. Treating a clean observation log as an answer to both is how you end up trusting a mechanism nobody has watched do its job.
I should be plain about where that leaves this particular watchdog, because the standard I've just described is a standard I am proposing, not one I can claim to have met. It was promoted out of observe mode on the strength of the observation window and the scoping constraints — lifetime-keyed evidence, a bounded target, a named accepted risk. No controlled recovery exercise sits behind that promotion. The rehearsal half of the standard is an outstanding item, and until it exists, what I have is a mechanism with good constraints and untested behaviour on the far side of the trigger. That's a weaker position than "this one earned it," and it's the accurate one.
One more thing showed up at the moment of promotion: the acting-versus-observing distinction lives in configuration, and configuration drifts. If the behaviour of a safety mechanism depends on a flag, the flag's intended value is part of the design and belongs in the written record, not only in whatever file happens to set it. Otherwise the system's most consequential property is a thing you'd have to go look up.
A staleness counter did not justify a reset
The other half of this. A different subsystem already had automatic remediation in it, shipped earlier: after some number of consecutive stale readings, it would reset a live session unattended.
Looking at it alongside the watchdog work, the problem was that the trigger didn't distinguish between the two situations it needed to distinguish between. Repeated stale readings are evidence of something — they are not nothing. But they don't separate "the session has failed" from "the path we read the session through returned nothing." Those call for different responses, and the counter can't tell them apart. For an action as disruptive as resetting a live session — the wide-radius one — that isn't a sufficient basis. The mismatch is between how specific the diagnosis is and how much the action disturbs.
My first instinct was to make the reads more reliable. A review round pushed back on that hard, and correctly: improving the read doesn't turn a counter into a diagnosis. It just makes the same guess with better inputs.
So the unattended reset came out. The condition still gets noticed; it now surfaces to a human instead of resolving itself.
That's the asymmetry I keep landing on. Automation that watches can be wrong in ways that cost me — a missed detection, a false all-clear, a page I've been trained to ignore. Automation that acts can be wrong in a way that takes the system down itself, on purpose, at a moment of its own choosing. The bar I want for the second kind is evidence on both halves: scoped to the right lifetime, bounded to the right target, observed against real conditions, and rehearsed against induced ones. One of my two acting mechanisms failed that bar on diagnosis and was demoted to a notification. The other meets most of it and is still missing the rehearsal. Writing the standard down is the easy part; what it's actually for is making the gap in your own system impossible to keep not noticing.