Dedupe Became a Lookup Once the Detector Named the Incident

I've fixed deduplication several times. Each fix held for a while, and then some new pattern of alerts got through twice, or two different problems got merged into one. This post is about the week I stopped patching it and changed the design.

The shape of the problem

The bot watches a lot of things about itself: connection health, whether its view of positions matches the broker's, whether scheduled jobs actually ran, and whether its own safety checks behaved. When one of those checks finds something wrong, it raises an alert. Some alerts become tracked issues so a human (me) can follow them to resolution.

A real problem rarely fires just once. A flaky dependency can trip the same check every few minutes for an hour. If each trip opens a new issue, the tracker fills with copies and the real signal gets lost. So I need deduplication: noticing that a new alert is really the same incident as an existing one, and adding to that one instead.

Until recently, that decision happened at the receiving end. The code that turned alerts into issues looked at what came in and asked, "have I seen this before?"

That question can't be answered well from where it was being asked.

Two ways of guessing, tested and dropped

When dedupe broke again, my first instinct was to improve the guesswork. I wrote up two candidate mechanisms. Both lived on the consumer side, and both tried to work out whether two alerts were the same incident from what the alerts looked like when they arrived.

This time I tested them before building either one. The method was simple. I put together a small set of recreated alert pairs modeled on the kinds of incidents the tracker had already seen. For each pair I wrote down the correct answer, "same incident" or "different incidents," based on what had actually gone wrong. Then I checked what each proposal would have decided. I was looking for failures in both directions:

Over-merging: two separate incidents collapse into one issue because their alerts look alike. This is the dangerous failure. A second, real problem hides inside a thread I already think I understand.
Under-merging: one ongoing incident splits into several issues because its alert wording shifts a bit between firings. It's noisy and annoying, but less dangerous.

The hard pairs fell into two groups. In the first, different checks, or the same check pointed at different things, produced messages from nearly the same template. On the surface they looked like one problem, but they were two. In the second, a single ongoing problem produced messages whose details changed from one firing to the next, such as a count or a description of the current symptom. On the surface they looked like several problems, but they were one.

Neither mechanism handled both groups. When I tightened a proposal so it stopped merging the first group, it started splitting the second. When I loosened it to keep the second group together, it merged the first. That tradeoff was the real finding. The pairs that were hardest to decide were the same ones in both proposals, and in each case the facts that settled the answer weren't in the alert.

Both proposals were working from the alert's surface: its text, its timing, its category. Whether two alerts are one incident depends on something only the detecting code knows: which condition it was checking, what it was checking it against, and whether this firing continues an earlier one.

For the approaches I tested, the consumer wasn't bad at judging. It was being asked to make a judgment its inputs couldn't support.

Moving the identity to where the knowledge is

The redesign has one idea at its center: the code that detects a problem also names it.

When a check finds something wrong, it produces a stable identity for the incident, not just an alert message. That identity is built from what the check actually knows, meaning the condition and the thing it applies to. It is not built from wording that might change. If the same condition on the same subject trips again, the check produces the same identity. If a different condition trips, it produces a different one, even when the messages happen to read almost the same.

With identity owned by the producer, the consumer's job changes from judgment to lookup. It no longer asks "does this resemble something I've seen?" It asks "do I already have an open incident with this identity?" That question has a yes-or-no answer.

The word open matters there. The identity names a kind of trouble: this condition, on this subject. It doesn't name one particular occurrence of it. Each incident record also has a lifecycle. It starts open, collects repeat firings while it's open, and closes once the condition is resolved. If a firing arrives and there is an open incident with that identity, the firing gets added to it. If the only matching incident is already closed, the firing starts a new incident. So the same failure coming back next week shows up as a separate occurrence with its own history, and it doesn't quietly reopen an old thread I've already stopped reading. The shared identity still links the two, so a recurring problem is easy to spot as recurring.

The four steps

I shipped it in four steps, each small enough to review on its own:

The identity schema. This defined what an incident identity is: which fields go in it, what makes two identities equal, and what a producer must supply. It comes first because everything else depends on the contract being clear. A loose identity scheme would just move the guesswork upstream.

Producers persist their incidents. Detecting code now records the incident it raised with its identity attached, instead of just sending an alert and moving on. I converted one health probe first to prove the path end to end before touching the others. Persisting also gives me a durable record that doesn't depend on the alert channel staying up.

Wiring identity into issue creation. The code that opens issues now receives the producer's identity and uses it as the dedupe key. The earlier similarity logic is no longer on this path.

A deterministic consumer. Finally, I built a consumer that reads the persisted incident records and processes them the same way every time. With the same inputs, it makes the same decisions, and there's no consumer-side similarity rule to tune.

The order matters. If the consumer had come first, it would have had nothing reliable to consume. If issue creation had been wired up before producers persisted anything, the key would have existed in one place and nowhere else.

How I checked it, and what that doesn't prove

The validation reused the hard cases from the proposal tests, this time as automated tests against the new path. A condition that keeps firing on the same subject produces one open incident that collects every firing. Two different conditions with nearly identical messages produce two incidents. A condition that fires, resolves, and then fires again produces two separate incidents that share an identity. The tests check that the new path gives the expected match for each of these chosen cases.

That's evidence the mechanism does what it's designed to do. It isn't a long-run measurement of a quieter tracker, and I'm not claiming one here. The design also has limits that tests can't remove. An identity is only as good as the check that defines it. If a check makes its identity too broad, it will over-merge. If it puts a changing detail into the identity, it will under-merge. The old failure modes are still possible, but now they belong to one specific check and can be found by reading its code, instead of coming out of a heuristic that no one could fully reason about. The lifecycle has a weak spot too: a condition that flaps between resolved and firing could open a string of short incidents. I'll deal with that if it turns out to be a real problem, not before.

What I actually learned

The broader lesson goes beyond alerts. If a downstream component keeps making the same kind of mistake, check whether it has the information it needs. I spent a long time making the consumer's guesses more sophisticated. Testing two concrete proposals showed that both got stuck on the same pairs, and those were the pairs where the deciding facts never reached the consumer. Making those two approaches smarter wouldn't have fixed that. The information had to come from somewhere else.

There's also a lesson about testing before building. Both rejected mechanisms seemed reasonable on paper. Writing them up and testing them against realistic cases took much less time than shipping them and finding out in production that they hid real incidents. In both cases, rejecting them was the useful result.

Last, this approach adds a responsibility for the people writing checks. Every new check now has to answer "what is the identity of the thing I'm reporting?" when it's written, not leave that to be worked out later. I think that's the right place for the question. The person writing a check knows best what it's detecting, so they're the one who should name it.

The system still raises plenty of alerts, but deduplicating them now depends on a lookup rather than a judgment call.