When 'submitted' doesn't mean 'done': four bugs at an asynchronous execution boundary

For weeks, one subsystem of the bot had a strange pathology: it would decide to do something, construct the request, submit it to a third-party API, log success — and nothing would happen. No completion. No error. Sometimes no trace of the request on the remote side at all. I had believed the decision logic and the execution machinery were cleanly separated concerns, with all the hard problems on the decision side. What this arc exposed is that the boundary between them was incomplete: everything between "do this" and "this is done" turned out to be an entire subsystem I had been treating as a function call.

This is the story of debugging that gap, in four acts.

Act one: the request that could never complete

The first bug was embarrassing in retrospect. The requests my code constructed satisfied every invariant I had written down — but the remote system has acceptance constraints of its own, and my requests were parameterized in a way that let them be accepted and then simply never progress. From my side, the behavior looked like indifference: submit, wait, nothing, forever.

The immediate fix was to validate a request against the remote system's acceptance model before submission, not just against my internal one. The more honest conclusion was that the boundary between decision logic and execution validation was incomplete: the code that decided what to do had quietly assumed that anything it asked for could be carried out, and no layer owned the job of checking that assumption.

Act two: the server-side checks I didn't know about

With that fixed, a new failure mode surfaced: the API began rejecting some requests with server-side validation errors — checks whose model of a valid request differed from my application's assumptions. I had treated my own validation as the whole story; the server disagreed, in a cryptic numbered error code, delivered asynchronously on a message channel rather than as a return value from the submit call.

That last part was the real bug. My first implementation couldn't even tell which request an error belonged to. Before I could reason about any individual rejection, I had to add explicit correlation between asynchronous error messages and the request that provoked them. Each rejection then had its own resolution — understand what the check guarded against, then adapt the request or the configuration accordingly — but none of that was possible while errors floated free of context.

The same episode produced a follow-up bug: adapting one request shape meant a field that had previously always been populated was now legitimately unset, and my reporting had been happily displaying that unset field as if it were meaningful data. Every consumer of a record carries its own assumptions about which fields are real.

Act three: the silently discarded request

The strangest bug of the arc: requests that were accepted and then vanished. No rejection, no completion — discarded by an intermediary. The cause was a configuration preset in the gateway software that sits between my code and the remote service: it silently drops requests that leave a particular optional attribute unset. My code had never set that attribute, relying on a default that the gateway's configuration quietly overrode. One attribute, always defaulted, never questioned, and the effect was indistinguishable from "the remote side ignored us" unless you went looking at status events.

The fix was one line — set the attribute explicitly, always — but the lesson was bigger: every field you leave to a default is a field whose behavior is owned by someone else's configuration.

Act four: assume it will hang

By this point I had stopped believing that any single fix would make submission reliable. So the last piece was structural: supervision. At an asynchronous boundary, a request that stalls without progress is not an edge case — it is a standing possibility — so the system needs a supervisor that notices stalls and drives recovery.

Two principles shaped that design. First, recovery must begin with reconciliation against the remote system's authoritative state: before retrying anything, confirm what actually happened to the original request, so you never redo work whose completion notification simply went missing. Real-time event notifications are the right primary signal — low-latency and cheap — but they are a signal, not the record; the remote state is what you reconcile against whenever anything is in doubt. Second, observability before automation: a new automated recovery path should first show you what it would have done before it is allowed to act.

The lesson

The through-line of all four acts: I had modeled this boundary as a fire-and-forget call — submit, assume done. In reality it is a distributed-systems boundary with every classic failure mode: requests that are accepted but can never complete, errors that arrive out of band and uncorrelated, third-party configuration mutating your semantics, and states that stall without a supervisor. It deserves what any such boundary deserves — explicit state machines, correlation of asynchronous signals, reconciliation against the remote source of truth, and observability before automation.

"Submitted" is a claim about what you sent. "Done" is a claim about the world. The distance between those two sentences is where a month of my engineering time went.

This journal documents software engineering on an automated trading system. Nothing here is financial advice, a trade recommendation, or a representation of trading results.