Insight
The monitor that went quiet.
Two of my own systems have been down without me knowing. The first for two days, the second for seven weeks.
Open the diagram description
A two row diagram comparing how an unattended system failed and how it was changed afterwards. Both rows run through the same three numbered stages, left to right: one, something notices; two, somebody is told; three, the work in flight survives. The upper row is labelled as it was built. Stage one is a solid panel reading, the supervisor notices: it checks a heartbeat every five minutes and restarts the bridge, and it keeps succeeding. Stage two is drawn as an empty box with a dashed outline and dimmed text, because this step never happened: nobody is told. It kept saying nothing, and restarting every few minutes and running normally look identical from outside. The line leading into that empty box is dashed and ends in a hollow, unfilled arrowhead instead of a solid one, which marks a signal that was expected and was never sent. A caption dropped below that gap reads, expected signal, not sent. Stage three of the upper row reads, history did not survive: down two days on one system, seven weeks on the other, and that history was not backfilled. The two durations are picked out in a warm tan colour. The lower row is labelled as it was fixed, and here the same three stages are joined by solid bright green arrows that all arrive. Stage one reads, it counts its restarts: three fast restarts in a row is the signal, and it keeps restarting slowly, once an hour. Stage two is the emphasised panel, outlined in bright green, and reads, it sends me an email: the test email was sent and checked, because a path nobody has watched work is not a path. Stage three reads, recovery is catching up: the work in flight survives, so recovery is catching up rather than reconstructing. Beneath both rows sits a quoted question, marked by a short green bar: when this stops, who finds out, and how long does it take?
Neither failed loudly. The first had a watchdog that restarted the service, and the watchdog itself stopped reporting, so the dashboard stayed green over a dead process. The second was held open by a zombie job that never released the session, and because nothing was expected to arrive on a schedule, nothing noticed that nothing arrived.
Why did a watchdog make the first outage longer?
The first system is a bridge that moves messages between Slack and WhatsApp. It runs on a machine I own, under a supervisor that checks a heartbeat every five minutes and restarts the bridge when the heartbeat goes stale.
That is the standard answer and it worked. It worked so well that when the bridge started failing for a reason a restart could not fix, the supervisor kept restarting it, kept succeeding at restarting it, and kept saying nothing. From outside, a service being restarted every few minutes and a service running normally look identical: both are up. The outage ran from 16 to 18 July, about two days, and it ended when I happened to look.
The failure was not in the bridge. It was in the assumption that a component whose whole job is to notice trouble will tell somebody when it finds some.
What actually fixed it, if not a better restart?
Making silence itself an alert.
The supervisor now counts its own restarts. Three fast restarts in a row is the signal that something is wrong in a way restarting does not address, and at that point it sends me an email. It does not stop there: it keeps restarting, slowly, once an hour. That combination is the part worth copying. A transient fault, a network blip, an API that was briefly unhappy, still heals itself without waking anybody. A real fault reaches a person while it is still two days old instead of seven weeks.
The test email was sent and checked, because a notification path nobody has watched work is not a notification path.
How does a system stay down for seven weeks?
The second system is a self-hosted message pipeline I run. It went down on 31 May and I found out on 19 July. Seven weeks.
The cause was mundane: a background job had died in a way that left it holding the session open, so the process looked alive and the session looked claimed, and nothing else could take over. What made it seven weeks rather than seven hours is the interesting part. Nothing arrived on a schedule. There was no run that failed to happen at nine o’clock, no daily digest that stopped landing, nothing whose absence had a shape. The system was quiet, and quiet was also what it looked like when it was working.
Seven weeks of history were lost. I did not backfill them, and I have written that down in the project’s own notes rather than quietly moving on, because the honest version of this story is more useful than the tidy one.
What do I ask before building anything now?
When this stops, who finds out, and how long does it take?
It is the first question, ahead of which tool, ahead of how many steps, ahead of what it costs. An automation that fails loudly costs an afternoon. One that fails quietly costs whatever accumulated while nobody was looking, and that number is not knowable in advance, which is exactly what makes it worth designing against.
In practice the answer has three parts. Something has to notice, and it cannot be the same component that is failing. Somebody has to be told, through a path that has been tested by sending a real message to a real inbox. And the work that was in flight has to survive, so that recovery is a matter of catching up rather than reconstructing.
The cheapest monitoring is the kind that only reports good news, and you find out what it cost much later.
I ask this about client systems because I have paid for the answer on my own.
Failure that makes no sound
Which of your systems could go quiet for a week?
Tell me what runs unattended and how you would know if it stopped. I will tell you what I would look at first.
Message Dan on WhatsApp (opens in a new tab)or email dan@burdetsky.xyz