Back to Blog
Engineering11 min read

A Monitor That Knows When It Is Blind

On 14 September our resolver stopped answering for a few minutes at a time, and we told five customers their perfectly healthy jobs were down. This is what we changed, and how the fix itself went quiet for forty minutes ten days later without anyone noticing.

The night we were wrong

A monitoring service has exactly one promise: when it says your job failed, your job failed. On 14 September we broke it. Eighty-four executions across thirty-seven jobs and five accounts were recorded as failures. None of them had failed. The customers whose phones went off that night had nothing wrong with their servers.

The cause was underneath us. Our worker resolves hostnames through Docker’s embedded DNS, which forwards to the host’s stub resolver, which forwards to the upstream recursors. Something on that path went intermittent. The Docker daemon log counts it precisely:

PeriodResolver timeouts logged
1 August to 11 September0
12 September1
13 September750
14 September558

They came in bursts of two or three minutes, hitting every hostname at once, including our own mail server. From inside the container it looked exactly like the whole internet had gone away, which, from inside the container, it had.

The bug that turned a blip into an accusation

A few minutes of flaky DNS should have produced a few retries. Instead it produced alerts, because of a classification mistake that is easy to make and expensive to keep.

Before each request we resolve the target hostname and check the address, so that a customer cannot point a job at our own internal network. That check is a security gate, and a security gate that throws is a block. When the resolver timed out, the lookup threw, the gate blocked, and the block took the one path that skips retries entirely, because an attempt to reach a forbidden address should not be retried.

So a temporary DNS failure was filed as a security violation, which meant no retry, which meant a consecutive failure, which meant an email. The lesson is narrower than it sounds: a lookup that cannot complete and a lookup that resolves somewhere forbidden are not the same event, and the code has to say which one it saw.

The principle we settled on

Once you have made this mistake, the design question is clear. A monitor has to tell apart two statements that look identical from where it sits:

  • •Your endpoint is not answering me.
  • •I cannot see anything at all right now.

The first deserves an alert. The second deserves silence and an apology to nobody, because the customer has done nothing wrong and cannot act on the information. Most monitoring tools, ours included until September, collapse both into the first.

Canaries, and why three operators

The worker now asks a small question about itself on a timer: can I still see? It probes six canaries, three by hostname to exercise resolver and egress together, and three by IP literal to exercise egress alone. The three IP literals belong to three independent operators.

That last detail matters more than it looks. If you probe a single provider, that provider’s bad afternoon becomes your declared blindness, and you go quiet while your customers are genuinely down. We declare blindness only when every canary in a group fails. One red canary is not an outage, it is a broken canary, and it gets logged as such.

A third outcome, next to success and failure

The interesting part was not detection. It was deciding what a run is worth when we cannot see. A run that we failed to observe is not a success, and calling it a failure is the exact lie we set out to stop telling. So it became a third status.

While we are blind, a run that cannot connect is deferred rather than judged: thirty seconds, then sixty, then one hundred and twenty. Most blips are gone inside the first rung. A run that still cannot connect after the whole ladder is recorded as inconclusive, with no alert, and it says plainly in the execution log that we could not see at the time. Nobody is woken up for our resolver.

Introducing a third status is more work than it sounds. Every chart, every success rate, every digest and every retry counter had to decide what to do with a run that is neither good nor bad. We chose to exclude it from success rates entirely rather than let it quietly drag an account’s numbers down.

Holding the scheduler, but never for long

If we genuinely cannot reach anything, dispatching jobs only burns through retries. So the scheduler holds. But a cron service that stops scheduling because its own canaries are unhappy has invented a new way to fail, so the hold is fenced on both sides.

It takes two consecutive bad rounds to hold anything, because one bad probe round is not grounds for stopping a platform. And the hold is capped at ten minutes per episode. Past the cap we dispatch anyway and let every job record its own inconclusive row. We would rather run and report honestly than sit still on the word of six probes.

What it was worth

The twenty-four hours after the change, measured against the same window before it:

OutcomeCount in 24 hours
Successful runs15,195
Failures35
Timeouts5
Errors1

Every one of those thirty-five failures was real and each was followed by a success, which is a customer backend having a bad second, not us having a bad network. Zero false alerts. We thought we were finished.

Ten days later, the monitor went blind for real

On 24 September at 17:54 UTC the host stalled. Not the network, the machine. The container runtime could not forward its own events and logged ninety-nine deadline errors in five minutes. Docker could not finish a container health check inside its fifteen second limit, on eight containers at once. For about four minutes nothing on that box could do anything in a reasonable time.

All six canaries timed out together. The gate did exactly what it was built to do and declared itself blind, with the reason recorded as egress, and sent the operator email. That part worked.

Then it never ran another round. For forty minutes the protection was switched off and nothing said so. The second email, the one the first email promises, never came, and that absence is the only reason anyone looked.

How a health check dies quietly

The rounds are coalesced, so that a burst of failing jobs cannot stampede the canaries. One round at a time: if a round is already running, everyone gets handed that one.

async runRound() {
  if (this.inflight) return this.inflight;   // one round at a time
  this.inflight = this.doRound().finally(() => { this.inflight = null; });
  return this.inflight;
}

Read that again with a round that never settles. The promise is never fulfilled, so the handle is never cleared, so every later caller is handed the same dead round forever. The timer kept firing every fifteen seconds and every tick returned immediately with the same corpse. Nothing threw, so nothing was logged: a rejection would have cleared the handle and left a line in the log. Silence was the only symptom.

We got away with it by luck. The gate holds dispatch only after the second bad round, and the second round never came, so the counter stayed at one and no job was ever held. Nothing was lost. Had the loop died one round later, every job on the platform would have been held for ten minutes.

Watching the watcher

Three changes came out of that, and the third is the one we would press on anybody building something similar.

A round now has a deadline. Past it the round is abandoned and counted as bad, not neutral. A worker that cannot finish six five-second probes cannot see the network either, and guessing green there is how this whole story started. It gets its own reason, so the operator email says the host is wedged rather than the network is down.

An abandoned round is fenced. Rounds are numbered, and a round compares its number before writing anything. The round we walked away from cannot come back twenty minutes later and declare everything healthy using probe results from a network that no longer exists.

A second timer measures the silence. It has nothing to do but notice that the first one has stopped talking. If no round has finished for ninety seconds it says so, once, and releases whatever is wedged so the loop restarts itself. There is something almost embarrassing about needing a watchdog for your watchdog, and we would still argue it is the correct shape: the one state a health check cannot report is its own absence.

What to take from this

If you run anything that judges other systems, three things are worth stealing.

Separate it is broken from I cannot see, in the data model and not just in the logs. If your schema only has success and failure, you will eventually record a lie, because there is nowhere else to put the truth.

Probe more than one operator, and require all of them to fail. Otherwise you have simply moved your single point of failure somewhere you do not control.

Make absence detectable. A component that goes quiet looks exactly like a component reporting good news, and the longer the quiet goes on the more confident the good news appears. Ours mirrors its state after every round with a short expiry, so a missing key means no round has run in two minutes. That one line is what turned a forty minute mystery into a five second check.

The same idea, pointed outward

Everything above is about noticing that something did not happen, which is also the hardest thing to notice about your own scheduled work. GitHub tells you when a scheduled workflow failed; it cannot tell you when a run never happened, and schedules go quiet for three documented reasons that leave no red mark anywhere.

We published the heartbeat side of this as a GitHub Action under MIT, with no dependencies, and it works with any service speaking the same ping contract rather than only ours. If you would rather read about the idea first, the heartbeat monitoring page explains what it catches that an uptime check cannot.

A Monitor That Knows When It Is Blind | CronJobPro