Why do the alerts never arrive, or arrive in spam?
In short
Monitoring that nobody hears is decoration. Alerts get lost in three ways — filtered by the receiving mail system, sent to one person who is away, or buried in so much noise that everyone stopped reading. The fix is routing, more than one channel for the few things that matter, and being ruthless about what is allowed to interrupt anyone.
A monitoring system has one job at the moment of failure: to reach a human being. Everything before that — the checks, the dashboards, the history — is preparation. And it is at that last step, the delivery, that most systems quietly fail.
Where does an alert get lost?
It was filtered. The message was sent, accepted, and put in a folder nobody opens. There is a story that goes round among people who run websites: an outage that lasted more than a day, and the alert was found afterwards, in spam, timestamped correctly. Nothing was broken. Everything worked exactly as configured.
It went to one person. The classic version comes from backup administrators: months without a valid backup, because the failure notices went to a single mailbox belonging to someone who had left the company. One address is not a destination, it is a single point of failure with a name on it.
It was buried. This is the modern one, and the hardest to admit. The alerts arrive, all of them, all the time, and at some point everybody stopped reading. A system that cries wolf forty times a week has trained its own audience to ignore the forty-first, which is the real one.
What should you fix first?
First, stop the noise. A failure should have to repeat before it is announced; a single lost packet is not an outage. Things that fail together — everything on one server, everything behind one line — should produce one message and one recovery, not nine of each. And when you are deploying, the alerts should be silenced on purpose, by you, rather than ignored by habit.
Then route. Group your monitors by who actually looks after them and give each group its own destination: this mailbox, that chat, this person. Where a contact is outside the company — the installer, the supplier, the person who opens the shop — give them their own alerts, in their own language. An alert nobody feels responsible for is an alert nobody acts on.
Then choose the ones that deserve to interrupt. Be honest and be brief: most services do not need to wake anybody. The two or three that do should reach a phone, because a ringing phone is the only channel that does not require someone to be looking at something. Set it up with restraint — only if the service is still down after a few minutes, never in the hours you have declared quiet, redial if the line was busy — and it stays credible.
Finally, keep a record. A phone call can be verified: whether it was answered, and how much of the message was actually heard before someone hung up. That is the difference between “we alerted you” and “you were told”.
And the reminders
The failure mode nobody plans for is the alert that arrives, is acknowledged at midnight by a tired person, and is then forgotten. A service that is still down deserves to say so again — after an hour, after six, then once a day — until somebody either fixes it or decides, deliberately, that it can wait.
What to do this week
- Send yourself a test alert on every channel you have configured, and find out where it lands. Half the time this is the whole exercise.
- Replace every “one mailbox” destination with a group, and write down who is actually responsible.
- Count the alerts of the last seven days. Anything you did not act on is either a threshold to change or a monitor to delete.
- Pick the services that justify a phone call. If the list is longer than three, it is a wish list, not a plan.
- Check that the people outside your company who need to know get their alerts in their own language.
The measure of an alerting system is not how much it sends. It is whether, at three in the morning, you still believe it.
What we cannot tell you
- We cannot guarantee that an e-mail reaches an inbox: once the message leaves us, whether it is delivered, filed or filtered is decided by the receiving mail system, and nobody can promise otherwise.
- We do not send SMS, and we do not plug into a separate on-call or paging platform: the channels are e-mail, chat, signed webhooks and a phone call, and that is the whole list.
- We cannot reach a phone that is off or out of coverage: we can tell you that we tried, when, and what the line answered, which is at least a fact you can act on.
- We cannot decide for you which failure is worth waking someone up for. We can make sure that when you have decided, the message actually arrives.
Questions people actually ask
Why do outage alerts end up in spam?
Because they look like the messages spam filters are built to catch: machine-generated, repetitive, sent at odd hours, often with a link. The filter is not wrong to be suspicious; it is simply making a judgement you never see. This is why an alert that matters should never depend on e-mail alone.
What is the difference between a channel and a route?
The channel is how the message travels — mail, chat, webhook, phone. The route is who receives it. Most alerting failures are routing failures: everything went to one address, and that address belonged to somebody who had left, or was on holiday, or had stopped reading. Groups of monitors should have their own destinations, and each contact their own language.
We get too many alerts and nobody reads them any more. Where do we start?
By removing, not adding. Require a failure to repeat before it is announced. Group what fails together — one server going down should be one message, not nine. Silence what you already know about, and silence everything during a deployment. Then check what is left: if it is still too much, the thresholds are wrong, not the people.
Is a phone call not excessive?
For most things, yes. For two or three services it is the only channel that works, because it does not depend on anyone looking at a screen. It is worth setting up properly: a call only if the service is still down after a few minutes, never during the hours you have declared quiet, and a redial if the line was busy or unanswered.
How do we know an alert was actually delivered?
For a phone call, precisely: every attempt is recorded with its outcome and how many seconds of the message were actually played, so you can tell a call that was answered from a call that rang out. For e-mail, that certainty does not exist, and anyone who tells you otherwise is selling something.