Avatar

The capacity upgrade that removed the capacity

Microsoft's email died last Thursday and stayed dead for about thirty six hours. Outlook, SharePoint, Copilot, most of Microsoft 365, with reports from seventy four countries inside the first hour. Estimates put the lost work somewhere around seven billion dollars.

Why Microsoft couldn’t fix its own servers for 36 hours...FireshipWatch Fireship from 1:36

The full post mortem is not out yet. What Microsoft has said already is enough, and it is this. The incident was introduced by an issue during an earlier network capacity upgrade.

The work they did to add capacity is the work that took all of it away.

That is the joke, and it is worth more than a laugh, because of the mechanism underneath it.

Every one of those servers sits behind a load balancer running a health check. The balancer pings each server, and any server that fails the ping gets pulled out of rotation. That is a safety feature. It is there so that your request never lands on a sick machine.

Now watch what it does when a lot of servers get sick at the same time. The failing ones are pulled out. Their traffic lands on the survivors. The survivors, now carrying more than they were built to carry, start failing their own health checks. So they get pulled out too. And their traffic lands on whoever is still standing.

The safety mechanism is the thing that spreads the failure.

Nothing was misconfigured. The health check did exactly what it was designed to do, at the setting somebody chose on purpose, and the result was a cascade. The design rests on one assumption that nobody writes down: that failures are rare and unrelated to each other. Pull one sick server out and you have protected your users. Pull two hundred out and you have pointed all of their traffic at the few that are left.

That is not a bug. It is a trade-off, made at design time, by somebody who was probably never asked to say it out loud.

This is why I keep going on about please advise me on the trade-offs. Nobody needed a cleverer engineer here. They needed one person, at the moment of that capacity upgrade, to ask what this costs and what it quietly assumes.

Efficiency is sharing. Reliability is not sharing. Every system you have ever used sits somewhere on that line, and almost nobody can tell you where it sits or who put it there.

My favourite detail is the one that says the most. The outage also took down the Microsoft 365 admin centre, so the administrators could not use the admin tools to find out why the admin tools were down. And when NBC News emailed Microsoft's PR firm for a comment, the email bounced.