News with an accent.

Campus de Microsoft en Redmond, Washington, en una imagen de archivo.Wikimedia Commons · Imagen de archivo
Internet

Microsoft blames 365 outage on a bug lurking since July

A Microsoft executive says a flaw in its network gateways, a Teams traffic shift and 14% of capacity already down collided on Nov. 12. A final report is due this week.

Three things went wrong at once in the Microsoft 365 outage on Thursday, Nov. 12, according to Charles Lamanna, a Microsoft executive vice president: a flaw hidden since July, extra Teams traffic pushed through the same front door, and 14% of that door's servers already out of service. It is Microsoft's own account, and no one else has verified it.

That front door is a network gateway that carries many Microsoft 365 services. Under certain loads its servers stopped responding, because internal processes kept getting in each other's way over the same resource (*lock contention*). The flaw came from a fix for those machines that began rolling out in July and ran for more than three months without trouble.

How it came together

About 15 hours earlier, Teams had moved its traffic to the main gateway, which also serves Exchange Online. Separately, roughly 14% of that gateway's capacity was down because of an unrelated problem the day before. At peak U.S. traffic, the gateway hit the July bug.

With servers down, traffic shifted to the ones still running, and they failed too. Within two hours the flaw had reached the whole fleet; at the low point, more than 80% of servers had stopped responding.

Our normal load testing did not detect the latent issue.

Lamanna says the usual load tests missed it because Microsoft's labs differ from live traffic and configurations, and the flaw gets worse on machines with more cores and newer hardware. He adds that the company is "deeply sorry" for the impact on customers.

Why Outlook took longer

Teams, OneDrive, SharePoint and the Office apps could be rerouted and came back sooner. Exchange Online can't be, which is why Outlook mail was the last to recover. For companies and schools in Mexico that rely on both, it matters for contingency planning: Microsoft 365 doesn't fail, or recover, all at once.

Recovery was slow, Lamanna says, because each server that returned was hit with all the backed-up traffic and overloaded again, automatic recovery wasn't built for a failure this broad, and a months-old change couldn't be rolled back quickly by the usual means. Microsoft shipped an update that restored the earlier code and restarted servers in batches of 50% or more of a region. It also concedes that the barriers between infrastructure partitions, meant to contain a cascading failure, didn't hold because of a separate fault.

Neither the flaw nor its fix affects other Windows installations, he says. The preliminary report was announced Sunday and is visible only to administrators, in their service-notice center (MO1550821 and EX1550543). The final report, Lamanna promises, arrives later this week. There is no total duration per service yet and no notice specific to Mexico, and Lamanna doesn't mention Copilot.

Keep reading

El Mediático counts visits without cookies, with its own measurement and with Google Analytics, which receives basic data about your visit. Only if you accept does Google Analytics also store cookies and, where advertising is active, show personalized ads. You can accept all, reject all, or choose what to allow. The cookie policy is available in Spanish only. More about cookies (Spanish)