Microsoft Attributes Global Microsoft 365 Outage to Months-Old Software Bug in Networking Gateways

Microsoft has confirmed that a software bug introduced in July triggered last week's worldwide Microsoft 365 outage, causing more than 80 percent of its networking gateway servers to become unresponsive. The nearly 24-hour disruption on November 12 affected Outlook, Exchange Online, Teams, and Copilot, blocking email access and delaying message delivery for users globally. The incident stemmed from three factors: the undetected bug, increased traffic following a Microsoft Teams failover to Exchange Online infrastructure, and reduced server capacity due to an unrelated issue affecting 14 percent of gateways. Microsoft explained that standard load testing failed to catch the defect because it used less powerful hardware with fewer cores than production systems, where the contention issue was amplified. Recovery proved challenging as restarting servers led to rapid re-overload from queued traffic, and regional safeguards did not contain the cascading failure. The company has since deployed a fix, increased capacity, improved monitoring, and added new safeguards, with a final post-incident report expected later this week.

BleepingComputerTranslated from English

Read at source

Microsoft has confirmed that a software bug introduced in July was the root cause of last week's global Microsoft 365 outage, which triggered a cascading failure leaving more than 80% of its networking gateway servers unresponsive.

The nearly 24-hour outage on November 12 disrupted Microsoft 365 services worldwide, including Outlook, Exchange Online, Teams, and Copilot, leaving users unable to access their email and causing significant delays in message delivery.

In a preliminary post-incident report shared in the Microsoft 365 Admin Center, the company said the outage was caused by three problems: a previously undetected software bug, increased traffic after a Microsoft Teams failover, and reduced server capacity.

The incident began at 11:20 AM ET on Thursday, November 12, and continued until 11:10 AM ET the following day, when Microsoft confirmed that services were fully restored.

According to Microsoft, the software bug was introduced in July through an update developed for Microsoft 365 networking gateway servers. While the update had been deployed for more than three months without issues, it contained a bug that could overwhelm servers under certain traffic conditions, causing them to stop responding.

Microsoft says its tests of the update did not detect the bug as it used less powerful testing hardware than the production servers. "Our normal load testing did not detect the latent issue because the traffic mix and SKUs used in our labs," Microsoft explained in its preliminary report. "The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on weaker hardware with fewer cores."

The bug was eventually triggered after Microsoft Teams automatically switched to the networking infrastructure used by Exchange Online approximately 15 hours before the outage, increasing traffic on those servers. Microsoft Teams also experienced a separate outage earlier that day, but it is unclear whether the two incidents are related.

To make matters worse, Microsoft says nearly 14% of the gateway's server capacity was already offline due to an unrelated issue. While the infrastructure had sufficient capacity to handle the reduction, the combination of increased traffic and reduced capacity triggered the software bug.

As servers stopped responding, Microsoft's systems redirected users to other available servers, which also became overwhelmed by the additional traffic and failed as well. "In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive," explained Microsoft.

Microsoft says safeguards designed to prevent problems from spreading between regions also failed to contain the cascading outages, allowing the disruption to spread worldwide. While other Microsoft 365 services, like Teams, OneDrive, and SharePoint, could switch to alternative infrastructure and recover earlier, Exchange Online could not make the same switch, leaving users unable to access their mailboxes.

Microsoft says recovery was difficult because restarting affected servers caused them to be quickly overwhelmed again by backed-up traffic. After identifying the bug, Microsoft engineers developed a fix and deployed it across affected servers, allowing services to stabilize throughout Friday.

Microsoft confirmed that active customer impact had ended by 11:10 AM ET on November 13. The company says it has since increased capacity, improved monitoring, and added safeguards to prevent traffic redirection from triggering similar cascading failures.

In a separate statement on X, Charles Lamanna, Microsoft's executive vice president of Copilot, Agents, and Platform, apologized for the disruption. "We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations," Lamanna said.

Microsoft says its investigation is continuing, with a final post-incident report expected later this week.

Meridian digestThe week's most consequential security stories, distilled into one briefing.See the digest

More in Vulns & Exploits

Microsoft Entra Connect Health Flaw Exploited to Access Tenant Metadata and Admin Emails
GnuPG Warns of Critical Double-Free Flaw in GPGME Exposing App Data to Malicious OpenPGP Keys
CISA Adds Four Critical Flaws in Apache Roller, Cisco Expressway, SAP NetWeaver, and VMware vCenter to Known Exploited Vulnerabilities Catalog