Microsoft wasn’t a cyber attack. They just didn’t properly test a bug fix before pushing it to production, and an edge case caused a cascading failure.
For those who haven’t read the initial findings, they do performance testing using their low-spec servers under the assumption that if the bad ones do fine, the rest will too. Months ago, they rolled out a patch that had a problem on systems with a large number of cores under heavy load, which they didn’t catch since the weak servers have less cores.
Their systems are designed to smoothly shift services if needed, so when two other issues took chunks of capacity offline, services got concentrated enough to trigger the bug, choking servers and causing them to fail. Repeat, repeat, repeat, and all their capacity was consumed. Then, automatic rebooting would bring individual chunks back online only to choke and die again moments later.
Since the change was months ago, it took them a while to figure out that it wasn’t a recent change causing the problem.
Microsoft wasn’t a cyber attack. They just didn’t properly test a bug fix before pushing it to production, and an edge case caused a cascading failure.
For those who haven’t read the initial findings, they do performance testing using their low-spec servers under the assumption that if the bad ones do fine, the rest will too. Months ago, they rolled out a patch that had a problem on systems with a large number of cores under heavy load, which they didn’t catch since the weak servers have less cores.
Their systems are designed to smoothly shift services if needed, so when two other issues took chunks of capacity offline, services got concentrated enough to trigger the bug, choking servers and causing them to fail. Repeat, repeat, repeat, and all their capacity was consumed. Then, automatic rebooting would bring individual chunks back online only to choke and die again moments later.
Since the change was months ago, it took them a while to figure out that it wasn’t a recent change causing the problem.