On reddit tons of people are like “hurr durr it was caused by a configuration error that propogated down causing failed health checks and remaining non-updated servers got overloaded durr” and that’s the “techy” explanation.
Maybe that’s enough for MS IT guys and no shade but, with e.g. crowdstrike we knew pretty quickly exactly what the cause was - an array out of bounds read because of an extra malware definitions file (or something to that effect i do not recall exactly).
It is suspect to me that this outage went on for this long, despite how much it affected, and how the recovery went along so haphazardly it seems. They are probably not done with RCA yet but i also wonder if it was a more negligent fuckup they want to keep under wraps to avoid bad rep and e.g. UK 999 system choosing a different tech corp to work with next time as some european govt agencies have already done.
What do you think?
I don’t really care what the reason is. I think looking at Microsoft and this case, isn’t going to solve anything. You can’t, since it is closed source software under the control of one provider - control here is limited to a use it or do not use it decision.
The problematic decision is that so many people rely on one product hosted by one service provider, where it should be a multitude of interacting services and products.
Application diversity plus self hosting would be key to more resilience.
Let’s take one step backwards to see the bigger picture: In this case it affected an application suite. The same applies to the other layers of the stack, too. I e. an outage of one of the very few CDNs (Cloudflare, AWS, mscdn) would affect many.
One other aspect is souverenity: the issue of jurisdiction. If one relies on services in another country one has to accept their political decisions. If you rely on their services, you could be forced to play by rules which are not in your own interest.I’m talking about DNS and payment providers which already have been used to shut down unwanted services.
Greed, and management being management.
Microsoft offers voluntary buyouts to 60,000 employees as AI leads to efficiency push
A few thousand missing engineers.
Also slop
Yep. They will never admit it though. Their alleged golden goose can’t be blamed.
Best scenario, in my opinion, they will admit the technical fault with not many details, whatever it is, but not that it existed because AI was forcefully fed to the system in a big scale.
And their long term solution not to let that happen anymore? MORE AI.
Infrastructure of this size is just code. It’ll spin up instances, route traffic and balance nicely.
They probably added a trivial timeout to their health check. Say, if an instance doesn’t respond within 60 seconds it gets cycled out.
Probably worked fine on Monday morning when they rolled it out.
It probably took them until Saturday night to remember that it takes two minutes for an instance to launch, so every instance will be cycled out before it tags in.
Just a wild guess about the kind of miss that might’ve caused it. I haven’t read it heard anything.
The root cause would be relying on technology not old enough to drink to manage systems that are old enough to drink.
Makes me want to drink.
I’m going for a drink
May is suggest “Microslop” v “M$”? They banned microslop on one of their forums so they really hate it.
My first thought when reading about it and how much was affected was “what if some of these outages are actually tests?”
It should be noted, though, that I don’t know shit.
I’m of the (completely speculative) opinion that the US just experienced a massive cyber attack (maybe Iran, maybe literally anyone else as USA has a lot of enemies) and that we’re all being kept in the dark about it.
“Never attribute to malice that which is adequately explained by stupidity”
https://en.wikipedia.org/wiki/Hanlon’s_razor
I personally believe it needs to be slightly modified to:
“Never attribute to malice that which is adequately explained by fucking idiots seeking profits.”
I disagree with Hanlon’s razor simply because stupidity and malice are often sold as a bundle.
There were thousands of people involved in fixing this. There is no way MS could keep an attack under wraps if they wanted to.
Yep.
Funny how the FAA comms outage was the next day.
This will be considered Day 1 of the first all-out cyber war.
How would MS and the FAA silence the thousands of people involved in fixing these incidents?
NDAs, money, threats, I do not know, just throwing stuff at the wall right now
Their status updates said they saw that a health check was pulling servers offline that were not unhealthy. The mitigated the health check but could not bring systems online without getting DDOS’d by all of the connections.
Hackers
I’m not saying it was an attack…but if it were, they would never admit it.
MS wouldn’t have to admit anything. There would be hundreds, if not thousands, of engineers shouting the truth to the heavens.
