On reddit tons of people are like “hurr durr it was caused by a configuration error that propogated down causing failed health checks and remaining non-updated servers got overloaded durr” and that’s the “techy” explanation.
Maybe that’s enough for MS IT guys and no shade but, with e.g. crowdstrike we knew pretty quickly exactly what the cause was - an array out of bounds read because of an extra malware definitions file (or something to that effect i do not recall exactly).
It is suspect to me that this outage went on for this long, despite how much it affected, and how the recovery went along so haphazardly it seems. They are probably not done with RCA yet but i also wonder if it was a more negligent fuckup they want to keep under wraps to avoid bad rep and e.g. UK 999 system choosing a different tech corp to work with next time as some european govt agencies have already done.
What do you think?
I don’t really care what the reason is. I think looking at Microsoft and this case, isn’t going to solve anything. You can’t, since it is closed source software under the control of one provider - control here is limited to a use it or do not use it decision.
The problematic decision is that so many people rely on one product hosted by one service provider, where it should be a multitude of interacting services and products.
Application diversity plus self hosting would be key to more resilience.
Let’s take one step backwards to see the bigger picture: In this case it affected an application suite. The same applies to the other layers of the stack, too. I e. an outage of one of the very few CDNs (Cloudflare, AWS, mscdn) would affect many.
One other aspect is souverenity: the issue of jurisdiction. If one relies on services in another country one has to accept their political decisions. If you rely on their services, you could be forced to play by rules which are not in your own interest.I’m talking about DNS and payment providers which already have been used to shut down unwanted services.