A few years ago I designed a way to detect bit-flips in Firefox crash reports and last year we deployed an actual memory tester that runs on user machines after the browser crashes. Today I was looking at the data that comes out of these tests and now I'm 100% positive that the heuristic is sound and a lot of the crashes we see are from users with bad memory or similarly flaky hardware. Here's a few numbers to give you an idea of how large the problem is. 🧵 1/5
Didn’t it just highlight how stable the software is?
I assume bitflipping crashes most softwares. If your software is so stable that hardware errors that effect everyone equally(which may be my erroneous assumption I’ll admit) then it is staying that if Firefox is crashing on you, it might be time to run some diagnosis on your hardware.
Fair question. I find it unnerving, because there’s very little a software developer can meaningfully do if they cannot rely on the integrity of the hardware upon which their software is running, at least not without significant costs, and ultimately if the problem is bad enough even those would fail. This finding seems to indicate that a lot of hardware is much, much less reliable than I would have thought. I’ve written software for almost thirty years and across numerous platforms at this point, and the thought that I cannot assume a value stored in RAM to reliably retain it’s value fills me with the kind of dread I wouldn’t be able to explain to someone uninitiated without a major digression. Almost everything you do on any computing device - whether a server or a smart phone relies on the assumption of that kind of trust. And this seems to show that assumption is not merely flawed, but badly flawed.
Suppose you were a car mechanic confronted with a survey that 10 percent of cars were leaking breaking fluid - or fuel. That might illustrate how this makes me feel.
Hmm thanks, also please massively digress if you would like to.
I interpreted it like 10% is a lot if it’s 10% of a million. That 100,000. So if there’s a million things that crash Firefox that’s a high number.
If Firefox only crashes 10 times a year because it runs that well, 10% or that 1 time it crashes from a bitflip is impressive that the rare bitflip takes up such a high percentage of total crashes because Firefox just doesn’t crash very often.
If your dread is found to be justified that won’t be too surprising, to me, if hardware is getting made less reliable these days thing. Enshitification being the norm, and tech being in everything nowadays
We obviously need more context from Mozilla, but this could be a canary in the mine type situation.
But it would be kind of neat if Firefox became something of a reliable test for bitflipping unintentionally
Well, that’s unnerving.
How so?
Didn’t it just highlight how stable the software is?
I assume bitflipping crashes most softwares. If your software is so stable that hardware errors that effect everyone equally(which may be my erroneous assumption I’ll admit) then it is staying that if Firefox is crashing on you, it might be time to run some diagnosis on your hardware.
A litmus test as a browser
Fair question. I find it unnerving, because there’s very little a software developer can meaningfully do if they cannot rely on the integrity of the hardware upon which their software is running, at least not without significant costs, and ultimately if the problem is bad enough even those would fail. This finding seems to indicate that a lot of hardware is much, much less reliable than I would have thought. I’ve written software for almost thirty years and across numerous platforms at this point, and the thought that I cannot assume a value stored in RAM to reliably retain it’s value fills me with the kind of dread I wouldn’t be able to explain to someone uninitiated without a major digression. Almost everything you do on any computing device - whether a server or a smart phone relies on the assumption of that kind of trust. And this seems to show that assumption is not merely flawed, but badly flawed.
Suppose you were a car mechanic confronted with a survey that 10 percent of cars were leaking breaking fluid - or fuel. That might illustrate how this makes me feel.
Hmm thanks, also please massively digress if you would like to.
I interpreted it like 10% is a lot if it’s 10% of a million. That 100,000. So if there’s a million things that crash Firefox that’s a high number.
If Firefox only crashes 10 times a year because it runs that well, 10% or that 1 time it crashes from a bitflip is impressive that the rare bitflip takes up such a high percentage of total crashes because Firefox just doesn’t crash very often.
If your dread is found to be justified that won’t be too surprising, to me, if hardware is getting made less reliable these days thing. Enshitification being the norm, and tech being in everything nowadays
We obviously need more context from Mozilla, but this could be a canary in the mine type situation.
But it would be kind of neat if Firefox became something of a reliable test for bitflipping unintentionally