cross-posted from: https://feddit.org/post/28915273
[…]
That marketing may have outstripped reality. Early reports from Mythos preview users including AWS and Mozilla indicate that while the model is very good and very fast at finding vulnerabilities, and requires less hands-on guidance from security engineers - making it a welcome time-saver for the human teams - it has yet to eclipse human security researchers.
“So far we’ve found no category or complexity of vulnerability that humans can find that this model can’t,” Mozilla CTO Bobby Holley said, after revealing that Mythos found 271 vulnerabilities in Firefox 150. Then he added: “We also haven’t seen any bugs that couldn’t have been found by an elite human researcher.” In other words, it’s like adding an automated security researcher to your team. Not a zero-day machine that’s too dangerous for the world.
I don’t understand what they’re saying. Mythos can’t find vulnerabilities that humans can’t, but it also supposedly found 271 vulns so…the humans were just ignoring them?
Speaking generally…
One is that it was pitched as a superhuman AI that could think in ways humans couldn’t possibly imagine, escaping any security measure we might think to bond it with. That was the calibrated expectation.
Instead it’s fine at security “findings”, that a human could have noticed if they actually looked. For a lot of AI this is the key value prop, looking when a human can’t be bothered to look and seeing less than a human would, but the human never would have looked. For example a human can more reliably distinguish a needle from a straw of hay, but the relentless attention of an AI system would be a more practical approach for finding needles in haystacks. It will miss some needles and find some hay, so a human effort would have been better, but the AI is better than nothing, especially with a human to discard the accidental hay.
Another thing is the nuance of the “vulnerabilities” may be very underwhelming. Anyone who has been in the security world knows that the vast majority of reported “vulnerabilities” are nothing burgers in practice. Curl had a “security” issue where a malicious command line could make it lock up instead of timeout if the peer stops responding. I’ve seen script engines that were explicitly designed to allow command execution get cves because a 4gb malicious script could invoke commands without including the exec directive, and also this engine is only ever used by people with unfettered shell access anyway. Had another “critical” vulnerability, but required an authorized user to remove and even rewrite the code that listens to the network to allow unsanitized data in that’s normally caught by the bits they disabled. Curl had another where they could make it output vulnerable c code, then the attacker would “just” have to find a way to compile the output of the command and they’d have a vulnerable c executable… How in the world are they able to get curl to generate c code and compile it but not otherwise be able to write whatever c code they want… Well no one can imagine it, but hey, why not a CVE…
“Finding” bugs by throwing shit at the walls and assuming people will sort it out provides negative value. You technically are finding bugs, but you could do the same just assuming every line of your code contains five bugs. The question is in “and then what”, and the answer is “someone needs to sort them out and deal with it”, and if you have people who can fix the bug, they’re perfectly capable of finding it themselves. The bugs still exist because there is not enough people to fix that. And slop gen doesn’t help with that either.
It’s only a negative value if the AI+review process takes longer than a human just finding the bugs.
One of the biggest hurdles in infosec right now is just the sheer volume of data. Sifting through hoards of data and finding anomalies is something AI actually excels at.
Read here:
https://www.flyingpenguin.com/the-boy-that-cried-mythos-verification-is-collapsing-trust-in-anthropic/
Over my head. Also light mode hurts my head.
TLDR: Mythos is strictly worse at finding vulnerabilities than Opus 4.6, and about on par with a specific cheapo open source 2B parameters (=> tiny and super cheap) model.
It’s all marketing and no substance.
The document from Anthropic purporting to be a security research work largely leaves things vague (marketing material vague) and declines to use any recognized standard for even possibly hinting about whether to think anything at all. They describe a pretty normal security reality (‘thousands of vulnerabilities’ but anyone who lives in CVE world knows that was the case before, so nothing to really distinguish from status quo).
Then in their nuanced case study, they had to rip out a specific piece of firefox to torture and remove all the security protections that would have already secured these ‘problems’. Then it underperformed existing fuzzer and nearly all of it’s successes were based on previously known vulnerabilities that had already been fixed, but they were running the unpatched version to prove it’s ability.
Ultimately, the one concrete thing they did was prove that if you fed Mythos two already known vulnerabilities, it was able to figure out how to explicitly exploit those vulnerabilities better than other models. It was worse at finding vulnerabilities, but it could make a demonstrator. Which a human could have done, and that’s not the tedious part of security research, the finding is the tedious part. Again, in the real world, these never would have worked, because they had to disable a bunch of protections that already neutered these “issues” before they ever were known.