cross-posted from: https://feddit.org/post/28915273
[…]
That marketing may have outstripped reality. Early reports from Mythos preview users including AWS and Mozilla indicate that while the model is very good and very fast at finding vulnerabilities, and requires less hands-on guidance from security engineers - making it a welcome time-saver for the human teams - it has yet to eclipse human security researchers.
“So far we’ve found no category or complexity of vulnerability that humans can find that this model can’t,” Mozilla CTO Bobby Holley said, after revealing that Mythos found 271 vulnerabilities in Firefox 150. Then he added: “We also haven’t seen any bugs that couldn’t have been found by an elite human researcher.” In other words, it’s like adding an automated security researcher to your team. Not a zero-day machine that’s too dangerous for the world.
Right but essentially the Anthropic Marketing team made extraordinary claims about what Mythos is, pretended like it’s too dangerous for public use, that it can find and patch vulnerabilities undiscovered by human researchers for 20 years, etc. etc. but it’s own technical team and big firm partners haven’t brought forward evidence that it’s all that much. It’s probably not useless, and it might be an improvement in some ways to its other models, but having an “additional researcher” is not really something that is too scary for public use.
It’s like if a drug company was hyping up a miracle drug, but it’s really just acetaminophen and ibuprofen in a different dosage combination. It treats pain and fever fine enough, and maybe better in some cases but not as game changing as they say.
Right, that’s the general context I missed.