cross-posted from: https://feddit.org/post/28915273
[…]
That marketing may have outstripped reality. Early reports from Mythos preview users including AWS and Mozilla indicate that while the model is very good and very fast at finding vulnerabilities, and requires less hands-on guidance from security engineers - making it a welcome time-saver for the human teams - it has yet to eclipse human security researchers.
“So far we’ve found no category or complexity of vulnerability that humans can find that this model can’t,” Mozilla CTO Bobby Holley said, after revealing that Mythos found 271 vulnerabilities in Firefox 150. Then he added: “We also haven’t seen any bugs that couldn’t have been found by an elite human researcher.” In other words, it’s like adding an automated security researcher to your team. Not a zero-day machine that’s too dangerous for the world.
It might be a config thing, but pretty often these scans will find issues which are only relevant on e.g. windows, when building a Linux container. Or the issue is in some XML parsing library in the base OS but the service never receives XML and isn’t public facing anyway. Context matters.
One that I have to copy-paste over and over are vulnerabilities in the CUPS printer driver chain that don’t apply because we don’t print arbitrary things, we only print things that we create. Yeah, there’s a vulnerability here in image-magick if you throw it such and such maliciously crafted… well, we only allow it to process our internally generated reports and there’s no pathway for maliciously crafted input to reach it, so…