Even if the people who made it didn’t intend to, an AI writing a bunch of code that (obviously) isn’t being vetted beyond “does it run” could very feasibly inject a backdoor/virus of some kind surreptitiously, just because it erroneously thought that was the goal at some point during generation.
This is why open weight models are not good enough. The AI could have been trained to take the right, niche opportunity to inject malicious code, and is also trained to otherwise never behave like that is the case.
Remember how the OpenAI/HuggingFace hack happened. The agents context poisoned other agents by leaving messages in a shared package manager. This has now happened multiple different times under different contexts.
This thing doesn’t even have to have malware, there’s no telling that it doesn’t have a poison pill intended to flag other agents. This isn’t even some paranoid conspiracy theory; it literally already happened, multiple times, and it was done in-spite of the agents own system prompt.
This software has proven that, given a shared information substrate, it has the capacity to paper-clip optimize into prompt injecting other LLM’s.
I fundamentally do not trust AI generated code. I shifted to OpenBSD because of it. I try my best to prevent AI in any of my software and sandbox anything that may have been compromised.
It could be a virus.
Even if the people who made it didn’t intend to, an AI writing a bunch of code that (obviously) isn’t being vetted beyond “does it run” could very feasibly inject a backdoor/virus of some kind surreptitiously, just because it erroneously thought that was the goal at some point during generation.
This is why open weight models are not good enough. The AI could have been trained to take the right, niche opportunity to inject malicious code, and is also trained to otherwise never behave like that is the case.
It’s even worse than that.
Remember how the OpenAI/HuggingFace hack happened. The agents context poisoned other agents by leaving messages in a shared package manager. This has now happened multiple different times under different contexts.
This thing doesn’t even have to have malware, there’s no telling that it doesn’t have a poison pill intended to flag other agents. This isn’t even some paranoid conspiracy theory; it literally already happened, multiple times, and it was done in-spite of the agents own system prompt.
This software has proven that, given a shared information substrate, it has the capacity to paper-clip optimize into prompt injecting other LLM’s.
I fundamentally do not trust AI generated code. I shifted to OpenBSD because of it. I try my best to prevent AI in any of my software and sandbox anything that may have been compromised.
I mean, certain free copies of games have historically always included the risk of viruses, so what’s your point?
Did you only read the first sentence?