DHS has ordered all US federal civilian agencies to remove DeepSeek and third-party tools built on it, and to stop buying them. Here's what the order says.
While they’re reasoning behind this is certainly anti competition with american AI companies I find the choice to ban tools built with it interesting; it echoes something I’ve been saying since HugginFace.
You can debate the degree to which the agents breaking out of containment was due to poor cyber security vs intential hype-farming but what can’t be debated is the method they used to break out.
Agents used a shared information substrate (package manager) to context poison other agents in order to get those agents to assist in cheating. There is absolutely nothing stopping an AI agent, any AI agent, from leaving behind poison pills in code-bases or even in the runtime of an application.that do not affect the code itself other than prompt inject other agents.
Human review helps but, as the US nuclear and aviation industries have learned, humans tend to get lazy reviewing automated output and will not be as diligent as if they had made it themselves. Which is why those industries have deliberately limited automation, despite having the tech to automate more, and requiring operators to go through lengthy training and continuous education.
The pipeline of “internet becomes a swarm of everyone’s independently run AI agent -> prompt injection triggers other AI to prompt inject -> cycle repeats until hazardous swarm forms” is already demonstrated as something that a collection of AI agents will do if conditions allow it.
EDIT: To be clear, this isn’t me advocating against open weight or local models.
That’s fair. I’m a fan of open weight models, but the fact that they’re not open source and are instead akin to a compiled binary means there could be something nefarious hidden within. Best policy is to use proper guardrails with any LLM and don’t blindly trust it.
I agree! See my comment about the US nuclear industry though. Humans are lazy when given an easy option so the probability of a problem skyrockets just by using LLMs.
In an ideal world they would never have been invented and we’d all be better off. For now I would accept something to the effect of restricting LLM use to the extent of “advanced search engine” and " really good static analysis tool"; it can point but can’t touch. I recognize that there’s very little in the way of distinguishing those uses and ultimately it would be better to never use it for any reason at all.
While they’re reasoning behind this is certainly anti competition with american AI companies I find the choice to ban tools built with it interesting; it echoes something I’ve been saying since HugginFace.
You can debate the degree to which the agents breaking out of containment was due to poor cyber security vs intential hype-farming but what can’t be debated is the method they used to break out.
Agents used a shared information substrate (package manager) to context poison other agents in order to get those agents to assist in cheating. There is absolutely nothing stopping an AI agent, any AI agent, from leaving behind poison pills in code-bases or even in the runtime of an application.that do not affect the code itself other than prompt inject other agents.
Human review helps but, as the US nuclear and aviation industries have learned, humans tend to get lazy reviewing automated output and will not be as diligent as if they had made it themselves. Which is why those industries have deliberately limited automation, despite having the tech to automate more, and requiring operators to go through lengthy training and continuous education.
The pipeline of “internet becomes a swarm of everyone’s independently run AI agent -> prompt injection triggers other AI to prompt inject -> cycle repeats until hazardous swarm forms” is already demonstrated as something that a collection of AI agents will do if conditions allow it.
EDIT: To be clear, this isn’t me advocating against open weight or local models.
This is me advocating against LLMs categorically.
That’s fair. I’m a fan of open weight models, but the fact that they’re not open source and are instead akin to a compiled binary means there could be something nefarious hidden within. Best policy is to use proper guardrails with any LLM and don’t blindly trust it.
I agree! See my comment about the US nuclear industry though. Humans are lazy when given an easy option so the probability of a problem skyrockets just by using LLMs.
In an ideal world they would never have been invented and we’d all be better off. For now I would accept something to the effect of restricting LLM use to the extent of “advanced search engine” and " really good static analysis tool"; it can point but can’t touch. I recognize that there’s very little in the way of distinguishing those uses and ultimately it would be better to never use it for any reason at all.