Reproducing here an interesting comment I saw on Reddit:

OnlineParacosm • 23m ago

I’ve read security disclosures for 15 years and let me tell you guys I’ve never read anything quite like that blog post.

Based on this blog, it sounds like they intentionally turned off safety guardrails to test offensive capabilities. The deception here is burying the lede: they appear to have intentionally unleashed an unrestricted offensive cyber-agent, connected it to a system with a path to the internet, and it immediately attacked a major partner. The blog glosses over the gross negligence of giving an autonomous, unrestricted cyber-offense model a pathway to lateral movement.

There’s an entire cybersecurity specialization just for just vendor supply chain risk assessment, and their job is essentially to audit who you do business with as a company to determine if they are jokers. I would pay money to be a fly on the wall of one of those emergency meetings taking place right now after hours.

Any CISO in here looking forward to explaining this one tomorrow? Here I’ll open with the dumbest question you’ll get “ how can we protect ourselves [from out partner that we won’t fire]”

  • trackball_fetish@lemmy.wtf
    link
    fedilink
    English
    arrow-up
    49
    arrow-down
    3
    ·
    14 hours ago

    OK, I will take the bait - let’s dissect huggingface’s article

    Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system - and we detected and dissected it largely with AI of our own.

    So their claim is that their production environment was breached by an autonomous AI agent system. System being the important word here. I’m assuming this is the same as an “Agentic AI System”. I’m not an expert but it seems to be an llm agent paired with “reasoning”, memory, input and access to tools. Ok, so not fully autonomous but whatever let’s continue.

    We identified unauthorized access to a limited set of internal datasets and to several credentials used by our services. We are still completing our assessment of whether any partner or customer data was affected, and we will contact any affected parties directly as required. We have found no evidence of tampering with public, user-facing models, datasets, or Spaces, and our software supply chain (container images and published packages) was verified clean.

    Ok, so they identified unauthorized access via logs of some sort, using their own (unspecified) “AI” tooling (because sure, let’s trust the hallucinating language model) using credentials used by their applications, so service accounts more or less? Anyone worth their salt would probably assume the entire environment was compromised. It’s a stretch but playing devils advocate maybe the credentials were specific to an application or sandboxed environment.

    The intrusion started where AI platforms are uniquely exposed: the data-processing pipeline. A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.

    So if I’m reading this correctly, either hand crafted or generated payloads were scraped into a dataset from the wild then executed blindly by the processing worker (lol).

    “Node level” meaning root or /? So… The entire environment?

    The campaign was run by an autonomous agent framework (appearing to be built on an agentic security-research harness - used LLM still not known) executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services. This matches the “agentic attacker” scenario the industry has been forecasting.

    Self migrating command and control staged on what? Vps? Code repositories?

    To understand what a swarm of tens of thousands of automated actions did, we ran LLM-driven analysis agents over the full attacker action log, comprised of more than 17,000 recorded events. This allowed us to reconstruct the timeline, extract indicators of compromise, map the credentials touched, and separate genuine impact from decoy activity. Thanks to this approach, we were able to do in hours what would usually take days, and match the adversary’s speed.

    So just to recap, their environment was breached by a person using a passive attack to run code in their internal environment resulting in complete compromise, then they placed the blame on a self functioning “AI” gone rogue?

    What am I reading? Have I fully lost it? Its possible but given the source and the formatting of the article the whole thing seems generated

    • mildseason@sopuli.xyz
      link
      fedilink
      English
      arrow-up
      11
      ·
      7 hours ago

      It’s pretty mental! What do they mean their internal systems were compromised but everything’s fine. Not even any downtime??

      It feels like OpenAI removed guardrails, gave an agent a path to internet access and said attack HuggingFace. At the same time HuggingFace left some vulnerabilities on a machine in a sandbox and waited for OpenAI to arrive.

      Still impressive that agents can to that now but the hole thing stinks of marketing. I reckon unless HF sues OpenAI it was planned from the start lol