I built KillSwitch, an esoteric language designed to confound LLMs and coding agents, and benchmarked the major models against it.
  • dan@upvote.au
    452·
    18 hours ago

    This is interesting because even though they designed the language to be impossible for LLMs to use, Claude still scored nearly 70% on their benchmark.

    • OldMrFish@feddit.dk
      233·
      11 hours ago

      Alternative headline: Claude scores 70% on benchmark in esoteric language it hasn’t been trained on. That’s basically an ad for Claude Code…

      • dan@upvote.au
        3·
        8 hours ago

        Not only was Claude not trained on it, but the language is actively trying to be hostile towards it.

    • Mirodir@discuss.tchncs.de
      222·
      13 hours ago

      And that’s without any of this being in its training set. I wonder if they realized how big of a self own that is.