I read the GitHub and tried it out. It basically works by recursively testing the output. It seems very expensive, and not specific in implementation. In the example given, the LLM can still make any changes it wants, as long as it doesn’t fail the test. They’re also designed to be written by the LLM itself, not humans. So it’s not really a replacement for natural language.

It reminds me of integration testing tools such as Cypress, except it’s an LLM running the test on itself. The “laws” are only in effect when the LLM understands the laws, and understands how it’s change will affect compliance with them.

In my opinion, in practice, LLMs will end up writing a 19,000 lines long impenetrable and unreadable “law handbook”, get confused about the laws, burn through a ridiculous amount of time and tokens recursively testing itself at every step of the journey, and people will end up removing it. It’s cool in theory though, but it has so many rough edges that seem inherit to the technology.