One problem I could see with the aggressive testing (which also happens in human-written codebases) is that it becomes hard to tell essential behaviour apart from details. If you nail everything down with a unit/integration test, nothing can be changed anymore.
I could imagine that this is part of the reason why LLMs love to duplicate code rather than adjusting existing code.
Well… That’s why you test the public API and not the internals. And if you need to change the API you should have a good enough reason that changing the tests is okay too.
Well, since it’s “just” LLM time writing the tests, test the internals too, but clearly mark which tests are the public API and which tests are intnernal, so it knows to never ignore / update a public API test without consultation, but it an internal test fails it can make the call: did this test start failing because I unintentionally broke something, or because the test was testing something that no longer matters - I have seen Claude go both ways, fixing its own mistake due to a test catching an unintended consequence of a change, and also just patching the test because the old test was… old.
One problem I could see with the aggressive testing (which also happens in human-written codebases) is that it becomes hard to tell essential behaviour apart from details. If you nail everything down with a unit/integration test, nothing can be changed anymore.
I could imagine that this is part of the reason why LLMs love to duplicate code rather than adjusting existing code.
Well… That’s why you test the public API and not the internals. And if you need to change the API you should have a good enough reason that changing the tests is okay too.
Well, since it’s “just” LLM time writing the tests, test the internals too, but clearly mark which tests are the public API and which tests are intnernal, so it knows to never ignore / update a public API test without consultation, but it an internal test fails it can make the call: did this test start failing because I unintentionally broke something, or because the test was testing something that no longer matters - I have seen Claude go both ways, fixing its own mistake due to a test catching an unintended consequence of a change, and also just patching the test because the old test was… old.