The benchmarking just shows the differences between the LLM models. Thus turning the benchmark into a LLM benchmark, rather than comparing it to other languages. I would like to know how it compares to similar languages that are similar in use case, popularity or esotericness. And how it works if we do a simple word replace to something LLMs have less problem.
The best performing model scores a 66.8. So does this prove LLMs can “understand” the code? I would argue that the answer to this is “no”.
But what is the peak value if they “understand” the code? We have no other languages to compare to. I also can’t find what exactly was benchmarked.
The benchmarking just shows the differences between the LLM models. Thus turning the benchmark into a LLM benchmark, rather than comparing it to other languages. I would like to know how it compares to similar languages that are similar in use case, popularity or esotericness. And how it works if we do a simple word replace to something LLMs have less problem.
But what is the peak value if they “understand” the code? We have no other languages to compare to. I also can’t find what exactly was benchmarked.