Intelligence and Imagination go hand in hand - as does the world model. If you give anyone intelligent a long list of coordinates, which only consists of 1-8 and a-h, they will also only choose out of these coordinates if you ask them to choose something legal, and they will also realize that a legal move has the prerequisite of choosing coordinates touching an enemy coordinate with one of my own chosen coordinates on the far end of the cluster - this does not involve imagining a “real” Othello Board.
Intelligence (and the world model) also implies that at some point the actual realization of the prerequisites for a legal move. These are pretty simple in Othello - actually, getting a computer to choose a correct move was solved decades ago, so we have actual proof that the playing entity does NOT need any kind of world model and therefore intelligence to ensure 100% of moves are correct.
If these things were intelligent (and as i insist on, world model and intelligence go hand in hand), they would be able to simply use the rulebook as training input and provide close to 100% legal moves. I’d like to see THAT paper, but papers with negative results seldom get published. (Sorry btw, i overlooked that they didn’t give it the rulebook - I was thrown off by the information that 2023 models all have the rules for common board games in their basic training data)
I cited your provided study because it is a very good example how research into LLMs is getting sensationalized. They did not even provide the prompting used in this study, so we have NO information about what they actually asked of the model - it’s simply non-available data, while being the actual seed used to get a result. We do not get information about the quality of the moves (which should probably be somewhere in the top 10% choices of possible moves when taking into account that this is championship data). We NEVER see the full actual output. Instead, we see a LLM having a 5% chance of doing random shit when fed with real world data. We are talking about a 8x8 board here, nothing that comes even close to the complexities of reality.
Edit: I just realized that we are also not shown the test cases themselves and the ratio of valid to invalid moves (and split in “not even touching another piece” and “touching, but invalid”), so we are missing the actual probability of making a valid move by random chance for those 2 variants. They only showed the 93% failure rate on an untrained model, which would be “i place a stone at random”. I’d not be surprised that every move made was at least touching another one - edge detection is one of the easier use cases for transformer models.
It’s the same with their mathematical proofs: we are never given the complete picture, only the result. We do not know what was used as training data, we do not know what “external” information was fed to it, we do not know the prompts used, and we do not know how much compute they have thrown at their room full of monkeys and typewriters to get a result.
Intelligence and Imagination go hand in hand - as does the world model. If you give anyone intelligent a long list of coordinates, which only consists of 1-8 and a-h, they will also only choose out of these coordinates if you ask them to choose something legal, and they will also realize that a legal move has the prerequisite of choosing coordinates touching an enemy coordinate with one of my own chosen coordinates on the far end of the cluster - this does not involve imagining a “real” Othello Board.
Intelligence (and the world model) also implies that at some point the actual realization of the prerequisites for a legal move. These are pretty simple in Othello - actually, getting a computer to choose a correct move was solved decades ago, so we have actual proof that the playing entity does NOT need any kind of world model and therefore intelligence to ensure 100% of moves are correct.
If these things were intelligent (and as i insist on, world model and intelligence go hand in hand), they would be able to simply use the rulebook as training input and provide close to 100% legal moves. I’d like to see THAT paper, but papers with negative results seldom get published. (Sorry btw, i overlooked that they didn’t give it the rulebook - I was thrown off by the information that 2023 models all have the rules for common board games in their basic training data)
I cited your provided study because it is a very good example how research into LLMs is getting sensationalized. They did not even provide the prompting used in this study, so we have NO information about what they actually asked of the model - it’s simply non-available data, while being the actual seed used to get a result. We do not get information about the quality of the moves (which should probably be somewhere in the top 10% choices of possible moves when taking into account that this is championship data). We NEVER see the full actual output. Instead, we see a LLM having a 5% chance of doing random shit when fed with real world data. We are talking about a 8x8 board here, nothing that comes even close to the complexities of reality.
Edit: I just realized that we are also not shown the test cases themselves and the ratio of valid to invalid moves (and split in “not even touching another piece” and “touching, but invalid”), so we are missing the actual probability of making a valid move by random chance for those 2 variants. They only showed the 93% failure rate on an untrained model, which would be “i place a stone at random”. I’d not be surprised that every move made was at least touching another one - edge detection is one of the easier use cases for transformer models.
It’s the same with their mathematical proofs: we are never given the complete picture, only the result. We do not know what was used as training data, we do not know what “external” information was fed to it, we do not know the prompts used, and we do not know how much compute they have thrown at their room full of monkeys and typewriters to get a result.