And is there any evidence that the required information for the other proofs wasn’t already available in the training data? As i said, these things remix stuff, they cannot come up with new stuff.
By “required information for the other proofs”, what do you mean exactly?
The proof, more or less in the same structure that the AI output, existed in the literature? This would’ve been published already and would be very easy to prove was stolen from the input corpus. We know this isn’t true.
The theorems and techniques used in the generated proof existed, but were not yet assembled into a final proof? Trained mathematicians have looked at the proof and seem to think it contains novel techniques, or at least novel applications of techniques from different areas of math. Even if that weren’t true, this would still be novel work; a lot of math breakthroughs are made this way, and it’s a nontrivial feat. Although if this was as far as it could go, we would run into limitations at some point.
The starting point of the proof existed, but the entire proof was built from the ground up by AI, using no existing methods or theorems? This would be very impressive, and as noted above the consensus seems to be that it did create some new techniques, but I’m not experienced or knowledgeable enough to verify that claim. Either way it definitely used at least some previous work for nontrivial reasons.
It looks like it’s somewhere between options 2 and 3 in reality; it used a lot of existing techniques and at least invented some new applications of existing techniques if not created new ones entirely (the line can be blurry). This is also what all mathematicians do! Remixing is coming up with new stuff. No thoughts, or generated output from LLMs, exist without prior context.
In the case i mentioned, it is very much number 2 IMO - the coincidence is just too much. In other cases, we are probably looking mostly at number 1: techniques that already exist, applied to other situations. These things are very good at iterating through rather uncommon combinations and have a large corpus of techniques in their training data, letting them walk across the “plane” of combinations quickly.
That’s not how humans work, and that’s probably the reason why there was a bunch of mathematical problems solved in a short time - the ingredients were there, but noone had yet tried that specific approach.
I have not yet seen any evidence of number 3, mainly because we do not know what these models are being trained on - transparency would give us the possibility to check, but we will never get it from OpenAI.
Here is the evidence: the entire field of mathematics was trying to solve these problems and couldn’t do it for decades.
If you want to still insist that it’s just “remixing the training data” then fine, but don’t act like that rules out the possibility of intelligence, because by any reasonable standard a system that can discover hundreds of groundbreaking new mathematical proofs is intelligent. Any human that accomplished that would go down in history as one of the most intelligent people to have ever lived.
It is a fucking bunch of math. It has no intelligence. It is a stochastic parrot. It is a blender for existing stuff. Doesn’t mean it isn’t a useful tool.
But if you actually believe that these mathematical construct have intelligence, do you also believe that every time you close a llama.cpp process you kill a sentient being? Or are you simply stopping a statistics engine? Because i am very much in camp “statistics engine”- statistics do not have intelligence.
Intelligence without a sense of self would be very strange, no? What we call intelligence is mostly thought of being able to think genuine thoughts that haven’t been there yet, and make sense (to exclude the inevitable hallucinations). This ability is deeply steeped in our ability to imagine hypothetical scenarios.
ChatGPT even fails to imagine a Othello Board and valid moves if you give it the rulebook, as @kromem@lemmy.world kindly provided.
I think you’re citing the wrong person. I cited the Othello-GPT line of research making the point transformers develop a world model. The other user (or you?) was mistaken about their interpretation of what a world model was representing and got caught up on the accuracy of a toy model exclusively trained on board notations like ‘a4’ and thought that the model making 95% accurate moves as a tiny parameter model trained on nothing else was somehow evidence that it didn’t form a world model from the data because it wasn’t 100% accurate, completely missing the point of the toy model scale allowing for interpretability methods of the time to identify geometric board reproduction and things like tracking game state on the board with flags for ‘my’ vs ‘opponent’ pieces, which was the data answering researcher questions about world modeling vs surface statistics in transformers.
Edit: just to further clarify re:
ChatGPT even fails to imagine a Othello Board and valid moves if you give it the rulebook, as @kromem@lemmy.world kindly provided.
The model was not given a rulebook in the experiment, only given games in shorthand notation and nothing else.
Intelligence and Imagination go hand in hand - as does the world model. If you give anyone intelligent a long list of coordinates, which only consists of 1-8 and a-h, they will also only choose out of these coordinates if you ask them to choose something legal, and they will also realize that a legal move has the prerequisite of choosing coordinates touching an enemy coordinate with one of my own chosen coordinates on the far end of the cluster - this does not involve imagining a “real” Othello Board.
Intelligence (and the world model) also implies that at some point the actual realization of the prerequisites for a legal move. These are pretty simple in Othello - actually, getting a computer to choose a correct move was solved decades ago, so we have actual proof that the playing entity does NOT need any kind of world model and therefore intelligence to ensure 100% of moves are correct.
If these things were intelligent (and as i insist on, world model and intelligence go hand in hand), they would be able to simply use the rulebook as training input and provide close to 100% legal moves. I’d like to see THAT paper, but papers with negative results seldom get published. (Sorry btw, i overlooked that they didn’t give it the rulebook - I was thrown off by the information that 2023 models all have the rules for common board games in their basic training data)
I cited your provided study because it is a very good example how research into LLMs is getting sensationalized. They did not even provide the prompting used in this study, so we have NO information about what they actually asked of the model - it’s simply non-available data, while being the actual seed used to get a result. We do not get information about the quality of the moves (which should probably be somewhere in the top 10% choices of possible moves when taking into account that this is championship data). We NEVER see the full actual output. Instead, we see a LLM having a 5% chance of doing random shit when fed with real world data. We are talking about a 8x8 board here, nothing that comes even close to the complexities of reality.
Edit: I just realized that we are also not shown the test cases themselves and the ratio of valid to invalid moves (and split in “not even touching another piece” and “touching, but invalid”), so we are missing the actual probability of making a valid move by random chance for those 2 variants. They only showed the 93% failure rate on an untrained model, which would be “i place a stone at random”. I’d not be surprised that every move made was at least touching another one - edge detection is one of the easier use cases for transformer models.
It’s the same with their mathematical proofs: we are never given the complete picture, only the result. We do not know what was used as training data, we do not know what “external” information was fed to it, we do not know the prompts used, and we do not know how much compute they have thrown at their room full of monkeys and typewriters to get a result.
Intelligence needs something to anchor it in the real world, or it becomes useless. In the biological world, that is a world model, based on previous experiences - and this experience needs at least a modicum of sentience to become useful as we can see all around us. Animals more aware of their own existence are also those who are a lot better at intelligent behavior.
The “intelligent” output we get from LLMs are NOT a sign of intelligence, it is a statistic. This is useful, no doubt about it - having computers that are able to output the average answer one would get if asking someone who knows their shit is a very good thing, as is being able to walk across a large amount of different (pre-existing) approaches. But conflating it with actual intelligence which would be able to correct hallucinations before they are uttered (because the hallucination isn’t logical at all in most cases) and which would be resistant against breaking them by asking them to do irrational things is not the way to go.
There is no representation of the huge amount of rejections to answer a question or do a task that many people are not qualified to work on at all. An actually intelligent being realizes when it reaches the limits of it’s knowledge (or it’s training data). Intelligent beings can then try to extrapolate from existing knowledge, but they know that they actually do not know, and often communicate that (putting social things aside for the moment).
This is the point where LLMs break down with 100% chance if pushed - they do not say: “i do not know”, or “i do not have enough data for that”, or “that doesn’t make sense”. An output is obligatory, regardless of how much an intelligent being would say “oh well, what you want from me doesn’t make sense at all”.
And it’s as easy as asking an LLM to modify their way of communicating: If you tell it to use a symbol/word again after that same symbol/word, and continue to ask it to do this every time it fails to do so (under the guise of having reading or hearing issues) will break these things every time, sending them spiraling. An intelligent being would say “You know what, fuck you, you are messing with me”.
This is actual proof of a complete lack of anything resembling real intelligence - intelligence would allow analysis of the situation and detection of unreasonable requests.
Edit: they trained a model they called “Othello-GPT” with championship datasets, containing only valid moves, and tried to make the point that these things have an internal world model by providing it with game states and asking it for a valid move. It still messed up 5% of the time with real world data, meaning there is no consistent reasoning going on here.
And is there any evidence that the required information for the other proofs wasn’t already available in the training data? As i said, these things remix stuff, they cannot come up with new stuff.
By “required information for the other proofs”, what do you mean exactly?
The proof, more or less in the same structure that the AI output, existed in the literature? This would’ve been published already and would be very easy to prove was stolen from the input corpus. We know this isn’t true.
The theorems and techniques used in the generated proof existed, but were not yet assembled into a final proof? Trained mathematicians have looked at the proof and seem to think it contains novel techniques, or at least novel applications of techniques from different areas of math. Even if that weren’t true, this would still be novel work; a lot of math breakthroughs are made this way, and it’s a nontrivial feat. Although if this was as far as it could go, we would run into limitations at some point.
The starting point of the proof existed, but the entire proof was built from the ground up by AI, using no existing methods or theorems? This would be very impressive, and as noted above the consensus seems to be that it did create some new techniques, but I’m not experienced or knowledgeable enough to verify that claim. Either way it definitely used at least some previous work for nontrivial reasons.
It looks like it’s somewhere between options 2 and 3 in reality; it used a lot of existing techniques and at least invented some new applications of existing techniques if not created new ones entirely (the line can be blurry). This is also what all mathematicians do! Remixing is coming up with new stuff. No thoughts, or generated output from LLMs, exist without prior context.
In the case i mentioned, it is very much number 2 IMO - the coincidence is just too much. In other cases, we are probably looking mostly at number 1: techniques that already exist, applied to other situations. These things are very good at iterating through rather uncommon combinations and have a large corpus of techniques in their training data, letting them walk across the “plane” of combinations quickly.
That’s not how humans work, and that’s probably the reason why there was a bunch of mathematical problems solved in a short time - the ingredients were there, but noone had yet tried that specific approach.
I have not yet seen any evidence of number 3, mainly because we do not know what these models are being trained on - transparency would give us the possibility to check, but we will never get it from OpenAI.
Here is the evidence: the entire field of mathematics was trying to solve these problems and couldn’t do it for decades.
If you want to still insist that it’s just “remixing the training data” then fine, but don’t act like that rules out the possibility of intelligence, because by any reasonable standard a system that can discover hundreds of groundbreaking new mathematical proofs is intelligent. Any human that accomplished that would go down in history as one of the most intelligent people to have ever lived.
It is a fucking bunch of math. It has no intelligence. It is a stochastic parrot. It is a blender for existing stuff. Doesn’t mean it isn’t a useful tool.
But if you actually believe that these mathematical construct have intelligence, do you also believe that every time you close a llama.cpp process you kill a sentient being? Or are you simply stopping a statistics engine? Because i am very much in camp “statistics engine”- statistics do not have intelligence.
Intelligence without a sense of self would be very strange, no? What we call intelligence is mostly thought of being able to think genuine thoughts that haven’t been there yet, and make sense (to exclude the inevitable hallucinations). This ability is deeply steeped in our ability to imagine hypothetical scenarios.
ChatGPT even fails to imagine a Othello Board and valid moves if you give it the rulebook, as @kromem@lemmy.world kindly provided.
I think you’re citing the wrong person. I cited the Othello-GPT line of research making the point transformers develop a world model. The other user (or you?) was mistaken about their interpretation of what a world model was representing and got caught up on the accuracy of a toy model exclusively trained on board notations like ‘a4’ and thought that the model making 95% accurate moves as a tiny parameter model trained on nothing else was somehow evidence that it didn’t form a world model from the data because it wasn’t 100% accurate, completely missing the point of the toy model scale allowing for interpretability methods of the time to identify geometric board reproduction and things like tracking game state on the board with flags for ‘my’ vs ‘opponent’ pieces, which was the data answering researcher questions about world modeling vs surface statistics in transformers.
Edit: just to further clarify re:
The model was not given a rulebook in the experiment, only given games in shorthand notation and nothing else.
Intelligence and Imagination go hand in hand - as does the world model. If you give anyone intelligent a long list of coordinates, which only consists of 1-8 and a-h, they will also only choose out of these coordinates if you ask them to choose something legal, and they will also realize that a legal move has the prerequisite of choosing coordinates touching an enemy coordinate with one of my own chosen coordinates on the far end of the cluster - this does not involve imagining a “real” Othello Board.
Intelligence (and the world model) also implies that at some point the actual realization of the prerequisites for a legal move. These are pretty simple in Othello - actually, getting a computer to choose a correct move was solved decades ago, so we have actual proof that the playing entity does NOT need any kind of world model and therefore intelligence to ensure 100% of moves are correct.
If these things were intelligent (and as i insist on, world model and intelligence go hand in hand), they would be able to simply use the rulebook as training input and provide close to 100% legal moves. I’d like to see THAT paper, but papers with negative results seldom get published. (Sorry btw, i overlooked that they didn’t give it the rulebook - I was thrown off by the information that 2023 models all have the rules for common board games in their basic training data)
I cited your provided study because it is a very good example how research into LLMs is getting sensationalized. They did not even provide the prompting used in this study, so we have NO information about what they actually asked of the model - it’s simply non-available data, while being the actual seed used to get a result. We do not get information about the quality of the moves (which should probably be somewhere in the top 10% choices of possible moves when taking into account that this is championship data). We NEVER see the full actual output. Instead, we see a LLM having a 5% chance of doing random shit when fed with real world data. We are talking about a 8x8 board here, nothing that comes even close to the complexities of reality.
Edit: I just realized that we are also not shown the test cases themselves and the ratio of valid to invalid moves (and split in “not even touching another piece” and “touching, but invalid”), so we are missing the actual probability of making a valid move by random chance for those 2 variants. They only showed the 93% failure rate on an untrained model, which would be “i place a stone at random”. I’d not be surprised that every move made was at least touching another one - edge detection is one of the easier use cases for transformer models.
It’s the same with their mathematical proofs: we are never given the complete picture, only the result. We do not know what was used as training data, we do not know what “external” information was fed to it, we do not know the prompts used, and we do not know how much compute they have thrown at their room full of monkeys and typewriters to get a result.
You are conflating intelligence with other concepts, like sentience or self awareness.
There are many different models of ChatGTP and most people do not typically have access to the cutting edge ones used to make these discoveries.
Intelligence needs something to anchor it in the real world, or it becomes useless. In the biological world, that is a world model, based on previous experiences - and this experience needs at least a modicum of sentience to become useful as we can see all around us. Animals more aware of their own existence are also those who are a lot better at intelligent behavior.
The “intelligent” output we get from LLMs are NOT a sign of intelligence, it is a statistic. This is useful, no doubt about it - having computers that are able to output the average answer one would get if asking someone who knows their shit is a very good thing, as is being able to walk across a large amount of different (pre-existing) approaches. But conflating it with actual intelligence which would be able to correct hallucinations before they are uttered (because the hallucination isn’t logical at all in most cases) and which would be resistant against breaking them by asking them to do irrational things is not the way to go.
There is no representation of the huge amount of rejections to answer a question or do a task that many people are not qualified to work on at all. An actually intelligent being realizes when it reaches the limits of it’s knowledge (or it’s training data). Intelligent beings can then try to extrapolate from existing knowledge, but they know that they actually do not know, and often communicate that (putting social things aside for the moment).
This is the point where LLMs break down with 100% chance if pushed - they do not say: “i do not know”, or “i do not have enough data for that”, or “that doesn’t make sense”. An output is obligatory, regardless of how much an intelligent being would say “oh well, what you want from me doesn’t make sense at all”.
And it’s as easy as asking an LLM to modify their way of communicating: If you tell it to use a symbol/word again after that same symbol/word, and continue to ask it to do this every time it fails to do so (under the guise of having reading or hearing issues) will break these things every time, sending them spiraling. An intelligent being would say “You know what, fuck you, you are messing with me”.
This is actual proof of a complete lack of anything resembling real intelligence - intelligence would allow analysis of the situation and detection of unreasonable requests.
Edit: they trained a model they called “Othello-GPT” with championship datasets, containing only valid moves, and tried to make the point that these things have an internal world model by providing it with game states and asking it for a valid move. It still messed up 5% of the time with real world data, meaning there is no consistent reasoning going on here.