There are exactly 2 't’s in the word
‘colonialism’.
C-o-l-o-n-i-a-l-i-s-m
Would you like to check the spelling or
character count of any other words? Let me
know!
LLMs break words up into chunks of letters which commonly appear - suffixes like “-tion” and “-ism” are obvious examples. They then predict which chunk comes next based on the ones before, or whether the word will end.
This is very useful for generating sensible-looking text while at the same time correlating concepts associated with different words. However, it also means that the dont really “see” the letters that make up each word, just the chunks of letters, which are stored as mathematical vectors. This is why they struggle so much with analysing the makeup of words.
However, with numbers they generally store each digit individually, so they shouldnt have as much of a problem saying how many 5’s are in 1,589,005, for example.
Short youtube video explaining why tokenisation causes this bug. It’s an older video, so it talks about tokens as being whole-word rather than chunks of words, which is how most modern models work.
The other persons explanation doesn’t acknowledge that emergent reasoning does kind-of exist in LLMs. That’s why theyre able to say how many 5’s are in a large number, despite never seeing that problem before. They dont ‘just’ repeat things they’ve been trained on, though they often do.
Of course, if that problem did exist significantly in the training data, it would be more likely to get it right. But you could say the same about any number of things an LLM doesn’t know.
Simply put, LLMs are great at forming sentences but can’t do math. Like, any math. If they get 60+21=81 right, it’s only going to be because it’s textually written somewhere in the training data that 60+21=81. However, it’s very unlikely for counting the number of Ts in colonialism to be in there, so it just hallucinates what it thinks is a correct response.
Is this actually real though?
Just tested with similar results, output was:
There are exactly 2 't’s in the word ‘colonialism’. C-o-l-o-n-i-a-l-i-s-m Would you like to check the spelling or character count of any other words? Let me know!
Wow, I didn’t think it was still that stupid
I don’t think this particular genre of stupid will ever be fully fixed in LLMs to be honest, it’s fairly structural
I hope so
What causes this?
LLMs break words up into chunks of letters which commonly appear - suffixes like “-tion” and “-ism” are obvious examples. They then predict which chunk comes next based on the ones before, or whether the word will end.
This is very useful for generating sensible-looking text while at the same time correlating concepts associated with different words. However, it also means that the dont really “see” the letters that make up each word, just the chunks of letters, which are stored as mathematical vectors. This is why they struggle so much with analysing the makeup of words.
However, with numbers they generally store each digit individually, so they shouldnt have as much of a problem saying how many 5’s are in 1,589,005, for example.
Two very different answers to this question…
Short youtube video explaining why tokenisation causes this bug. It’s an older video, so it talks about tokens as being whole-word rather than chunks of words, which is how most modern models work.
https://youtube.com/shorts/7pQrMAekdn4
The other persons explanation doesn’t acknowledge that emergent reasoning does kind-of exist in LLMs. That’s why theyre able to say how many 5’s are in a large number, despite never seeing that problem before. They dont ‘just’ repeat things they’ve been trained on, though they often do.
Of course, if that problem did exist significantly in the training data, it would be more likely to get it right. But you could say the same about any number of things an LLM doesn’t know.
Simply put, LLMs are great at forming sentences but can’t do math. Like, any math. If they get 60+21=81 right, it’s only going to be because it’s textually written somewhere in the training data that 60+21=81. However, it’s very unlikely for counting the number of Ts in colonialism to be in there, so it just hallucinates what it thinks is a correct response.