LLMs break words up into chunks of letters which commonly appear - suffixes like “-tion” and “-ism” are obvious examples. They then predict which chunk comes next based on the ones before, or whether the word will end.
This is very useful for generating sensible-looking text while at the same time correlating concepts associated with different words. However, it also means that the dont really “see” the letters that make up each word, just the chunks of letters, which are stored as mathematical vectors. This is why they struggle so much with analysing the makeup of words.
However, with numbers they generally store each digit individually, so they shouldnt have as much of a problem saying how many 5’s are in 1,589,005, for example.
LLMs break words up into chunks of letters which commonly appear - suffixes like “-tion” and “-ism” are obvious examples. They then predict which chunk comes next based on the ones before, or whether the word will end.
This is very useful for generating sensible-looking text while at the same time correlating concepts associated with different words. However, it also means that the dont really “see” the letters that make up each word, just the chunks of letters, which are stored as mathematical vectors. This is why they struggle so much with analysing the makeup of words.
However, with numbers they generally store each digit individually, so they shouldnt have as much of a problem saying how many 5’s are in 1,589,005, for example.