strawberry problem

The tendency of large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. to miscount the letters in a word — canonically answering that 'strawberry' contains two r's rather than three — and, by extension, a shorthand for simple character-level tasks that models fail despite strong performance elsewhere. The main cause is tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with.: models receive text as subword tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. — 'strawberry' typically arrives as two or three multi-letter chunks — so individual letters are not directly visible and must be inferred from memorized knowledge of spellings. The question became the archetypal illustration of jagged intelligencejagged intelligence, n.: The uneven pattern of AI abilities: today's systems can solve very hard problems yet fail at some a child could manage, so their skills form a spiky profile rather than a smooth one. and a quick test users ran on every new model; reasoning modelsreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning. that spell a word out step by step, or that call a code tool, largely solved it.

Before we argue about AGI, it should at least pass the strawberry problem.