PaLM
In plain wordsA very large Google language model from 2022 that showed sudden jumps in reasoning, and even explained jokes, as it grew. Its successor powered Google's Bard chatbot before the Gemini models replaced it.
Google Research's Pathways Language Model, announced in April 2022: a dense, decoder-only transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. with 540 billion parameters, then among the largest of its kind. Training consumed 780 billion tokens on 6,144 TPUTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud. v4 chips spread over two TPU pods, coordinated by Google's Pathways system. PaLM set new results on many benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., including BIG-bench, showed abrupt improvements on some tasks as it scaled, and with chain-of-thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate. prompting carried out multistep arithmetic and commonsense reasoning and explained jokes. PaLM 2 (May 2023), smaller but trained on more data, powered Bard and, by Google's count at launch, more than 25 other products and features; specialized versions included Med-PaLM for medical questions and the embodied PaLM-E. The line was succeeded by GeminiGemini, prop. n.: Google's family of AI models and its chatbot assistant, launched in December 2023. Gemini models handle text, images, audio and video and power many Google products. from December 2023.
For many researchers, PaLM's joke explanations were the first sign that scale was producing reasoning-like behavior.
Origin
Described in 'PaLM: Scaling Language Modeling with Pathways' by Aakanksha Chowdhery, Sharan Narang, Jacob Devlin and colleagues (April 2022). PaLM 2 was announced at Google I/O on 10 May 2023.
Usage note
DeepMind's Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model., also published in 2022, implied that PaLM was undertrained for its size, a lesson reflected in the more data-heavy design of PaLM 2.
Related
- GeminiGemini, prop. n.: Google's family of AI models and its chatbot assistant, launched in December 2023. Gemini models handle text, images, audio and video and power many Google products.
- chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
- Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model.
- emergent abilitiesemergent abilities, n.: Skills that seem to switch on suddenly as AI models grow: small versions fail at them, then larger ones past a certain size succeed. Some researchers argue the suddenness is partly an artifact of how skills are measured.
- TPUTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud.
Sources
- Chowdhery, A. et al. (2022). PaLM: Scaling Language Modeling with Pathways. arXiv preprint; Journal of Machine Learning Research (2023).
Mentioned in
GeminiGemini, prop. n.: Google's family of AI models and its chatbot assistant, launched in December 2023. Gemini models handle text, images, audio and video and power many Google products.