audio language model
In plain wordsAn AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words.
- 1
Audio understanding
A multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type. that accepts audio as input, and sometimes produces it, so that it can transcribe, translate, summarize and answer questions about speech, environmental sounds and music, and follow spoken instructions. The usual design attaches an audio encoder (often WhisperWhisper, prop. n.: A free OpenAI model that writes down what is said in recordings in many languages, and can translate it into English; it learned from hundreds of thousands of hours of online audio.'s encoder or another pretrained model operating on mel spectrogramsmel spectrogram, n.: A version of a sound picture whose pitch axis is squeezed to match human hearing, giving more detail to low sounds than to high ones; the usual way AI models take in audio.) through a projection or adapter layer to a text large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., which then treats the resulting audio embeddings like tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.; training mixes speech recognition, audio captioning, question answering and instruction data. Early examples include SALMONN and Qwen-Audio (both 2023); by 2025 several frontier multimodal models, such as GPT-4o and Gemini, accepted audio natively. Benchmarks probe spoken content, speaker and emotion cues, sound events and musical reasoning. Such models underpin speech-to-speechspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice. assistants, meeting analysis and audio search.
We ask the audio language model whether the caller sounds confused, not just what they said.
- 2
Audio generation
A model that generates audio by next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers. over discrete audio tokens, treating sound as a language: 'semantic' tokens from a self-supervised speech model typically capture content and structure, while 'acoustic' tokens from a neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece. capture voice and recording detail. Examples include Meta's Generative Spoken Language Modeling (2021), Google's AudioLM (2022) and codec-language-model text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps. such as VALL-E (2023); the same approach underlies much music generationmusic generation, n.: Using AI to compose or produce music — from melodies written as notes to complete songs with vocals — often from a short text description like 'upbeat jazz with saxophone'..
Origin
Language modeling over discrete speech units was pioneered in Kushal Lakhotia and colleagues' 'Generative Spoken Language Modeling' at Meta (2021); Zalán Borsos and colleagues' AudioLM at Google (September 2022) popularized the name. Instruction-following audio-understanding models followed from 2023 as multimodal LLMs extended to sound.
Usage note
Terminology is unsettled: 'speech language model' often means models restricted to speech, while 'audio language model' also covers music and environmental sound. Not to be confused with a text language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. that merely reads speech transcripts.
Related
- multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type.
- speech-to-speech modelspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice.
- neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece.
- WhisperWhisper, prop. n.: A free OpenAI model that writes down what is said in recordings in many languages, and can translate it into English; it learned from hundreds of thousands of hours of online audio.
- speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video.
- sound event detectionsound event detection, n.: Teaching computers to recognize everyday sounds in a recording, such as a dog barking, glass breaking or a siren, and to say exactly when each one starts and stops.
Sources
- Borsos, Z. et al. (2022). AudioLM: A Language Modeling Approach to Audio Generation. arXiv:2209.03143. arxiv.org/abs/2209.03143
- Lakhotia, K. et al. (2021). On Generative Spoken Language Modeling from Raw Audio. Transactions of the Association for Computational Linguistics 9.
- Chu, Y. et al. (2023). Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models. arXiv preprint.
Mentioned in
speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video., multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type., speech-to-speech modelspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice., WhisperWhisper, prop. n.: A free OpenAI model that writes down what is said in recordings in many languages, and can translate it into English; it learned from hundreds of thousands of hours of online audio., sound event detectionsound event detection, n.: Teaching computers to recognize everyday sounds in a recording, such as a dog barking, glass breaking or a siren, and to say exactly when each one starts and stops.