Whisper
In plain wordsA free OpenAI model that writes down what is said in recordings in many languages, and can translate it into English; it learned from hundreds of thousands of hours of online audio.
A family of open-weight speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video. models released by OpenAI. Each is a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. encoder–decoderencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output. whose encoder hears audio in fixed 30-second windows, resampled to 16 kHz and turned into a log-mel spectrogrammel spectrogram, n.: A version of a sound picture whose pitch axis is squeezed to match human hearing, giving more detail to low sounds than to high ones; the usual way AI models take in audio., and whose decoder writes text tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. one at a time. Control tokens opening the output choose the task (transcribing, translating into English, identifying the language, timestamping or detecting no speech), so one network covers several pipeline stages. Its training data were 680,000 hours of web audio with loosely matched transcripts ('weak supervision'), roughly a third of it in languages other than English, and it generalizes zero-shot to new datasets far better than models fine-tuned on a single benchmark, approaching human robustness to accents and noise in English. Sizes range from tiny (39 million parameters) to large (1.55 billion); large-v3 (2023) was trained on several million hours, including pseudo-labeled audio, and 'turbo' (2024) shrank the decoder for speed. Code and weights are MIT-licensed, which made Whisper a default for open-source transcription and spawned many faster reimplementations.
We run Whisper turbo locally so interview recordings never leave the newsroom.
Origin
Released by OpenAI on 21 September 2022 and described in 'Robust Speech Recognition via Large-Scale Weak Supervision' by Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey and Ilya Sutskever (arXiv December 2022; ICML 2023).
Usage note
Whisper can hallucinate fluent text during silence or noise, a documented risk in medical and legal transcription; filtering with voice activity detectionvoice activity detection, n.: Detecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk. and human review reduce it. Its text normalization affects measured word error rateword error rate, n.: The standard score for speech recognition: the share of words a system gets wrong compared with a correct transcript, counting wrong, missing and extra words. Lower is better..
Related
- speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video.
- mel spectrogrammel spectrogram, n.: A version of a sound picture whose pitch axis is squeezed to match human hearing, giving more detail to low sounds than to high ones; the usual way AI models take in audio.
- word error rateword error rate, n.: The standard score for speech recognition: the share of words a system gets wrong compared with a correct transcript, counting wrong, missing and extra words. Lower is better.
- encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output.
- weak supervisionweak supervision, n.: Training AI with labels that are cheap but imperfect, such as rough rules of thumb, partial tags or noisy guesses, instead of carefully hand-checked answers.
- audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words.
Sources
- Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C. & Sutskever, I. (2023). Robust Speech Recognition via Large-Scale Weak Supervision. ICML 2023. arxiv.org/abs/2212.04356
- OpenAI. Whisper: code and model card. GitHub repository. github.com/openai/whisper
- Koenecke, A., Choi, A. S. G., Mei, K. X., Schellmann, H. & Sloane, M. (2024). Careless Whisper: Speech-to-Text Hallucination Harms. ACM FAccT 2024.
Mentioned in
OpenAIOpenAI, prop. n.: The American company that makes ChatGPT and the GPT models. It began in 2015 as a nonprofit research lab and is now a for-profit company overseen by a nonprofit foundation., speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video., Alec RadfordAlec Radford, prop. n.: An American AI researcher who led OpenAI's first GPT language models and its CLIP and Whisper systems, key steps toward today's chatbots and multimodal AI., audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words., encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output., mel spectrogrammel spectrogram, n.: A version of a sound picture whose pitch axis is squeezed to match human hearing, giving more detail to low sounds than to high ones; the usual way AI models take in audio., word error rateword error rate, n.: The standard score for speech recognition: the share of words a system gets wrong compared with a correct transcript, counting wrong, missing and extra words. Lower is better., LibriSpeechLibriSpeech, prop. n.: A free collection of about 1,000 hours of volunteers reading English audiobooks, matched with the book text; it became the standard yardstick for speech-recognition systems., wav2vecwav2vec, prop. n.: A family of Meta speech models that learn the sounds of language from thousands of hours of unlabeled recordings, so that only minutes of transcribed speech are needed to build a recognizer.