speech recognition
In plain wordsTechnology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video.
The automatic transcription of spoken language into text. For decades the dominant design was a statistical pipeline: acoustic features such as MFCCsMFCC, abbr.: A short list of numbers, usually about 13, that summarizes the shape of a brief slice of sound in a way that mimics human hearing; for decades the standard input to speech recognizers. computed every 10 milliseconds; an acoustic model, a hidden Markov modelhidden Markov model, n.: A model of a sequence, such as speech or DNA, in which hidden states — say, the words being spoken — change step by step at random, and each state gives off signals that can be observed, such as sounds. with Gaussian-mixtureGaussian mixture model, n.: A model that explains data as coming from several overlapping bell-shaped clouds, working out where each cloud sits, how spread out it is, and how likely each point is to belong to it. outputs, scoring how well the sounds matched phonemes; a pronunciation lexicon; and an n-gramn-gram model, n.: An early type of language model that predicts the next word from just the few words before it, using counts of how often each word followed those words in a large collection of text. language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own., combined by searching over possible word sequences. Around 2010–2012, deep neural networks replaced the Gaussian mixtures and cut error rates sharply. End-to-end models then mapped audio directly to characters or subword tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., using CTCconnectionist temporal classification, n.: A training method that lets a network learn to transcribe speech or handwriting without being told exactly where each letter or sound begins and ends., attention-based encoder–decodersencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output. or RNN transducers (favored for streaming), often with ConformerConformer, prop. n.: A neural-network design for speech recognition that combines a transformer, which relates distant parts of a recording, with convolutions, which pick up fine local sound patterns. encoders that combine convolution and self-attention. Self-supervisedself-supervised learning, n.: A way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image. pretraining on unlabeled audio (wav2vec 2.0wav2vec, prop. n.: A family of Meta speech models that learn the sounds of language from thousands of hours of unlabeled recordings, so that only minutes of transcribed speech are needed to build a recognizer., 2020) and weakly supervised training on hundreds of thousands of hours (WhisperWhisper, prop. n.: A free OpenAI model that writes down what is said in recordings in many languages, and can translate it into English; it learned from hundreds of thousands of hours of online audio., 2022) brought robustness across accents, noise and languages, and by the mid-2020s audio language modelsaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words. transcribed speech as one skill among many. Accuracy is measured by word error rateword error rate, n.: The standard score for speech recognition: the share of words a system gets wrong compared with a correct transcript, counting wrong, missing and extra words. Lower is better.. Applications include captioning, dictation, voice assistants, call-center analytics, meeting notes and voice interfaces for agents.
Speech recognition handles the doctors' dictation well but still stumbles on drug names, so we added a custom vocabulary.
Origin
Bell Labs' 'Audrey' (1952) recognized digits spoken by a single talker, and IBM's Shoebox (1962) about 16 words. DARPA's Speech Understanding Research program (1971–1976) and hidden-Markov-model work at IBM and Carnegie Mellon in the 1970s–80s led to commercial dictation in the 1990s; a 2012 paper by Geoffrey Hinton, Li Deng, Dong Yu and colleagues marked the shift to deep learning.
Usage note
Speech recognition concerns what was said; speaker recognition (speaker verificationspeaker verification, n.: Checking whether a voice really belongs to the person it claims to be, as when a bank confirms your identity by the sound of your voice on the phone., speaker diarizationspeaker diarization, n.: Working out who spoke when in a recording, splitting a meeting or phone call into turns labeled Speaker 1, Speaker 2 and so on, without necessarily knowing who the speakers are.) concerns who said it, and 'voice recognition' is used loosely for either. Claims of 'human parity' on particular benchmarks (Microsoft, 2016–2017) did not extend to noisy, accented or conversational speech in general.
Related
- word error rateword error rate, n.: The standard score for speech recognition: the share of words a system gets wrong compared with a correct transcript, counting wrong, missing and extra words. Lower is better.
- WhisperWhisper, prop. n.: A free OpenAI model that writes down what is said in recordings in many languages, and can translate it into English; it learned from hundreds of thousands of hours of online audio.
- wav2vecwav2vec, prop. n.: A family of Meta speech models that learn the sounds of language from thousands of hours of unlabeled recordings, so that only minutes of transcribed speech are needed to build a recognizer.
- connectionist temporal classificationconnectionist temporal classification, n.: A training method that lets a network learn to transcribe speech or handwriting without being told exactly where each letter or sound begins and ends.
- LibriSpeechLibriSpeech, prop. n.: A free collection of about 1,000 hours of volunteers reading English audiobooks, matched with the book text; it became the standard yardstick for speech-recognition systems.
- speaker diarizationspeaker diarization, n.: Working out who spoke when in a recording, splitting a meeting or phone call into turns labeled Speaker 1, Speaker 2 and so on, without necessarily knowing who the speakers are.
- text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps.
- audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words.
- speaker verificationspeaker verification, n.: Checking whether a voice really belongs to the person it claims to be, as when a bank confirms your identity by the sound of your voice on the phone.
- text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps.
Sources
- Davis, K. H., Biddulph, R. & Balashek, S. (1952). Automatic Recognition of Spoken Digits. Journal of the Acoustical Society of America 24(6), 637–642.
- Hinton, G. et al. (2012). Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups. IEEE Signal Processing Magazine 29(6), 82–97.
Mentioned in
natural language processingnatural language processing, n.: The field that teaches computers to handle human language — to read, interpret, translate, summarize and produce text and speech. It is one of the oldest and largest areas of artificial intelligence., text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps., audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words., conversational AIconversational AI, n.: The technology that lets computers hold conversations with people in everyday language, by text or voice. It powers chatbots, voice assistants and automated customer-service lines., end-to-end learningend-to-end learning, n.: Training one AI system to go straight from raw input, such as camera images, to the final output, such as steering commands, instead of chaining hand-designed steps., hidden Markov modelhidden Markov model, n.: A model of a sequence, such as speech or DNA, in which hidden states — say, the words being spoken — change step by step at random, and each state gives off signals that can be observed, such as sounds., mel spectrogrammel spectrogram, n.: A version of a sound picture whose pitch axis is squeezed to match human hearing, giving more detail to low sounds than to high ones; the usual way AI models take in audio., pattern recognitionpattern recognition, n.: The ability of a computer, or a brain, to spot regularities in data and sort things into categories, such as recognizing handwritten letters, faces or spoken words., perceptionperception, n.: How an AI system or robot turns raw sensor signals, such as camera pixels, microphone sound or laser scans, into an understanding of what is around it: objects, people, words and distances., Ray KurzweilRay Kurzweil, prop. n.: An American inventor of reading machines and speech technology who became the best-known prophet of the 'singularity', predicting human-level AI by 2029., sequence-to-sequence modelsequence-to-sequence model, n.: A model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step., speaker diarizationspeaker diarization, n.: Working out who spoke when in a recording, splitting a meeting or phone call into turns labeled Speaker 1, Speaker 2 and so on, without necessarily knowing who the speakers are., speaker verificationspeaker verification, n.: Checking whether a voice really belongs to the person it claims to be, as when a bank confirms your identity by the sound of your voice on the phone., spectrogramspectrogram, n.: A picture of sound that shows time from left to right, pitch from bottom to top, and loudness as brightness or color, so that speech and music can be seen as patterns., speech-to-speech modelspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice., streamingstreaming, n.: Sending an AI's answer piece by piece while it is still being written, so the words appear on screen one after another instead of all at once at the end., virtual assistantvirtual assistant, n.: Software you speak or type to, like Siri, Alexa or Google Assistant, that answers questions and does simple jobs such as setting timers, playing music or controlling smart-home devices., Viterbi algorithmViterbi algorithm, n.: A method that finds the likeliest hidden sequence behind some observations — such as each word's grammatical role in a sentence — by keeping only the best path to each point instead of trying every combination., voice activity detectionvoice activity detection, n.: Detecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk., voice agentvoice agent, n.: An AI system you talk to out loud, for example on a phone call, that listens, answers in a natural-sounding voice and can carry out tasks such as booking appointments or checking an order., wake wordwake word, n.: A word or phrase, such as 'Alexa' or 'Hey Google', that a smart speaker or phone listens for constantly and that switches it into full listening mode., WhisperWhisper, prop. n.: A free OpenAI model that writes down what is said in recordings in many languages, and can translate it into English; it learned from hundreds of thousands of hours of online audio., word error rateword error rate, n.: The standard score for speech recognition: the share of words a system gets wrong compared with a correct transcript, counting wrong, missing and extra words. Lower is better., ConformerConformer, prop. n.: A neural-network design for speech recognition that combines a transformer, which relates distant parts of a recording, with convolutions, which pick up fine local sound patterns.