vocoder
In plain wordsA device or program that turns a compact description of a voice back into actual sound; in modern speech AI, the final stage that produces the audio you hear.
- 1
Neural speech synthesis
The component of a text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps. or voice-conversion system that converts an intermediate acoustic representation, usually a mel spectrogrammel spectrogram, n.: A version of a sound picture whose pitch axis is squeezed to match human hearing, giving more detail to low sounds than to high ones; the usual way AI models take in audio., into an audio waveform. Because a mel spectrogram discards phase and fine frequency detail, the vocoder must infer them. The classical Griffin–Lim algorithm (1984) estimates phase iteratively but sounds metallic; neural vocoders learn the mapping from data: autoregressive models such as WaveNetWaveNet, prop. n.: A DeepMind model that generates speech and other sounds one tiny audio sample at a time, producing voices far more natural than earlier computer speech. (the vocoder of Tacotron 2) and WaveRNN, the flow-based WaveGlow, and much faster adversarially trained GANgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people. vocoders such as MelGAN (2019), HiFi-GAN (2020) and BigVGAN (2022). In systems that generate neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece. tokens, the codec's decoder plays the vocoder's role.
The acoustic model is fine; the buzzing on high voices comes from the vocoder.
- 2
Telecommunications and music
Originally a 'voice coder': a system that analyzes speech into slowly varying parameters, such as the energy in each of a bank of frequency bands plus pitch and voicing, transmits them with far less bandwidth than the waveform needs, and resynthesizes speech at the receiver. Vocoders were used for secure and low-bit-rate telephony, and the robotic effect of imposing a voice's spectral envelope on another sound, such as a synthesizer, became a staple of popular music.
Origin
Developed by Homer Dudley at Bell Telephone Laboratories in the 1930s and described in his publications of 1939; the name blends 'voice' and 'coder'. Dudley's related keyboard-played synthesizer, the Voder, was shown at the 1939 New York World's Fair, and vocoders protected Allied telephone links (the SIGSALY system) in the Second World War. 'Neural vocoder' entered speech synthesis around 2017–2018.
Usage note
The musical vocoder effect is often confused with Auto-Tune pitch correction, a different technique.
Related
- text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps.
- mel spectrogrammel spectrogram, n.: A version of a sound picture whose pitch axis is squeezed to match human hearing, giving more detail to low sounds than to high ones; the usual way AI models take in audio.
- WaveNetWaveNet, prop. n.: A DeepMind model that generates speech and other sounds one tiny audio sample at a time, producing voices far more natural than earlier computer speech.
- neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece.
- spectrogramspectrogram, n.: A picture of sound that shows time from left to right, pitch from bottom to top, and loudness as brightness or color, so that speech and music can be seen as patterns.
- generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people.
Sources
- Griffin, D. W. & Lim, J. S. (1984). Signal Estimation from Modified Short-Time Fourier Transform. IEEE Transactions on Acoustics, Speech, and Signal Processing 32(2), 236–243.
- Kong, J., Kim, J. & Bae, J. (2020). HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. NeurIPS 2020.
- Dudley, H. (1939). Remaking Speech. Journal of the Acoustical Society of America 11(2), 169–177.
Mentioned in
text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps., mel spectrogrammel spectrogram, n.: A version of a sound picture whose pitch axis is squeezed to match human hearing, giving more detail to low sounds than to high ones; the usual way AI models take in audio., spectrogramspectrogram, n.: A picture of sound that shows time from left to right, pitch from bottom to top, and loudness as brightness or color, so that speech and music can be seen as patterns., WaveNetWaveNet, prop. n.: A DeepMind model that generates speech and other sounds one tiny audio sample at a time, producing voices far more natural than earlier computer speech., mean opinion scoremean opinion score, n.: A quality score from 1 (bad) to 5 (excellent) obtained by asking many listeners to rate audio, such as a synthetic voice or a phone call, and averaging their ratings., neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece.