neural audio codec
In plain wordsA neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece.
A learned encoder–decoder that compresses audio waveforms into a low-rate sequence of discrete codes and reconstructs audio from them. A convolutional (sometimes transformer) encoder downsamples the waveform to tens of frames per second; each frame is quantized, typically by residual vector quantization (RVQ), in which a stack of codebooks successively encodes the error left by the previous stage, so bitrate can be traded against fidelity by using more or fewer stages; and a decoder resynthesizes the waveform, trained with reconstruction, spectral and adversarial losses. Besides compression — SoundStream spanned 3–18 kilobits per second with a single model — codecs serve as the tokenizers of audio generative modelsgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them.: language models over codec tokens power zero-shot voice cloningvoice cloning, n.: Using AI to copy a specific person's voice from recordings, so a computer can say new things that sound like them — useful for dubbing and accessibility, and abused in scams. (VALL-E), music generationmusic generation, n.: Using AI to compose or produce music — from melodies written as notes to complete songs with vocals — often from a short text description like 'upbeat jazz with saxophone'. (MusicLM, MusicGen) and full-duplex speech-to-speechspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice. systems. A tension between acoustic fidelity and tokens that are easy to model has led to 'semantic' tokens, distilled codecs and single-codebook designs.
Origin
Google's SoundStream (Neil Zeghidour and colleagues, posted July 2021) introduced end-to-end neural codecs with residual vector quantization, and Meta's EnCodec (Alexandre Défossez and colleagues, October 2022) followed. Earlier neural codecs (2017–2019) paired conventional compressed features with WaveNet-style neural decoders.
Related
- vocodervocoder, n.: A device or program that turns a compact description of a voice back into actual sound; in modern speech AI, the final stage that produces the audio you hear.
- VQ-VAEVQ-VAE, n.: A network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate.
- image tokenizerimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures.
- voice cloningvoice cloning, n.: Using AI to copy a specific person's voice from recordings, so a computer can say new things that sound like them — useful for dubbing and accessibility, and abused in scams.
- music generationmusic generation, n.: Using AI to compose or produce music — from melodies written as notes to complete songs with vocals — often from a short text description like 'upbeat jazz with saxophone'.
- speech-to-speech modelspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice.
- tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.
Sources
- Zeghidour, N., Luebs, A., Omran, A., Skoglund, J. & Tagliasacchi, M. (2021). SoundStream: An End-to-End Neural Audio Codec. arXiv:2107.03312. arxiv.org/abs/2107.03312
- Défossez, A., Copet, J., Synnaeve, G. & Adi, Y. (2022). High Fidelity Neural Audio Compression. arXiv:2210.13438. arxiv.org/abs/2210.13438
Mentioned in
text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps., tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., voice cloningvoice cloning, n.: Using AI to copy a specific person's voice from recordings, so a computer can say new things that sound like them — useful for dubbing and accessibility, and abused in scams., audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words., generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people., music generationmusic generation, n.: Using AI to compose or produce music — from melodies written as notes to complete songs with vocals — often from a short text description like 'upbeat jazz with saxophone'., speech-to-speech modelspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice., SunoSuno, prop. n.: A popular AI service that writes and performs complete songs — music, lyrics and singing — from a short text description., image tokenizerimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures., vector quantizationvector quantization, n.: Compressing data by replacing each item with the nearest entry from a small catalog of representative examples, so that only the catalog number needs to be stored or sent., vocodervocoder, n.: A device or program that turns a compact description of a voice back into actual sound; in modern speech AI, the final stage that produces the audio you hear., VQ-VAEVQ-VAE, n.: A network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate.