mel spectrogram
In plain wordsA version of a sound picture whose pitch axis is squeezed to match human hearing, giving more detail to low sounds than to high ones; the usual way AI models take in audio.
A spectrogramspectrogram, n.: A picture of sound that shows time from left to right, pitch from bottom to top, and loudness as brightness or color, so that speech and music can be seen as patterns. whose frequency axis has been mapped to the mel scale, a perceptual pitch scale that is roughly linear below about 1 kHz and logarithmic above it. The power spectrum of each frame is multiplied by a bank of overlapping triangular filters (commonly 40 to 128) spaced evenly in mels, so that each output channel sums energy over a band that widens with frequency; the result is usually log-compressed (a log-mel spectrogram). This shrinks each frame from hundreds of frequency bins to tens of channels while keeping the detail that matters for perceiving speech and music. Log-mel spectrograms are the standard input to modern speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video. models (for example 80 channels every 10 milliseconds in the original WhisperWhisper, prop. n.: A free OpenAI model that writes down what is said in recordings in many languages, and can translate it into English; it learned from hundreds of thousands of hours of online audio. models, 128 in Whisper large-v3), audio classifiers and sound event detectionsound event detection, n.: Teaching computers to recognize everyday sounds in a recording, such as a dog barking, glass breaking or a siren, and to say exactly when each one starts and stops. systems, and the standard intermediate target of neural text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps., where an acoustic model predicts mel frames and a vocodervocoder, n.: A device or program that turns a compact description of a voice back into actual sound; in modern speech AI, the final stage that produces the audio you hear. turns them into a waveform. Applying a further discrete cosine transform gives MFCCsMFCC, abbr.: A short list of numbers, usually about 13, that summarizes the shape of a brief slice of sound in a way that mimics human hearing; for decades the standard input to speech recognizers..
The TTS model predicts 80-band mel spectrograms at 86 frames per second, and HiFi-GAN turns them into audio.
Origin
The mel scale ('mel' from 'melody') was proposed by Stanley Smith Stevens, John Volkmann and Edwin Newman in 1937 from listening experiments on judged pitch. Mel-spaced filterbanks became standard in speech recognition with MFCCs in the 1980s, and log-mel spectrograms became the default neural-network input for audio in the 2010s.
Usage note
The formula is one common approximation, converting frequency f in hertz to mels; several variants exist. Mel spectrograms discard phase and some fine detail, which is why a separate vocoder is needed to make audio from them.
Related
- spectrogramspectrogram, n.: A picture of sound that shows time from left to right, pitch from bottom to top, and loudness as brightness or color, so that speech and music can be seen as patterns.
- MFCCMFCC, abbr.: A short list of numbers, usually about 13, that summarizes the shape of a brief slice of sound in a way that mimics human hearing; for decades the standard input to speech recognizers.
- vocodervocoder, n.: A device or program that turns a compact description of a voice back into actual sound; in modern speech AI, the final stage that produces the audio you hear.
- WhisperWhisper, prop. n.: A free OpenAI model that writes down what is said in recordings in many languages, and can translate it into English; it learned from hundreds of thousands of hours of online audio.
- text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps.
- speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video.
Sources
- Stevens, S. S., Volkmann, J. & Newman, E. B. (1937). A Scale for the Measurement of the Psychological Magnitude Pitch. Journal of the Acoustical Society of America 8(3), 185–190.
Mentioned in
text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps., audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words., spectrogramspectrogram, n.: A picture of sound that shows time from left to right, pitch from bottom to top, and loudness as brightness or color, so that speech and music can be seen as patterns., WhisperWhisper, prop. n.: A free OpenAI model that writes down what is said in recordings in many languages, and can translate it into English; it learned from hundreds of thousands of hours of online audio., Fourier transformFourier transform, n.: A way of breaking a signal — a sound, an image, any pattern over time or space — into the simple waves of different frequencies that add up to it, much as a prism splits white light into colors., MFCCMFCC, abbr.: A short list of numbers, usually about 13, that summarizes the shape of a brief slice of sound in a way that mimics human hearing; for decades the standard input to speech recognizers., sound event detectionsound event detection, n.: Teaching computers to recognize everyday sounds in a recording, such as a dog barking, glass breaking or a siren, and to say exactly when each one starts and stops., vocodervocoder, n.: A device or program that turns a compact description of a voice back into actual sound; in modern speech AI, the final stage that produces the audio you hear.