speech-to-speech model
In plain wordsAn AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice.
- 1
Conversational models
A model that takes spoken audio as input and produces spoken audio as output within a single network, instead of chaining speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video., a text large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. and text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps. in a 'cascaded' pipeline. Audio is represented as continuous features or as discrete tokens from a neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece., and the model is usually built on a pretrained language model extended to read and write audio tokens, often also predicting time-aligned text tokens as an internal 'inner monologue' that preserves reasoning and content quality. Compared with cascades it offers lower latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system. (replies within a few hundred milliseconds), keeps paralinguistic information such as emotion, emphasis, laughter and accent, and supports natural turn-taking. Full-duplex models listen and speak at the same time, so they can handle interruptions and back-channel sounds such as 'mm-hm'; Kyutai's Moshi (2024) reported about 200 milliseconds of latency in practice. Commercial examples include the voice mode of OpenAI's GPT-4o (2024) and the real-time APIs that later powered voice agents. Drawbacks include weaker tool use and instruction following than text models, harder evaluation and moderation of audio, and new risks such as voice impersonation.
Moving from the cascade to a speech-to-speech model cut our response latency from 1.8 seconds to about 400 milliseconds.
- 2
Translation
A model that translates speech in one language directly into speech in another, without an intermediate text transcript, as introduced by Google's Translatotron (2019), which could also carry the original speaker's voice into the translation. Later systems such as Meta's SeamlessM4T (2023) combine speech and text translation across about a hundred languages, and streaming variants interpret while the speaker is still talking.
Origin
Direct speech-to-speech translation was introduced by Ye Jia, Ron Weiss and colleagues at Google in Translatotron (arXiv April 2019). Conversational speech-to-speech models spread in 2024, when OpenAI demonstrated GPT-4o's voice mode (May) and Kyutai released Moshi (September), and underpinned many voice agents in 2025–2026.
Usage note
In voice-AI products 'speech-to-speech' can also mean voice conversion, re-speaking a recording in a different voice while keeping its words and delivery (see voice cloningvoice cloning, n.: Using AI to copy a specific person's voice from recordings, so a computer can say new things that sound like them — useful for dubbing and accessibility, and abused in scams.). Many deployed voice agents still use cascades because they are easier to control and audit.
Related
- audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words.
- speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video.
- text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps.
- neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece.
- voice activity detectionvoice activity detection, n.: Detecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk.
- latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system.
- voice cloningvoice cloning, n.: Using AI to copy a specific person's voice from recordings, so a computer can say new things that sound like them — useful for dubbing and accessibility, and abused in scams.
- text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps.
Sources
- Jia, Y., Weiss, R. J., Biadsy, F., Macherey, W., Johnson, M., Chen, Z. & Wu, Y. (2019). Direct Speech-to-Speech Translation with a Sequence-to-Sequence Model. Interspeech 2019. arxiv.org/abs/1904.06037
- Défossez, A. et al. (2024). Moshi: A Speech-Text Foundation Model for Real-Time Dialogue. arXiv:2410.00037. arxiv.org/abs/2410.00037
Mentioned in
text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps., audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words., conversational AIconversational AI, n.: The technology that lets computers hold conversations with people in everyday language, by text or voice. It powers chatbots, voice assistants and automated customer-service lines., multimodal AImultimodal AI, n.: AI that can take in or produce more than one kind of information, such as text, images, sound and video, so you can show it a photo and ask questions about it., multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type., streamingstreaming, n.: Sending an AI's answer piece by piece while it is still being written, so the words appear on screen one after another instead of all at once at the end., voice activity detectionvoice activity detection, n.: Detecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk., voice agentvoice agent, n.: An AI system you talk to out loud, for example on a phone call, that listens, answers in a natural-sounding voice and can carry out tasks such as booking appointments or checking an order., neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece.