Skip to content
Last Dictionary
Plane
Vision, Speech & PerceptionAI Engineering & MLOps

speaker diarization

nounAlso: diarization, diarisation, speaker diarisation, who spoke when, diarization error rate, DER, pyannote

In plain wordsWorking out who spoke when in a recording, splitting a meeting or phone call into turns labeled Speaker 1, Speaker 2 and so on, without necessarily knowing who the speakers are.

  1. The task of answering 'who spoke when' in an audio recording: dividing the speech into stretches and grouping together those produced by the same voice, so that each speaker receives a consistent label, anonymous unless matched to enrolled voices. A classic pipeline applies voice activity detectionvoice activity detection, n.: Detecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk., splits speech into short segments or detects speaker changes, computes a speaker embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. for each segment (i-vectors, then neural x-vectors and ECAPA-TDNN embeddings), and clustersclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size. the embeddings, agglomeratively or spectrally, to decide how many speakers there are and which segments belong together, before refining the boundaries. Overlapping speech, frequent in meetings, defeats pure clustering, so end-to-end neural diarization (from 2019) and hybrid systems such as pyannote.audio predict several active speakers per frame. Accuracy is measured by diarization error rate (DER): the fraction of speech time that is missed, falsely detected or attributed to the wrong speaker. Combined with speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video., diarization yields speaker-attributed transcripts of meetings, interviews, call-center conversations, broadcasts and medical consultations.

    Diarization merges the two interviewers whenever they talk over each other, so questions get attributed to the wrong person.

DER=Tfalse alarm+Tmissed+TconfusionTtotal speech\text{DER} = \frac{T_{\text{false alarm}} + T_{\text{missed}} + T_{\text{confusion}}}{T_{\text{total speech}}}

Origin

From 'diary', a record of events. The term became established in the 2000s through the US NIST Rich Transcription evaluations of broadcast news, telephone and meeting speech, which scored 'who spoke when' separately from transcription; its earliest use has not been traced. The DIHARD challenges (from 2018) later targeted difficult real-world audio.

Usage note

Diarization groups speech by speaker but does not identify people; naming them requires speaker verificationspeaker verification, n.: Checking whether a voice really belongs to the person it claims to be, as when a bank confirms your identity by the sound of your voice on the phone. or identification against enrolled voices, with the privacy implications that brings.

Related

See also
  • speaker verificationspeaker verification, n.: Checking whether a voice really belongs to the person it claims to be, as when a bank confirms your identity by the sound of your voice on the phone.
  • voice activity detectionvoice activity detection, n.: Detecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk.
  • speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video.
  • clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size.
  • embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
  • sound event detectionsound event detection, n.: Teaching computers to recognize everyday sounds in a recording, such as a dog barking, glass breaking or a siren, and to say exactly when each one starts and stops.
Contrast with
  • speaker verificationspeaker verification, n.: Checking whether a voice really belongs to the person it claims to be, as when a bank confirms your identity by the sound of your voice on the phone.

Sources

  1. Anguera, X., Bozonnet, S., Evans, N., Fredouille, C., Friedland, G. & Vinyals, O. (2012). Speaker Diarization: A Review of Recent Research. IEEE Transactions on Audio, Speech, and Language Processing 20(2), 356–370.
  2. Park, T. J., Kanda, N., Dimitriadis, D., Han, K. J., Watanabe, S. & Narayanan, S. (2022). A Review of Speaker Diarization: Recent Advances with Deep Learning. Computer Speech & Language 72.

Mentioned in

speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video., speaker verificationspeaker verification, n.: Checking whether a voice really belongs to the person it claims to be, as when a bank confirms your identity by the sound of your voice on the phone., voice activity detectionvoice activity detection, n.: Detecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk., source separationsource separation, n.: Pulling apart a mixture of sounds into its separate parts, such as isolating one voice in a noisy party or the vocals, drums and bass in a finished song.