Conformer

A neural-network encoder architecture for speech that combines self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to., which models long-range dependencies across an utterance, with convolutionconvolution, n.: An operation that slides a small grid of weights across an image or signal, multiplying and adding at each spot, to measure how strongly a pattern appears at every location., which captures local acoustic patterns. Each Conformer block places a multi-head self-attention module (with relative positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model.) and a convolution module (pointwise convolution with a gated linear unit, depthwise convolution, batch normalization and a Swish activation) between two half-step feed-forwardfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge. modules, a 'macaron' arrangement, with residual connectionsresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable. and layer normalization throughout; convolutional layers first downsample the input log-mel features in time. Introduced for speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video., it reported 2.1% and 4.3% word error rateword error rate, n.: The standard score for speech recognition: the share of words a system gets wrong compared with a correct transcript, counting wrong, missing and extra words. Lower is better. on LibriSpeechLibriSpeech, prop. n.: A free collection of about 1,000 hours of volunteers reading English audiobooks, matched with the book text; it became the standard yardstick for speech-recognition systems. test-clean and test-other without an external language model, better than transformer-only and convolution-only models of similar size. It became a default encoder in production and open-source recognizers, paired with CTCconnectionist temporal classification, n.: A training method that lets a network learn to transcribe speech or handwriting without being told exactly where each letter or sound begins and ends., transducer or attention decoders, and is also used for speech translation and other audio tasks.

Our streaming recognizer is a Conformer-transducer with limited look-ahead to keep latency under 300 ms.