How voice typing works

How speech recognition turns sound into words

Why speech is hard for computers, how classic recognizers with acoustic, pronunciation and language models worked, what end-to-end models (CTC, encoder-decoder, transducers) changed, and why streaming text can change while you speak.

MacWindowsiPhoneAndroid

In short

Speech has no clear gaps between words, and everybody pronounces things differently, so a recognizer never "reads" words off the audio. It estimates the most likely word sequence from the sound and from patterns learned in training. Older systems used separate acoustic, pronunciation and language models; most modern ones learn the whole path from audio to text in one network.

Messages

You said

i scream you scream we all scream for ice cream

SayVelio typed

I scream, you scream, we all scream for ice cream.

Why recognizing speech is hard

Written text has spaces between words. Speech doesn't. When you say "ice cream", the sounds run together, and "I scream" sounds almost the same. Sounds also change depending on their neighbors, people speak at different speeds and pitches, and accents shape every vowel.

So the same stretch of audio can fit several word sequences. A recognizer doesn't decode words with certainty: it weighs the options and picks the most probable one. That's why a result can read perfectly well and still be wrong.

The classic approach: three models and a decoder

For decades, speech recognition was built from separate parts:

  • An acoustic model linked short slices of audio to speech sounds (phonemes).
  • A pronunciation dictionary listed how each word can be pronounced as a sequence of those sounds.
  • A language model scored how likely a word sequence is in the language ("ice cream" after "we all scream for" is far more likely than "I scream").
  • A decoder searched through the combinations to find the best overall match.

This design is still the clearest way to understand the problem, and some systems use parts of it. But it isn't how most modern recognizers work.

Classic recognizers combine acoustic, pronunciation and language models; end-to-end models learn the whole path in one network
Two ways to build a recognizer: separate models with a decoder, or one end-to-end network

End-to-end models: one network from audio to text

Modern systems learn a direct mapping from audio to text in a single neural network trained on huge amounts of transcribed speech. There are three common families:

  • CTC (connectionist temporal classification) predicts a character or word piece for every short audio frame, including a special "blank" symbol, then merges repeats and removes blanks. It doesn't need anyone to mark where each word starts in the training audio.
  • Attention-based encoder-decoder models first turn the whole recording into an internal representation, then write the text piece by piece, "looking back" at the relevant parts of the audio for each next piece.
  • Transducers (RNN-T and similar) combine an audio encoder with a component that remembers the text produced so far, which makes them well suited to producing text while you're still talking.

Real products mix these ideas, and some add extra steps such as context hints for rare words (see how context helps recognition). An app that offers dictation doesn't tell you which architecture it uses, and it doesn't need to: what matters to you is accuracy, speed and what happens to your data.

Whole recordings vs streaming

Recognition can run on the complete recording after you stop, or continuously while you speak:

  • Streaming shows words almost immediately, but early guesses can change as later words add context. "Recognize speech" may briefly appear as "wreck a nice beach" and then correct itself.
  • Whole-recording recognition waits until you're done, but sees the full sentence at once, which often helps with accuracy and punctuation.

The delay you notice also includes detecting that you've stopped, network time and any cleanup step. In SayVelio you hold the key while you speak, so the end of the recording is exact, and the text is inserted once it's final, not word by word.

Why the output is a probability, not a fact

Because the system always picks the most probable option, its mistakes look plausible: a real word, spelled correctly, that you didn't say. Names, numbers and rare terms are most at risk, because the model has seen them less often. Two practical defenses:

  1. Teach the system your words with a custom dictionary.
  2. Read important text before you send it, especially amounts, dates and "not".

Questions

Does speech recognition understand what I mean?

Not in the human sense. It estimates which words were said. Meaning comes in only through patterns learned from text, which helps it choose between similar-sounding phrases.

Why does my dictation get a word wrong every time?

Usually because the word is rare or sounds like a more common one. Adding it to your dictionary, with how it sounds as a variant, fixes most of these cases.