James Baker and Fred Jelinek Bring Hidden Markov Models to Speech Recognition

The 1970s is known as a "winter" for the learning tradition of AI in the West. But even brutal winters have the occasional bearable day.

Learning research in these early days was still disjointed (and would be for decades). In speech recognition, Markov models and neural nets were first rivals, then partners. By around 2012 systems started using them both. But back in the 1970s, they were sub-rivals within the learning tradition. You must remember that "neural nets" were widely discredited still at this time, largely thanks to Marvin Minsky's Perceptrons.

But what are these things? A Markov model is a simple rule that what comes next depends only on what came just before. The odds are taken from counting real examples. But the catch is they can’t see far back and don't understand anything. Neural nets are the smarty pants that can understand. They learn their own patterns from data, so they can find much richer relationships and use much longer context.

Hidden Markov models taught machines from piles of recordings to predict what most likely came next. Jelinek’s IBM team carried that approach into statistical translation and next-word prediction. Today’s large language models still do the same basic job of predicting the next word from probabilities learned from data, only over far longer stretches of text, to put it lightly.

The disjointed nature of AI research in these days meant this breakthrough came from researchers in three places and took years. First, James Baker at Carnegie Mellon described DRAGON, a speech recognizer built on “a probabilistic function of a Markov process.” Then, in 1976 Fred Jelinek of IBM Research published statistical methods for recognizing continuous speech. Both built on theory from Leonard Baum and colleagues at the Institute for Defense Analyses. Yes, military funding always seems just around the corner from AI research.

In this further blow to our poor neural nets, Markov models would remain speech recognition’s preferred method well into the 2000s. Minsky, you devil.