Supporters of Marcus Endicott’s Patreon can access weekly or monthly consultations on this topic.
By the turn of the millennium, the neural network had become an embarrassment. The field had tried it, found it slow and temperamental, and moved on to cleaner machinery — support vector machines, kernel methods, the elegant convex guarantees that let a researcher prove things rather than merely hope. To still be training multi-layer networks in 2000 was to advertise that you had not gotten the memo. Yoshua Bengio had not gotten the memo. Working in Montreal, in a lab he had founded a few years earlier and would eventually grow into one of the world's great centers of machine learning, he belonged to a small and unfashionable confraternity — alongside Geoffrey Hinton in Toronto and Yann LeCun in New Jersey — who kept building neural networks through the years when almost no one else would. They would later call the period a winter. Bengio has credited Canada's tolerance for curiosity-driven research with letting him do unfashionable work in plain sight, and the work he chose was language.
The problem he set himself was the one that had quietly defeated everyone before him. A language model, in the statistical sense inherited from Shannon, is a machine for assigning probabilities to sequences of words — for knowing that "the cat sat on the mat" is a likelier English sentence than "mat the on sat cat the." The reigning method counted: tabulate how often each short run of two or three words appeared in a vast corpus, and use those frequencies to predict what comes next. These n-gram models worked, after a fashion, and powered the speech recognizers and translation systems of the 1990s. But they were brittle in two specific ways that Bengio could name precisely. They could not see past a word or two of context, because the tables required to track longer spans grew impossibly large. And they treated every word as an atom, a discrete symbol bearing no relation to any other. To such a model, "cat" and "dog" were as unrelated as "cat" and "Tuesday." Having seen a sentence about a cat taught it nothing whatsoever about dogs.
Bengio gave this failure a name borrowed from an older corner of mathematics: the curse of dimensionality. To model the joint probability of, say, ten consecutive words drawn from a vocabulary of a hundred thousand, one faced a space of roughly ten-to-the-fiftieth possible combinations — more sequences than could ever be observed, so that almost any real sentence a model encountered in the wild would be one it had never seen in training. Counting could not save you from a space that vast. The only escape was to generalize, to make a sentence the model had seen lend its probability to the countless similar sentences it had not. And that required a way for the machine to know that two sentences were similar in the first place.
His answer was deceptively simple to state and consequential beyond anything its modesty suggested. Instead of treating each word as an isolated symbol, assign to every word a vector of real numbers — a point in a continuous space of perhaps thirty or a hundred dimensions — and let the network learn where each word should sit. Words that behave alike would drift toward one another. Then express the probability of a sentence not in terms of the words themselves but in terms of these learned coordinates, and train the whole apparatus at once, so that the network simultaneously discovered the right place for each word and the right way to combine those places into predictions. The illustration he offered has become the canonical one. Suppose the model has learned, from training, that "The cat is walking in the bedroom" is a plausible English sentence. It should then assign reasonable probability to "A dog was running in a room," a sentence it has never encountered — simply because it has come to place "dog" near "cat," "a" near "the," and "room" near "bedroom," and so recognizes the new sentence as a near relation of the old. A single observed sentence could now inform the model about an exponential number of unseen neighbors. Generalization, the thing counting could never buy, fell out of the geometry.
These coordinates would later acquire a name that has since become ordinary: word embeddings. The underlying intuition was not wholly new, and Bengio said so, tracing it to Hinton's work in the 1980s on distributed representations, where concepts were encoded not as single units but as patterns spread across many, and to the recurrent networks with which Jeffrey Elman had coaxed grammatical structure out of raw prediction. What Bengio's group did was weld that old connectionist idea to the hard statistical problem of modeling real word sequences at scale, and show, on the Brown corpus and on millions of words of Associated Press news, that the resulting model genuinely outperformed the best n-gram systems of the day. The neural network's predictions were measurably less perplexed by held-out text — by roughly a quarter on one corpus, less on the other — and the gains were largest when the two approaches were blended.
The honesty of the paper extended to its cost, and the cost was brutal. Computing a probability meant, at the final step, evaluating a score for every word in the vocabulary, an operation that consumed nearly all of the network's running time. Training the larger model required spreading the work across a cluster of forty processors connected by a fast network, and even then it ran for more than three weeks to complete a mere handful of passes over the data. This is the detail that explains everything about what happened next, and it is why the chapter's title is finally a half-truth worth correcting. The idea was not unread. Within the small community that cared about neural language models, it was known and built upon almost immediately — adapted for speech recognition within two years, extended by Hinton's students, and carried forward by the researchers who, later in the decade, would establish pre-trained embeddings as a general tool for natural language. The paper was respected. What it lacked was not readers but a world ready to run it. The hardware was punishing, the available text too thin, and the broader field still in love with methods that neural networks could not yet beat on anything that paid. The drama is not that the paper sat unopened. It is that the idea sat waiting for the machines to catch up to it.
They caught up around 2012, when graphics processors and large datasets arrived together and the long-dormant neural program detonated into the dominant approach in all of artificial intelligence. The most direct vindication came the following year, when a young researcher named Tomáš Mikolov released a method called word2vec that made Bengio's embeddings cheap and fast enough to compute at enormous scale, producing the now-famous party trick in which the vector for "king," minus "man," plus "woman," lands near "queen." Mikolov was forthright about the lineage: he had read Bengio's paper, understood that good word vectors could be learned from rather simple models, and set out to do exactly that. From there the principle spread until it became invisible through ubiquity. Every modern system that processes language — every translator, every search engine, the transformer architecture itself, and the models that would eventually answer questions in plain prose for a hundred million people — rests on the same foundational move Bengio's paper made: words are not discrete symbols to be counted but dense vectors to be learned, points in a space whose geometry encodes meaning. The future-work section of that 2003 paper, with its sketches of hierarchical outputs and recurrent context and sampling tricks to tame the cost, reads in hindlight less like a list of loose ends than like a table of contents for the fifteen years that followed.
Bengio shared the field's highest honor, the Turing Award, in 2018, with Hinton and LeCun, the three holdouts of the winter crowned together. By 2025 he had become the first living scientist whose work had been cited more than a million times. But the neural probabilistic language model remains the purest emblem of his particular kind of patience — a correct idea, fully formed, set down in the open at a moment when the world had neither the appetite nor the silicon to use it, and left to wait. It was not that nobody read it. It was that the answer arrived a decade before the question everyone else was asking.