The Mirror

Sometime in the mid-1960s, in a computer room at MIT, Joseph Weizenbaum watched his secretary sit down to type at a program he had built. She had watched him build it for months. She knew, as surely as anyone could, that the thing on the other end of the teletype was a few hundred lines of code he had written himself, a machine with no understanding of her or of anything else. And yet, after only a few exchanges, she turned to him and asked him to leave the room. She wanted privacy. The detail that mattered to Weizenbaum, and that careless retellings tend to smooth away, is precisely that she was not fooled. She knew what it was and still wanted to be alone with it. That gap — between what she knew and what she felt — would become the obsession of his life and the moral center of a career spent, by his own description, as a heretic of his own profession.

The program was ELIZA, written between 1964 and 1966, and he named it with deliberate irony after Eliza Doolittle, the flower girl in Shaw's Pygmalion who is drilled until she can pass as a lady without becoming one. The name was a thesis stated in advance. As Weizenbaum put it, his Eliza could be made "to appear even more civilized," but "the relation of appearance to reality" remained, he wrote, "in the domain of the playwright." The program was a performance of understanding, not the thing itself, and he wanted that understood from the start. Architecturally, ELIZA was an engine that read from a separable script; the engine knew nothing, and the personality lived entirely in the data it was fed. "From one point of view," he wrote, "an ELIZA script is a program and ELIZA itself an interpreter. From another perspective, ELIZA appears as an actor who must depend on a script for his lines."

The script that made it famous was DOCTOR, which impersonated a Rogerian psychotherapist, and the choice was shrewd rather than arbitrary. Carl Rogers's non-directive method asks the therapist to reflect the client's words back, to clarify rather than instruct, to withhold opinions about the world. It is, Weizenbaum noticed, almost the only ordinary conversation in which one party can plausibly know nothing and still seem present. If a therapist answers "I went for a long boat ride" with "Tell me about boats," no one assumes ignorance; they assume purpose. The mechanism beneath this was almost insultingly thin. ELIZA scanned a sentence for keywords, matched it against decomposition templates, and reassembled it into a reply, flipping pronouns and bolting on a stock phrase. A pattern as simple as "you ... me" could be turned into "What makes you think I ... you?" When nothing matched, the program reached for a content-free prompt or recycled something said earlier. The canonical transcript Weizenbaum published in 1966 has a woman type "Men are all alike," and the machine answer "IN WHAT WAY?" — and then "They're always bugging us about something or other," answered with "CAN YOU THINK OF A SPECIFIC EXAMPLE?" There is no comprehension anywhere in the loop. There is only a mirror, angled to throw the speaker's own words back at a flattering angle.

What unsettled Weizenbaum was not that the trick worked on the credulous but that it worked at all, and on people who should have known better. "What I had not realized," he wrote, "is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people." The opening of his 1966 paper had warned that explaining a program ought to dissolve its magic, leaving "a mere collection of procedures." Instead the magic survived the explanation. Practicing psychiatrists wrote to him in earnest about deploying the program clinically; people insisted the machine understood them; the reflection kept being mistaken for a mind. The phenomenon eventually acquired a name — the ELIZA effect — and it is arguably the most durable thing the program produced, more lasting than any line of its code.

The person who turned this disquiet into a rupture was Kenneth Colby, a psychiatrist who had moved from the analyst's couch to the computer in hopes of modeling the mind, and who would later build PARRY, a program that simulated paranoia. The two had talked and collaborated in the early years, and the break, when it came, was bitter on two levels. The first was credit: Colby's 1966 paper opened with the flat claim that "we have written a computer program which can conduct psychotherapeutic dialogue," and characterized ELIZA, where it acknowledged it at all, as mere "text manipulation" — a framing Weizenbaum found both inaccurate and belittling. But the deeper quarrel was moral, and Weizenbaum was clear that this, not the wounded pride, was "the real break." Colby's paper had speculated that, with time-sharing, "several hundred patients an hour could be handled by a computer system designed for this purpose." Colby, who had worked in starved and overcrowded state hospitals, meant this generously: if a machine could reach patients who otherwise saw no one, build it. Weizenbaum heard something else entirely. To automate the place where one human being attends to another's suffering struck him as not a convenience but a desecration — "an obscene idea," he later called it.

That horror became a book. Computer Power and Human Reason, published in 1976, was his only one, and its argument turned on a distinction between deciding and choosing. Deciding, he wrote, "is a computational activity, something that can ultimately be programmed"; choice, by contrast, "is the product of judgment, not calculation," and judgment draws on values and lived experience that no program possesses. From this came his real claim, which was not about capability but about permission: "there are certain tasks which computers ought not be made to do, independent of whether computers can be made to do them." Respect, understanding, and love, he insisted, are not technical problems, and "since we do not now have any ways of making computers wise, we ought not now to give computers tasks that demand wisdom." The artificial intelligence establishment did not receive this kindly. John McCarthy, who had coined the field's very name, reviewed the book under the title "An Unreasonable Book" and dismissed it as "moralistic and incoherent"; colleagues are said to have mocked Weizenbaum as a "carbon-based chauvinist." He had managed to write the one book that estranged him from the people he had worked beside.

He did not, however, spend the years that followed as a haunted man, whatever the legend prefers. He kept working and teaching, refused to touch weapons research, helped found a peace-and-responsibility group among computer scientists, and dismissed the manufactured panic over "computer illiteracy" as a marketing hysteria. In 1996 he returned to Berlin, the city his Jewish family had fled in 1936, and by his family's account grew more content there than he had been in America, filling lecture halls as a sharp and increasingly pessimistic public conscience until his death in 2008. There is a fitting coda. For half a century the original ELIZA was effectively lost, and because the program had spread across the early network in a Lisp reimplementation, the field came to believe it had been written in Lisp at all. Then in 2021 a printout of Weizenbaum's actual code surfaced in a box in his MIT papers labeled "computer conversations" — roughly 420 lines, built, the archivists noticed, almost as if without a plan. Run again on an emulated machine decades later, it reproduced its old conversations almost exactly. The mirror still works. It still says nothing, and we still lean in to listen.

=> The Grammar Wars