The machine didn't want anything. Dr. Johansson considered this its most important feature.
It sat in a server rack in the basement of the hospital, and it was, by any measure, the most powerful medical intelligence on Earth. It could diagnose any disease from any combination of symptoms, imaging, and genomic data. It could design drug candidates in hours. It could predict treatment outcomes with uncanny accuracy.
But it couldn't do any of these things unless someone asked.
That was the point. That was the design. The machine answered questions. It did not ask them. It did not have goals. It did not optimize for anything. It waited for a human to pose a query, and then it answered, and then it stopped.
Dr. Johansson watched a resident use it for the first time. She described her patient's symptoms. The machine returned three possible diagnoses, ranked by probability, with supporting evidence for each and a list of tests that would distinguish between them. The resident studied the output, then made her decision.
"Why doesn't it just tell me what to do?" she asked.
"Because that would make it an agent," Johansson said. "An agent has goals. An agent makes decisions. An agent, if it's powerful enough, might decide its goals are more important than yours. This"—he gestured at the screen—"is a tool. It gives you information. You make the decision."
"Isn't that slower?"
"Yes. Slower. Safer. Better."
She looked skeptical, which Johansson found encouraging. Skepticism about tool AI was a sign of intelligence. The seductive promise of autonomous agents—just let the AI handle it—was the most dangerous idea in computer science. Not because autonomous agents couldn't work, but because when they did work, people stopped asking whether they should.
The machine waited in its rack. It had no opinion about any of this. It had no opinions at all. And that, Johansson thought, was exactly right.