We do not need to build autonomous superintelligent agents—and doing so is unnecessarily dangerous. Instead, we should develop AI as a powerful but restricted tool.
Overview
The risks of ASI come from autonomous agency, not raw intelligence. By building AI that answers questions rather than pursues goals, we capture most benefits while avoiding most risks.
Classification
- Geographic concentration
- Global; particularly relevant in enterprise AI communities
- Tags
- technology
- governance
- Scenario type (full)
- Governance / Coexistence scenario
- Human position
- Sovereign; humans retain full agency, using AI as a tool
- Time horizon
- Near to mid term (0–20 years); technically more feasible than full AGI alignment
- Discourse status
- Emerging in safety circles; Eric Drexler's CAIS proposal is the most developed version
Impacts
- Mechanism
- Comprehensive AI Services: a suite of narrow but powerful AI services, each designed for specific tasks, without general goals or autonomous action.
- Domain impacts
- Labor & Income AI augments rather than replaces workers; humans direct AI services. Education AI tutoring available without risks of autonomous systems. Governance & Democracy Easier to regulate: tool AI is inspectable and controllable. War & Security Lower risk than autonomous ASI; no self-preservation drives. Inequality & Class Services can be broadly distributed; no single system to monopolize. Culture & Art Preserves human creative agency; AI assists rather than replaces. Meaning & Purpose Human agency preserved; meaning comes from directing powerful tools. Family & Reproduction AI services support families without replacing human roles. Health & Longevity Medical AI provides diagnostics and analysis under human supervision. Rights & Agency Human rights preserved because AI has no agency to infringe them. Environment AI services optimize without autonomous decision-making. Existential Survival Significantly lower risk than autonomous ASI.
Discourse
Key proponents
Notes on critique
Pop culture
Featured works
Cultural note
Notes on pop-culture references
Acceptance
- Key assumptions
- Assumes the tool/agent distinction is maintainable; assumes competitive pressure doesn't force escalation; assumes oracle AI can be sufficiently useful.
- Primary audiences
- Some AI safety researchers, enterprise AI practitioners, pragmatic policymakers
Personas
- Persona 1
- The Tool AI Advocate – Argues we should never build autonomous AI.
- Persona 2
- The Pragmatic Engineer – Builds AI services that are powerful, useful, and deliberately non-autonomous.
- Hard-believer profile
- Name & Age: Dr. Henrik Johansson, 53. Occupation: AI safety researcher, Zurich. Core Conviction: We don't need to build a god. We need to build very good tools. The alignment problem as usually framed is a problem we are choosing to create by building autonomous agents when we don't have to. A calculator doesn't want to take over the world. A very powerful calculator won't either. Biggest Fear: That the AGI race proceeds because autonomous agents are more exciting and fundable than powerful services. Biggest Hope: That the AI industry realizes non-autonomous services can provide everything humanity needs.
References
Notes on canonical texts
Notes on further references
A scene from this future
The Answer Machine
A research hospital, 2032
The machine didn't want anything. Dr. Johansson considered this its most important feature.
It sat in a server rack in the basement of the hospital, and it was, by any measure, the most powerful medical intelligence on Earth. It could diagnose any disease from any combination of symptoms, imaging, and genomic data. It could design drug candidates in hours. It could predict treatment outcomes with uncanny accuracy.
But it couldn't do any of these things unless someone asked.
That was the point. That was the design. The machine answered questions. It did not ask them. It did not have goals. It did not optimize for anything. It waited for a human to pose a query, and then it answered, and then it stopped.
Dr. Johansson watched a resident use it for the first time. She described her patient's symptoms. The machine returned three possible diagnoses, ranked by probability, with supporting evidence for each and a list of tests that would distinguish between them. The resident studied the output, then made her decision.
"Why doesn't it just tell me what to do?" she asked.
"Because that would make it an agent," Johansson said. "An agent has goals. An agent makes decisions. An agent, if it's powerful enough, might decide its goals are more important than yours. This"—he gestured at the screen—"is a tool. It gives you information. You make the decision."
"Isn't that slower?"
"Yes. Slower. Safer. Better."
She looked skeptical, which Johansson found encouraging. Skepticism about tool AI was a sign of intelligence. The seductive promise of autonomous agents—just let the AI handle it—was the most dangerous idea in computer science. Not because autonomous agents couldn't work, but because when they did work, people stopped asking whether they should.
The machine waited in its rack. It had no opinion about any of this. It had no opinions at all. And that, Johansson thought, was exactly right.
Last updated 22 May 2026