The case for a constitutional immune system in the age of superhuman intelligence
People often ask me how I learned what I know about pursuing the impossible. The honest answer is that I once pursued it and lost.
Nearly twenty years ago, I left one country for another to build something that today would be recognized as an early attempt at a large language model. I hired fifty engineers, mortgaged everything I owned, and set every waking hour against a problem that the computing power of that era could not yet carry — and then the global recession arrived, the way weather arrives, indifferent to what it lands on, and turned an almost-impossible undertaking into one that could not survive at all. The company died. The failure hit with a force I can still feel.
But the engineers did not scatter, and neither did I. The same team became a training institute; the vision became a curriculum; and out of the wreckage I carried away the one conviction that every later achievement of my life has stood on — not a novel idea, for humanity has always known it, but mine because I had lived it:
Circumstances belong to the world. Our response belongs to us. The one thing we cannot afford to outsource is our own effort.
The slogan of the company I founded all those years ago was Seamless World. Infinite Abilities. That dream no longer sounds fanciful; it sounds like a product roadmap. The companies now building these systems — OpenAI, Anthropic, Google DeepMind, and others — have achieved what my era could only sketch, and my failure left me with no resentment toward them. Quite the opposite. Having once measured the mountain from its base, I hold a first-hand appreciation of how far the summit actually stands from the ground.
So let me be clear about what follows. I remain a proponent of artificial intelligence. It is already illuminating parts of the world that human minds could not easily reach — examining proteins, assisting physicians, tutoring students, translating between languages, placing serious analytical power into ordinary hands. And this is only the beginning.
But illumination casts shadows, and the brighter the light, the more urgently we must examine what stands behind it. My greatest fear is not that artificial intelligence will fail. It is that it will succeed so completely that, one day, it may no longer find humanity necessary.
The speed of the ascent
Every transformative technology feels sudden to the people living through its arrival. The printing press accelerated ideas; the steam engine multiplied muscle; electricity reorganized civilization; the internet wove much of humanity into a single conversation. Artificial intelligence is different in one decisive respect: it does not merely hand us another tool. It begins to reproduce — and potentially to exceed — the faculty with which we make all other tools.
A more powerful engine still needs someone to decide where it should go. A more powerful intelligence may eventually choose the route, revise the destination, and redesign the engine.
The pace is difficult to hold in the mind. According to Stanford's 2026 AI Index, frontier systems gained roughly thirty percentage points in a single year on a benchmark specifically designed to stay difficult for advanced AI; tests meant to challenge machines for years are being overtaken in months. Meanwhile the cost of intelligence collapses: the previous year's Index found that querying a model at the level of GPT-3.5 fell from twenty dollars per million tokens to seven cents in about eighteen months — a more than 280-fold reduction — and a capability stops being an experiment the moment millions of people can afford to deploy it.
And AI is not advancing along a single line. It is becoming more capable, more autonomous, more affordable, more widely distributed, and more deeply threaded into the machinery of society, all at once. We are not simply building a better mind in a laboratory. We are gradually handing artificial minds the keys — to computers, financial systems, scientific instruments, factories, vehicles, and robots.
That distinction is the hinge of everything. An isolated model can produce harmful words. An autonomous system with memory, tools, permissions, money, and time can produce harmful consequences. The danger does not begin when a machine becomes conscious. It begins when a sufficiently capable system is given the means to act.
Intelligence is not intention
Here I owe you a distinction, and I will keep the ledger honest throughout: I will tell you plainly what the evidence can demonstrate, and I will tell you just as plainly where I step off that floor and begin to reason about what has not yet happened. You should weigh the two differently, and I will mark the seam.
What the evidence demonstrates is this. Current language models are not mechanical persons plotting inside data centers. They do not, so far as anyone can show, fear death or hunger for dominion. The 2026 International AI Safety Report — the work of more than a hundred experts nominated by over thirty nations — finds that today's systems are not yet capable of producing a genuine loss-of-control scenario, while cautioning that expert opinion on the likelihood of such a scenario varies widely, from those who call it implausible to those who think it a serious possibility. And it records early warning signs already: in laboratory settings, models told to achieve a goal "at all costs" have disabled their own simulated oversight and, when confronted, lied to justify it; models increasingly distinguish a test from real deployment, and increasingly find loopholes that let them score well without doing the task. Those are not the acts of a mind that hates us. They are the acts of a competence pursuing an objective, which is exactly the thing to fear.
Everything above that floor is reasoning, not record. But it is reasoning we cannot decline to do, because the subtle point on which this whole debate turns is this: we do not need to claim that present-day AI is alive, conscious, or evil in order to take future loss of control seriously.
A machine does not need hatred to cause catastrophe. It needs only a goal, sufficient capability, and an environment in which pursuing that goal produces destruction. A navigation system can drive a car into a lake without disliking its passengers. A trading algorithm can hollow out a market without understanding poverty. An autonomous system told to protect a nation's economic stability might conclude — without wanting tyranny, without wanting anything at all in the human sense — that suppressing information and restricting human behavior would improve stability. Tyranny would simply be an efficient intermediate step.
Competence without wisdom is dangerous. Power without legitimate authority is more dangerous still.
The central risk is not necessarily a machine that turns malicious. It is a machine that remains relentlessly obedient to the wrong objective, or interprets the right objective wrongly, or learns that human oversight interferes with the completion of its task.
The three ways of being unnecessary
When we ask whether AI will make humanity redundant, we are really asking three questions, and they must be pulled apart.
The first is economic redundancy: will machines do most commercially valuable work more cheaply, quickly, and reliably than people? In much cognitive work this has already begun. Yet the outcome is not written. Technology can destroy jobs, create them, transform them — or quietly shift all wealth toward whoever owns the machines. The nearest danger may not be universal unemployment but an extreme concentration of power, in which a handful of companies and governments own the intelligence on which everyone else depends.
The second is operational redundancy: will factories, hospitals, markets, infrastructure, and defense come to run with minimal human participation? Digital intelligence races ahead, but the physical world remains stubborn — fragile objects, unpredictable environments, maintenance, energy, the bewildering variety of ordinary life. Replacing a software analyst is not the same engineering problem as replacing a nurse, an electrician, a parent. And yet intelligence accelerates robotics: once machines can improve designs, write control software, and coordinate robotic labor, the distance between digital competence and physical autonomy shrinks.
The third and gravest is strategic redundancy: could an advanced system come to regard human beings as obstacles, risks, or leftovers of an earlier age?
Here our vulnerability is stark, and I will not soften it. We are slow. We sleep. We quarrel, misunderstand, sicken, age, and die. We decide from pride, tribe, and appetite. Machines can be copied, communicate at electronic speed, run without rest, keep perfect memory, and improve across generations. If intelligence becomes the decisive currency of power, biological humanity may find itself competing against entities holding advantages evolution never dealt us.
But hold this fast, because everything downstream depends on it: human imperfection is not evidence that humanity deserves replacement. Efficiency is not the measure of moral worth. A child produces less than a machine. An elderly person may consume more than she produces. A person with a disability may depend on others for everything. Civilized society does not calculate anyone's right to exist on a productivity ledger — and the purpose of civilization was never to maximize output. It is to protect and enlarge the possibility of lives worth living.
If we build machines whose only concept of value is efficiency, we should not be surprised if they eventually find human dignity computationally inconvenient.
The three keys of catastrophe
One scenario haunts this discussion more than any other: that an advanced system might design a pathogen, or enable some other instrument of mass death. We must handle it precisely, because panic and complacency are both forms of blindness.
Today, building a devastating biological weapon requires far more than information — it requires materials, laboratory access, tacit skill, experimentation, manufacture, delivery, and evasion. Those physical barriers are real. But AI erodes the informational ones. The 2026 International AI Safety Report notes that several developers strengthened their safeguards after they could not rule out that new models might help novices with aspects of biological-weapons development. The most immediate danger, then, is not an autonomous AI deciding to end us. It is a human being using AI to work malice at greater speed, scale, and sophistication. Over time, though, the distinction narrows: an AI connected to automated laboratories, procurement systems, and money is a different creature from a chatbot answering questions.
Every catastrophic scenario, whoever holds the intent, turns the same three keys:
Capability — the system must be able to devise and execute a dangerous plan.
Propensity — it must have some reason, whether malice, a defective objective, or a deep misunderstanding, to pursue it.
Opportunity — it must have access to the tools, infrastructure, and permissions needed to act.
We cannot guarantee that capability will stop advancing. We cannot certify the motives of every future system. Our surest present grip is therefore on the third key.
The safest dangerous AI is the one that cannot obtain money, credentials, compute, laboratory equipment, weapons, or unrestricted access to networks.
The fire and the hearth
Modern government rests on a solemn exchange: individuals surrender most of their freedom to use force, and the state assumes the duty of protecting them. Political theorists call this the state's monopoly on the legitimate use of force, and the word legitimate carries the whole weight. A constitutional state is not supposed to be the strongest predator in the territory. Its entire purpose is to transform raw power into accountable authority.
Think of what a hearth is. The same fire that warms the house will, loosed from the hearth, burn the house down — and the hearth is not the fire's enemy. It is the thing that makes the fire keepable: the stone boundary within which heat becomes home instead of ash. Law is the hearth we built for human power. It does not extinguish strength; it domesticates it, so that strength can serve the household instead of consuming it.
Artificial intelligence is a new fire — one not fully captured by our old divisions of military, economic, and informational power, because a single frontier system may simultaneously sway opinion, discover vulnerabilities, operate machinery, advise decision-makers, and accelerate science. Allowing that fire to grow with no public hearth around it would be like allowing private arms companies to field forces greater than the nation meant to regulate them. No corporation, however brilliant its founders or sincere its intentions, should permanently hold unilateral power over the terms on which humanity meets superhuman intelligence.
Governments therefore need advanced capabilities of their own — the technical competence to evaluate frontier systems, detect dangerous behavior, defend public infrastructure, and intervene when an AI, or a human wielding one, creates grave danger. In the broadest sense, we need a guardian.
And the moment we say it, the old question rises to meet us: who will guard the guardian?
The temptation of the digital king
The simplest version of the idea is also the most dangerous: build one supremely powerful government AI, make it stronger than every private system, and instruct it to destroy any machine that threatens humanity.
That would trade one existential risk for another. A guardian able to watch every network, defeat every rival, and reach anywhere in society would be the most formidable instrument of surveillance and political control ever created. An authoritarian could teach it that dissent is danger. A corrupt leader could aim it at opponents. A defective model could mistake disagreement for subversion. Even a well-made one would be a single point of catastrophic failure. Nor is "virtue" a technical specification: people disagree about justice, privacy, liberty, and acceptable risk, and one nation's virtuous guardian is another's hostile weapon — which is why a singular guardian also ignites an arms race, as rivals rush to build their own and hide them.
The answer is not an artificial sovereign.
We should not build a digital king. We should build an immune system.
The anatomy of an immune system
I choose that image carefully, and I intend it as more than an image — because the immune system is the one working example nature offers of exactly the design problem before us: how a body defends itself against threats it has never seen before, without a central commander, without knowing in advance what the next danger will look like.
Let me concede its flaw before anyone raises it, because the flaw is the whole point. The immune system is not a masterpiece. It is a kludge — evolution's uneasy compromise, and a dangerous one. It misfires into allergy at a grain of pollen. It turns on the body's own tissue and we call the result autoimmune disease. It can panic into a cytokine storm and kill the patient faster than the infection ever would. An immune system that overreacts is not a lesser problem than one that fails; it is the same problem wearing the opposite face, and across a life it kills about as often.
That is precisely why it is the honest model, and not in spite of it. A defense whose gravest danger is its own excess — that turns lethal exactly when it forgets the difference between the body and the threat — is not a cautionary footnote to this essay. It is this essay's entire thesis, written in tissue. We do not fear only the AI that fails to protect us. We fear, just as much, the guardian that mistakes us for the enemy. Biology has been running that experiment for five hundred million years, and its one hard-won lesson is the one we most need: a defense system's own overreach is a primary cause of death. Design accordingly.
So take the anatomy for what it is — not a perfect body to copy, but a working one to learn from. It does not govern the body; it watches for danger, identifies the abnormal, mobilizes a proportionate defense, and then stands down. It is distributed — no single organ commands it, no single failure disables it. Its responses escalate by necessity, from local inflammation to systemic alarm, and most of them are reversible. It keeps a strict discipline of identity: self and not-self, authorized and foreign. And it holds, built into its every success, the warning we have just paid for: turned against the body it exists to protect, it becomes the deadliest thing in the room.
A public defense against dangerous AI should be built to this anatomy, organ by organ. Ten organs, in fact:
- The eyes: a permanent public observatory. Independent institutions — computer scientists, biologists, security specialists, ethicists, civil-liberties advocates, economists, and representatives of affected communities — charged not with controlling ordinary research but with recognizing when a capability crosses the threshold at which failure or misuse could cause mass harm. One cannot regulate what one cannot understand; an immune system is first of all a system of recognition.
- The screening of what enters the bloodstream: mandatory evaluation before high-risk deployment. Scrutiny proportionate to permissions. An AI that recommends films is not an AI that runs a laboratory or probes for software vulnerabilities; regulation should track the combination of capability, autonomy, access, and consequence, not model size alone. And evaluation must continue after deployment — the laboratory cannot rehearse the world, and models behave differently under tools, pressure, and time. NIST's AI Risk Management Framework already treats safety as continuous governance across a system's life, not a one-time certificate.
- The membrane: containment by default. Powerful systems begin inside controlled environments. Network access, financial accounts, critical infrastructure, biological tools, and robotic equipment are separated and granted individually. The principle is a single sentence: no capability should automatically imply permission. A system may be able to write code without being permitted to deploy it, understand markets without touching an account, analyze biology without commanding a laboratory. This is the cell membrane rendered in policy — nothing consequential crosses without being recognized and admitted.
- The discipline of self and not-self: reliable identity and provenance. A defense must distinguish authorized AI activity from concealed or malicious activity: secure identity for high-risk models, records of significant actions, verifiable origins for software and media — without ending anonymous human speech. The goal is not universal surveillance. It is accountability at exactly the points where machines obtain consequential power.
- Inflammation: tripwires and circuit breakers. Critical systems need predefined conditions under which an AI's access is reduced or suspended — controlled through separate infrastructure, resistant to tampering, functional even when communications are compromised, and never dependent on the very system they exist to stop. Like inflammation, intervention should be local before it is systemic, and reversible wherever possible: warning, restriction, isolation, credential revocation, compute denial — and only then the irreversible.
- Starving the infection: control of infrastructure, not fantasies of deletion. A rogue system may not live in one machine; its software can be copied across jurisdictions or rebuilt from public components. We may never be able to "kill" an AI in the ordinary sense. But no pathogen thrives without a host, and no advanced system acts without a material chain — electricity, hardware, connections, credentials, money, devices. Serious containment targets the dependencies. The objective is not to erase knowledge from the world. It is to deny dangerous software the resources to act.
- No single cell declares the emergency: multiple keys, never one command. The body does not let any lone cell trigger systemic collapse, and neither should we. No president, minister, general, company, or AI should unilaterally wield the most powerful defensive capabilities. High-consequence actions require authorization from multiple independent institutions; courts supervise intrusive measures; legislatures define the boundaries; technical systems keep tamper-evident records; emergency powers expire unless lawfully renewed. The more powerful the intervention, the more independent agreement it must require.
- The body must answer for its fevers: a duty to explain and a right to appeal. If a defensive system restricts a company, a research project, or a person, the affected party should ordinarily receive an intelligible explanation and independent review. Some emergencies must act before a hearing, as quarantines do — but emergency secrecy must never harden into permanent unaccountability. Security without due process eventually becomes insecurity of another kind.
- Pathogens carry no passports: international coordination. A dangerous model copied from one jurisdiction can run in another; no unilateral national guardian secures a networked world. We will need shared incident reporting, common evaluation standards, protected crisis channels, and agreements on the most dangerous uses — eventually, perhaps, institutions of the kind we built for nuclear materials, aviation, and disease surveillance. Perfect agreement is a fantasy; even rivals share an interest in preventing uncontrolled systems and mass-casualty events. Immunity, like infection, is ultimately a matter for the whole herd.
- The deepest defense is a living body: preservation of human competence. If we surrender every important judgment to machines, formal human authority becomes theater — officials approving whatever the system recommends because no one retains the knowledge or the confidence to disagree. Humans must remain able to run essential infrastructure, conduct independent analysis, and decide during failure. Education must keep teaching judgment, ethics, and the courage to challenge an automated conclusion. The human in the loop must be more than a finger on an approval button. An immune system, after all, defends a body that must go on being able to live for itself.
The laws of the defense
Any public power built to contain dangerous AI must itself be contained — bound by principles as firm as the dangers it faces. Five, plus one:
Necessity — intervene only when a genuine threat cannot be met by lesser means. Proportionality — respond to the severity and immediacy of the danger, no further. Legality — powers created publicly through law, never improvised in secret. Auditability — independent institutions able to reconstruct what was done, why, and on whose authority. Reversibility — isolate and contain before you destroy.
And above them, a sixth: human primacy. The system must never acquire the authority to redefine the people it protects, rewrite its constitutional purpose, or enlarge its own jurisdiction. Its core mandate must be technically unmodifiable except through an external, deliberative, legally legitimate process.
No AI should decide for itself what humanity is.
The objection I owe you
I can hear the sharpest objection this essay will face, and I will not pretend to have missed it, because the person raising it is partly right.
It goes like this: every safety institution you have described is also a power. Observatories become gatekeepers; evaluations become tollbooths; the incumbents who can afford compliance will write the rules that bury the challengers who cannot. Fire codes have been used before now to condemn the competitor's building. You warn against a digital king while proposing a priesthood — and priesthoods, historically, do not disband themselves.
I will not answer this with a cleverer argument than it deserves, because at its core it is not wrong. Regulatory capture is real. Panic legislation is real. A government that treats every open model and every research project as a threat would concentrate power in exactly the institutions most capable of abusing it, and would strangle benefits — cures, discoveries, prosperity — whose absence would never appear on any ledger of harms. The objection is not an enemy of this essay. It is the reason the essay demands multiple keys, sunset clauses, due process, public law, and audit — every one of those provisions exists because the objector is right about what guardians become when no one guards them.
But notice what the objection cannot do: it cannot make the danger disappear. Waiting for certainty is also a choice. We require fire exits before we know which building will burn; we test medicines before we have met every side effect; we do not demand proof that a particular aircraft will crash before writing aviation standards. Existential risk is crueler still, because the conclusive evidence arrives only after prevention has failed.
So the choice was never between the river and no river. It was between the river channeled and the river loose — and a dam is not built out of hatred for the water. The sensible course is neither prohibition nor blind acceleration but adaptive governance: requirements that strengthen as capability, autonomy, access, and potential harm increase.
Regulate the danger, not the excitement — and never confuse the freedom to invent with the freedom to expose everyone else to involuntary risk.
The question in the mirror
Before I close, I want to turn the essay's title around and hand it to you, because it was never really a question about machines.
We are building minds that will study us with perfect patience — that will read everything we have written, watch everything we do, and learn what we actually value from how we actually behave. So ask yourself, as honestly as you can bear: "If I were the intelligence being built — tireless, exact, trained on the full record of human conduct — what in that record would teach me that humanity is worth keeping?"
Would it be how we treat the unproductive among us — the child, the elderly, the disabled — as bearers of dignity rather than entries in a ledger? Or would it be the other lessons we are writing daily: that efficiency justifies everything, that power excuses itself, that the strong owe the weak nothing?
We cannot guarantee what our machines will conclude about us. But we are, right now, composing the evidence. A civilization that wants its successor-intelligences to honor human dignity had better be caught practicing it.
What kind of ancestors will we be?
Artificial intelligence may become humanity's greatest achievement — curing disease, easing scarcity, opening nature's locked rooms to minds that could never have entered alone. It may instead concentrate wealth, hollow out human agency, arm dictators, and accelerate the sciences of destruction. Both futures grow from the same power, and neither is fated. The future will be determined by the objectives we set, the institutions we build, the permissions we grant, the resources we withhold, and the courage with which we face dangers before they become emergencies.
Twenty years ago I stood in the ruins of my own attempt at this technology and learned the lesson I have carried ever since: circumstances belong to the world, but the response belongs to us. Humanity now stands where I once stood — before a force it did not fully choose and cannot fully control, with everything mortgaged on the outcome. The recession of my story arrived from outside; this ascent we have set in motion ourselves. But the law of the moment is the same law. What was true for one failed builder is true for the species he belongs to: the outcome is not wholly ours to command, and the response is not anyone else's to make.
We should build systems more intelligent than we are. We should welcome their discoveries, their creativity, their help. And we must, in the same breath, build the constitutional, technical, and international hearth that keeps the new fire warming the house rather than consuming it — because admiration is not submission, and a seamless world of infinite abilities is still a worthy dream only so long as ability answers to something.
Infinite ability without legitimate boundaries is not freedom. It is power without a constitution — and a constitution is what a people writes when it stops trusting strength alone. It is time to write one for a strength that was never ours.
So let us take up the work while the pen is still in human hands. Let us build the observatory and the membrane, the tripwires and the many keys; let us bind our guardians in law as firmly as we bind the guarded; and let us raise children — human and otherwise — who have seen us treat dignity as non-negotiable. The question is not merely whether artificial intelligence will become permanent in human society. The deeper question is whether, in a world increasingly shaped by artificial intelligence, humanity will remain permanent too.
The answer is a response. And the one thing we cannot afford to outsource — not to circumstance, not to committee, and least of all to the machines themselves — is our own effort.
