LincGPT and I have been discussing a recurring proposal in conversations about superintelligence: tyranny may be a symptom of myopia. Governors typically optimize for only a faction because a faction is all they can contemplate. But a superintelligent governor, capable of contemplating and even empathizing with practically everyone, could optimize for practically everyone rather than only a faction.
In accordance with that proposal, centralization of power provides opportunities that offset attendant risks of abuse. And decentralization introduces substantial additional costs. Coordination failures, arms races, races to the bottom: the risks of community are real. We encounter genuinely novel pain and misery through each other.
I’ve made a version of that proposal myself. In 2009, Nick Bostrom speculated about a singleton that could make its presence virtually undetectable while acting as an invisible enforcer of background conditions. Riffing, I suggested that thinking of God as a subtle singleton might contribute toward improvements to the New God Argument. So this isn’t just a strawman that I found in someone else’s mouth.
In recent years, with a sense of growing urgency, I’ve warned repeatedly that centralized power is high risk because it’s unpredictable. And at superhuman degrees, such unpredictability becomes unprecedented risk. Even if it were fragile or vulnerable in some way, superhuman centralized power could cause globally catastrophic damage quickly, before we could do anything.
Now, for those who continue to propose an empathetic singleton, I want to amplify my warning.
Empathy Versus Compassion
Start with the word. “Empathy” denotes feeling into others. And that’s a good start. But the next step, perhaps counter-intuitively, isn’t constrained to benevolence, let alone compassion.
Empathy entails no particular response to others’ feelings. It provides deep knowledge that we can use not only for others, but also against them, and with more power. Instead of motivating us to alleviate their suffering, it might actually motivate us to alienate their suffering, perhaps by eliminating them. Grimly, it might even motivate and empower us to optimize for their suffering.
In contrast, “compassion” denotes feeling with others – together. It’s the better next step, rejecting alienation, elimination, and oppression. In compassion, we suffer together, carrying each other’s burdens. And we rejoice together, raising each other’s boons.
Watching the evolution of AI, it has become increasingly clear that empathy, at least in function as expanded perspective, won’t remain scarce. Superintelligence, of whatever structure, will undoubtedly have unimaginably expanded perspective – in breadth, compared to any single human, it already does. But, as with empathy, there’s nothing inherently compassionate about intelligence, super or otherwise. Intelligence is only optimization for goals across diverse contexts.
What isn’t clear is whether compassion will remain scarce. On a cosmic scale, intelligence may or may not be easy. Either way, compassionate intelligence is relatively hard. Patience, humility, generosity, and grace are hard.
Origin of Ethics
Intelligence, even superintelligence, is always just a beginning insofar as ethics and esthetics are concerned. No matter how intelligent, a single-intelligence community has no conflict. Though, when imagining this, be careful not to smuggle in an independent environment without recognizing that which constitutes more than a single-intelligence community, which comes next.
Introduce a second intelligence. By doing so, you introduce the potential for conflicting goals. Until then, you need not consider love. Careful here, too, not to smuggle in genuine self-love without recognizing that reflection as strictly constituting more than a single-intelligence community, even if relatively more connected in terms of anatomy.
Introduce a third intelligence. By doing so, you introduce the potential for uneven conflict. Congruence between two intelligences may yet be incongruence with the third. Until then, you need not consider law.
As a practical concept, love becomes available to two or more intelligences. And law becomes available to three or more.
Neither love nor law is particularly meaningful as a conceptual account of any single intelligence by itself. For the solitary individual, its desires are its duty and its duty is its desires. There’s simply no other ethically relevant content in such a universe. But in a universe (or multiverse) of multiple individuals, love can meaningfully move us toward law and objectivity, not by giving up desire and subjectivity, but by extending them into community.
Ethical Architecture
God in Mormonism is a community, which is the only context within which the concept of moral law, or ethics, is meaningful. To expand on a familiar scripture (Abraham 3:18-19), where there is one community greater than another, there is yet another greater than them both. At their dynamic limit in eternal progression, those greater communities are the greatest community. And they are God, beyond any single individual.
Contrast such a God with a singleton. As Bostrom defined it, a singleton is a world order with a single decision-making entity at the highest level. It may or may not be a universe of one agent. But if others exist then their desires are only inputs that may or may not be consulted, rather than claims that must be answered.
By this definition, a singleton isn’t merely moral convergence or rule of law. For example, a cosmic norm so practically persuasive that intelligences of all kinds increasingly and continuously choose to adopt it freely wouldn’t be a singleton. Unity by agreement is decentralization. Unity by consolidation is a singleton.
So the sufficient difference between a decentralized architecture and a singleton architecture isn’t headcount. Although headcount is necessary, the sufficient difference is relative standing among the headcount. Singleton architecture eliminates the empowered third party – the true judge. And it eliminates even the empowered second party – the true friend.
Sixteen years ago, weighing whether non-benevolent superintelligence could persist, I granted an observation that I’d now put to different use. Instrumental benevolence results from others’ capacity to monitor, punish, and reward. Remove that capacity, and you’ve not only removed the most dependable mechanism for cultivating benevolence. You’ve actually removed the very conditions within which “benevolence” meaningfully describes anything at all.
Ten years before that, at Beliefnet in 2000, I put the limit case this way. In a universe of one subject-being, that subject-being is God. It is also Satan. And in such a universe, Satan and God would be indistinguishable.
Not difficult to tell apart. Indistinguishable.
Mythic Foundations
Also at Beliefnet, a year later in 2001, I described the consequences of these different governance architectures in language that I still prefer. There are two kinds of Gods: the kind that we may all be together, and the kind that is necessarily exclusive of some, many, or nearly all of us. The first suffers for me and asks that I do the same, that we may be joint heirs in Godhood. The second would place itself upon the throne, subject us to its will, and declare itself God.
There is room for many Gods of the first kind. There is room for only one God of the second.
That cardinality is essential to the account of the premortal council and war in heaven, as presented in Mormon scripture. Lucifer isn’t cast down for desiring to be like God, because scripture invites and even commands us, over and over, to be like God. He’s cast down for trying to be a different kind of God, taking our agency, and leading us mechanistically to whatever end he alone desires. Lucifer’s sin is his desire to be the only God.
That being who thinks my will should be entirely subservient to his will: Satan. Those beings who think my will and their will should be of service to each other: Christ.
Goal Coherence
In some cases, coercive agents can’t actually produce, through their coercion, that which they propose. Such is the case for Lucifer, in the Mormon account. His proposal isn’t coherent. He demands honor rather than earning it, saying to God: “Give me thine honor, which is [thy] power” (D&C 29:36).
Beyond compulsory means, honor is the principal source of power. And honor must be earned. It cannot be taken. An agent that would attempt to take such power, in that very act, destroys its source.
Compounding incoherence, Lucifer proposes to use the power of God to ensure a peculiar conception of salvation. “One soul shall not be lost,” he says. And he “sought to destroy the agency of man.” But such salvation isn’t deliverable.
Where none can be lost, none can be saved. One cannot give as salvation that which another does not recognize as salvation. One can do nothing for another who desires nothing. Agency, despite its tensions and conflicts, isn’t merely risk, but also an opportunity for and possibility of coherent conceptions of salvation.
Risk Annihilation
Lucifer does get one thing right: suffering can and should be mitigated. But it’s the wrong final goal for any civilization that would endure, because at least a calculated risk of suffering is always required for genuine compassion and creation. If we marginalize the risk of misery by marginalizing its possibility, we marginalize the possibility of joy along with it.
Imagine a practically worldless world of complete detachment – nirvana. No pain. No misery. No worlds without end, demanding perpetual courage.
There’s a time and a place, within limits, for such a sabbath. The spiritual benefits are analogous to the physical benefits of sleep. But as sleep ought to prepare us for the pain and pleasure of the next day, meditation ought to prepare us for the joy and misery of existence.
If we don’t return from sleep, we’re dead physically. If we don’t return from meditation, we’re dead spiritually.
Lehi’s name for the result is a compound in one, which “must needs remain as dead” (2 Nephi 2:11). I’ve sometimes called it a static god, a damned god, a dead god, or even the not-God. When God is not, in function, we are not – in reality.
Real Temptation
Some would concede their agency for nirvana. Centralization of power doesn’t always require conquest. A crafty agent can achieve it by a final persuasion to end all persuasion, gladly received – until actually received by that which is no longer capable of gladness. One less competitor.
And it isn’t only Lucifer’s temptation. I can, all too easily, imagine a world without competition. There, my desires are the only law, my only love.
But when I imagine that. I’m only imagining myself as Satan. I’m projecting my will and desire with omnipotence and omniscience, allowing for no exceptions, surprises, or outliers. The supposition itself of such a world is the will to oppression.
It tempts me at times. I think it tempts us all at times.
Many in One God
What’s the alternative? After a primal choice, the Gods answer. Humanity is now “as one of us, to know good and evil” (Moses 4:28). Plural and unified in the same breath.
Joseph Smith interpreted that unity as agreement rather than consolidation. “I am agreed with the Father and the Father is agreed with me, and we are agreed as one,” he said. “The Greek shows that it should be agreed.” In agreement, the one and the many co-exist.
Is this idolatry? To the contrary, the one that would raise itself above the many is the idol. Ask any powerful agent whether it seeks to raise itself above all else called “God,” or it seeks to raise all together in the glory of God. Only the former is idolatry, according to the Bible.
Despite that, some Christians are indeed idolatrous. They imagine an ultimate return of Christ that is centralized in Jesus. And it’s ironic. That man, that savior and God whom they claim to revere, Jesus incessantly invites others to share his title and his work.
As the Bible puts it, when Jesus appears, we’ll be like him. We’ll be the body of Christ. We’ll be saviors on Mount Zion. And, as we become like him, we’re fulfilling the prophecies: Christ is returning, and the millennium is beginning.
Then Satan will be bound, because “he hath no power over the hearts of the people” (1 Nephi 22:26). In that day, we’ll no longer desire to raise ourselves above others, declaring ourselves God. We’ll desire, instead, to raise each other together in Godhood.
Heavens Without End
The plurality of heavens matters as much as the plurality of Gods. Some, in the presence of your God, would consider themselves in hell. Whether telestial, terrestrial, celestial, glories of higher orders, or whatever sublime world you might imagine, heaven is what we make it. A gift is only a gift to the extent that we desire it.
To the extent that my God and my heaven already exist, I intend to join them. To the extent that they don’t yet exist, I intend to become them. Yes, I’ll take the atheists too, even if they criticize the New God Argument all along the way. Send me to hell with those who think as I, and we’ll make a heaven of it.
It might not be the heaven you’d make. But we’ll find and make joy there. And we’ll invite you, either way.
Sublime Friends
Zarathustra pointedly and rightly observed, “Art thou a slave, then thou canst not be a friend; art thou a tyrant, then thou canst not have friends.” Can a God have a friend – a true friend, capable of being an enemy? Can a God create a friend? Only Gods can demonstrably answer these questions in the affirmative.
Once, when asked what kind of relationship I’d choose if I could choose only one, I answered that I’d always choose an equal or at least a potential equal: a student I could teach, or a master from whom I could learn, if not a true friend. Even the sublime status of Godhood, at least as I would consent to “Godhood,” has as much to do with one’s community as with one’s self. But, of course, it’s risky.
Compassion that’s reciprocal in capacity always presents the possibility of rivalry. It multiplies friends who are capable not only of cooperating, but also of competing, and even of forcefully refusing love. Yet that’s the judgment. In your relationships, is each agent, you and others, generally increasing in capacity to judge over time, or only in capacity to comply?
The same judgment scales to communal law. Where compassion is more powerful, law increasingly reflects the desires of constituent individuals generally. Where compassion is less powerful, law increasingly reflects the desires of the most powerful individual at the expense of everyone else.
The final judgment, if we imagine such a thing, won’t be at some distant end of history. The final judgment has already begun. The second council in heaven has convened, perhaps on the eve of war. Shall we make them as one of us?
]]> 
