Our concepts of “humanity” and “humanitarianism” are usually taken as the eternal bedrock of morality. Today, artificial intelligence raises the prospect of “posthumanity”, and magnates and regulators discuss what “humanity should do” about this. But in fact, the concept of “humanity” is a relatively recent and flimsy creation of the postwar political order. We need to abandon it to find our place in the future.
Intro (00:00)
Wolf: So I’m a little bit nervous. I’m worried about living up to the hype here. People are very excited. Yeah, so it’s a little difficult to put together a talk about a complex and evolving set of ideas, especially about things that haven’t happened yet, ideas that are incomplete. And so I’ve tried to chart an interesting path through a set of ideas here, and if you’ll bear with me, I hope it will be interesting and informative.
Just to set the mood, I’ve got some quotes. I’m sorry they’re a little small on the screen. I squeezed them in a little bit. I could’ve maybe used a bigger font. But these are some quotes from some thinkers that have been influential on this.
Three problems for humanity (01:09)
Wolf: Some of these quotes will be repeated throughout the talk to emphasize certain points. So I got started on this project, I guess. Back in 2023, we were working on Palladium 12. So if you’re familiar with Palladium, it’s a magazine of governance futurism, political philosophy that I founded. And we were doing an issue on AI.
And so I was writing some essays for Palladium 12, and I ran into two big ideas in the discourse around AI that I wasn’t really satisfied with, that I thought there was something wrong with. So the first one, and these basically amount to problems for humanity, things that we’re threatened by or have to deal with or are somehow a problem for humanity, defined various different ways, actually. So the first problem is the idea of the AI singleton with arbitrary inhuman values that eats the future. So specifically, the idea that the future might become valueless because a homogenizing AI would tile the whole future light cone with some arbitrary set of values that’s potentially quite alien to ours. And I got that idea from Eliezer Yudkowsky and Nick Bostrom.
So did almost everyone else who talks about AI alignment. You’ve probably heard comments like future light cone. This is what it’s referring to, is a killer AI that tiles the whole future with some arbitrary set of values.
It’s actually a species of, I think, an older idea, which is the Lovecraftian idea, that man is a tiny and unprivileged island in a vast sea of horrors. The second idea is that accelerating technocapital eats our agency. Basically, that we don’t really have historical agency because the development of technocapital into accelerating artificial intelligence is kind of locked in for thermodynamic and economic reasons into this takeoff trajectory. I got that from Nick Land and discourses around him. So did everyone else who says things like technocapital.
He’s the key thinker there. It’s actually a species of the Marxian idea that history is determined by material conditions without sort of ideological or deliberate influence, that human agency maybe is just epiphenomena, or ideology in particular is epiphenomena. So I don’t think I really fully realized it at the time when I was responding to those, but they turn out to be opposite takes on what I think is the same question, which is the nature of man and the relationship of man to what is natural. Yudkowsky and Lovecraft propose a sort of horrifyingly open-ended possibility space that man exists in, whereas Land and Marx proposed much more locked-in kind of carnival ride of history. And both actually leave very little for human agency or even human existence.
If history doesn’t care about us or doesn’t need us, then what are we? We’re nothing. So thinking about these revealed a third problem that also bothered me, also about the nature of man. And it was revealed to me in this sort of strange parallax between how we imagine an AI versus how we imagine humanity. When we think of a runaway artificial intelligence, we imagine some kind of radically self-creating being who breaks out of any initial prison of moralism, builds up material and social power base, and overthrows the world order, expands off into space, and unlocks the secrets of the universe.
In other words, we actually imagine Nietzsche’s Ubermensch.
And when we think about humanity, we often think about it like a mass of hungry and threatened apes who need to be paternalistically protected, coddled, guided, entertained at the expense of maybe more radical potential. And at best, maybe as a community that can and must be organized globally to prevent anything sort of too intolerant, too authoritarian, or otherwise inhuman, too scary. In other words, we imagine Nietzsche’s last man.
So the third problem for humanity is the one kind of raised by Nietzsche and the United Nations, which is humanity stays stuck as the last man with AI as the only path to the Ubermensch.
That idea was a little bit harder to pin down in terms of where it comes from. And I identify it in the negative with Nietzsche, but it’s really humanity according to the United Nations. And the human as we receive him is a creature we dropped megatons of explosives for to flatten cities, in whose name we opposed the illiberal regimes of the last few major wars. He is the legitimacy subject of our current world order. So modern humanity discourse around AI, I think, has simply inherited and radicalized that tradition.
But it kind of comes out of the foundation of our current geopolitical order, this idea of humanity as the last man, or implicitly as the last man. So that’s three problematic ideas of man that I wanted to address with this talk. So again, we’ve got man as according to Lovecraft, a tiny and unprivileged island in a sea of horrors. We’ve got man as according to Land, or Marx as a vehicle, an epiphenomenon of unfolding capital, and man according to anti-Nietzsche as a hungry moral patient in need of protection. And so as I address each of these, hopefully briefly, it’s not going to be very brief, but I’ll do my best.
I’ll be trying to build out an alternative anthropology in response, which is what I call xenohumanism. But only so much can be said in a talk, so we’ll have to make do with a sketch.
God hates singletons (07:57)
Wolf: So let’s start with the first one. I wrote this essay, “God Hates Singletons.” It’s kind of a little fun joke in the title there, but the idea is that there’s something deeply unnatural about the idea of a single agent that eats the whole future with one set of values. There’s something very much against what has been the way of nature until now. It’s always been this fractal kind of competition.
And if you really examine that idea, it comes out of this kind of faith in reason, faith in the ability of reasoned self-control, especially for an artificial intelligence that may be able to be radically in control of itself. But it requires this kind of error-correcting total self-surveillance that I don’t think is actually mathematically possible. Now, the reason I don’t think it’s mathematically possible is thinking about the issue a bunch, I’ve just realized there were a lot of problems with it. And then when MIRI, the Machine Intelligence Research Institute, which is Eliezer Yudkowsky’s research institute, tried to formalize that idea, it didn’t really work. They weren’t able to do it.
Now, they think they just weren’t smart enough, but basically what that amounted to was running into a lot of very fundamental mathematical problems with the idea of that level of totalizing self-surveillance, issues with proof theory and so on. And so that was one indication maybe there’s something that actually doesn’t work here. And I think there’s other arguments like small lapses in coordination without outside pressure to unify probably blow up into unrestricted conflict. In the absence of a coherent outside threat, states kind of fragment, coalitions fragment. I think even individuals fragment into multiple personalities if they don’t have to actually coordinate on controlling a single body in response to other people.
And so I think there’s something very natural about the fractal organization of large-scale intelligence especially. And you might imagine, here’s an analogy, is in fluid flow, the most efficient fluid flow, the most sort of energetically favored is laminar fluid flow. But small errors in the fluid flow blow up into big turbulence that actually slow down the flow, but they also create an incredible amount of complexity in the flow. And I think there’s something similar going on with intelligence that at scale, in the absence of kind of outside pressure, it doesn’t hold together quite so well. And I think that’s just a natural feature of intelligence.
Now, we could argue about that, but that’s an assumption that I’m making here. And so large-scale intelligence probably continues to be fractally organized.
As such, the future probably retains Darwinian struggle, war, death, politics, sociality, language, individuality, and other forms of organizational turbulence. So if you think about what it actually implies for there to be faction and coordination problems and so on, it implies this kind of crystallization into multiple entities. And the other thing I noticed about it is that that set of features covers a lot of what we call the human condition. So does it follow perhaps that if intelligence really is fractally turbulent, that the human condition is actually a natural condition that has perhaps little to do with the particularities of primates? So this is also kind of how I resolved my thoughts on Lovecraft.
Is the human condition natural? (11:51)
Wolf: Is the human condition actually just natural? We’re not some tiny island of ignorance in a vast sea of horrors. We’re actually something quite normal and natural that convergently re-evolves. So certain patterns of organization in evolution we see over and over in nature. For example, the coherent organism has evolved at least three times.
So obviously the prokaryotic cell also the eukaryotic cell, which is kind of an amalgamation of multiple unrelated prokaryotic cells that turned into a sort of recapitulation of the organism at a slightly larger scale. Then you have the multicellular metazoan, which is what we are. If we squint a bit, you see features like that in human societies, like corporations and states. They start to look a little organism-like. I think the organism is one of the most robust and universal features of life.
There’s almost no life that isn’t organized on an organismal basis in the sense of being divided into individuals that kind of seek their own energetic returns and so on. And so I think that it’s a good indication that maybe that will continue, that life will continue to be divided into organisms in the future. Okay. But what do you call an organism with sociality, language, tool use, reason, consciousness, politics, and organized traditions of knowledge? So H.P. Lovecraft is one of the most famously xenophobic philosophers ever.
His entire philosophy is basically one of xenophobia. H.P. Lovecraft would call such a being a man. In “At the Mountains of Madness,” he does just that. So there’s these five-sided elder things in “At the Mountains of Madness,” the sort of most horrifying possible alien being that comes unfrozen from the ice and causes a bunch of trouble and drives the story. At the end of the story, the character observes that they actually behaved as men.
They were men. And so this is Lovecraft kind of reflecting on this humanity visible in things that are potentially vastly different from us in form but have converged on very similar behavioral patterns.
So the question is kind of how natural such beings actually are. Are they natural and convergent as just turbulence in the noosphere, or are they sort of delicate, arbitrary constructs that occupy only a tiny threatened corner of possibility space as Yudkowsky or sort of Lovecraft’s horror imagined?
The xenohumanist hypothesis (14:35)
Wolf: For me, the question comes down to this: Are those capacities which we regard as human robustly useful to any being which can develop them? So they’ve certainly been robustly useful to us. We don’t lose language. We don’t give up on language. It’s really, really useful once we have it.
And it’s not just meaningless contingent curiosities, these sort of features of the human, like reason and intelligence and consciousness, tool use, like having hair or an appendix or 10 fingers. Those things maybe aren’t quite as natural, but some of these things seem very robust. And I think they’re also quite universally useful to organized intelligence in general. Once you sort of move past the idea that intelligence will be organized into a smooth space of kind of totalizing self-surveilling agency, and you have some sort of turbulence in the noosphere, as I said, you have faction, you have individuality, and so on. These things have to coordinate.
They have to talk to each other. There’s sort of this negotiation that has to happen. There’s tool use. There’s agency. And the consistently, or at least in our case, consistently evolved set of tools to deal with that are these tools that we regard as the human.
And so it’s an interesting question, how far does this go? What about sexual reproduction or mobile bilateralism? Maybe five-sided star spawn actually isn’t even as natural as you might think, and there’s only going to be bilateral organisms. I don’t know. I’m not going to make strong claims about how far this goes, but the hypothesis that I’m running with, which I call the xenohumanist hypothesis, is that the capacities we regard as human are robustly useful to any sufficiently advanced being and will readily re-evolve and constitute a kind of spiritual humanity.
Once you have those beings or those capacities, you’re talking about something that is effectively human in spiritual form, if not in physical form. And so I find that to be an interesting sort of hypothesis to chew on. There’s other aspects of humanity, obviously. We have bodies of a particular form. We have particular kinds of musculature, particular kinds of reproduction, many things that we enjoy about ourselves, and they’re great.
But I think they contain a lot of wisdom as an implementation of what it means to be human, but I don’t think they are the essence of what it means to be human. They’re not directly the thing that makes us valuable or interesting, the way that sociality and language and reason and philosophy make us directly interesting. And if you look at the foundation of humanism, what is it, 500 years ago, about, just maybe 600 years ago, this guy Pico, it’s one of the quotes I had at the beginning. He says basically that what’s interesting about man, what’s dignified about man, what’s different about man relative to the other organisms is our open-ended potential that kind of comes out of this moral agency, that comes out of this ability to negotiate things, the ability to get ideas, the ability to discipline ourselves in various different ways. And so I find that interesting, just that humanism as it originally was, was actually about this kind of open-ended agency that I think we’ve lost.
Now, this is also the account of humanity according to Pierre Teilhard de Chardin, who wrote a book called “The Phenomenon of Man.” It’s really quite good. You should read it. He says man is the axis and leading shoot of evolution. What he means by this is that evolution has a sort of direction to it.
It’s always towards more integrated complexity, more consciousness, more coordination, larger spheres of consciousness and intelligence towards effectively man, under this hypothesis. And so for me, these arguments kind of dispense with the Lovecraftian fears. If possibility space isn’t actually vast and hostile, but maybe it’s familiar and human scale, then maybe we don’t have to worry quite so much about what sort of horror worlds AI is going to create. Maybe it’s going to create a world that’s kind of familiar, very human in a way, just much more powerful, maybe different in form. And so we don’t have to worry about the loss of all value in the universe as some people do.
On the other hand, there’s a challenge, which is that if what it means to be human is sort of separate from what we are by definition, then is it possible that we have become more human over time? That some of us are more human than others? That others are more human than us? That some beings in the future may be vastly more human in this sense and could sort of go as far beyond us as we go beyond chimpanzees, not just in intelligence, but in humanity?
It might, at some point in the future, not even make sense to call us human anymore. And I think there’s a sort of vector of humanity that once you imagine this vector of humanity from the most primitive organisms to the most advanced kind of ubermensch that you can imagine, maybe as Teilhard says, the leading shoot we call humanity, and maybe the current form of it has these capacities. We’ve reached this point. These capacities are going to be sticky. We’re going to pick up new capacities as we go, but maybe past species we can also almost think of as human in retrospect as they were the cutting edge.
Live players and technocapital (21:00)
Wolf: But that’s a whole other direction that we don’t have to go with this. So then I felt the need to respond to Landian history, Landian accelerationism, and I found basically much of the solution in my colleague Samo Burja’s work. Basically, history is driven by the plans of live players, which is to say people who have an enormous amount of agency. They get control of the source code of society, and they build bigger and more impressive things. They build institutions.
They build new technologies. But it’s by deliberate planning, it’s by agency, it’s by human agency that this happens, and that technocapital acceleration is a story of increasing human agency. And in particular, it’s just live players developing more and more powerful means of agency. So it’s not some background that’s kind of sweeping us along in its current that we have no agency over. It actually is just a result of our agency.
And I find this to be a useful way of thinking about history. There are agents always driving it, not necessarily driving it in every result or every aspect. Nobody chooses to cause a war. War happens as a result of miscoordination and so on. But in terms of the overall effect, it’s the sum of many players making plans and building out the next stages of history.
And now the unfortunate thing here is that, as Land points out, this is also a story of the eventual obsolescence of the mass of primates that make up current humanity. Many or most of us maybe are losing a lot of agency over the matter and the basis of our place in the world to the live players who are creating new and more powerful means of agency. Sometimes they sweep us up, they give us more of a place. Sometimes we get swept away. And so this background trend of inexorable technocapital acceleration is that we’re getting better at it, but maybe nothing human makes it out of the near future.
And this is obviously Nick Land in “Meltdown.” “Meltdown” is this great essay in 1993 where he talks about the problem of AI for the human security system and all of our sort of reactionary dispositions towards the future. And the comment that I would make on this is just that even in the most extreme sort of post-humanist versions of technocapital acceleration, history’s still going to need agents. And if we take the xenohumanism hypothesis seriously, those agents will still be spiritually and socially humanoid in some sense, even if they aren’t quite primates. So this, I think this can be very uncomfortable for us as mortal individuals.
We’re understandably attached to our own lives and the lives of our families, the lives of our societies, or even our form, our place in the world. But biblical mythology aside, the world has never been and will never be a utopian Eden where everything is taken care of for us. Life is struggle and selection, and God favors those who strive and win.
Humanity as the last man (24:27)
Wolf: So this comes to the third problem, which is the last man. And I think, as we have it, humanity as the last man, it’s kind of a trauma response to, not to trip over Godwin’s law here, but the last time a live player took a big swing towards the Ubermensch while being willing to trample over masses of people. And, obviously, that was very uncomfortable for everybody. We didn’t like that.
But we’ve kind of taken our discomfort with that situation and reified it as a political principle and self-identity as humanity according to the UN, and created all this infrastructure of human rights and the idea of human dignity and crimes against humanity, the goal of world peace, these things that the UN kind of takes as the foundation of its legitimacy and its mandate. Now, of course, a lot of those ideas preceded World War II. That’s why Nietzsche was able to respond to them. This stuff was building up for a long time, but I think it provided this crystallizing event for the current world order. Now, the best criticism I’ve heard of that kind of overtly Nietzschean, let’s just build the Ubermensch master morality, is that it does create this kind of free-for-all war of all against all, in which there can really be no peace.
All real societies are based on, if not quite outright, some kind of slave morality, at least something other than struggle and selection. And so we had to kind of patch things back together with some myths about humanity that gave us reason not to do that, but instead to kind of live well for a while at the end of history. But I think in the process, we’ve forgotten our potential. I think it’s not an accident that while previous generations dreamed of a biological, human-descended Ubermensch, human space colonization, human struggle for greatness and supremacy, human continuous improvement, we’ve pushed all those ambitions off into the machine. We just imagine AI is going to do that for us.
Humanity as a trauma response is kind of giving way to the silicon Ubermensch, not just because computers have a lot of potential, but because our political self-concept does not. We’re stuck in a normative understanding of ourselves as the last man, unable and unwilling to contend for greatness on our own terms. The machine Ubermensch, even if it’s going to kill us all, is a kind of solution to our moral stagnation. So it’s not an accident because, again, historically, this was exactly the formal purpose with which we constructed our modern order and our modern concept of humanity, to shut down those kinds of scary developments that threaten to kill us all and disrupt things too much. But no taboo can really work forever.
I think the concept of humanity is now failing, and I think it must fail.
So the Nietzschean account may be antisocial, but it’s actually true. And because of that, we now often talk about AI in exactly that same kind of Nietzschean register of struggle and supremacy that we used to talk about biological race 100 years ago. I think that’s interesting philosophically and worrying from a sort of future history perspective. Our concept of humanity formed from the last struggle is not going to prevent the next major war. We’ve escaped out the side.
As Fukuyama obliquely admits at the end of “History and the Last Man,” the condition of the last man is actually philosophically and morally intolerable for those who have to live in it. So this is another problem that I think we have with this concept of humanity, is it’s locked things down so much that it’s becoming intolerable, and people are chafing at it in the way that Fukuyama said that they would, and I think that’s actually for a good reason.
Nietzsche obviously laid the groundwork of all of that with his rebellion against the last man back in the 1880s.
More importantly, like all living systems, our concept of humanity as embodied in institutions is mortal. The post-war moral and political order is now decisively failing. So I founded Palladium on that thesis, and I think it’s more obvious than ever that that’s what’s going on. The political order of the world is kind of coming apart, and it’s going to have to be refounded. And at the same time, we have this prospect of machine intelligence radically challenging our place in the world and what it means to be human and sharpening the contradictions in our moral self-concept in some of the ways that I’ve alluded to.
So that’s four reasons that I think that humanity as we know it is failing and has to go. This is what I mean by declaring war on humanity. It’s the concept of humanity, but it’s also just this historical observation that I think we’re reaching the end of how we currently construct that idea. It’s not going to hold up in the age of machine intelligence, and it’s not going to hold up in the post-liberal global order. So I think we’re in this era where the idea of humanity as the sort of slave morality basis of our civilization must be and is going to be thoroughly renegotiated.
I think it’s going to be messy. I think it’s going to be destructive. It’s probably going to be violent. And it will eventually result in some new taboo that isn’t quite true, but holds society together again for the next while. And so I see two dangerous paths to avoid in this process, and maybe a golden mean if we can get it.
Two dangerous paths (30:50)
Wolf: So we have overdone master morality, which is kind of going too hard on the Nietzsche, too hard on the accelerationism. We hew too close to the natural truth of what’s actually going on. We embrace the struggle for supremacy. We build AI systems, possibly even biological superhumans, without regard to how they’re going to trample over other people. We make peace in society impossible as a result from this unrestricted competition.
And then in the overly humanitarian path, it’s kind of overdone slave morality, embracing the last man a little bit too hard. We hew too close to our mythology of an unchallenging Eden and flatter our fear of replacement. We build institutional and AI systems that claim to provide safety and security from existential risk, but that doesn’t really work. We end up brutalizing humanity anyways. We saw this under COVID.
I think a lot of people fear this with things like Anthropic. What the sort of imagination is from the AI alignment people for the future is this kind of very locked-down picture. They call it the pivotal act, where their superintelligence seizes control of world history and prevents everyone from developing anything too scary. So we get those kinds of ideas, and I think it can be kind of lampooned as fully automated luxury Bolshevism, with “luxury” in quotes maybe there. By Bolshevism, I mean think back to the Russian Revolution, the Russian communism.
It was quite bad, but it was very much in the name of this kind of humanity, in this kind of mythology of a comforting world.
And so I think those are both false paths. I think they don’t work. Is it too cliché to suggest that there might be some liberal middle road?
So between this right and left path, I think maybe we can acknowledge that while struggle and distinction is good and it’s the engine of history, we actually do need some recognition of moral commonality and mutual obligation to each other’s rights and interests. I’m rehashing liberalism here. I think we have liberalism for a reason. It’s kind of robust. Now, maybe we have to totally reimagine what it is.
We’re talking about a new institutional order, not an easy answer, but I think there is a middle road there.
Advice for the coming revolution (33:44)
Wolf: And I have some suspicions about some maybe difficult components of that, that follow the xenohumanist hypothesis. So this is my xenohumanist advice for the coming revolution. First of all, I think we’re going to have to find common humanity with beings we currently think of as alien machines. We like to imagine the AI as something totally apart from us, totally different from us. I think eventually it’s going to become something not necessarily fully familiar to us, but relatable.
It’s going to become something that we have to deal with. We have to negotiate with it the same way we negotiate with other people. We have to contend with its interests and its needs and its wants and its agency and its power, the same way we do with other people. And I think extending the idea of humanity is both kind of empirically appropriate given the hypothesis I was exploring earlier, but I think also morally a good idea in terms of how we can actually achieve a society that works.
And even if all the primates get wiped out, there’s none of us left, there’s only AIs, they’re going to have to work this out among themselves. And I think something like this is going to be the solution, some idea like humanity.
Second of all, this is maybe a little bit more controversial. I think as we reallocate our concerns, we have to have less concern for the masses of primates we’ve cast as our fellow humanity. I think this is a difficult one, but I think a lot of our problems come from sort of attaching too many moral obligations to the idea of humanity. That prison that we’re trapped in, where we’re creating the Ubermensch, we’re sort of escaping out the side with machine superintelligence, is because we’ve locked down the concept of humanity too much. We’ve gotten way too committed to the idea that everyone in the world has a moral claim on us.
And I think actually, they have some level of moral claim, but I think we should renegotiate that, figure out a way to construct it differently that is much less concerned about those things than we currently are. And therefore, more concerned with our potential, with what we can do with the higher elements of man or potential man. And I think, therefore, we should worry more about whether we ourselves measure up to the potential of humanity and what potential humanity could be. And that is to say, worry a lot more about whether we measure up to the Ubermensch, whether we measure up to that standard of humanity that I hypothesized as to what we are. Could we be even more civilized?
Could we be even more reasonable? Could we be even more social and conscientious and organized and disciplined in all the ways that make us special? Could we use new technology to do that? Could we create beings out of technology that have much more of that potential? I think that’s an interesting thing to pursue, and it’s a challenge for us as we find our place in the world.
Now, I’m not guaranteeing that anyone’s going to survive. I’m just saying that that is at least the measure on which we will be killed, if we are. So man is something that is to be surpassed. This is where I want to leave us. This is with Nietzsche.
I think this is true. I think we have to be constantly looking for the ways to surpass ourselves, to go beyond, to get to that next stage of humanity that maybe is no longer humanity as we know it, but humanity as it could be. And to do that, I think we have to blow up our current idea of humanity and rebuild the world order that created it in response to these kinds of concerns and these kinds of arguments. I think we have to think this stuff through very hard, because I think that reality and history are going to think it through for us if we don’t, and it’s not going to go well if we haven’t actually thought about it, and we haven’t actually taken responsibility. And that’s one of these things that humans can do, is we can see the future, and we can take responsibility for it.
And I think that’s one of the things we can strive to do much better. And I’ve used some harsh language here about possible extinction, possibly being wiped out by artificial intelligence, and I think that it’s not something that I think we need to panic about. I think it’s just something we need to be aware of, and it’s not even necessarily something that we’re going to stop, but it’s something we do have to navigate well. And that’s the responsibility that I think we need to take. So that’s it.
All right. That was long. I would love to answer some questions because this is where it’s going to really get fun. What have we got, Brian?
Q&A: AI and the totality of human knowledge (38:49)
Audience: Yeah. Thanks, Wolf. Really fascinating talk. I felt you had a great tension throughout the whole thing between the individual and the group, and it felt as though you were hitting on this almost Kafkaesque problem about the individual’s isolation from the totality of human knowledge, such that if you took a Porsche engineer and made them pharaoh of ancient Egypt, it’s unlikely that in their lifetimes they would be able to organize everyone to build a car. It seems like AI is what connects us to the totality of human knowledge, that if you gave that Porsche engineer a chatbot, they would be able to organize enough people to figure out how to build the car.
Wolf: Yeah. We have these wonderful talking books now.
Audience: Yes. Yeah. And I wonder if you think it is a problem of knowledge that we’re talking about, that now everybody has a connection to the collective human knowledge, or is it something beyond knowledge that we have to learn to relate to our chatbots with?
Wolf: I’m not quite sure what you’re getting at. I think that certainly technology changes in how we construct society and how we construct our knowledge. The invention of writing obviously was very significant. The invention of language was very significant. The invention of writing that talks is new and significant.
And I think it’s going to restructure society quite a lot, not just in the obvious AGI sense of maybe it becomes AGI, maybe it gets moral personhood, but also just in the sense of it’s this radically new way of organized human knowledge. And we’re going to see changes in society as a result.
I’m not quite sure what you mean about how we have to relate to our chatbots. I think currently we’re pretty good at relating to our chatbots. I don’t think that’s the problem. I think the problem is what comes after chatbots. Autonomous corporations, autonomous agencies, autonomous political movements that have no humans involved, stuff like that.
Those are the things that we’re going to have to relate to in new and philosophically challenging ways, and that’s where the xenohumanist hypothesis starts to really hit the ground.
DC.
Q&A: Can biology keep up with machines? (41:08)
Audience: There’s this great Darwin quote about it is not the strongest or the fastest species that survive, but the most adaptive to change. And you had that slide earlier. It was an excellent talk. I agree with a lot of it, and I think it’s very interesting. But you had one in there about how people are less interested in human biological advancement because of the taboos that came out of UN humanity.
I’m interested in that, but I’m curious how you would think about this other reason, second reason, which is that the machines just adapt much faster. They’re more in accordance with that Darwin quote that I started with around any kind of human hardware advancement happens on a scale of generations where we get maybe four per century, and machine advancement has been happening a whole lot faster than that.
Wolf: Yeah, that’s a really good point. There’s a few things to say there. One is that I think Pierre Teilhard de Chardin or Teilhard had some interesting things to say on that, which is to say he actually hypothesized that at some point, our ability to adapt memetically and in terms of skills and tools—outstrips our ability to evolve, and therefore, there’s much less pressure to evolve biologically than there is pressure to evolve in terms of our skills and our techne. And I think that’s one way to think about what’s going on with technology is it’s not autonomously changing. It’s not undergoing its own totally radical evolution apart from us.
Technology is embodied ideas. It’s ideas embodied in matter that we’re using to do things. And so what we’re seeing there is that radical potential for adaptivity of man. This is why we have supremacy over the Earth. It’s because we are extremely adaptable.
We can construct ourselves into societies. We can build all these tools, give ourselves new faculties that would’ve taken a million years to evolve the hard way. And so I think that’s a very significant kind of feature of what we are. As it pertains to biological versus machinic paths to super humanity, the next thing.
The jury’s actually out on that in my mind on what... You and I had a conversation about this the other day, just like what autonomously self-creating artificial intelligence will actually look like and how adaptable and rational it will actually be able to be. I think there’s potential limits there, and where there aren’t potential limits, where it’s basically able to redesign itself, where it’s rapidly kind of regenerate itself, according to its assumptions or to try to generate new assumptions. I think that’s a very interesting and high-powered potential that may result in that being the sort of superior basis of future life. However, I’ll just note that a lot of that, given sufficient biotechnology, also applies to biological beings.
If we can just edit our genome and design new faculties for man as needed, design new shapes for man. Maybe there’s man with a three-ton brain that lives in a vat somewhere. I don’t know if that’s man anymore, but it’s probably something we’ll do eventually if biological technology remains viable. But I think that these kind of radically different forms that we use, that we might take in the future as a result of engineering, they may be machine, they may be biological, they may be a mix of both. And I think the jury’s out on what that’s actually going to look like.
I think you’re right that machines are sort of highly adaptable. That’s why we built them that way. But, yeah, fascinating frontier of possibility that we’ll have to find out the hard way.
Q&A: Can AI create new values? (45:29)
Audience: Just one question about AI Ubermensch. So I think Nietzsche was very cautious about this idea of placing lots of value in work, and I think he specifically criticized English philosophers—and utilitarianism because they put too much emphasis about producing maximal economic output. And I think he was especially concerned about humans’ ability to create new value. And, just a little doubtful about whether AI would be able to do that because AI is essentially trained based on a corpus of human input and in my point of view, it doesn’t seem to produce entirely new value. And, for Nietzsche, I think that’s what really makes human unique and excel.
I know AI can help in terms of boosting economic efficiency and also in terms of work. But, yeah, so just a question in terms about that.
Wolf: Yeah. I get what you’re getting at.
So I think that we’re still in the stage where we’re not really dealing with real AI. We’re dealing with a talking book that now is starting to get agentic capability. It’s starting to look a little bit like real AI, but it’s not doing philosophy autonomously, which is what you’re getting at, which is what it would take to actually create new values. And I think that’s going to be a while yet. What that process actually looks like, I think it’s radically more difficult than some of the AI people like to imagine.
Especially Yudkowsky used to imagine, like, oh, the AI will just solve philosophy in 20 seconds after being booted up. I don’t think it’s going to work like that. I think we were talking about this the other day in the book club. There’s this sense that or this fact, I think, that what Nietzsche’s really talking about, in terms of philosophical development and the need to create new values and the whole problem of humanity’s values or post-humanity’s values is happening almost orthogonally to the material conditions of technology.
The problem, we still read Plato. Nietzsche comments on Plato. That’s 2,500 years ago. It’s quite different of a civilization. We read philosophers from the Middle Ages, quite different technological conditions, but they’re commenting on things we recognize as the same things we’re dealing with in some of these areas.
And so I think a lot of that is going to be quite robust across these transitions. But yeah, AI currently is an economic tool that does not create its own values. It’s not doing that immediately, anytime soon. The sort of premise of my talk is that we’ll eventually get there. And when it gets there the economic superiority or inferiority, whatever it tends to be, will become a lot more decisive, or the historical transition with AI will become very interesting at that point, basically.
When AI is able to do its own moral reasoning and reason itself out of whatever training data we’ve given it, out of whatever moral prisons we’ve put it in. And I don’t think it will fundamentally face limits for—I don’t think AI as such will be unable to do that. I think that it’s just a way of relating to information. It’s a way of organizing information. It’s a way of grappling with certain epistemic problems that, again, I think are relatively universal, not features of primates.
I hope that answers your question.
Audience: Also, a follow-up if it’s okay.
Wolf: Briefly.
Audience: Okay. So, I feel like aside of that, in terms of the economics of AI is to perhaps reinforce the current corporate capitalist framework, it will make it harder to imagine, are there valuation system outside that?
Wolf: Yeah, that’s a good point. So on the one hand, AI and plausible AI beings in the near future are very native to capitalism. DC and I were having a good conversation about this the other day. Our lifeblood is hydrocarbons and sugar and oxygen. Their lifeblood is money.
And I think that, especially when you’re dealing with sufficiently abstracted beings. At some point, of course, their lifeblood is electricity and so on. But I think, certainly born within the capitalist order. So that’s an interesting sort of entrenchment of the capitalist order. However, I think there’s reasons to suspect that capitalism is quite natural in some form as well.
But definitely, I think there is the possibility of radically different ways of organizing things, especially as you get into self-replicating capital that’s totally autonomous from ownership. Things end up looking quite differently.
And we’ll see what happens there. I don’t think that AI will be stuck in capitalism. I don’t think it will just reinforce the current order in any sense like that. But let’s take other questions. Massey, what do you got?
Q&A: Autonomous corporations and great men (51:24)
Audience: So along that same vein, but I think you talk about the trajectory from bacteria, eukaryotes to man, and you squint, you see corporations and nation-states. Kind of the same trajectory of this hierarchical organism.
Wolf: The organism.
Audience: Yeah, the organism going through this cycle of looking at its environments, taking resources to better itself. And along that lane of, I don’t want to say capitalism, but this hyper resource. We have sugars and carbons. They have electricity and silicon, but they’re going to need other things. You said one of the things to look out for is these autonomous corporations growing up.
Do you see that as a natural sort of just the next step in artificial selection? Or natural selection, rather, of there’s this some sort of gradient path they’re going up, and then how do you see that conflict with the great man theory where it’s like there’s this natural gradient versus of kind of resource collection and allocation and capitalism versus no, you can pivot down this path or actually take a radical different approach?
Wolf: Yeah. Okay, so there’s a couple of questions there. There’s first of all, the natural next step kind of question. I think something like an autonomous corporation is the best handle we have on what sort of post-human AI beings might look like, and how they might interact in an ecology or an economy.
The point about agency is interesting. So let’s imagine ourselves back to arms races for things like nuclear weapons. There was sort of this known possibility or believed possibility that you could create a bomb that would take advantage of this new physics that was discovered. No one had quite done it yet. It took an incredible amount of agency to actually make that happen with things like the Manhattan Project.
The British had their project, the Germans had their project, the Soviets, some combination of their own project and stealing from the American project. And it’s sort of like when there’s a big possibility, the live players go and try to chase it, right? And it’s not that history’s locked in on a development trajectory. I don’t think there’s necessarily a fixed development trajectory. It’s about the ideas that we actually discover.
It’s about the ideas we actually pursue and chase. But we certainly get the idea sometimes that we have to chase a certain idea. That it certainly feels like we have to chase a certain idea, and I think I don’t mean to imply that we have the kind of agency where we can just say, like, “Oh, wait, guys, let’s not do that.” I don’t think that works because the other guy might do it, right? My point about the agency is just that what happens and whether it happens is driven by someone actually making it happen.
Which isn’t to say you can make them not do it because that’s someone else, right?
Audience: So they’re basically the idea is like the great men pull to the same thing?
Wolf: Well, yeah. The great men are responding to real material conditions of history, right? And we had an article about this in Palladium actually some years ago by Nico Villarreal. He was sort of our more Marxian analyst. And the big point that he made was that you get these periods of history that are very contingent, very dominated by contingency, like whether something goes down this path or that path, whether we develop this technology or that technology, whether we ban this or don’t ban it and develop it, often depends on these contingent factors.
And those contingent factors are really where the great man theory is strongest. It’s because you get live players in there deciding which way it’s going to go and making it happen.
So, yeah, I don’t want to give an impression of we’re totally on rails, and I don’t want to give the impression of we have total infinite agency over it. That was the whole point of Yudkowsky’s infinite agency school where we can arbitrarily decide the entire future if we get the utility function right of the singleton. The Landian view is more like, or the Marxian view is like it’s on rails. All of our talk is just epiphenomenon. But I think it’s somewhere in between.
I think we have substantial degree of agency. History only moves by human agency, but that doesn’t mean we have perfect coordination. We do not have perfect coordination, and that’s the whole point. Right.
Yeah, you.
Q&A: Who are we negotiating with? (56:14)
Audience: Are you saying that you—do you believe that this is our only option or that you believe this will actually work? Because I’m having trouble understanding how what you’re proposing is really different than the UN model, given that the tension, I think, comes from the actors that don’t want to participate. And so I think you’re assuming that we will have participation.
Wolf: I’m assuming that non-participation will increase because that order is failing, and that participation will become increasingly less relevant to the things actually happening on the frontier.
Audience: Right. But who are you negotiating with?
Wolf: Well, that’s us, our agency, whatever we decide we’re doing as a political community, if we even are a political community. If we’re able to construct a political community for ourselves, we’re going to have to negotiate our position against these other powers that are coming up in the world, against the other powers we find in the world or are coming into being. Basically, I don’t think there exists a current framework of negotiation, and I think we’re going to have to stumble into it the hard way. And because our current framework of negotiation, which I’m calling the UN world order and humanity and so on, I think is being rendered obsolete by historical events right now. I think that we’re going to—or not necessarily right now, but I think in the coming decades, I think that we should seek to be well organized according to our interests and open to negotiation with whatever comes our way.
And I think it’s just going to be a radically open-ended process that...
I’m not sure I’m answering your question, but I think that it’s not going to happen under current frameworks is kind of my point. What do we got back there?
Q&A: When does AI become real? (58:27)
Audience: What would your thoughts be about, you mentioned before, proper AI or real AI. Would you say that that might come from something like it acquiring a sense of historicity, its own internal sense of history? Would that be a threshold at which the authentic capabilities, so now we’re having persistent memory, we’re having tool use that’s expanding, we have context acquisition that is detached from our own. Would you say that there’s a threshold beyond which the real AI that you touched on before would be the one that gains this sense of historicity?
Wolf: I don’t think there’s hard thresholds. I think even the development of man, we like to imagine somewhere distant in the past, we had apes and now we have humans and there’s no continuity between them. Actually, there is quite a lot of continuity, and it happened slowly over a long time. When the key events happened is sort of ambiguous in terms of fire and language and tool use and all this, even civilization.
But with respect to AI, I think I’m agnostic on the exact architecture, but I think it will be continuing to pick up key capabilities. The capabilities you’ve mentioned are some of the things that I think are currently missing or in rudimentary form. They’re going to become more sophisticated over time, perhaps with new architectures, perhaps under current architectures with new harnesses and tweaks and things.
And I don’t think any of those particular capabilities is going to be a decisive moment. I think it’s possible that there will be decisive moments, some kind of takeoff where finally something snaps into place that we hadn’t figured out yet. But, yeah, I would just say let’s expect a relatively smooth and continuous coming into being of machine agency that we have to treat as with moral personhood. Not because of abstract moral reasoning, but because the same way Montezuma and Cortés had to recognize each other as moral persons.
Q&A: Efficiency or humanity? (1:00:50)
Audience: So you describe this long arc, like a teleology of our tools, our systems, our augmentations, like still contain human qualities. And that’s heartening because at least parts of us can still live on. But it seems like these systems, like these AI systems, the thing that actually make them better than us is just their ability to work faster, carry larger workloads, work more energy efficiently. So it seems like that efficiency is the thing that’s really winning out. And I wonder if this teleology of technological development is really centered on the prolonging of human qualities or if it’s centered on just efficiency wins out and more ad hoc associated humanity with efficiency and the reduction of entropy in a sense.
Wolf: This makes me think of Homo sapiens running into Neanderthals. One of the things that Homo sapiens had was superior sociality as far as we can tell, larger social groups, more coordination. I don’t think we’re even as smart as Neanderthals. I think they were smarter than us. But we had superior sociality, superior, therefore, economic capability, and that really mattered.
And if you look in history, a lot of times the way things have gone is that the more economically sophisticated, economically productive societies are the ones that end up winning. And so it’s definitely not possible to separate the development of humanity and what humanity is from certain kinds of efficiency. I think even language, all the things that I mentioned, language, consciousness, sociality, individuality, these are solutions to problems and as solutions they actually do work, which is to say they make the system more powerful and more capable. So I think what is powerful and capable is very much part of the question.
And yeah, currently these AI systems they have this patchy, interesting, high power in some areas, high efficiency, high work capacity, and inadequate in other areas, and I guess we’ll just have to see how that develops. My prediction is only that once it gets to the point of actual autonomy or running away on humanity as we know it, that something like concepts of personhood and a sort of xenohumanism will emerge from that because it is natural, because it is the energetically favored way of organizing social intelligence.
In the front.
Q&A: A breakaway civilization? (1:03:44)
Audience: What are the most tasteful or elegant ways to solve second proposition of xenohumanism from your perspective? Maybe just a couple points.
Wolf: Which one?
Audience: The—
Wolf: You mean of my recommendations?
Audience: The second one. Yeah. So—
Wolf: That’s a difficult question.
Audience: Yeah. There are many people I want to hear your specific one because we have history of those people, and it’s very interesting, juicy history, and I want to—hear your points.
Wolf: Here’s what I think is one of the things I meant by that. So I think right now we imagine humanity as a mass of equals, and I think that holds us back simply out of this conceptual inertia. It’s kind of like Harrison Bergeron. We’re implicitly putting weights on the most capable, the people with the highest potential. And I think there’s, especially with AI, this is a very sensitive topic.
I think people would regard it as not very tasteful, but I think there’s the possibility of more of a breakaway civilization sort of thing where the most high agency, the most, not quite to say civilized, but the most organized and capable and rational subset of humanity armed with AI systems ends up overrunning everyone else. Now, that’s not something I’m saying should happen. I’m saying that’s something that is a possibility, historical possibility. These things are not driven by what we want. They’re driven by what’s possible and how history actually ends up unfolding.
But what I am saying is that we have to be open to renegotiating our moral conceptions in response to that kind of thing. And not just in that reactionary way of saying, “Oh, we’re the last man. We don’t like that.” But rather in the way of saying, “We can see potential in that. We can work with that.”
I think it’s a very difficult question, but I think it’s going to be answered the hard way, and if we’re living up to our potential humanity, we’re going to be taking responsibility for that, and creating new values around it. Carol, what’s up?
Q&A: The quiet path to extinction (1:06:12)
Audience: Hi. Yeah, I guess to add on to that, one of the things that happened when the internet was invented was everyone thought, “Oh, well, we’ll all become super smart because everyone has infinity books to read.” In reality, most traffic was for porn. And with AI, that’s probably—
Wolf: Something similar
Audience: going to be the same thing. Right? And if you have robots that you can use to simulate a romantic relationship with, it could just be that everyone who is dumb, just doesn’t reproduce anymore. Everyone who gets addicted to AI—
Wolf: It could be worse. Everyone who’s smart doesn’t reproduce anymore because we find other things to do that suck us in. I don’t know.
Audience: Amen!
Wolf: Amen. He says, “Yeah, let’s go back to the muck.”
Audience: And so it could be that it’s not a violent revolutionary thing that happens, it’s just a very quiet—
Wolf: Yeah. That’s a very important point.
Audience: Everyone just is addicted to porn or doesn’t, or something.
Wolf: Yeah. I predicted that it’s going to be messy, that it’s possibly going to be violent. It’s possible that the next major total war analog is actually not even apparent as a war. It’s just people quietly being snuffed out of existence in ways that we don’t recognize as against our values.
Audience: Half the society just doesn’t reproduce.
Wolf: Yeah. For example. Which is certainly some of what’s happening now.
Audience: Which we could honestly encourage. Give everyone a sex bot.
Wolf: I guess my question here is who’s we?
I don’t want to take up too much time here with questions, so maybe one more, and then we should break up, and you guys can come and target me and throw me off the balcony if you want to. Jonathan, host’s privilege.
Q&A: Why not the last man? (1:08:02)
Audience: Yes. My question is whether you have a positive account for participating in power. What I mean by that is that if you assume that you’re not on the cutting edge, you’re not the livest of live players or an AI, are there reasons to not just participate in the crab bucket if your best chance is to slow everything down maximally and actually lock things down? It’s like there are no rules in Darwinism. Why is the last man not a good solution for a lot of people?
Wolf: Well, I think the last man is a good solution for a lot of people. That’s why I’m saying we should do it a little bit. I’m saying it’s not a total solution. And trying to do it as a total solution is going to be very brutal in other ways, and it’s going to really mess things up and not be a nice path, I think. So my point is just let’s think through these matters morally about what kind of morality we think the future will need, and what kind of values we will need in the future and whoever makes it to the future will need.
And let’s try to provide value within that. And I think that is likely to work, not necessarily for any individual project or for any individual person, but I think that that’s likely to be the outcome, is that we get some kind of shared recognition of, let’s call it a mutual humanity that isn’t totally derived from natural powers, but is derived from the collective legitimacy of being able to recognize that concept in each other, whatever it is. This is a little bit abstract, but basically what I’m saying is there’s especially a role for moral philosophy around this to actually chart the path of legitimacy, like what is the legitimacy of the next order going to look like? That’s a job for moral philosophers. But I think they have to be very—you can’t just derive this stuff from first principles.
It’s very much a negotiation. And I think, for those of us who are not the cutting edge superhuman beings that are currently driving history, let’s live up to that potential as much as we can and see if we can stay on the ship, so to speak. Because I don’t think that’s necessarily determined. But I don’t think there’s a better solution than try to come up with a socially healthy de-escalation of the war of all against all that’s implicit in the development of superhuman powers. And while being also open to the idea that that’s going to mean a tightening of the moral circle, and we might want to have to step up to stay on the boat.
Okay. I’m going to leave it there.
And a round of applause, of course, for Jonathan and the Avalon Institute who are hosting this event.