Richard Ngo — What Makes Hierarchy Legitimate (Human Or Not)

Richard Ngo argues against Wolf Tivy’s xenohumanism lecture. What is the moral ground for cooperation and hierarchy? We need the answer to build a good society today, or to build a good AI future tomorrow.

---

Transcript

Intro (00:00)

Wolf: Welcome, Richard Ngo. So you’ve been working in AI alignment research for many years now. You worked at OpenAI, and you’ve been also, I think, one of the most interesting sort of rationalist-adjacent posters on Twitter in terms of your engagement with a wider set of political ideas and ethical ideas, and even technically, I found your work to be very interesting. And so the other day, I gave my talk on xenohumanism, provocatively titled “The Problem of Humanity” and where I declared war on humanity.

You said that we should fight afterwards, and so here we are.

Richard: Great.

Wolf: So I’m obviously very interested in your take on the things. Let’s hold the things that I said in that talk relatively lightly. I will defend what I think is solid, and we’ll sort of try to find the crux of where the interesting dialogue actually happens here. Because I think you have interesting thoughts on AI alignment that are different from, let’s say, the Yudkowskian consensus that has kind of been dominant up until recent years. I have thoughts that are different than that, but they’re different from each other as well.

So this is—maybe an interesting discussion to have. So Richard, why don’t you give us an introduction to either what you think is wrong with my view, or maybe you could just—ask me to give more information on any part of it—just so we get us a better grounding. Or start at where you think is interesting.

Against radical egalitarianism (01:51)

Richard: Yeah. So I’ll start with the place where we definitely agree, which is there’s some kind of toxic, radical egalitarianism that—has implemented various taboos across the West. And it’s both anti-empirical, but also, in some ways, it’s contentless. And you can trace this back a long way. You can trace it back to the Founding Fathers, exactly.

Wolf: Yeah. “All men are created equal.”

Richard: Right. What does that mean? You can trace it back to Christianity.

And so, I think very reasonable to reject that. To me, it seems a little like you’ve polarized against that view. So I want to reject it, and—then I want to say, now that we’ve discarded it, where do we go? How should we think about things? And it almost felt like your xenohumanism talk was saying, “Okay, what’s the opposite of that, and how do we do it?”

Wolf: Yeah. I didn’t actually mean it to be the opposite. I guess there was two sort of positions that I was laying out that are a little bit difficult to relate. So I guess, as background, one thing that I claimed was that our concept of humanity is very close to Nietzsche’s last man right now. And this is one version of that claim about this toxic egalitarianism that has kind of taken root in our culture, is that we sort of imagine ourselves as this hungry, incurious kind of being that just needs to be protected.

And this is why I think that there even is so much hype and investment into AI, in particular, is like it’s a way, or this is one of my bold claims that I made, is that the AI hype is not just that AI is very promising, not just that it’s an economically, potentially very productive technology and so on, but it’s that it’s a place where we can put our higher ambitions towards—conquering the stars, becoming more intelligent, totally revolutionizing the world. Those sorts of ambitions have been kind of blocked off by our political self-conception, and so we displace them into something that is inhuman and therefore not subject to our self-limitations. So with that background, there was kind of two ideas that I’d like to separate in my view of things. So the first one is what we can just call the raw Nietzschean view. Right?

And the raw Nietzschean view, as I said in the talk, I think is just more or less correct in terms of its view of how the world works and what’s going on. And the problem with that is that it doesn’t appear to be possible to build a workable society on that basis. It’s too harsh. In particular, it’s a lot about this kind of radical competition and this sort of drive to overcome and so on, that if implemented as a social principle, doesn’t seem to actually produce a sustainable society. And maybe that’s what I’m going to identify with what you’ve said, that, okay, we’re just polarizing against egalitarianism.

Well, yeah, that thing does just—polarize against egalitarianism. But what I said, and what I would say, is that, okay, we can’t go there. Even if that’s kind of true about the nature of the world, we have to find some other basis for cooperation that we agree on. And this is me putting on my cynicism hat and not trying to directly embody or inhabit the kind of new mythology that we need in some way, but I claim that we do need some kind of new mythology of some type of kinship, some type of relationship with each other that isn’t just based on our immediate utility and our immediate power over each other and so on. It can’t just be the Nietzschean, raw dominance worldview, which isn’t really a good representation of Nietzsche, but it’s the stereotypical representation of Nietzsche.

And so, the way I put it explicitly is we need a new slave morality that isn’t the current slave morality. And, because the current one is quite broken, I think. And even if it wasn’t broken in the past, I think it’s quite obsolete.

So we have problems now that I think need to be solved with some kind of new social contract—so to speak. But that’s distinct. I guess I want to separate those things because I think you can have an analysis of the world that’s based on power and coordination and cooperation, but having that analysis doesn’t get you an actually good cooperation structure. It gets you maybe a clearer picture of things, but not a clear picture that you can use for social cooperation. And so you need this other thing, which is more like the, some concept, at least as we understand it now, some concept of shared humanity—or sort of—equality of some kind, some kind of moral reciprocity.

And what I proposed, in my view, which is, again, there’s two different things going on here. One is just the empirical claims about the nature of humanity and the nature of cooperation and power in society. The thing that I proposed, though, was that we should deliberately try to find some kind of moral kinship on a basis other than the last man mutual apeness—that is going to be flexibly able to incorporate AI agents into that—but then also potentially walk back these overly egalitarian commitments of current humanitarian ideology. Which is separate from the empirical claims about what I think is going on with the concept of humanity in—cosmic history.

Richard: Cool.

Yeah, let’s see. So I guess we’ve got this kind of kinship that you lay out. And maybe in a little bit I’ll stake out my own view, but first I want to ask what this kinship gets us because—

Wolf: That’s a good question

Richard: so suppose I say, “Oh, these AIs, I view them as my kin. They’re much smarter than me.” “Great, they can be in charge. They can do whatever they want because they’re my kin.” At the very least, you want a sense of reciprocity there, right?

You don’t want to count something as kin when they don’t count you as kin.

Wolf: No. Yeah, it has to be reciprocity, right? And I guess I’m not even going so far as to claim, like, “Oh, we should just let the AI off the leash and let it rip.”

Against astronomical stakes (09:43)

Wolf: I guess one of the implications of my view is that we should... Let’s say the first big radical implication with respect to the orthodox Yudkowskian view—is that there isn’t the whole value of the future light cone at stake, because that’s not on the table. No one is actually going to control that.

My thesis is basically the same patterns of life and value are going to re-evolve, recapitulate in various forms as life continues to unfold, and that’s just how the future is going to look. And so the question then is not how do we seize the future light cone of value in the universe and—buy a galaxy or whatever. It’s how do we live well now? How do we achieve good results now? How do we sustain the things that we value in terms of the wisdom and wealth and cultural inheritance that we have now?

How do we sustain those things through—the future, through the next while in the future, insofar as we’re able to control it, which is not the entire future, right?

Richard: Yeah. So I am with you on that. I think there’s something a little suspicious here about the talking... So, you’re saying it’s not the whole value of the future that’s up for grabs. I feel suspicious about the concept of the value of the future.

It seems like it’s taking a view from nowhere.

Wolf: Yeah. That’s one of the problems. That’s what I’ve been criticizing. Since I published “God Hates Singletons” in Palladium, 2023—the point of that essay was, okay, first of all, the whole value of the future is not really up for grabs for any given utility function. The secondary implication of that is that you shouldn’t even think about value in those ways, which is, I think, what you’re getting at.

And, so yeah, I wish to put that aside. Basically what I’m saying is this astronomical stakes argument that goes around in AI alignment circles—is wrong. I think it’s not the way we should think about the future. The way we should think about it is much more in terms of our immediate interests right now, our immediate agency, our immediate alliances and so on. And so, okay, coming back to this matter of what does it buy us to have some kind of moral ground for cooperation with—or I guess really the question is is like, what is our moral ground for cooperation with each other—as we build new coalitions—going into the future?

The moral ground for cooperation (12:11)

Wolf: Because you do need some kind of moral ground for cooperation. This seems to be a reliable fact of—how social groups work, how political coordination works. It’s not possible to construct, again, to come back to the earlier argument about Nietzsche, it’s not possible to construct a purely individual interest based—cooperation framework that actually sustains a society. Societies are always based on some kind of shared cosmology and moral framework and set of taboos even—that hold in place a worldview and a way of seeing things that everyone involved knows how to cooperate with and—knows their place within.

Richard: I would almost go further and say, before you can coordinate, you need to figure out who you are, what your identity is. And you’re going to be part of many different groups in many different ways. I am an individual. I’m part of my family. I’m part of my nation.

I’m part of humanity. So to me, when I think about cooperation, a lot of that is in relational terms in the sense of I’m trying to figure out, yeah, which larger scale entities I identify as part of, and then how can they flourish—and how can they interact with each other.

Wolf: So I would claim that those questions have to be resolved simultaneously. The problem of what kinds of coordination are we actually engaged in, what kind of coordination could we be engaged in, is very closely related to what we are and who we are. Right? This is one of Nick Land’s methods. Is rather than starting from value and then back chaining from your utility function to what you should be doing now, you start from the available set of options—and the structure of the available set of options.

And then identify value as a phenomenon within that. Like the way that, oh, if I take this action, then I get more agency to do more things. And so therefore, there’s this axis of value there where I can accumulate more wealth or power or whatever it is, or intelligence that you pull the nature of the kind of life that you’re able to live out of your actually existing options landscape. And then that is the empirical reality of what kind of being you are or what kind of life you’re capable of, and then that structures your value. It’s like a radically opposite way of looking at value.

Richard: Yeah. I’d go even further and I’d say, I’m just very suspicious of any attempt to try to cash things out in terms of a single spectrum of value. And—

Wolf: Yeah. There’s a lot of choice involved. There’s a lot of strategic ambiguity, where it’s not clear whether it’s this way or that way, and you have to make these leaps of faith and kind of construct it—actively. But what are you actually getting at there?

Richard: Okay. So, I guess, yeah, let me lay out a little more of a positive picture. So I think of the future as you’ve got these large scale entities like humanity or—America or so on.

Insofar as I’m part of them, I would like them to grow and flourish. This is extremely hard to define in terms of a simple objective, because being a healthy—organism is just... You can’t measure the health of a human body. You can’t measure the health of a nation or anything.

Wolf: Yeah. Not easily, for sure.

Richard: Yeah. And sometimes it’s useful to be able to look at two entities and say, “Well, on this metric, one’s doing better than the other.” But like—it’s not like—

Wolf: There’s nothing definitive about those metrics.

Richard: Right. And suppose I were to say, what’s the value to you of being an American versus being, I don’t know, Japanese? At that point, there’s no way to measure the value to you, because if you were Japanese, you’d care about different things.

Wolf: Yeah, you’d just have totally different worldview.

Richard: You’d have a different identity. There are a bunch of agents, by default, we can’t measure what’s good for those different agents on a single standard. But I would like those agents to grow and thrive. And so, there’s some version of the future in which humanity grows and thrives in a way that we construct our successes in the same way that I am constructing my future self—as we talk. And everything I do is, in some ways, aimed at making my future self the person I want them to be.

So, in a similar way, I want humanity, and also each subgroup and individual within humanity, to be able to construct the future self that they want to have.

Wolf: Let me challenge the concept of humanity there.

Richard: Great.

Wolf: So, if we name something, like even the concept of the United States is a little bit wishy-washy. But at least the United States has some kind of formal command structure that does, in fact, do things. And But humanity does not, and humanity doesn’t really have any organized agency at all. I sort of, let’s say, deny on empirical grounds the reality of this concept of humanity.

It’s like, yeah, okay, there’s the species of, or the genus of homo, right? The existing members of genus homo. But why is that a basis of any kind of collective agency if there’s not, in fact, any organization among them—of acting as humanity, right? What there are is—groups of humans, simply because humans are a good thing to build groups out of. And various levels of organization and kinship and comradeship and the various ways that we construct our relationships with each other.

And those things seem like the real things to me. And the genus homo seems like this kind of... It’s like, yeah, okay, it’s an interesting biological taxonomy, but it’s not a political thing.

Richard: Yeah. Well, I’d say that—

Wolf: Or it’s maybe not even a moral thing.

Richard: Yeah. I’d say that these coalitions get bound together much more tightly when there is at least some opposing coalition to distinguish them from. And so, yes, humanity as a whole right now. You can already see ways in which people are able to coordinate on things for the good of humanity. Like everyone agrees that nuclear war would be bad for humanity.

But then you might say, okay, well, they’re just coordinating.

Wolf: But it would also be bad—it would be bad for the individual actors as well. I would claim that the only coordination that happens on that basis, there’s sort of two types of coordination there. One is the sort of actual UN Security Council type, like, “Let’s sit down and agree not to destroy each other, because that would be bad for everybody.” And that, you get some level of coordination out of that. Obviously, it’s imperfect.

And I think it’s imperfect for structural reasons—not just like, oh, humans are bad at coordinating or something.

Then the second type is, let’s say, broadly speaking, coordination on a religious basis within the West. If we take Moldbug’s Cathedral model and we have—this huge set of institutions that forms the power structure of what we’ve called the West, which has somewhat acted as a coherent entity—against the rest of the world and against its own internal dissenters. That thing has a bunch of organized agency, and it does things like, “Oh, let’s go solve this global warming problem because that’s going to be bad for us.” And bad for our sort of—ideological conception of humanity.

Richard: Great.

Wolf: Or like, let’s solve this or that problem. But to be sort of realistic about it, that thing, it has as its own legitimacy narrative about itself that it kind of serves all of humanity. In fact, it is a narrower entity than that.

Richard: But those two things aren’t inconsistent. You can serve all of humanity by serving your own interests.

You might say we are doing scientific research for the sake of all humanity and also for our own sake. And then, I guess the question comes, when is coordinating as humanity useful? And the answer is when AIs are also intelligent agents, right? So, basically the category of humans as a coordination device is just going to become far more salient over the coming decades because... Yeah.

Wolf: Yeah. And I guess my claim would be, so yes, I think people are starting to get a lot more touchy about the concept of humans and humanity and these things because—of the rise of AI. But one of the things that I think is going to go on there is the opposite case, which is people are also going to become less reverent about the concept of humanity as they realize they have more shared interests with some faction that includes AI agents than they do with some other faction of humans that includes AI agents. You’re going to get—the empirical reality, or this is my prediction, is going to be humans plus AI versus humans plus AI is going to be much more of a salient—vector of political opposition than humans versus AI.

Richard: I think that’s possible. I think it depends partly on how people choose to organize. And I think that humans plus AI versus humans plus AI is pretty rough as a way of organizing precisely because the optimal strategy there is to give all the power to your AIs, and they give all their power to their AIs. And so, right.

Wolf: Which is the nature of warfare, right? In war, in arms races, what happens is the people or subsystems or entities that are providing the most kind of power—like war power, the sort of external power capability, get the most internal power. We saw this in the World Wars. You go from a society where science is kind of sitting on the outside and the high technocratic kind of modern state is—sort of this distant idea to one where post-World War II—the scientists are sitting at the center. The high technocratic modern state is sitting at the center, and they’ve completely marginalized everything else.

And that’s because that set of possibilities, those social technologies, the people organized in that way were able to provide the crucial types of power—during the war. And so yeah, in these conflicts, I think what ends up happening is you just give power to whatever is the most effective. And I don’t think there’s any avoiding that.

Richard: No, there absolutely is.

Wolf: Why?

Richard: Well, because in the long term, it’s bad for everyone involved. And so, you can look at a slippery slope and then try and coordinate to not go down the slippery slope. And I think that a lot of people who are worried about AI risk have the mental habit of seeing a slippery slope and then feeling like it’s impossible to avoid slipping down it, and thereby, being the first ones to jump down it.

Wolf: That is definitely true.

Richard: And I think you might just be doing that with xenohumanism. I think it’s like, cool, there is a slippery slope. Man, it’s hard to coordinate as humanity when everyone is sort of going to be tempted to identify as, “Oh, I am human plus AI against all of this other crowd.” But it seems like a very clear demarcation of are you human or not. Humans tend to have a lot of interests in common with each other.

We really care about the habitability of Earth.

I think that we care about...

Yeah, it’s just a very natural coalition to form, and so I think it’s worth trying to figure out how to aim towards that. Or at the very least, pursuing strategies that don’t actively undermine that coordination.

Wolf: Yeah. So I guess this is the question is, what level of possibility does that actually have? What does that actually look like? Because certainly, if it is possible for humanity, under whatever current definition, to sort of come together and stop going down that—slippery slope of development of more advanced AI technology or something that would—eventually supplant us, then that may be a valid move to make, right? It’s like, okay, well, we are organized, and we choose not to go this way because that’s—going to basically extinguish all of us.

We can see—where that’s going to go. The question, I guess, comes down to is that a possible form of coordination?

To put it more pointedly is, can you sustain cooperation of a large group like that against—a sort of hypothetical counterfactual enemy?

Richard: Yeah. Great.

Wolf: Because it’s like, I don’t know—

Richard: But increasingly non-hypothetical.

Wolf: Increasingly non-hypothetical, but I think the degree to which it becomes non-hypothetical is also the degree to which you’ve already kind of—lost some ground, right? But it’s like, can we stop it from going further—by coordinating among ourselves? And this is, I guess, the central question of cooperation. And I don’t think it is possible. I look at that and historically—institutionally—in terms of the ways cooperation tends to work, it doesn’t look like the kind of thing that is going to happen.

Richard: Cool. Yeah.

Wolf: It’s like I don’t see...

You can get things that are relatively more limited like, “Yeah, no one’s dropping nukes right now. Nukes are expensive. Nukes aren’t really getting us very much. Let’s not push the nuke thing too much further.” But it’s sort of harder to imagine, especially given the way the AI thing has heated up, it’s harder to imagine that there could actually be any kind of mutual disarmament on that front—between the various players.

Richard: Cool.

Wolf: So that feels like a crux of—what is the actual set of possibilities here.

Richard: Yeah. And I just think it’s not that...

The future feels very open to me.

Taboos on unfettered conflict (29:14)

Richard: We talk about coordinating on the basis of shared humanity, and I think one reason that you might be skeptical of this is that people have misused that in the past to mean coordinating in an egalitarian way. But when I say coordinating on the basis of humanity, I really mean Xi Jinping and Trump or his successor talking to each other, being like, “Hey, you’re human so I would prefer you to be in charge rather than your AIs to be in charge.” And the other one says, “Yeah, me too.” And they’re like, “Okay. Let’s figure out some way to make that happen.”

Wolf: Yeah. No, this is a very different form. The actual shared coalition of genus homo is based on real material facts. Like the fact that we prefer the air temperature to be certain parameters—and certain parameters of gases and—

Richard: But also just humans are less—

Wolf: not toxic, and there’s food

Richard: scary. If China were run by a superintelligent AI, then that sure seems much worse for the US’s prospects than it being run by a non-superintelligent human. Yeah.

Wolf: So this is the way that actually the two concepts kind of run into each other, because the other concept is kind of our current definition of humanity as being this kind of limited, non-scary thing that we try to make limited and non-scary. And I think China run by humans that are unfettered by the concept of humanity, even if we put aside AI. I think that would be more scary. And maybe even taking humanity out of it because it’s a little bit confusing with—the sort of shared interest of like being apes and so on. I think there’s things that all powers in the world today are doing, or nearly all powers, or at least the most advanced ones are not doing that could make them much more fearsome and much more powerful.

But they’re not doing it because there’s this kind of custom of not violating these kinds of taboos, I guess—around what kinds of coordination, what kinds of national interest you’re considering. And I think this sort of came out of the Second World War for obvious reasons. Basically, no one’s really doing nationalism—in the world. And—

Richard: China’s doing nationalism

Wolf: They’re sort of doing it. They’re doing a limited form of it where they don’t think too hard about what the nation is. And—I don’t think they’re doing it

Richard: feel skeptical of that. I think I wouldn’t know how hard they were thinking about what the nation is from the fact I don’t have a great sense of their internal discussion, but I would be pretty surprised if that were an accurate summary.

Wolf: That’s a tangent. But I’ll leave it aside. I do think that they’re ideologically, in fact, more communist than nationalist.

But—

Richard: Do they care about the century of humiliation, on your view? About who got humiliated for a century.

Wolf: The Chinese people. But that’s different from the kinds of nationalist organization, the kinds of nationalist goal seeking that they could be doing, I think. It’s not that distinct at this stage in history, but I think it becomes different as they become more powerful. But I just—I think that the way they’re coordinating, they’re maybe the closest to doing the real thing, but most countries in the world are actively holding themselves back from this. I don’t think that’s going to last very long.

I think one way or the other—these taboos are very temporary. But, and to your point about the future is open, it is possible to set up these kinds of customs. We did this obviously with nationalism. We did this—with nuclear power to a large degree. People still build civilian nuclear plants and—

Richard: Not really

Wolf: non-civilian nuclear plants that are—or nuclear plants that are maybe enriching some uranium. But we haven’t gone wholesale into the nuclear future the way—that the economics would imply—if you were sitting there in 1965 being like, “What’s the future going to be like?”

And so we could imagine we treat AI the way we treat nuclear power—or nationalism. Right? It’s like, okay, we’re not actually going to go there. That’s dangerous. But yeah, I think that the existing taboos somehow don’t support that.

And I think that you don’t get that—without a major war or something. And now, one of the comments I made in my talk was that we talk about AI right now, the same way we talked about race 100 years ago. Which is to say, it’s like this vast sort of frontier of power—and overwhelmingly important kind of sphere of competition—that we’re basically gearing up to have a major war over. In some kind of, 100 years ago, it was kind of this racial ideology arms race. Now it’s this—

Richard: Within Europe, or?

Wolf: Yeah, within Europe.

Richard: Yeah, okay. Fair.

Wolf: Just the ways that people were talking about it in terms of very—again, kind of very Nietzschean—very, very much about supremacy and so on. I think we’re talking about AI in that way now because AI is not fettered with these kinds of—any existing taboo against having that kind of ambition. Right? And this is part of my overall thesis. It’s like we successfully locked down the frontier on nuclear power—and racial politics to some extent, but it’s escaping out the side in various ways—into AI and a different kind of racial politics.

And I don’t think that’s going to get stopped without some kind of major conflict that’s probably going to reopen a bunch of other fronts—that are no longer quite so scary or something. But I expect, I guess, what happens in a war like that is that it’s not going to be that everyone... It’s only after a war, that after someone has achieved supremacy, that you get to establish taboos about how far we’re going to take this stuff. It’s like during the war, we’re going to get the war, it looks like, and during the war, people are going to accelerate AI capabilities to try to give them a hand up in the war. And this is already what’s happening in Ukraine.

China and the US haven’t really connected it directly to war aims, like active war aims yet. But you could see that happening. And then even within the United States, let’s say it’s not an external war, China and the US. Hopefully we can avoid that one. I don’t think it really serves anyone very well to have that war.

But, ongoing political breakdown within the United States—there could be some kind of major internal political conflict that also accelerates these AI possibilities. And so, yeah. I guess the larger point I’m making is, there’s a bunch of reasons that I’m skeptical that there’s actually going to be any kind of possibility of coordination.

Richard: Cool.

Wolf: I appreciate that the, like, oh, coordination’s not possible, therefore, let’s throw ourselves into the recursive self-improvement arms race, is bad and dangerous reasoning. I’m not saying let’s throw ourselves into the recursive self-improvement arms race. But I am saying we do have to think carefully about what might come out of that and prepare ourselves philosophically for it.

Outside the master-slave dichotomy (37:54)

Richard: Yeah. So can I try diagnosing the crux between us? So there’s the Nietzschean view that morality is defined by this contrast between the aristocratic class and the lower classes. And the taboos, the egalitarian taboos are basically agreeing with this view and just saying we want to invert it, right?

Wolf: Yeah. Well, since Nietzsche certainly it’s turned into that kind of—agree and amplify sort of approach.

Richard: And so there’s this dichotomy here where it’s like, well, either we have egalitarianism or we have master morality. And I think that there’s some sense in which I think you just agree with the taboos too much. You don’t agree that the taboos should be in place, but you do agree that without them, we head towards master morality. And I think I want to say, well, I have at least two alternatives to that. The first one is the Randian view, which is the pursuit of excellence in a way that is not defined at all, in contrast to lower classes or the bad or whatever.

You’re just purely orienting to the good. And maybe as one point in favor of that view, I would say that I don’t think colonialism was actually good for the colonizers.

Wolf: No.

Richard: And so you can think of colonialism as a mistake of master morality. In some ways—a kind of understandable mistake, like, “Wow, we conquer a bunch of stuff, surely that should help us somehow.” It’s just like, no, it doesn’t help you at all. And in the long run—it really fucks you up.

Wolf: No, it’s...

Richard: And so, there is this more Randian approach, I could say of it would’ve just been better for each European country to live in splendid isolation, focus on pursuing some vision of—what a really good society—

Wolf: Or—

Richard: looked like

Wolf: pursuing the kind of territorial ambitions that were played out in Australia and—and North America, as opposed to the kinds of things we did in India. But, okay, let me just respond to the Randian—the Randian view.

Richard: Go for it.

Wolf: So, first of all, I know Rand didn’t like this idea, but she really looks like a Nietzschean to me, in terms of very much valuing and venerating the creator, this idea of this visionary sort of superman kind of creative person who is the way that we should imagine ourselves and the way we should go. There’s a whole bunch of Nietzschean aspects to the way she looks at things.

Richard: Well, she’s not—

Wolf: But she’s not about—

Richard: She’s not defined in terms of the contrast and—and maybe Nietzsche also didn’t define it in terms of contrast.

Wolf: Yeah, I think exactly. To that extent, that’s not quite a good characterization of Nietzsche.

Richard: Yeah. So, maybe I should say, there’s the master slave dichotomy, and that’s the thing that people are stuck in. Rand, and to some extent Nietzsche, are out of the dichotomy. So that’s at least the first step that you want to have.

Wolf: Yeah. And so I agree that we should not be stuck in that dichotomy. There’s a bunch of reasons for that, and one of those dimensions is the problem of colonialism and slavery is the other thing I would bring up there. With precisely the problem that you brought up, which is that they were bad even for the master. They actually aren’t a good collective strategy as a society.

They don’t work at creating long-term wealth or power. They mostly just end up taking on a bunch of liabilities, in the short term. I think of them much more as socialized liability than as a productive strategy. It’s like, oh, we’re going to go and acquire a bunch of territory and a bunch of people that now we’re morally and politically entangled with. And all the other people in my society are going to have to deal with the consequences of that—whereas I just get to take the profits.

Right? And that’s the kind of social contract of those kinds of experiments.

Richard: Right. No, no.

Wolf: And that’s like there’s a caricature of master morality that would be like, “Oh, yeah, let’s go out and be slave masters.” But I think that’s certainly not what I would say. Whether or not that’s what Nietzsche would say, that’s certainly not what I would say.

Richard: Cool. That makes sense. Yeah, I don’t know if you’d endorse it explicitly. I do think there’s something, it felt like there was something creeping in of like, well, the people who are scared about getting rid of the taboos are right, that if we’re not egalitarian we move much more towards master morality.

Wolf: Well, I think they’re right that if we got rid of the taboos, some things that they’ve been suppressing would come back—very hard, and it would be very uncomfortable for them and all of their clients. I think that’s very different from, again, the caricature of master morality of colonialism and slavery and stuff. I think, in particular, the thing that I would argue for does include a substantial component of egalitarianism. But it’s egalitarianism within a coordinated group, as opposed to egalitarianism among people who are not—countrymen in that way. They’re not a part of the same society.

Healthy hierarchical relationships (43:20)

Wolf: I think within a society, you actually do need some kind of egalitarianism. You don’t want this hierarchical domination of caste systems and slavery and all this. And to the extent a society goes down that path, as we are now in America—and have in the past, it doesn’t work. It just causes a bunch of problems. You actually do want all men are created equal within a society.

Especially—if you restrict that to all Englishmen are created equal. That’s one way of interpreting what Jefferson said there.

Richard: Yeah. So I don’t like that.

Wolf: That works better.

Richard: Yeah. So, that’s where I want to talk about the second option. So, the Randian option is one step out of this dichotomy.

The Randian option is sort of what happens to people who aren’t creators in this way? What happens to somebody who isn’t cut out to be a Randian hero? I guess there’s still some sense in which you can be a great craftsman and so on, but realistically, it’s a long road to most people getting into that head space. The second option is what I’m going to call healthy hierarchical relationships. And I basically think I would characterize them as being constituted by bonds of loyalty and trust—between people who are in charge in some sense and—the people that they’re leading.

This sort of seems to me like the crucial missing ingredient when we are talking about any of this stuff. Because especially as we go into the wild transhuman future, we already have massive capability differences between different people. It’s just going to be far vaster. And you really just need some kind of ideal to point towards to say, “Hey, when you are in this kind of hierarchical relationship, here’s what it looks like to be a—

Wolf: Yeah. So you get this more—

Richard: good elite.” Yeah

Wolf: more like the kind of feudal social contract where there was unequal reciprocity.

Richard: But I think you want something. Yeah. I think feudalism was not doing us great because it was way too close to slavery. So it’s like—

Wolf: Yeah. No, and I would agree with that. Let’s maybe take a closer example, which is something I would much more be close to endorsing—is early Massachusetts Bay Colony, you had Winthrop, the governor—wrote this essay and gave a sermon called “The Model of Christian Charity.” It’s where the idea of the city on a hill comes from—in American mythology. But basically, “The Model of Christian Charity” is him laying out the social contract—for how we should be in North America, or how the—sort of Puritan Englishmen should be in North America.

And it’s basically this view of everyone’s kind of responsible for their own stuff, but then there’s also these bonds of charity that you have to have to each other—and when do you help people out who are actually in need of help? When are they just being indigent? And like—when is it a business loan versus a charity loan, and what are your various—obligations in these things? So he goes into all this detail about how to actually arrange those relationships in the context of—17th century commercialism and social relationships, and technology. But I think that’s the kind of, I don’t know, egalitarianism that I’m kind of talking about, which is that everyone involved in society is expected to be able to handle themselves in some sense.

But there is a very explicit recognition. He starts out the essay with, “God, in his wisdom, has created some high and in positions of wealth and fortune, and others mean and in poverty.” And like—basically, he goes into why this is even a good thing. But what he’s not saying is, “Oh, this is some kind of caste system. We have to lord it over them.”

It’s like there’s obligations owed to each other. There’s nonetheless a kind of shared standard of—humanity that everyone there is expected to meet. And so that’s what I mean. I think, in some sense, that’s a slave morality. It involves this element of Christian slave morality, but it’s not the full-on—resentment, tear down all hierarchies kind of thing.

I wouldn’t quite call it a middle ground, but it’s a very wise equilibrium between these different problems. And that’s the kind of thing that I think is necessary to make a society work, is some kind of social contract among the members where they actually all do have obligations to each other. And it’s not just like, “Oh, you’re on the down and out, therefore we can get rid of you. You’re turfed.” Right?

People have to be able to trust in each other that it’s like, “Okay, I can fall on hard times, and like-” “my neighbor’s going to help me.” And so I guess as we go into radically more technically empowered future where people are going to be even more unequal—and there’s going to be even more inequality in the world, rediscovering what that kind of relationship looks like seems like the core problem. And, so finding, I guess one of my interests within the xenohumanist project is to find what is it that we might recognize in each other there, and what are the obligations of that, if especially as we get to the point of some of those agents—not necessarily being human as we think of it, right? Of not being apes, not being—genus homo. And that branch of reasoning isn’t the whole of the project, but it’s certainly one of the branches of the project.

But that’s separate. That’s assuming we go into that world, right? But there’s also this question of do we go into that world, given your quibbles—on the problem of could biological humans as biological humans choose not to go into a world where we become obsolete.

Richard: Well, I don’t think of it as a binary in that way. It’s more like the more collective bargaining power biological humans have, the more capable we are of steering towards a world in which we’re confident that the AIs have this kind of healthy hierarchical relationship with us. So yeah, there’s a path thing going on here. And also, the better we lay out a vision of what it looks like. So a lot of my research right now is focusing on maybe the three most relevant concepts are trust, loyalty, and respect.

I think that there’s something about our standard frameworks, and especially economic frameworks, that just sort of makes it extremely difficult to reason about these concepts on anything like a principled level. And so I think there’s just a bunch of formal research, like game theory logic, that stuff which helps clear the way to think about those concepts in principled terms.

Wolf: Yeah. And this is a great example of the kind of reasoning that I think needs to be done is re-grounding a bunch of these things that we might think of as moral concepts in technical reasoning about the kinds of relationships you can have between—agents. Like putting aside particular religious frameworks or particular ideas of man as ape or natural being or whatever. Just what does it mean for two agents to be loyal to each other? What does it mean—for an agent to have honor?

What does it mean for agents to respect each other? I think those questions do have answers. I think the, I don’t know, the orthodox Yudkowskian sort of view would be that those concepts don’t mean anything outside of the context of the tiny sort of placid island of ignorance in the Lovecraftian sea of horrors, right? We have our little humanoid slice of mind space. And then the vast majority of possible minds are totally alien to all that.

I don’t think that’s true. I think there’s quite a lot of natural content—like that, that is the sort of moral concepts we associate with humanity are in fact natural concepts associated with agency and intelligence as such.

Richard: Yeah. Great. Feels like we’re mostly on the same page here. Two paths we could go down. One is I can say a little more about the critique I’ve been developing of the Yudkowskian, but also just mainstream academic and economic perspective, and then why they make it hard to pin down these concepts, these more relational concepts.

Another path we could go down is we both agree that healthy hierarchical relationships are very important, and then there’s this question of, okay, but how do you possibly stop that from turning into a caste system? Has anyone actually managed to stop it from turning into a caste system? Not really. Or maybe there are some types of caste systems that are—

Wolf: Historically, let’s just look historically, right? So they do often turn into caste systems, especially when the nature of power is such that you can get a very small number of actors to have power over everyone else. But nonetheless, still finds use for everyone else, right? It’s like slavery is effectively just a relation of various kinds of actual inequality, like responding to various kinds of actual inequality.

Which isn’t to say like, “Oh, we’re just treating the people different.” It’s like, well, the people are different, or they have vastly different amounts of power due to the technological conditions or whatever. And so you end up with this situation where it’s like, well, I can extract labor from the lower classes. And I’m going to keep myself insulated in my bubble—and use my power to extract the labor from the lower classes and enrich myself. And because I don’t really share any kind of deeper social relation with the lower classes, I actually don’t care who they are or their—quality as neighbors or whatever, so I can just bring in whatever foreign slaves or conquer these people—who are already foreign.

I go and—install myself. And that seems to be the sort of thing that definitely happens in history. It tends not to go well in the long run for the society of the masters, but it’s kind of this empirical condition. And then the thing that makes it go the other way is sometimes the technological conditions shift the other way. Like Napoleon and modern mass production of rifles and other kinds of guns.

I don’t think it was rifles quite yet, but modern mass production of guns makes possible this mass army of loyal people, who were formerly the peasants. Now they become equal, in some sense, more equal citizens—within a political order that’s based on the ability to mobilize mass numbers of people—in some kind of army or shared political act. And that gives rise to various kinds of social egalitarianism. It’s like, okay, well now because the power is shared much more widely, you have to have relationships that are much more egalitarian in a way. And it means that marriage taboos are going to break down.

You’re going to have people marrying across class lines. You’re going to have people having various kinds of society across class lines. And so the question is, first of all, which kind of future do we think we’re going into, and how do you respond to that—appropriately? There’s how people will naturally respond to that, and then there’s, is there a wise way to respond to that? I guess I’m curious if you would echo these thoughts.

What fills the vacuum left by decline? (56:53)

Wolf: It looks like we’re going into a vastly even more unequal future—with the nature of technology. Now, that may be somewhat complicated by what actually ends up happening with these bureaucratic systems that we have that underpin the property rights that are—the basis of the inequality. It’s not entirely clear to me that things are as anti-democratic as people seem to think in the future. Or undemocratic would really be the right word. I think everyone expects that like, oh, the state can just suppress everyone and impose its will, and democratic power doesn’t mean anything.

Which is to say, the power of actual groups of people who can coordinate with each other, who aren’t—the powerful elites. But I’m not sure about that. But it does look like, certainly default null hypothesis is that we’re going into a vastly more unequal future. Yeah. So let’s start with that.

What do you think about that idea? Are we in fact facing a future of massively more inequality?

Richard: So, I think one of the main things that grounds my thinking here is exit rights. And the thing about exit rights is that you actually... There are two reasons why it might not happen, like a group might want to exit and not be able to. One is just the infeasibility, logistically, of coordinating and physically, the wealth required to exit. And then the second one is oppression.

Now, it does seem possible to me that the first one starts to go away over time. For example, right now, suppose you and a bunch of 5,000 friends wanted to set up your own city state in some part of Africa that would let you buy the land from them. It wouldn’t be that hard, I think, to accumulate enough money that some African government would be happy to take that money in exchange for the land.

Wolf: Money is not the bottleneck, though. It’s the license to coordinate—that’s granted by the “New York Times” and the UN. Because people tried various things like this in the 20th century, and it was like UN boots on the ground within 72 hours of declaring sovereignty for some section—if they don’t like the project.

Richard: Yeah. So I think that’s an ideological thing.

Wolf: Yeah. Well, it’s not just ideological. It’s about the conditions of power, right?

Richard: But that’s—

Wolf: If there is a massive international power structure that’s able to prevent such things, someone will come up with ideology to justify its use. Now, the ideology does shape what kinds of use it gets. But—

Richard: Yeah. It feels like a side point. I think we probably agree on this. But the other problem is just like, man, it’s miserable to be living in the middle of Africa, and it’s going to be kind of rough to set up a city of your own. And that second problem maybe just goes away with technology.

Maybe it does actually become... How long does it take you to build a city the size, about as nice as San Francisco, for 5,000 people with a bunch of money? Right now, it takes a long time. With robots who are not going to be improving San Francisco because of the city laws, it maybe takes much less time. Maybe it takes five years or something—with enough robots.

Wolf: If you look at, let’s say, the development of Chinese cities now as the proxy on how—long it takes to stand up a new city if you have the capital—and the political license to do so, and the coordination and so on. They happen pretty quickly.

I would also expect that to kind of go the other way. Like, as technology makes more things possible, it doesn’t necessarily just make sort of bringing up the bottom possible. It also pulls the top away, which means that it might be—

Richard: Which is why San Francisco is going to be so beautiful in 15 years

Wolf: Well, I’m not saying San Francisco’s going to be any different. I’m saying that in whatever places actually do manage to take advantage of the new technology, those places are going to be so compelling that it’s sort of comparatively harder from a lifestyle perspective to leave them than it is now. There’s sort of this, let’s say, varying kinds of technological and social conditions that determine the comparison between going off and living in a sort of self-reliant way with you and 5,000 of your closest friends in the wilderness, versus toughing it out in a big city, or thriving in the big city, depending on the sort of historical period. This sort of goes up and down based on technological and social conditions, similar to the power of the elites versus the masses.

Richard: This feels fine because it’s actually we’ve just created these places that are really nice to live, where you can raise kids, for example. And you can have a good education system and be able to walk around without being harassed. And that seems great. Cool. Job done.

Or we don’t, and then that’s the case that I’m interested in where I’m talking about exit rights. And then—maybe there’s just very few places in most countries where it is possible to have the amount of autonomy that you want. And so then the existing order faces two choices. Either it comes in and just suppresses your new city—which right now is not that hard, because it’s sort of hard to build a new city. But once you’ve actually built the new city, maybe it doesn’t take you that long, and then the apparatus is slow to react, and like, okay, is it just going to come in and bomb you?

So maybe the point I’m making here is something like it does seem possible to me that exit rights actually become easier given a like... You have to be sort of more egregiously oppressive in the future in order to suppress exit rights than you do today.

Wolf: Yeah. I’m skeptical because I think, again, the baseline increasing power of centralization and inequality as a result of scaling of technology—is going to be sort of both source and consequence of what you’re saying in terms of technological improvements or how easy it is to do things like build a city. But that also relatively empowers the large central power structure that might be trying to suppress you and react to you. Like, okay—you get this fast new AI technology. You can move faster than any human institution.

Okay, well, so can they, right? But they have a bigger budget, and now they can hunt you down with Stasi Claude that like—

Richard: Maybe

Wolf: you have your personal Stasi agent AI that’s monitoring you for every little thought—out of alignment.

Richard: This all seems possible. I guess to me, it seems like the concern about all institutions becoming last man-ified and just losing all sense of desire and—ability to coordinate and so on. It’s a mixed blessing, right? Obviously it’s bad, but like one of the things—

Wolf: That’s maybe the other thing that we should predict about the future—is it certainly looks like the current crop of institutions are in their decay cycle—where they become more and more apathetic and unable to coordinate. But then that just means, okay, you get some new generation of institutions like, oh, well—

Richard: Cool

Wolf: maybe the Chinese spies actually are going to become more powerful in America than we’d hope, and the Mexican cartels—and the various ethnic lobbies, right? Take the space that the state leaves behind.

Richard: Or you and your 5,000 friends.

Wolf: And me and my 5,000. Yeah, well—

Richard: If we’re talking about a world where there are new powers rising, then I’m like, great, that’s the kind of world where new powers can rise.

Wolf: Yeah. The problem is when it’s like, to speak specifically with respect to Western people like us, our power socially is very caught up in the power of the existing state, and this was the thing we created to be the vehicle of our power. And as part of that bargain, it has power over us disproportionately. And as it goes into decline, a lot of the way it goes into decline is like autoimmune disorder, like going to war with its own people in various ways or like—just finding that it has less power, and so the only power it has is over its own people now. And it becomes this suppressive, like it—becomes the instrument of our own oppression, whereas other—people who are not historically bound up in it don’t have that same—relationship to it.

And so it’s like, yeah, you have the declining state, and that means a room for new things. That doesn’t mean we become free. Because we might just become even more oppressed by that system as it declines—because we have the historical relationship to it. And so that’s maybe the scenario I fear, right? It’s like how do we get out from under the state is kind of this crucial question.

And Nietzsche talks about this. We’ve been reading Zarathustra. He says basically where the state ends, that’s where the bridge to the overman begins. And it’s like there’s this—this essential need to get out from under the state because it has this last manifying effect—effect on you.

Richard: So, again, I think we might just be agreeing. I’m like, “Yep, there are bad possibilities, and there are also good possibilities.” And the thing I’m pointing at is more like, huh, we have two problems. One is that the state has become... Has turned against the kinds of principles and people that it used to be for.

And the second problem is that AI will dramatically disrupt the landscape of power. And I’m like, well, okay, maybe we have two problems, or maybe we have zero problems, or maybe they cancel each other out. And I’m not saying that like—

Wolf: Yeah, we don’t know how those are going to interact. I guess, yeah, this definitely feels like one of the crucial sort of future historical problems that we are faced with is, okay, we have the AI situation. AI is rapidly accelerating. There’s all sorts of things going on there. The technology is growing.

It certainly looks like we’re going to get more of that. And then on the other hand, we have this decaying sort of Western nation concept, or the Western nation state is kind of decaying, and you’re getting it. It’s—possible morph into something else that’s scarier or less scary or it gets taken apart for parts by various ethnic lobbies or becomes—some other version of itself or whatever. And how those pictures interact is the essential question, right? It’s like—are we going to see, like you say, they add up?

The one way they could add up is the state is getting worse and it’s getting more powerful—right? Because as it picks up the ability to use Claude accelerated Palantir to chase you down for every little opinion that you have—versus maybe the state just falls apart, and we’re able to use these technologies to kind of escape its power. I guess I’m more pessimistic than many people on this front.

Richard: Yeah.

I guess I, and I don’t know if this is cope, but the categories of optimism and pessimism sort of feel a little fake to me—in the sense that things really do feel pretty up for grabs. And especially even with Trump and Brexit and what’s happening in Europe, that does just give me a bunch more faith in democracy than I used to have. I’m like, “Oh, yeah.”

Wolf: No, likewise, I think that’s why I was adding kind of caveats earlier when we were talking about how democratic is the future actually going to be. I think there is this sense in which the kind of, let’s call it the broader extended elite consensus, has gone way more undemocratic than might actually be warranted—relative to the actual balance of power.

Richard: Yeah. And I do think that there is some culture and some set of ethics and principles and sort of intellectual understanding of what the hell is going on—that I think does just solve a bunch of these problems. In the same way that wokeness swept through all the institutions, it’s like, no, the good version of it can also sweep. Right. So it’s like a—

Wolf: And the question is—

Richard: ideology design problem

Wolf: the question is what kind of search process is going to create that ideological possibility, and what does the existence of that search process also create?

Are norms designed individually or collectively? (1:11:23)

Richard: Yeah. So should I tell you about my search process?

Wolf: Yeah. Well, let’s hear about it.

Richard: Cool.

Wolf: Because that is definitely of interest.

Richard: Cool. So I think the core problem that I’m trying to tackle is the sense in which existing frameworks are implicitly a kind of... They don’t know how to handle multi-agent dynamics. Basically our frameworks for understanding intelligence and rationality and so on are sort of—extremely hacky in any multi-agent case. So, of course, there’s expected utility maximization, where it’s like, “Here I am. I’m the decision-maker. Everything else is the world around me. I’m just going to optimize over it and try and control it. Okay, well, you’re in the world, so I’m going to try and control you.”

So that’s like baked into the framework.

Wolf: But didn’t that paradigm break as soon as it encounters game theory, like as soon as it encounters other agents? It’s been known to be broken for a long time, even though we still use it.

Richard: Yeah. So then I’d say game theory doesn’t actually solve this problem. It just gets around the problem. And the reason for this is game theory adds the concept of equilibria, but it doesn’t tell you how to get to an equilibrium. So we’re starting off, we’re playing a game.

Here are five different equilibria we could be in. There’s just no answer in game theory for how even in principle we should navigate between the different equilibria. And you can have equilibria. And there’s folk theorems in game theory that say, look, in iterated games, you can have basically any equilibrium. You can be in the heaven equilibrium where everyone is arbitrarily nice to each other because that’s what makes...

Because it’s stable for everyone to be arbitrarily nice. You can be in a hell equilibrium where everyone is arbitrarily cruel to each other because if they’re less cruel, then other people will be more cruel to them, and so they just start hurting each other. And so any equilibrium is possible. And so in that sense, game theory has failed to describe the process of like, “Okay, we’re playing a game. What should we actually do? How should we move towards an equilibrium?”

Yeah, so maybe the sort of core problem that I’m trying to solve is the problem that equilibria are a hack in the game theoretic setting.

Wolf: Yeah. And so how do you move beyond that? Because this does feel like the essential thing. You run into this game theory stuff.

One of the things I’ve noticed as you think through game theory and sort of, I don’t know, ethics in response to game theory and the kinds of decision theories that come out of that sort of stuff within the rationalist discourse and elsewhere. You end up reinventing a lot of concepts that sort of had traditional wisdom around them. The concept of honor I mentioned earlier, right? It’s like something that you can actually give something of a game theoretic foundation to. It’s like, well, I’m going to insist on certain types of treatment and—certain types of commitment to my own word and so on, not because in some naive utility maximizing way that maximizes my utility, but because the kinds of interactions that it incentivizes and makes possible with other agents is a much better set.

So that’s one way that you end up kind of recovering a bunch of interesting virtue ethics kind of philosophy out of things like game theory. But I’m curious where you would take—like going beyond game theory or getting those insights or how you would structure that kind of thinking.

Richard: Yeah. So I think there is a path to talking about things like honor in a game theoretic setting. I don’t think we’re there yet. And the reason is that one of the core reasons is this concept of the commitment races problem, which is just like, cool, I’m going to commit myself to being honorable. And now you’ve just got to take that for granted.

But then, surely then you should first commit to ignoring whether or not I’m being honorable so that then—there’s no point in me committing to being honorable. There’s a kind of—

Wolf: Yeah, historically speaking, it seems to be something that comes into being as a social equilibrium—rather than as an individual agent—unilaterally doing it.

Richard: Right. Yeah, and so in some sense the gap we need to bridge is between, like, hey, this is a stable equilibrium, but then how do the rational agents get themselves into the social equilibrium?

Wolf: Well, how much of that is just that a lot of these things kind of inherently have to come from outside of the agency of the agents involved, and it’s like there’s either some founder lawgiver figure or some kind of divine intervention—of some kind where the thing gets into the good equilibrium or the bad equilibrium, not for any reason that has to do with the agency of the agents involved. And because the challenge suggested by that is that, well, if they could do it by their own agency, they can also kind of undo it by their own agency or subvert it by their own agency.

Richard: Well, why would they?

Wolf: Well, why would they? But if they have the power to change the equilibrium, then maybe one of the agents hasn’t—now you’re playing a different game, right? It’s no longer the original game. You’re playing a game where at least one of the agents has the power to change the game in a way that—potentially—upsets that, and now you have a whole new set of—sort of problems and equilibria. And so it’s like whenever you sort of introduce the power onto the playing board of the ability to change the thing or the ability to resolve these kinds of coordination problems, that is itself this overpowered thing that changes the game.

And so my—my hunch is that as you run into that problem, at some point you just have to sort of recognize this kind of from outside quality of—of our ability to get into these things.

Richard: Cool. Yeah. So I disagree.

Wolf: Great.

Richard: And I think this is the sort of core crux between the classic conception of game theory in which it’s not able to talk. You’re in an equilibrium—and then it’s held together because everyone’s move is a rational best response to everyone else’s move. And so what if I do a different thing? It breaks. There’s no way to talk about why would I do a different thing.

We’ve assumed that I’m rational. Whereas I think that maybe most sociological phenomena are best understood as attempts to part of a process of making equilibria robust. So that’s why when you have norms, it’s not just that we’re all doing the same thing, it’s also that if somebody deviates, part of the norm is that you punish them for deviating. And so sometimes people can just through sheer force of will, set up a new system of norms by just refusing to get punished, and attracting others to follow them. But it’s actually a very hard process because the norms have this defensive—process baked in.

So that’s what makes it a robust equilibrium. That if any individual agent tries to change the norms, they’re pushing their way uphill.

Wolf: But someone set that up in the first place, and they had—

Richard: No

Wolf: they had some kind of large power.

Richard: No, that’s the whole thing about norms is you can’t mandate them. They grow via the actions of many people who each individually push a little bit in the direction of the norms they would like to see.

Wolf: I would say that, well, here’s a claim. The functional ones come out of someone designing it, and the ones that just grow are often quite dysfunctional, or they’re sort of one of the bad equilibriums or the middling equilibriums. Um—

Richard: So if we went and looked at a hunter-gatherer tribe, and they had some reasonably functional norms that said, “Here’s how you distribute the meat, and here’s how we decide who’s in charge,” and so on. Do you think they were designed?

Wolf: I think there’s probably some kind of authority figure that—would hold that together or something like that. Like—

Richard: But even—

Wolf: There can be—good equilibria that come out of the sort of natural process where if you throw people into the situation, and they’ll—find their way into the good equilibrium because it’s just sort of rational and it’s not that complicated of a problem. But I think historically, when you’re looking at kind of the laws of a society or the rules under which everyone’s coordinating—and the norms, it’s like they tend to be founded by particular political events where someone got a huge amount of power and like—

Richard: Ah, cool

Wolf: really reshaped things and built that. And that’s what I mean by this process of revelation, right? As far as anyone who’s living under that is concerned, that person might as well be divinely inspired, right? They sort of come in as this founder figure who—lays down the laws, and then everyone else is just kind of stuck in them.

Richard: Cool. Yeah, so I think I want to say that you’re reintroducing our master-slave dichotomy again, that I want to break out of. So, you’re like, well, look, either people are just bopping around in an egalitarian fashion, or you have this authority figure that lays down the law. And I want to say, well, actually, a part of the way that people choose what norms are in place is that they allocate prestige and respect to the authority figures that they trust the most. And what it means for a person to have enough power to lay down the law.

No individual has ever had the ability to—

Wolf: Yeah, they can’t—

Richard: like change things individually

Wolf: they can’t do it against—they can’t do it against the will of everybody. They do it because they’ve accrued the social authority—where everyone accepts their authority—on whatever basis.

Richard: Yeah. And so that process, I claim, is also part of the healthy process of individuals shaping norms—because they all push—

Wolf: That’s—that’s the process that you’re kind of interested in. I was trying to introduce, but there is that crux figure, who—who has to be the lightning rod of the equilibrium. And how do they end up in that position? What does it mean that that kind of person can exist? It can’t be that anybody can just do that willy-nilly.

But they do exist historically. That is where a lot of these things get figured out. But they end up in those positions of, I don’t know, folk legitimacy—because they incredibly demonstrate their ability to do it well—and their goodwill to do it well.

Richard: Yeah. Maybe William of Orange being the canonical historical example.

Wolf: Sure. There’s many. There’s many.

Yeah, we talked about that the other night at Jonathan’s talk.

Towards non-marginalist economics (1:22:56)

Wolf: So what kinds of theory are you working on that helps us—better understand that process?

Richard: Two things. One’s a high-level theory, one’s a very fundamental theory, I guess. The first one is I’m trying to figure out what it would look like if economics were not marginalist. That is to say, if it were illegitimate to talk about things like, well, let’s look at the effect of my action by holding everyone else’s actions fixed and just saying, “There’s going to be a small change on the margin.” Now, this is a big change because it gets rid of things like, well, here’s the share price of a company, right, and it’s defined on the margin.

It’s like—the marginal buyer and seller. And then, because you have a share price, you can multiply it by the number of shares to get the market cap of a company. And I’m—

Wolf: Yeah, which is a really stupid idea.

Richard: Right. So I’m saying if you get rid of marginalism, you can no longer talk about the market cap of companies, and that’s a good thing—because it’s a bad concept.

And ideally, we’d eventually want to be able to get another precise way of talking about how valuable is it to own or control a company. But like—

Wolf: Yeah. Or better yet, how valuable it is for the overall economy to have that company or something. That’s sort of what I imagine as the non-marginalist economics, is an economics where we can take the various parts and say, “Here’s what this thing is doing, and its—inputs and outputs, and why is that valuable?” And, “What are its capital stocks, and what are its liabilities?” In a way that is not as much about the buying and selling of the ownership shares—but about the underlying—activity that—it’s doing.

Richard: And I think this would look ultimately more like a question of how valuable is France for Europe or something. Where it’s like, okay, sure, you can measure some—things in monetary terms, but also just Europe without France would be very different. Maybe it’s—helping anchor the culture. Maybe it’s helping defend the continent, whatever.

Wolf: Mm-hmm. Do you think it’s possible to have kind of crisp concepts there, or is this inherently a question of, broadly speaking, the humanities, where it inherently is kind of qualitative reasoning that is very hard to pin down?

Richard: I think it will, at the very least, be as crisp as something like evolutionary biology, where you can’t measure...

Most traits of animals, it’s not like, “Oh, here’s a way of saying this is how much it contributes to...”

Wolf: Yeah, you can’t easily measure the relative fitness of—

Richard: Yeah. And the concept of relative fit or the concept of fitness in general is not very well-defined. Much less well-defined than I used to think in biology. But nevertheless, if you understand evolution, you just understand everything in biology far better, and you can think sensibly about all sorts of stuff that you couldn’t previously think sensibly about. So, I think non-marginalist economics, which will end up being moving towards a formal theory of sociopolitics—is ideally getting rid of a bunch of the precision, but replacing it with much more correct concepts.

So, that’s one research project.

Wolf: Yeah. And do you think that that relates to some of our earlier discussions about the question of interactions between humans and AI, or coalition dynamics one way or the other on those kinds of questions? Do you think that with such theory, we could achieve better coordination against slippery slope arms races? Or maybe that would just give us a crisp reason to understand why we can’t or—

Diving down the slippery slope (1:27:13)

Richard: Yeah. No.

Wolf: Or like—

Richard: Well, I think the whole concept of the reason that people slide down slippery slopes is because they’re using marginalist reasoning of like, “Well, it doesn’t matter if I go a little bit down the slippery slope because...” And I have this retrospective of the field of AI alignment. It’s like 20,000 words long. I just put out the second part of it a few days ago. And that is basically this detailed play-by-play account of how all of the AI safety people at these AGI labs were using marginalist reasoning to be like, “Oh, it doesn’t matter if I build the frontier models because somebody else will anyway.”

And it turns out that at all three AGI companies, the people who are building the frontier models were the ones nominally motivated by safety who were using that reasoning, right? It was literally this fucking comedy. Well, it would be comedic if it weren’t so tragic of just all of them individually saying, “Hey, somebody else will do it anyway, so I may as well get the safety benefits for myself.”

Wolf: Yeah. Well, this sounds a lot like being stuck in a bad equilibrium in game theory, right?

In the prisoner’s dilemma, it’s like, “Oh, well, the other guy’s going to defect, so I have to defect,” or, “At least the other guy’s decision is independent of mine, so I should just defect.” Right? And—

Richard: But the important thing is that if they had said, “Hey, the AI safety community is stuck in a prisoner’s dilemma with itself-” that would be very different. The thing they actually were saying is, “Oh, the AI safety community is stuck in a prisoner’s dilemma,” or stuck in a tragedy of the commons, I guess, with all of the AI capabilities people, and that just wasn’t true to anywhere near the extent.

Wolf: There were no such people.

Richard: Yeah. There were people. Obviously, I’m not saying all of the frontier systems, and specifically I’m thinking early Claude, ChatGPT, and then Sparrow at—DeepMind. It’s not like all of the work was done by safety people, but a lot of the work was done by safety people.

Wolf: Yeah. This is a very interesting topic, actually. One of the things that I observed was that the arms race with China preexisted any substantial results in China.

Richard: Right. Yeah. It’s insane.

Wolf: By which I mean the arms race in China, the meme—of the arms race in China preexisted any apparent attempts by the Chinese to dominate in AI or even to compete—in AI. And so, yeah, the Western labs were using “Oh, we have to beat China” as a justification for competing among themselves—in this uncoordinated way.

Richard: Well, it’s even worse than that. It’s not just that the people who are trying to push AI capabilities were using it as an excuse. It’s also that the EAs and AI safety people were playing up the threat of China—because they thought that that was the way to get taken seriously in DC, and then along with all the fear-mongering about China, they could also slip in a bunch of safety recommendations. And so, Situational Awareness, the memo is the worst offender here. Dan Hendrycks with Superintelligence Strategy memo, I think is doing some of this.

AI 2027 and AI 2040 are doing some of this. Yeah, and so it’s just this really incredibly destructive process of being like, “How do we cozy up to what we think the political consensus is,” while in fact being the people who are pushing the political consensus that they don’t want. Yeah.

Wolf: Yeah. Right, this kind of self-destructive anti-coordination is—is a very interesting phenomenon, and I think this is maybe getting to another interesting crux is, I’ll ask, why do you think that it is possible to come to a crisper theory that would be able to avoid this? What evidence do we have that that’s possible versus the alternative, which is that the crisper theory just formalizes why this always happens under any kind of... Maybe it’s not just marginalist economics, right? It might be that—this vulnerability exists or some analogy to this vulnerability exists within any sort of—worldview of coordination.

Richard: Yeah. Well, so I think one thing is that intellectuals are making this marginalist mistake far more than most people. And that seems related to the fact that economics is the dominant paradigm and—

Wolf: Yeah. Well, this—

Richard: economics is marginalist. So it’s like—

Wolf: This is really fun result in surveying various professions on game theory, and the economists do the worst because they have this like—this defect mindset.

Richard: Right. Exactly. And so it’s just like at the very least, if you make people take economics much less seriously, then you’re getting rid of one of the main forces pushing towards this destructive mentality. And, but then like—I’ve been told—

Wolf: No, and I think that’s true. Certainly, our society has been much more broadly than just in AI, torn apart by the sort of economic justification for looting—broadly speaking. There’s just all these ways that the economics profession amounts to just apology for various kinds of looting—and defection. And so, that I very much take seriously. I guess where I start to become interested is, yeah, we could have a better sort of basis for coordination, but—can we have crisp theory that actually avoids this failure mode or some analogy to it?

Can there be a final victory of truth over corruption? (1:33:20)

Wolf: Or is this a result, is the existence of economics kind of like in this format, the result of some kind of latent demand within the sort of space of possible coordination for this kind of corruption, right? Like the—is it just that you always have corruption and then corruption comes up with ideology for itself? And then to propagate itself. Or I think you can probably have healthier and less healthy versions, and it’s like, okay, maybe we’re further in the direction of being unhealthy on some of these dimensions—relative to maybe where we were previously or—other possible societies. But I guess I’m very skeptical that any of these problems will ever be permanently solved by better theory.

I think at best you get a temporary sort of, oh, we all see things in a less corrupt way for a while, and then someone figures out how to corrupt it. But—

Richard: Yeah. I mean, this seems like—

Wolf: But I’m curious what you make of that sort of line of argument.

Richard: Yeah. This seems like a bet against intellectual and scientific progress in the sense that, sure, no scientific theory is ever final, or none so far.

Wolf: Well, it’s certainly a bet against final victory in—coordination. But I wouldn’t say it’s a bet against intellectual progress because you could just as easily flip that around. It’s like, are we really going to bet against the intellect of the corruptors? They are also engaged in much progress, right?

Richard: But so yeah. So if I give some examples of success, so I think the invention of probability as a concept, the invention of computation as a concept. Now, I agree that these things can be corrupted, but it is just kind of hard to corrupt.

Wolf: Well, they massively improve our capabilities in some area. But that’s different from massively improving our coordination, or the possibilities of coordination apart from there’s technical means where—the coordination improves because now we can structure things in this new way where it’s not reliant on this kind of relationship, and instead we can lean on that kind of relationship. But it’s different from better theory solving coordination in particular. I guess my view is that coordination in particular, because it’s an adversarial game, it’s like the good equilibrium versus the corruption. Right?

And because it’s an adversarial game, there’s some way that the progress is symmetric. And—

Richard: Oh, I see. Yeah, definitely not.

Wolf: You think? Oh, you think so. Why?

Richard: Because you can just describe the corruption process. You can describe the forces—

Wolf: You can describe any corruption process. Yeah, sure. We could describe even the corruption process perfectly. But does that make it easier or harder to be corrupt? I don’t know.

Richard: Oh, it may—

Wolf: Or maybe it doesn’t change anything.

Richard: Yeah. I think bringing all the stuff, I believe in the shining the light on things cleans them up. So I’m like, yeah, if everyone understands the dynamics by which people get internally...

Yeah, by which this corruption happens, both on an internal psychological level, like the shadow, for example, from especially Jung’s concept of the shadow, and then also stuff like the way that norms tend to be corrupted. Even the concept of slippery slopes is an interesting piece of social technology that—

Wolf: Right. Okay, my claim would be that you can close off certain avenues—but it inherently kind of opens up other avenues. And we find it very difficult to imagine the sort of possibilities because to some extent, these things are happening outside of our ability to theorize them. It’s only—in retrospect that you can diagnose the cancer once you’ve really seen the result and seen how it worked and so on. But at the time, there was sort of no preexisting mechanism that could have recognized—that that was cancer because it occupied the blind spots of the system, and there were blind—spots of the system, and sometimes you can’t even tell that there’s blind spots of the system.

But my claim is there’s always going to be blind spots in the system.

Richard: You can make them smaller and therefore easier to combat.

Wolf: I don’t know. I’m skeptical of that.

Richard: I’m very confident of that. I’m just like, “Yep, I’m not betting against intellectual progress.” We just understand a bunch more stuff, like probability and computation and concepts and information, like other concepts we’ve formalized. They don’t help that directly with coordination, but that’s partly because they’re not about coordination. Whereas we start formalizing concepts related to coordination, and we start to be able to talk about what a norm is or what a—healthy relationship is.

Wolf: Part of the problem is that your epistemic environment, you can’t assume an uncorrupted epistemic environment when you’re dealing with these things because the corruptors are going to be trying to mess with your epistemic environment, and therefore the basis of your knowledge of these things. So as soon as you’re trying to say, “Okay, we have theory that allows us to solve these problems,” well, that’s equivalent to saying—there’s some theory that could be turned the other way as well, right? If someone got control of that epistemic process or corrupted the epistemic process in some way, where now it’s not... And this is almost what has happened in our society—is we have all these ideas of what it means to prevent corruption and prevent discoordination—and achieve coordination in society, and we have all these checks and balances. And those checks and balances now, largely speaking, exist to—facilitate corruption.

Richard: Right. ACLU is suppressing free speech and so on.

Wolf: Yeah. And so how do you... I think the cancer is as smart as you are, right? Is the basic problem.

Richard: No. Yeah. I think that’s just not true. I think the cancer is dumb by virtue of, and wokeness is dumb by virtue of being wokeness. Right?

It’s just like you can’t be—

Wolf: It’s dumb but it’s powerful, right?

Richard: Right. And there are viruses—

Wolf: And it’s done by smart people.

Richard: Sure. But that’s—

Wolf: It’s perpetrated by smart people.

Richard: Yeah. So viruses can spread. There are mind viruses.

Wolf: No, I don’t think it’s a mind virus. I don’t think it’s a mind virus. It’s that there’s people with interests that diverge, right? Someone finds a way—to get more out of the system for themselves by making things a little worse for everyone else, and they decide to do that. Maybe they see it clearly, maybe they don’t.

But in deciding to do that, now they have more power, and therefore more power towards the type of thinking that creates—that kind of problem. Right? And so as soon as you have this sort of—never mind the rational agents and the equilibrium, it’s just that there’s this kind of flywheel effect of—the defector gets more power, and then now you’re closer to an equilibrium of defection. And I think that this seems to be universal to life. Now, maybe it’s because we just haven’t discovered—

Richard: Yeah. We’re not—

Wolf: the true singleton operating system yet

Richard: all made out of cancer. Are we?

Wolf: No, we’re not. Because the cancer systems die and are replaced. There’s an other asymmetry on the other end, which is that once things are sufficiently decayed, the thing that it’s no longer able to enforce, it’s the decay, right? And so life is able to circumvent and come back. And so there’s this cycle between growth and decay, where when the growth has happened for a while, you’ve got a bunch of resources laying around that have been coordinated, accumulated in coordination.

There’s suddenly this huge opportunity for subverting that coordination, and that can be a self-compounding process. But on the other hand, once you’ve had self-compounding corruption for a while and now you’re back in the dark age, now the only thing to do, the only way forward is to create virtue and to create life. And so you get this—balance between them, but you never get a victory of one or the other because I think—it’s like we would like things to be nice and coordinated and honest because we’re engaged in the coordination and the honesty. But that doesn’t mean, and therefore, we’re going to apply our intelligence to solving things that way, but that doesn’t mean that someone else applying their intelligence doesn’t have other interests. Right?

Then they actually get more out of corrupting the system in a particular way. And so it’s not quite so simple just to say intellectual progress because the intellectual progress exists within the institutions under question and are influenced by the ways that corruptors can subvert these ideologies. I would even argue that maybe this is what happened with economics. Why is economics so bad for coordination?

Richard: Oh, yeah. Absolutely.

Wolf: It’s because people figured out, “Oh, we can come up with a sort of intellectual sounding justification for corruption.” And maybe economics was actually a lot healthier before that happened—

Richard: Oh, absolutely

Wolf: than I think it was. And I’m like, “Okay, let’s close off that avenue with economics.” That’s certainly a worthy project. But I think that’ll be temporary, even if you succeeded. And this comes back to the question of the singleton, right?

This is—why I think that singletons don’t work. I don’t think there actually is a final solution to corruption. There’s always more sort of blind spots somewhere, and that because it’s a adversarial game where you can’t assume that all the intelligence is on your side—you get these self-compounding corruption effects once you have a big ripe utopian society where everything’s coordinated. At that point, there’s this huge bounty for anyone who can corrupt it and crack it open from the inside. And so you get this feedback balance between growth and decay.

And maybe this should be a theorem in your new science of coordination, but—

Richard: I’m open to... Well, yeah. Right. Unfortunately, it’s the kind of thing that... I’m very open to the idea that there’s no way of locking in a good outcome.

I’m just like, “Yep, that seems right.”

You know what this reminds me of? Kuhn and his original work on scientific revolutions. He talked about paradigm shifts. And then he said that the next paradigm is not necessarily any better than the last paradigm. And that feels like the thing you’re saying to me.

I’m like, “We’re going to have a new paradigm.” And you’re like, “Listen, the new paradigm, it’s different. It’s corruptible in different ways.” And I’m like, “No, no, no. The new paradigm, it’s better than the last paradigm.”

“It still is going to make a bunch of mistakes, but it’s going to make smaller mistakes. And then the next paradigm, it’s going to make even smaller mistakes.” I’m just like, “Yeah.”

Wolf: Yeah. I would say with respect to coordination paradigms and ideology and the social basis for coordination and so on, the next paradigm is usually better than the previous paradigm, but they get worse over time. And so—I don’t mean the absolute progression is worse. I mean, you establish a paradigm, it works for a while, but then it decays as people find more ways to subvert it. And the result of this, anyway, to bring it all the way back to the beginning—the result of this is, I think that, I’m very skeptical of things like, again, the sort of coordination of humanity as humanity against—these highly valuable—technologies or potential political allies—that could crack open that coordination.

And so I expect that we’re going to go into that world. And now this, again, may be the sort of the traitorous kind of reasoning that you’re -

Richard: A little

Wolf: you’re attacking—as marginalist economics. And I recognize that. That is a very valid criticism. I’ve felt like that before, but just on reflection, I think, I don’t think it’s true that we can actually occupy that equilibrium. And so it’s something that I’m more interested in exploring to say, how then do we theorize the not necessarily arms race equilibrium, but the competitive equilibrium where there isn’t any kind of global coordination?

What is that actually going to look like, and what kind of ethics comes out of that?

Richard: Yeah. I’m reminded of Meditations on Moloch—which was a very powerful post. I think it’s mostly wrong. I think competition between nation states, competition between religions, basically all kinds of large-scale competition, are non-Molochian. There’s only one player.

Wolf: Well, I think they’re, broadly speaking, they create many good things even.

Richard: Yeah. I would say that the one place where there’s the most Molochian competition—in the world is amongst the group of people who take Meditations on Moloch seriously, namely the people at OpenAI and Anthropic. And Sam Altman said—

Wolf: How ironic

Richard: it is the one blog post he thinks about the most often. And EA culture is infused with Meditations on Moloch. And so I’m just like, “Yep, Scott summoned Moloch.” That’s just what happened. Not he’s created, right?

Many people involved, but just people read this, and it is self-fulfilling to the extent that, yeah, I think basically the single competition out of the large-scale competition, out of all large-scale competitions that ever happened, the one between Anthropic and OpenAI is best described as sacrificing everything in pursuit of this kind of competitive efficiency. Right? Because every other competition is between these nation states or these religions or things that are held together, even ethnic groups, they’re held together by non-economic forces. And—

Wolf: Yeah. Or they’re not about the competition. They’re not about trying to win the competition. Yeah.

Richard: And then this one is the purest, and so I’m like, “Wolf, watch out, there.”

Wolf: No, I completely agree with that diagnosis of the—concept of Moloch. Let me throw you something maybe strange here, which is that I think what he described in that process is not Moloch, but God. And—the result of taking that definition of what is effectively God, which is, if you take a sort of a Landian view on theology, which is that—

Richard: But why would you do that?

Wolf: Well, I do.

Richard: Yeah, but why would you do that?

Wolf: Basically the idea that... Well, I can answer that question, I guess. I think good things exist in the world. I think the good things were created by competitive processes. And in fact, that’s where the good things come from, broadly speaking, is the creative value in the world, the creation of new value and new good things comes out of competitive processes.

Richard: And cooperative processes working together.

Wolf: And cooperative processes. Mostly competitive processes in terms of—

Richard: Well, that’s just a vibe, man.

Wolf: I’m just saying, in terms of the way that evolution has worked, it’s like—it’s which things work, right?

Richard: Wait, have you—

Wolf: Which things work is the crucial—grounding.

Richard: Yeah. So this is the—

Wolf: And that is where good things come out of.

Richard: Yeah. I think that’s—

Wolf: And so what Scott tried to do is take a sort of appreciation for how you can get this defection equilibrium—and say that that was the source, that he generalized that way too far, and generalized it to the point of basically denouncing competition as the source of good, broadly speaking. And made all competition into a form of treason against this sort of Edenic equilibrium—that I think doesn’t exist. And in doing so, I think he is effectively engaged in some kind of rebellion against God, because he’s in fact defining himself against the creative process that has created all this goodness. And—what—what ends up happening when you do that is that you get that kind of like, I think you’re right, that worldview does summon into existence the kind of purely sort of, we must win the competition for its own sake kind of escalation—that is sort of treasonous in the way that he described. I think there’s this deep irony in that.

And this is one of the reasons that I like—try to attack the basis of that particular competition—because I don’t think they’re fighting over anything real. I don’t think any of them are actually going to get control of the future light cone. I don’t think there is any such thing as getting control of the future light cone. I don’t think there’s any such thing as astronomical stakes in terms of whether the future is going to be happy and utopian, or horrible. I don’t think that those are actually the possibilities on the table.

And I think if people took those kinds of possibilities less seriously, they would also engage less in that kind of purely, let’s call it—Molochian competition, and more in the kind of competition that I think is much more valuable, which is let’s pursue constructive projects, even against other people, but for our own ends that are not about the competition. And I think that’s been kind of my angle on that—discussion. I think the AI competition would be a lot healthier for everyone involved if they were pursuing their own ends rather than—just purely the competitive end of winning.

Richard: Sure. Absolutely. I think, well, a couple of things. My sense is that you’ve gone a little...

Yeah, let’s see. So I agree with the criticism of Scott. I’m thinking especially of his post, “The Goddess of Everything Else.” Which has the goddess of cancer as this kind of competitive force, and the goddess of everything else as a cooperative force. And it’s like, well, there should be this synergy between competition and cooperation.

It’s not like one’s just cancer and the other one is all the good shit. So there’s something off in that worldview. Land’s worldview goes way too far in the opposite direction, though, by sort of just biting the nihilist bullet almost. It’s like, oh, competition—extreme competition is extraordinarily destructive. So right, you want some synthesis—

Wolf: Well, this is where I’ve tried to kind of, I did try to respond to Land’s worldview, or the naive Landian—kind of accelerationism on that front, the nihilistic accelerationism. That I’ve sort of found that Pierre Teilhard de Chardin is a good antidote there. It’s a very similar worldview, but putting on the optimism glasses—in a way. It’s like, oh, this is actually beautiful and good, and this is what creates good things, and let’s actually take that seriously. And also—love is the basis for all order in the universe, and love is what’s ultimately being created by this process.

And that’s also true. And so Teilhard de Chardin’s “The Phenomenon of Man” is one sort of good antidote. Another good antidote is recognizing the importance of agency in the thing, because there’s, again, I’ve mentioned a few times, in the naive sort of nihilistic Landian worldview, there’s a discounting of agency, the sense that we don’t have control over the future. We’re just on rails, right?

Richard: But that’s also what you’re doing, right? You’re saying that there aren’t astronomical stakes. That it’s not possible to make the difference.

Wolf: Yeah. But I’m saying there are stakes. We do have domains—of agency—that we should act within, and we should try to make what happens in those domains of agency as good as we can. But I think one of the things that creates really bad equilibria and causes you to destroy the things in your domain of agency is to think that there’s some bigger value outside of your domain of agency that you’re going to try to seize. Right?

You sacrifice your actual domain of agency—for the sake of an imaginary utopian endpoint. And that’s what’s going on with the AI lab competition, right? Is they think they’re going to get control of the future light cone and have astronomical stakes, which means that any amount of destruction of our actual domain of agency—is justified on—the utility calculus there.

Richard: Look—

Wolf: And so I think that—you have to be clear about what is your actual domain of agency and treat it as your beautiful garden, right? It’s like, how do I make my actual domain of agency good? And this is where I’ve tried to provide the antidote to the problems of the Landian worldview.

Richard: Yeah. Right. And then I think I would just say, it is correct to focus on smaller domains of agency. I think there are ways to do that without calling the larger scale agency fully illusory. I don’t think—

Wolf: Well, sometimes it is, and sometimes it isn’t.

Richard: Ah. Right. But it’s hard to know.

Wolf: Because sometimes you think it’s illusory, but that’s actually an illusion itself. And it’s like—it depends on your own ambition that—

Richard: Yeah. Exactly. And so I kind of want to say, actually, it’s like, yes, maybe the universe is up for grabs.

Wolf: The universe is up for grabs, but not by any coherent—

Richard: Well, that’s just the lack of ambition

Wolf: coordinated thing. No, well, this is the thing, this is the hard challenge, right? Is to tell the difference between delusion and ambition. And I think it is epistemically very difficult, but I think on reflection, the light cone eating singleton kind of idea is on the delusion end and not on the ambition end.

Richard: Yeah. That’s important.

Wolf: I think it’s not actually a real thing, and I think that believing in it causes these sort of terrible coordination outcomes.

Richard: Sorry. Right. Maybe the synthesis here is something like, actually the universe is up for grabs if you are capable of being principled and high integrity and virtuous in a way which manifests, for example, in not just trying to barrel ahead—under the grip of these competitive forces. It is true that the things we do now will have repercussions that ripple throughout the entire future of humanity. And also, the only—

Wolf: But not so much arbitrarily. The thing I—

Richard: Maybe arbitrarily, but only if we’re actually able to follow good cooperative principles now. Yeah, I think that’s where I—

Wolf: What I expect is that, yes, life will continue to unfold and eat the universe and so on, and things will change on the margin depending on sort of what we do at crux times, but not so much and not in any predictable way. You might get stuff like QWERTY, right? Where maybe QWERTY is just locked in for all time. I don’t know. But—I think fingers are probably going to go away at some point, and then QWERTY will no longer be a thing.

But things like that, right? Unix maybe is locked in for all time or whatever, right? English language may be locked in for all time. But they tend not to matter, right? They’re like these arbitrary little technical details—that aren’t the important things.

I think what you don’t get is massive unbounded loss over the far future because it’s just not really possible to maintain the kind of order on that timescale or on that distance. And if it’s not, then we can kind of dispense with those sorts of mythologies. The only question as far as I’m concerned is like—do we sort of make it through the great filter or not? Or do we knock a big chunk off of our potential by slowing ourselves down for another thousand year dark age or something? Those kinds of things suck.

Let’s not do that. But I guess I’d put it as a quip like, if it keeps going, we win, right?

I think the best far future outcome that we actually have control over is whether we get anything there at all—versus just kill ourselves in some way. Which death does appear to be a possibility. We could in fact wipe out all life on Earth if we—did totally the wrong thing. But yeah, I think it’s just a way more realistic, and therefore healthier view to say, okay, there are limits to coordination, there are fundamental limits to coordination, therefore the kind of astronomical stakes arguments don’t work. Therefore, we should be focused on more immediate domains of agency and what we would like to see in those domains of agency for their own sake and not for the sake of their future impact.

And I think that is the only domain of value actually accessible to us, and if we sort of competed over that, I think we’d get much better outcomes—in that sort of sphere of value—which I think is the only one.

Richard: I don’t think I disagree that much with your direct prescriptions here. I think that getting to those prescriptions via the route of this is the only one is going to come back to bite you, and yeah. And so I’m like, look, we’re broadly on the same page, but I think it feels like there’s something that is currently load-bearing here. It’s sort of like, oh, the way you get rid of infinities is by multiplying by zero. Or infinite stakes.

Oh, but it’s got to be, like, the only way to get rid of this infinite stake over here is to say it’s impossible. Like zero probability. Great, now we can focus on this other stuff. And I’m like, “Oh, multiplying by zero, that’s kind of rough. It’s very dogmatic.”

I think you can focus on the more immediate domain, while maintaining—

Wolf: Well, where did—

Richard: agnosticism about the larger scale stuff, and just like, “Look, there’s going to be...” I think there’s—

Wolf: I guess my claim is that the focus on the infinity in the first place came by dividing by zero, so to speak. In the sense that you sort of... There’s an assumption—

Richard: But it’s not—

Wolf: There’s an assumption that you can drive coordination error down to zero. And if you can drive coordination error down to zero, then you get this huge infinite result.

Richard: Right, it doesn’t need to be infinity. It just needs to be very large. And I think you’re—

Wolf: No, but I—

Richard: trying to multiply it by something that’s small enough to bring it down to a manageable degree.

Wolf: I’m not... Historically, where I came onto this kind of thinking was I was very much in the, let’s say, orthodox Yudkowskian camp. I wrote a blog post called “Capturing Gnon”—which was a sort of view on... It was like a Landian interpretation of Yudkowsky’s project. And that blog post directly inspired “Meditations on Moloch.”

If you read—

Richard: Oh, wow

Wolf: if you read “Meditations on Moloch”, one of the things—

Richard: That was you?

Wolf: it cites that post that I wrote.

Richard: That’s wild. I love that

Wolf: I chewed on this stuff for 10 years—of believing that there were astronomical stakes—and that therefore various things. And because of that overwhelming importance, you have to focus on, how do we have to get there? How solid is this actually? And what does this actually look like? What possibilities are there?

And all this. And eventually, I just realized that the whole thing had kind of divided by zero. It was not actually a correct assessment of the kind of agency we had over the future. And I credit this to Nick Land. I got the antidotes—from Nick Land.

And I’ve tried to kind of overcome the problems in that worldview as well. But, so in terms of my historical evolution as a thinker, yeah, it’s like coming to believe that this sort of set of possibilities was impossible. And then—that, I had to kind of go and update my whole—value cosmology and so on as a result of that. Now, it might be that after you get through all this, you can come up with a much better framing of it. I agree, probably there’s a much better framing that doesn’t come as a result of explicitly like, “Oh, that’s closed off to us, so it drops out as a term.”

I think that’s my response to the Yudkowskian view, simply because I developed that idea as a response to the Yudkowskian view. And we’re talking about these Yudkowskian AI labs, right? But in terms of my actual worldview, I think how I would justify it more directly, I think there’s probably a much healthier way to do that. We should probably talk about that sometime. I think that’s another discussion.

But, I think there is an optimistic, directly stated view of what kinds of agency are available to us—and why that’s good, why it is that way, and what the good things we can do with that are. I think that you can just frame it all entirely positively. And I think that’s a useful project. It’s just not something—

Richard: I think it’s—

Wolf: It’s not something I emphasize in response to the Yudkowskian stuff—because it’s not as directly—applicable to that, and maybe it’s also more incomplete, right?

Richard: Yeah. And I think that’s basically my project and this thing you were describing as what’s the healthier version that doesn’t need to shut the door off to astronomical stakes is—what I’m going for.

Wolf: Well, I think—

Richard: And so I feel pretty good about just like, yep, we’re both seeing the shape of something here. I have some sense that your current framing of xenohumanism is optimizing a bit, trading off a bit against that something. But maybe, yeah.

Wolf: Maybe. These things always need more work. I guess the one big crux question that does matter is, does the integral blow up or is the integral finite? In terms of that long tail of future impact possibility.

Richard: I don’t believe in adding things up.

Wolf: Well, maybe that’s a way of saying the integral’s finite. But—

Richard: No. No.

Wolf: No, because it’s like you have this problem of... Maybe this is, again, the problem with the expected utility framework, right? But if you take anything like an expected utility framework—then there’s like—

Richard: We’re done. Yeah

Wolf: if there’s the possibility of an infinite—utility or a dominating utility that has non-vanishing possibility, then it blows up your integral, right?

Richard: Yeah. So we get out of that framework.

Wolf: Yeah. So let’s get out of that framework. I don’t really like that framework either. But it’s for reasons of tending to blow up in that way. Some of which I think are fake, like this one, and some of which maybe are real in the sense that if you define your utility function like that, it blows up.

But I guess that is the sort of... It sounds like, then, you’re on my side of that problem, which is the let’s not act as if the integral blows up.

Because that’s the mistake that I think the frontier AI labs are making—

Richard: Well, yeah, but I just want to—

Wolf: is they just think the integral blows up.

Richard: I just want to say, no, and I think you need to go one step further and be like, “No, there is no integral. The integral is a lie, Wolf”

Wolf: Well, that sounds like fun. We should have that next time.

Richard: Let’s do that next time. Yeah.

Wolf: Yeah. I think we’re probably out of time, though.

Richard: It seems, yeah.

Wolf: But yeah. Responding to utilitarianism and providing a coherent alternative to utilitarianism does sound like a good project as well that we should do.

Richard: Yeah. Oh, yeah.

Wolf: Well, this was a lot of fun.

Richard: Thank you, Wolf.

Wolf: We wandered all over the place, but I think there’s a coherent thread.

Richard: I think so.

Wolf: All right. Well, thanks a lot.

Richard: Cheers.

Wolf: Until next time.