Samo Burja and Wolf Tivy cover the political economy of AI: How AIs will reshape bureaucracies, how some governments will rely on AI labs like petrostates rely on oil, and how AI apocalypticism will struggle against careerism.

Subscribe now

Samo’s essays: https://samoburja.com/essays/

Samo on Twitter: https://x.com/SamoBurja

Wolf on Twitter: https://x.com/wolftivy

The Students is sponsored by Zcash

Jason Crawford and Ben Landau-Taylor cover the Industrial Revolution, the acceleration of technology, and how our increasing awareness of technological progress as a historical force has altered society.

Subscribe now

Jason on Twitter: https://x.com/jasoncrawford

Roots of Progress: https://rootsofprogress.org

Ben on Twitter: https://x.com/benlandautaylor

The Students is sponsored by Zcash

Danielle Fong and Ben Landau-Taylor talk about invention—how new technology depends on specific individuals, successfully not burning down the science shed while doing the early experiments, taking advantage of AI energy demand, and more.

Subscribe now

Danielle on Twitter: https://x.com/DanielleFong
Lightcell Energy: https://www.lightcellenergy.com/
Ben on Twitter: https://x.com/benlandautaylor
The Students is sponsored by Zcash.

Duncan Umphrey and Ben Landau-Taylor cover the power of book clubs, how and why you should start your own, and how new ideologies develop from these seeds.

Subscribe now

Duncan on Twitter: https://x.com/dunc914
Ben on Twitter: https://x.com/benlandautaylor
Join The Students at our book groups and other events in San Francisco: https://luma.com/thestudents

Guilherme Freire explains why his elite training program uses the classics to cultivate virtue and action in business, and how this can lead to social and political rejuvenation.

Subscribe now

Guilherme on Youtube: https://www.youtube.com/@guilhermefclfreire
Guilherme on Twitter: https://x.com/guilhermefclfr
Wolf on Twitter: https://x.com/wolftivy
Join The Students at our upcoming events in San Francisco: https://luma.com/thestudents

Transcript

Training a new aristocracy (00:00)

Wolf: Welcome back to The Students. So I’m here again in Brazil with Guilherme Freire. He’s a philosopher and something of an influencer, and he runs a company that’s very interesting and unique in Brazil, which is training the future elite of Brazil in philosophy as applied to business. Now—what’s interesting about this is that it’s an approach for how to restore civilization by training a new elite—in an aristocratic pattern, in a philosophical pattern. So I found this very interesting.

So Guilherme, I’d love to hear, first of all, why don’t you give kind of an overview of how do you create a new elite to restore our civilization? I think Brazil has many of the same problems that we have up in the United States.

Guilherme: Yeah. That’s one of the biggest problems we have in the world right now actually, because we lost a lot of the ethos to create a new elite in the world. People don’t want to be associated with the idea of, “I’m a member of an aristocracy.” But really what aristocracy means, and Aristotle used it that way, is just the government of the best. It’s just striving for excellency and having people of excellency ruling.

And so the first thing you need to have an aristocracy is to actually trying to prove your own life to live virtues, to live a life of self-sacrifice, to grow in temperance, self-control to our pleasure, like fortitude, which is to face adversity and increasing adversity. Justice, which is the sense of having proportion and doing your duty, and prudence, which is practical wisdom. Now, if you have all the very famous cardinal virtues, you’re starting to control yourself. Now, if you can control yourself, you can actually achieve a lot of things, and you can start to spread this through the next generation throughout a family. So the family unity, since the Roman times, one of the foremost schools of leadership, of ruling because you need to have the sense of duty to raise the next generation.

In our company is a very natural extension of that because you have people who really, all the good things we have in the world come from labor, come from work, come from the fact that people sacrifice themselves for excellency to actually have to build things. And this culture of builders, once you have someone who’s actually striving toward virtue, they can actually assemble people of excellency within the company. So, the company structure can have a sort of what Aristotle called a mixed monarchy. Now you have a monarch, which is the owner, the CEO of the company. The aristocracy, which is the people of high capacity, next to the CEO, and then you can have a system of merit, of sacrifice, that is a republic.

Because republic actually means—a system in which you grow in responsibility insofar as you grow in service to the organization and in merit. Now, if you have this sort of mixed structure, then the company works as a school of self-sacrifice, and it has to be agile. It has to actually work. You need to deliver stuff, otherwise the company goes bankrupt. Now, if you think of the state, if you have people who are living those virtues, who are tending to the next generation, and who are actually making things work by having high-capacity people, being highly productive and looking toward the common good, towards self-sacrifice, then entrepreneurs can come together to start changing all sorts of things.

Like media could start telling the truth. Imagine that. Media that strives for the truth. Like education institutions, the whole HR system that’s actually today very geared toward bureaucracy and cancellation and whatever. And if they change that, and they’re actually striving toward excellency, then entrepreneurs together can change a whole lot of things, like from local politics to the whole nation.

And so what you’re forming is a new type of aristocracy. It’s not people with wigs in 18th century France. It’s actually you’re selecting people to have a position of power based on how much they want to actually change things for the better, to actually assume responsibility for things. And people have been taught in this bourgeois sort of human rights, bureaucracy, UN sort of the machine mentality, like you shouldn’t strive to be excellent. You shouldn’t strive to be this leader of society.

You actually should hide somewhere, make your own money, turn away from anything that strives toward the good. But this is a very unreflected life. It doesn’t work within the companies because if you just let the bureaucrats run the company, they ruin the company. So it’s not going to work. It doesn’t work in the family because if you just delegate your education to the random public school in your neighborhood, your kids are going to be terribly educated.

And it doesn’t work in a country because you’re just outsourcing everything, and you’re destroying your own country. So, Tocqueville, in the second part of “Democracy in America,” he said how America needed an aristocracy to protect it from the subtle tyranny that could arise from Washington. Now, people don’t tend to think those terms because they tend to think, “Well, but aristocracy, isn’t that destroying my liberty? Isn’t that destroying my freedom?” And it’s quite the opposite because it’s actually people who are highly capable that can protect other people’s freedom or the common good.

If you don’t have people who actually have power, who have capacity, you can’t protect common people. It’s the same in a company because one of the functions of the monarch, Aristotle actually speaks of that in the “Politics” in, I think it’s book 12 or something that he talks about it. One of the functions of the monarch is to actually protect people who are striving toward the common good from the evil elements, from the bad elements within an organization. Because the bad elements, they can lie, they can manipulate, they can destroy things. And if you don’t have a monarchical figure, who is to do away with the oligarchs within an organization?

You don’t have the mechanism to get away from that. So by teaching responsibility, monarchical thinking and responsibility thinking and leadership in this classical sense within companies, the company leader can get rid of the oligarchs, and actually start building beautiful stuff and promoting actually competent people. And if they come together and do that in society, and they’re already doing, our students are doing that in Brazil, they can at least at the local level, start changing things beyond what they thought they were capable of.

The founder against the bureaucrats (07:23)

Wolf: Yeah. So implicit in all of this is something of a thesis of what’s wrong with society and what can be done to fix it. So the diagnosis as I understand it is we have this over-bureaucratization. We have maybe the wrong class of people in charge, this kind of HR bureaucrat. You mentioned the human rights, UN kind of consensus-driven—managerial society that is not only reflected in our political institutions, it’s reflected in the businesses and the corporations.

And there’s actually an alternate way that things could be run, which is maybe kind of an older form of capitalism in a way. Which is where you have the founder, owner, director—character actually as a much more powerful political force within the companies and also within the country. And you’re sort of identifying this—class of people that could be elevated into an aristocracy—and then teaching them the virtues and the skills that would be necessary to first of all elevate them in their companies—and improve their control over their companies—improve the governance of their companies. But in so doing, actually elevate them as a class—relative to the bureaucratic class in society.

Guilherme: Yes. Yes, exactly. And there is some beautiful synergy between what formed Alexander the Great, what made the American growth in the Gilded Age, and what makes a startup grow or all sorts of things. Because Steve Jobs actually, when people ask him what he wanted from AI, he started talking about the mentorship that Aristotle gave to Alexander the Great. Now, this is very interesting because Greece was in this very decadent, read Plato’s dialogues, they’re talking about sex, drunk, and they’re not going anywhere.

And in Greece, and always the civil wars and this self-destructing mentality. And then Aristotle becomes the private tutor of Alexander, and Alexander changes everything. The Book of Maccabees even begins by saying everything was Alexander. Alexander became this new reality, like the Colossus of Rhodes, the beacon of Alexandria, the Library of Alexandria. They come from this Hellenistic world in which the Greek culture just became this dominant culture, like 300 years after Alexander.

They had Greek coins in India. So you’re actually coming from this semi self-destructing manner to this—it’s not like Alexander was perfect. He had a lot of problems, but you have this sudden rise of authority, of competence that happened. And Julius Caesar, when he was actually conceiving the Roman Empire, he based himself on Alexander the Great. Now, Aristotle of course taught his classical political philosophy and taught those virtues, but it’s very interesting because when you look at how a startup works, you actually need authority, agency, you need accountability.

You need this sort of guy who goes in front of the troops like Alexander went in front of the troops. The guy who’s risking his neck to make things work. I went to visit this very rich friend of mine, and he was sleeping at a mattress on the floor because he was moving, he was changing his house, and I was like, “Well, finally. First time I see you with an entrepreneur’s house.” And this sort of mode of self-sacrifice to actually be a father is very similar because the kid’s crying, you have sleepless nights.

The kids, they twist on the floor and you want them to stop eating candy, actually do math, and they spin on the floor like crazy. And so you have this really tough challenge to face because as James Burnham wrote in “The Managerial Elite,” now once oligarchy takes place, it’s very hard to get rid of it. And so within a company or the oligarchs, they attract people who don’t work They’re like those big fishes that have those small fishes beside them, and they’re trying to traffic influence to grow within the organization instead of actually putting in the hard work or innovating or having the best people. So the monarch needs really a lot of inner resilience to expel the oligarchs, and this is actually habitual. You’re fighting entropy on a daily basis.

This is what a lot of what running a company is about is that. And then within a family, you’re always fighting entropy. Your kids want just another party, just another donut, whatever. You’re always fighting entropy. You’re fighting entropy against yourself.

Your cravings are against what’s going to forge you. I practice jujitsu. There’s not a single day I went to jujitsu that I didn’t feel lazy about going, that I “Oh, really, I need to get a beating today. I need to go.” So actually, you are always in a situation where you have to grow in this resilience to change things.

And the same problem happens in politics because people are seeing government as something that’s entirely different. It has nothing to do with. But if you look at the managerial elite, the current oligarchical system, it’s a bunch of parasites taking hold of institutions that were supposed to—some of them were badly created, but some of them were supposed to be good in their origin. We even lost that origin already. And you lost the sort of feeling of Baron Stanford wants to make a university out of his farmhouse, and the university just starts to pop out, like—this sort of free spirit.

And even more so, if you actually start to get rid of the bad elements in a company, why couldn’t you do that the same in the country? Well, you could say the challenge in the country is quite big. Yeah, sure. But that’s the whole point. Once you have a lot of people who have a lot of influence and who are doing that in their organizations, and they’re coming together instead of just fighting for their egos, and they’re like, “Yeah, we’re having a shared knowledge project,” they’re capable of a lot of stuff.

They’re capable of infinite types of changes.

Getting business elites to act together (14:30)

Wolf: I’m curious about that point in particular. How do you get the stereotype of the business elite kind of person who does really well in business, who succeeds at that, is that they are going to be very sort of selfish. They’re going to have a hard time coordinating with other business elites. There are a lot of big egos involved. And is that sort of how you see the role of the kind of self-sacrificial development of those virtues that they enable that kind of coordination?

Or more generally, how do you overcome that kind of coordination issue? Because I think, Peter Thiel had some interesting ideas about this, sort of reflecting on PayPal and other companies, that you can get these really A-player guys—but you usually can only get one in any given organization—because they have a very hard time coordinating with each other. Can you comment a little bit on that? How do you actually get these—business elites to not just act as business elites acting selfishly, but act together—as an aristocracy?

Guilherme: Yeah. So, in some of our groups of students, and today in Brazil, we have 2,000 business owners here as students, and out of the 20 biggest entrepreneurial families in Brazil, 12 have been through the classes. So, those are very prominent people. People didn’t believe it was going to be possible because when we started, we opened our headquarters in Faria Lima, which is the financial district here. It’s like Wall Street, the equivalent of Wall Street in Brazil.

And actually, when I decided to go to Faria Lima was in one of the dinners, and one guy was like, “Yeah, Guilherme, very nice, virtuous, changing the country, but Faria Lima is lost. That place is lost. There’s no hope.” And I was like, “Oh, great. So I’m going to open a company in Faria Lima.”

So we’re going straight to Barad-dûr. Where the ring was forged, that’s where it’s supposed to be destroyed. And your point about it’s something that’s very real because there’s this sort of ghost of Atlas Shrugged, because the whole point of Atlas Shrugged is that they’re going away, they’re leaving. “Yeah, I’m very competent, so I don’t care about you guys. Bye.”

But this is, of course, the opposite of the Socratic ethos, because, in Plato’s cave, if you leave the cave, the next step that you go back to cave to take the other guys outside of the cave. That was Plato’s whole point, because there is no point in just living a life in which you opt out, you tap out of the battle. If you want business battle, you definitely, first of all, you never win the business battle because bigger battles will always be happening. You definitely have to win the family battle also, but you have to win the societal battle because the company is within the country, and the country is within a civilization. You can’t escape from the problem.

You can’t just think, “Oh, I’m going to change countries,” because the next country is going to suffer from the same problems, and you’re always leaving. Of course, in the United States, there are people who leave New York because, “Oh, New York’s declining, and so we’re going to Texas.” And I was in Texas, and the guy was like, “I’m going to Montana because Texas is declining.” Then you’re going to, of course, leave Montana because Montana is declining. And then it doesn’t end.

So—what do we do for our students? Instead of thinking of where do I live, how do I escape to the safer point, how do I stop escaping, stop planning my ski trip to Europe and actually do the fight. And I actually posted the students some battles. There is a stereotype here, it’s a funny one, that they fire the HR director when they start studying here. It’s not necessary, but hey, you can’t really have an HR director that’s against you—in the company, and doesn’t have a concept of virtue, of excellency, of human capital, if he doesn’t understand that.

But your company’s self-destructing whether you realize it. And that’s one of the reasons the guy just wants to leave his own company, is that his own company starts to become unbearable for himself, which is wild. But it’s true, and he has to prevent that. And once he sort of, “Yeah, I fired my HR where I selected a new HR, which is actually good HR. My company’s actually looking toward those good people.”

And I think part of why the A players, they leave, is that, there is a problem, talent is rare, I think there is that, but part of the problem they leave is that the bureaucracy within those companies, they start fighting against those a players. And they don’t want to fight against the machine. And actually, the entrepreneur should be fighting against the machine for the A player. It’s a—it’s a different mentality. And what we teach our students that, okay, so you fired your HR, you’re actually hiring the great kids that were not being hired anymore.

Sales is not a bad word, merit is not a bad word anymore. Virtue is not a bad word anymore. You bring back all those stuff. But now you have this other friend who’s also entrepreneur. Now, you guys are studying together because that’s the whole point of creating the classes.

You’re meeting, it’s in person, so forth. Why don’t you guys buy a school or create a school, get rid of all the bad teachers because they are indoctrinating your kid. They’re giving the worst education possible to your kid. They’re teaching whatever to your kid, like George Washington was George Floyd. They can create all sorts of weird combos—for kids.

And yeah, we’re actually going to teach math and the great classics and actually stuff that you use at a company. Not that you’re going to create a utilitarian education, but some of our students, they started buying schools and reforming schools, and the kids are playing classical music, fighting jujitsu, reading great books, and studying hard math. And what happens now is that your kid’s not being indoctrinated, so half of your problems toward the next generation, they’re stopping. And then you can buy a local media, and the media starts telling the truth. So another huge part of your problems, they’re not there anymore because, A, you’re actually very rich, and your friends are very rich, and the other guy who’s in class is very rich, and sort of the journalist who’s trying to destroy you, why don’t you hire good journalists for the same local media, the piece of media?

And now you have good local journalists, and why don’t you ask of your local politicians, they actually select competent people and not machine weirdos. And at a local level, everything starts to change. Your whole life around your company changes really fast. So the company changes, your family changes, everything changes really fast, but you’re not going to be able to do that without a serious study, without really going and delving deeply into the correct books for that. Because it’s not like you have to be a full-time intellectual to do those reforms, but you need the right theory, and you need to justify that.

There’s this dialogue by Plato called “Laches,” and Laches is a very brave general, he lives courage, but he doesn’t know how to argue in defense of courage.

And then he faces a sophist, and a sophist just wrecks Laches in a debate. But you can see that Laches is actually the right guy. He’s correct, but he’s very easily prey to the sophist. And so this dialogue is actually Plato’s whole point is that you need the philosophical education. Now, we use the word actually formation.

People use formation for military purpose, like in the United States, but the idea is the German word bilden, like to forge people, to actually assemble them. And there was this really beautiful idea in Prussia of bilden, of we’re going to forge this next generation of Junkers, of those young aristocrats, and they’re going to work in industry and the military, and they’re going to fight all those things. And Prussia really punched above its weight because of that. And so this idea that, “No, I’m going to create this junker class,” it’s very powerful. So if you start going to this direction with the right theory, and Confucius implanted a sort of system that decayed into bureaucracy in China, but in some moments, it was really about forging those virtues to create the best people to run, and China became a dominant power with that.

I’m talking about Song dynasty or—whatever. But if you have all those principles rightly articulated in the right theory, then you can do all that without creating as much of a noise. Because if you’re just using the crass language of current-day politics, you’re doing things that are simultaneously less radical, and they are being less effective, and they’re drawing more attention than if you’re actually doing this like Aristotle would do, in a more—intellectually savvy manner. And I think a lot of our students, we’re not interested in philosophy, like an academic pursuit, like abstract pursuit. We’re actually very interested in reading Spengler and applying that to business, and reading Plato and applying that to business, and the practical application of that.

And that intellectual formation is in the community that’s formed. It’s a synergy of a lot of stuff, right? But the synergy of all those things, they actually tend to transform the business leader from just planning my ski trip to like, “No, I’m leader, at least of this community.” At least of the, I don’t know, tech guys from my city, I actually getting them together to do incredible projects. So, we break that by the formation.

Reversing the abdication (25:23)

Wolf: Right. Interesting. It’s almost the opposite of the abdication process that is—One way to read the decline of aristocracy, the decline of monarchy over the past several hundred years is this kind of progressive abdication as we face these difficulties. The trench fighting against the bureaucracy and so on. In the case of Brazil, obviously, we’ve got the old flag here of—Pedro II—who abdicated because—

Guilherme: Abdicated. He didn’t fight, yeah

Wolf: He just couldn’t be bothered to fight these new-fangled bureaucrats who were—trying to oppose him. But what you’re doing, in a way, is kind of reversing the abdication.

Guilherme: Yes. Yes. Yeah, and half of my school kid colleagues were Japanese, right? And at a very early age, I became obsessed with martial arts. And so Brazil is very famous for jujitsu, and I got into jujitsu.

I started in karate, but I went to jujitsu. And there are a lot of stuff you learn by doing jujitsu, like you have to keep your cool when you’re being strangled. I think that’s one of the—that’s a very interesting thing. And actually, Plato started by doing wrestling. And old leaders were formed like that because they have to keep their cool when everything was falling apart.

And so we got away from this fighting spirit. One of the Valar of “The Silmarillion” of Tolkien, Tulkas is the one, he laughs when he sees the devil because he’s so used to fight that he doesn’t care about one more fight. And the loss of religion, the loss of the sense of mystery, to use the Spenglerian term, really created a society where people fear death above all. And the fear of death weakens them because they’re not seeing beyond death. They’re not thinking of, as Plato would say, the immortality of the soul.

Now, what does that have to do with business? Well, to create a startup, you have to sort of die in a certain sense to see your work come to life. For some years, you go into the cave. You have to emerge from the cave with the company. It’s a cyclical death of the project, of everything.

And you have to actually be predisposed to kill everything you’re building to build it better. There’s this sort of very adventurous mentality. And Brazil was actually founded by the Great Navigations project, which was this entrepreneurial mentality. But Camões, the great poet in Portuguese, he said the Portuguese have two defects. They don’t value their heroes enough, and there is the old guy from Restelo.

The old guy from Restelo is the guy who says everything’s going to not work. Everything’s not working. They’re like, “You’re going to the Great Navigations.” Restelo is the port in Lisbon, and they were leaving for the Great Navigations. And the guy is like, “Yeah, your wives are going to cheat on you when you are outside. The boats are going to sink. You guys are not going anywhere.”

So there’s this sort of defeatist mentality, and the Portuguese have this sort of idea of we have to just appease everyone. Let’s appease everyone. Let’s be friends with everyone, which in some ways came from the alliance with the British, and that’s a long story. And Dom Pedro, he tried to be the appeasing guy. Instead of the heroic spirit of the past, let’s try to appease everyone, and they just exiled him because that’s what when you try to appease evil, evil just kicks you in the butt and exiles you.

So after we lost our monarchy, Brazil fell into this really decadent, oligarchical, agricultural, stagnated structure. Although it’s a very rich country, it’s still a very rich country, it’s run by those socialists, like mob mentality. And this tends to fight against the great entrepreneurs. Now, that’s all well and good, but decadence does happen in society. Julius Caesar came out of the decadent Republic.

Alexander came out of the decadent Athens, or post the decline of Athens.

Charlemagne came after, actually, after Charles Martel, the great empire was fragmented. Alfred the Great, the origin of England, came after the old kingdoms were fragmented, and he had to unite them. So there’s this sense in which actually the call of leadership is the call in the face of adversity. And people in the West, since they were looking at it from a too materialistic, hedonistic, which are bad ideas, even for a company, even for profit, because if you’re too only for the money, you can’t create a company. It just doesn’t work.

You can create a bad company, but you can’t create a great company if you’re only for the money. You have to be for the profit. You have to be for sales. You have to make a profit. But you have to strive for something more than just profit.

And so if you overcome a bit of that, and you’re actually thinking in a sense of purpose, then you’re leaving your own ego, your own little world, and now you’re back into the real world. In the real world, yeah, they have the replicants. You’re sort of plugged to the matrix. They have all sorts of weird stuff going on. But yeah, what if we start like, yeah, we unplug to the matrix, turn away the machine, and start actually breaking some machines.

“Oh, no, you shouldn’t break machines. Oh, that’s terrible.” Go back to the... No, no, actually, I think if I break three or four, this might start working again. And you break three or four machines, and people start to thank you.

They’re like, “Yeah, thank you so much.” So I actually learned that in a company, that you could that in a state, and you could that in your life, because I saw those oligarchs in those big companies, and I was director in a company, and I saw those terrible oligarchs. And you have to expel the bad oligarch. People, they just tell you it’s doom. “You’re going to get a lawsuit. Oh, it’s going to be terrible. You can’t run a company without me,” and whatever.

And you fire the oligarch, and people start thanking you, like, “I didn’t know how you could live like that.” And once you do that, and you actually start promoting competent people, and you see, wow, that actually works, you start thinking, “Why don’t I do that with the whole of society?” Even more profound, Alexander is great. I love it. Instead of trying to vote every election and hoping Alexander rises, how about you be sort of Alexander-like?

At least within your power.

Elite formation vs. mass movements (32:16)

Wolf: So your approach is very, I don’t know, we could call it elitist, right? It’s aristocratic. You’re very much focused on the formation of an elite. You’re focused on what these relatively small number of—very powerful people—are able to do to reform society. Now, that makes a lot of sense, but various other movements, ultimately tackling the same problems, have taken populist approaches—or very more mass approaches, trying to create a big movement.

One that I’m very fond of is Partido Missão here in Brazil. Renan Santos, I think has done a great job kind of building this inspiring, vertically integrated—what is really a mass movement, even though—it’s sort of a segment of the youth right now. And I’d love to hear you comment on the outlook of that kind of strategy versus what you’re talking about, which is this more elite formation strategy. Do you need both? Do they need to work together?

Is yours just dominant? How do you see that relation?

Guilherme: Those two things are very complementary, right? They’re two different angles. But the good thing I see about elite formation, focusing on the elite, is how fast things can move. Like, for instance, I have this student, he’s in mining. He’s mining the rare minerals that have been used for microchips and so forth.

A lot of those are in Brazil, by the way. That’s a big topic because China is trying to grab all of those, and it’s a huge topic right now in Brazil, and one of my students is doing that. So I was talking about create your own school. Stop sending your kids to those schools who are indoctrinating your kids. And he created a school in one month, like from scratch.

Because he has capital. He knows people. He knows how to do business. So if you actually get one guy who’s not like, “Yeah, I’m getting all my money, and I’m spending it for myself and putting my money in some weirdo bank in Switzerland, so people don’t mess with my money. If the United States fall, I leave it.”

What if the guy actually starts doing crazy stuff like that for real, like with his actual money? Now, things start changing really fast. Now, the problem is that there’s a game theory problem because no one wants to be the prisoner who’s the—it’s like an inverted prisoner’s dilemma. No one wants to be the guy who speaks alone, right? In the prisoner’s dilemma, you should be quiet, but let’s invert it, yeah, and you actually are saved if you speak, right?

Inverting it. Now, if you are saved if you speak, no one wants to be the guy who speaks alone, because if you speak alone, you’re going to get persecuted. But that’s a self-fulfilling prophecy because the whole point is that the entrepreneur, he fears to do more than just make immediate money, because he fears he’s doing it by himself. But the other guy, he’s fearing the same. He’s actually your friend, and you can actually come together and start doing stuff together.

There’s nothing impeding you from doing that. And I think what we do here is to create the environment and to actually bring the pieces that glues everything together. Now, once you do that, you just can’t change everything because the business guys, they own the media, they own all sorts of stuff. So, and the problem is that the popular mass movements, they’re actually going to be scripted by the previous institutions. There’s no agglomeration of people in Montana.

They have no way to beat “Harvard” or “The New York Times” or be an alternative in any way, shape, or form. The idea doesn’t even apply. They can’t even think in those terms. So it’s like, it’s a different creature. They’re not operating in the same world.

And this sort of problem, then if you have a problem with “The New York Times,” then everyone’s going to suffer for that, and you can amount all the mass movements you want. You’re going to still be attacked by the same problem. And so we have to actually change that, and the business leader who’s been taught by the consulting agents like, “Yeah, just leave your company, man. It’s okay. You’re an investor. You have nothing to do with the company. Leave the bureaucrats in charge. We know what we’re doing.”

People are actually paying a lot to hear that. To feel like everyone who has more testosterone than 200 points has to leave the company. That’s how people are being taught. So yeah, sure, but the problem with the low T approach to business is that anyone who actually built anything in their lives know this doesn’t work in the field. It’s like Ernst Jünger says that you actually understand your life when you’re about to lose it, and you’re in the battlefield.

And of course, every time I’m strangled in jujitsu, I learn a bit of that. I get strangled a lot because I fight those monsters here in Brazil, like the UFC monsters, like I have some of them as students. And the thing is that if you’re up for the challenge, in some sense, you become an aristocrat because you are learning the warrior virtues, the intellectual virtues, the virtues of the other castes that you’re sort of being deprived of because that’s the whole thing. Our society is actually trying to separate those virtues, and if you have your warriors not be your entrepreneurs, not be your intellectuals, those virtues are all fragmented. People lose a sense of unity and lose a sense of leadership.

Now, the idea that elitism is bad comes from the sort of democratic propaganda type of mentality, like, “Oh, but you think you’re better than anyone else.” But about that, there’s an interesting topic. First of all, the idea that created the United States, it can’t be really a democratic idea in the sense that you have a Republican idea. There’s actually, why don’t you guys take charge? That’s the whole point.

That’s quite the opposite of the democratic ethos. And in a company, if you have a democratic ethos, that amounts to the destruction of the company. Actually, usually a company begins as heroic monarchy because it has to, and virtue and the success of the company, they’re very connected. Then it reaches its apex as a mixed monarchy, and you have the CEO, the engineering team, the board of the directors, and whatever, and then the complex hierarchy that leads up to this and the merit hierarchy within the company. Now, when the company starts to decline, it becomes more oligarchical, and then in the late stages of its decline, it becomes democracy.

Then that’s unionizing. It’s my group versus your group. It’s my sector. It’s not your sector. And they start fighting among themselves.

Until once upon a time, the union connected to Alphabet, to the company, they started breaking toilets. It might be a myth, but it’s a good myth. In Italian, they say “If it’s not real, it’s at least well invented.” And the employees started breaking the toilets for the right to have Japanese automatic toilets. And yeah, sure.

That’s like hyper-democracy, like ultra-hyper-democracy. The world owes everything to us, and you see a lot of that with some elements of Gen Z who are actually in the company. There are great Gen Z-ers, and actually, most of my guys are Gen Z-ers, and I’m pro Gen Z, but the more system-inclined Gen Z-ers, they go into the companies, and they feel like the whole point of the company is giving them more. It’s not like what they can sacrifice for whatever is there because they feel that toward their family, and they feel that toward the country. It’s the same mentality.

Now, once an organization becomes a democracy, it’s going to either go bankrupt or go to private equity. It’s about to be liquidated because once democracy takes over, it can be a monopoly, and its rust takes a lot of time. That can happen. You see that in states, actually, not the United States, in any state, in any country. It can go bankrupt.

So it just starts rusting out. It slowly starts sinking in its own weight. Nothing works. They can’t create any project. But in an organization, it might start to catch fire a bit faster, and so they start buying other companies to raise the valuation of the company, but they’re thinking of stock value, but they’re not thinking of the actual results and product.

And you can’t make a Gen Z university indoctrinated robot kid to actually go back to heroic culture. So the problem is that you have to institute heroic culture within an organization as soon as possible. Otherwise, it’s going to fall into this democratic phase, and it’s going to self-destruct. You see the same cycle with countries. They tend to begin as heroic monarchies, like George Washington is going to fight whoever to create the country, and then you create this sort of mixed monarchy system, and then it starts to decline, and people become more democratic as time passes.

Plato says that the last stage of decline is tyranny. Polybius actually puts kleptocracy, that’s organized crime ruling. That’s very symptomatic, like—Brazil, we’re seeing like a kleptocracy happening, and the good thing I see is that all through that is happening, a lot of people are waking up, and once you start to understand, to read Plato in this manner or start in this manner, they’re not dead books like in a college professor, “Oh, in 1800s, the great professors spoke of how...” It’s not that. They’re alive.

They’re alive back again. You’re now in Greece, and yeah, people, they’re drinking, and they’re destroying the country, and they are not caring about anything. And now we need Alexander the Great to come in. We need something to happen, and people need to start to live those virtues right now because things are happening right now. So, we actually make those things alive.

It’s like making a statue alive again. You have to take those ideas and instead of thinking of them as museum pieces, you have to think of them as, yeah, you are the guy in the idea. You are the guy. You are the guy in the cave myth. You are the guy.

Your company is actually a little fiefdom in the great scheme of things, and you created it because you’re building stuff. You’re actually advancing society. So, if you are not the guy in “The Foundation” and you’re not the guy in “Dune,” right, if you don’t want to be poor traders, and you don’t want to start the foundation, you’re going to be the guy in “Blade Runner.” There are not a lot of... You’re going to be the guy in “Terminator,” not the revolutionary, like Mexican revolutionary model of John Connor.

You’re just going to be wrecked. And the problem is that you actually need this mentality of I’m going to restore things and fight entropy all the time to run a company. If you understand that, you should apply the same mentality to everything else.

How the program works (44:20)

Wolf: Mm-hmm. So, we’ve said or hinted at many times here, the company that you run, the contents of your course—how you teach these virtues to business elites, but can you give us more directly, how do you create this new elite for Brazil—with your course, with your program? How does it work?

Guilherme: Yeah. We started the company because I only believe in companies. I don’t believe in NGOs because they’ll be—begging people for money, and—that would create an... I’m not against NGOs. I’m just saying it would create, for this kind of thing, it wouldn’t work at all.

It wouldn’t work because you need a sort of autarky. You need to be able to be self-reliant. And then I created a startup to do that and started hiring talented kids. Those talented kids that I started stealing them from finance because—you can just sit and be part of the bureaucracy, or you can start changing the world. What do you want?

And so the crazy kids, they want to come work, and they start reading the great classics, the great books, like living this culture. And so we start calling. It began with dinners for entrepreneurs. They started calling their friends, and then we created the whole system. We built our headquarters here in Faria Lima.

We created sort of a club, right? So, you have a two-day immersion, and then you join the club. So, we have encounters for people in financial markets, encounters for people in industry, encounters with... And they read the books. They see the classes.

They go through all those different themes. And then we have one-on-ones with a lot of them to actually make those projects concrete, and we started opening those classes in different cities in Brazil. My company has three years, right? So, people get shocked that it’s three years because you’re visiting it. It’s like things are growing fast, but it’s how they should be.

And so I reinvest most of the money the company makes into the company, and into finding new talent. And my idea is that, well, every single one of those students is part of the class, so we need the best people we can assemble together, because if you don’t have the best students... The fact that you invite the best students is also part of the formation in itself, so we actually grab coffee with the students, invite in one-on-one their famous students, known cases in Brazil that they... And then they start reforming their companies, changing a lot of stuff in their companies. We actually teach at the companies, too.

And then after it starts working on their company, sometimes the student is like... It’s very interesting because sometimes he’s not really that much into it. At first, he’s like, “Wow, it’s going to be like a classics department at a university. Is that real?” And then he hears this sort of wild red pill at one of the classes, and he’s like, “Yeah. Wow, that is real.”

And then he applies one idea to his company, and that works. And then he’s like, “Man, I’m up for the task. What do you want? What do you need of me?” Yeah, this stuff actually works because the thing about creating something new, like bringing the classics to the business world today, is a different approach.

It doesn’t exist. So, there is a downside and a good side to that. The downside is that since it’s a new creature, people don’t know what to expect, and they have to see it to actually participate. But there is the good side. The good side is that once they take a bite of it, they go crazy to just multiply it.

So, most of our students, they’re invited by the other students. It’s not Freemasonry. It’s not a cult. It’s not any of that. It’s just because the business guys, they know the right guys that need to hear this content they know who’s the sort of guy who’s actually going to, and this is a very important thing about our program, we are focused on people who actually call the shots.

I have nothing against mass education sort of initiative, but if the guy’s not taking the shots, he’s not really going to understand what I’m talking about. It’s not going to make any sense. He’s going to think it’s too weird. He’s saying, “Yeah, but I have to deal with my boss,” whatever. Now, if he’s the boss and he’s together with these other guys, I remember this very funny episode.

I have a lot of episodes, but there’s this big hedge fund manager and there’s this other entrepreneur, and they’re sitting together at a table, and he’s like, “I’m a very busy entrepreneur. How could I read Plato? I don’t have time for that.” And the other guy in front of him, it was a famous hedge fund manager everyone knows about, and of course, he’s a more busy person than he is, and he’s like, “Oh, I’m in my sixth Plato book, ‘In Search of Truth.’” And the guy’s like, “Man, if this guy’s reading that, I need to know what’s going on.”

And of course, a lot of his motivation is that he’s applying of that. He’s not just reading for the sakes of showcasing his classics Cambridge program or whatever. So they actually gave a lecture in Cambridge, and it was very funny because the guys in classics, they were like, “Wow.” One of the guys, an English guy, he’s very funny, he was like, “I could actually apply the things I’ve been studying my whole life.” And I was, “Yeah, you can apply that. You should apply that.”

Actually, Plato wrote that stuff for you to apply that, like those virtues. When I go to train jujitsu, it’s not because I like to be strangled, it’s just that I’m trying to forge the virtue of fortitude. Because I’m trying to change this country. And it’s for real. It’s actually for real.

And when I try to, like the kids, people change their diets, it’s because they want to grow in temperance. And when I’m reading Plato, it’s because I want to grow in prudence. And the whole point is changing things, because if I watch my own country be destroyed and I did nothing, then I am the problem. I am part of the problem, part of the whole oligarchical thing. And this is a very interesting idea in Aristotle.

Aristocracy and oligarchy, they’re actually polar opposites of the same coin, right? Because the same guy who becomes an aristocrat becomes an oligarch. So you’re always on this fine edge where you can become an oligarch, and you can become an aristocrat, and it does depend on your spirit. So what we’re doing is like, “Yeah, so you’re a business owner. Let’s apply this in your business. It works. Now let’s read that. Now let’s fight jujitsu. Now let’s improve, I don’t know, your diet.”

That’s not the main point, but whatever. Like your T is going to rise. It’s falling in 1% every year in the United States. It’s pretty much like one American today has like 50% of the sperm count of one American in ‘60s. It’s a pretty big problem, right?

It’s impacting a lot of stuff. So this is a minor problem, but if you start looking at those problems in perspective, then how about if you do all of that? “But Guilherme, that’s too crazy. You want me to be a good father, a good entrepreneur, to make my company have a profit, to just choose the good kids, to get away of all the kids, to actually change politics, to change media. It’s too much.”

Well, that’s the point. Hey, you are lost. You came to the wrong program. If you want me to just massage your ego, you came to the wrong classes. So, actually, there’s a lot of challenge to the students.

It’s a challenging idea. I don’t want to say kind truths. Or I’m sorry, kind lies. It’s all about unkind truths. It’s entirely about that.

And it’s about having a company of unkind truths. I hate the PowerPoint BS stuff, the presentation, like, oh, look at the pastiche from AI that they sent. The whole corporate bureaucracy is destroying the world. And if you work in government, actually work in government, this bureaucracy devoured everything. It’s like the nothingness of the Neverland that’s eating everything up.

And to rage against that and to be a beacon of light, to be like in this cult of Apollo, of I’m actually going to shine some light in this goddamn darkness, and I’m going to start changing it. It’s something so far away from the imagination of those entrepreneurs because they think, “No, company’s for money, personal money.” It’s not even collective money because Aristotle talks about, I don’t know how to translate that into English. I know in Greek, I know in, its magnificent—Magnificence yeah, virtue. It’s to make great sacrifices with money, to use your money for the public good.

Magnanimity? Yeah, it’s not magnanimity because it’s magnanimitas. It’s a difference. Magnificence. Magnificence.

See, magnificitas is you’re making great sacrifices with private money for the public good. I think we saw a bit of that when Elon Musk bought Twitter. Imagine if that was the rule and not the exception. Now that’s a bomb. That’s a nuke in the current system because, in Aristotle, he speaks of that all the time.

The Aristotelian project is this, because he’s actually thinking that the guys he’s educating, they’re going to be highly successful. It’s not a loser’s program, okay? It’s like a program for people who are going to be highly successful. And so since this, to be magnanimous, is connected to that. But I think people, entrepreneurs tend to be magnanimous in the sense that they strive toward doing great things.

I think that’s part of the entrepreneurial mindset, but they’re not necessarily thinking of how can I use that for the common good. Now, if you’re doing all of that, you are an aristocrat. You might not like the term. You might say, “Oh, but America was created to destroy the aristocrats.” It doesn’t make any sense because America was created to create a company, like the Gilded Age, you’re having business guys.

That’s aristocracy. You might not like the term. You can call it whatever, but every time you have a culture of excellency and having the best ruling, that’s aristocracy. You can put lipstick on a pig, but it doesn’t become a woman. So you can call things by the wrong term, but they don’t change.

It’s like people, they said, “Well, you shouldn’t call companies monarchies. You shouldn’t call your project aristocracy because people are going to have the wrong idea.” But it’s better to teach the right term than to try to deal with the bad terms because we’re living in—Elizabeth Noelle-Neumann has this thesis about the spear of silence. You’re silent about what’s happening. He’s silent.

The other guy’s silent, and it’s a spear. It keeps growing because everyone is silent, so the terrible idea grows every day stronger. Now, how do you break this spear of silence? You break the spear of silence by saying the right words, like if they are enchanted, like they have actual power. You have to use the right terms to call things as they are.

And I think this sort of egalitarian, if egalitarian ideology in universities, the spirit of this political religion, as Eric Voegelin used to call, is collapsing. It’s dying. It’s a dying religion. It’s like a shaman trying to do a ghost dance to invoke a dead god from the past. It’s dead already.

It’s cargo cult. And since the spirit is dying, what could be born out of this? What could be reborn out of this? Now, you might think, well, but aren’t those old ideas? I think those are the best fitting ideas for a startup world.

I think those are the best ideas for a world in which people are not having kids, and they’re having a demographic collapse. And actually, our students start having kids. Start having a lot of kids, actually, because they’re saying, “You know what? I’m a wimp. I’m going to have 10 kids, and I’m going to create my own private army of kids, and I’m going to make my own...”

This mentality, I say, yeah. And this is like, how many kids do you have? Five. And you’re like: Wow, you’re not going to stop and say, “Hey, I’m going to have five kids more for everyone who questions how many kids I’m going to have.” And then you’re going to multiply because people who are highly competent, if they had mentality of having less kids, then they are with—Spengler speaks of that.

You’re trying to destroy yourself. You’re in a death cult of your own civilization. So fertility is very much connected to creating new companies. And creating new companies is very much connected to restoring countries, which is very much connected to restoring your soul, to actually thinking in terms of this life is just a scrapbook of the whole eternity that is behind the world. So there is a metaphysical component to it, but it’s not a religious program.

But we’re trying to make students think in terms of purpose. Because you can’t create a family, a company, if you don’t have a sense of a higher purpose.

Plutarch, Caesar and great men (58:54)

Wolf: Yeah, and connecting those things together. It seems like people can do one or the other of those without the kind of holistic mindset. But to do them all together, to do them as a joint project.

Guilherme: People talk about Plutarch, but they talk in a dead way because Plutarch was concerned about that. About great men, and his whole point about great men was like, yeah, they changed history for real. They’re not great men because they want to showcase for their friends in some sort of private party. They’re great men because they, for real, change history. They are real.

They are not deracinated. They are inserted into actual historical reality. They’re actually doing things. So Charlemagne was obsessed with St. Augustine. Julius Caesar went to have mentorship at a Greek island, at Rhodes, if I’m not mistaken.

And actually, the oligarchical structures like Sulla and the Roman Empire, they feared Greek philosophy because they said, “Oh, this is going to bring monarchical ideas back into Rome.” So they feared that. Of course, you were having all those great thinkers, and Napoleon, he actually fought for his conquests to come, and Machiavelli was more important than anything else. And people should definitely read the “Commentaries on Livy” by Machiavelli. That’s actually the great book.

It’s not “The Prince.” People read “The Prince” and they’re like, “Wow, that’s like the rules of power.” No, it’s not. That’s not what Machiavelli is about. It’s all in the “Commentaries of Livy” because for a great man in the past, you had to comment Roman history to be great man.

Now, why do you have to comment Roman history? It’s not for you to be the great guy at parties that recite Roman classics. It’s because you’ve seen the downfall and the rise of regimes multiple times. And you actually understand what makes it rise and the downfall. The bad founder in a company, the bad business guy.

I actually use that for myself to invest, but it teaches students that and they use it, so it’s good. But I use that to invest. He’s like Pompey, because Pompey, when he fought Julius Caesar, he’s the insecure guy. He’s asking his wife, “Oh, what do you think, honey, about this?” He’s asking the oligarchs.

He is hesitant. He’s always insecure. He never really takes the front of the fight. Now, Caesar, and that’s the opposite of a lot of the politicians we see today. His bite is very strong.

He doesn’t really bark. He barks very softly, but he bites—like crazy. And a lot of politicians, they are the opposite. They bark all the time. It’s like, “Yeah, man, I’m going to change the...”

They don’t bite. And Caesar, like the great entrepreneur, he’s very workaholic, always trying to get. The Senate, there is this moment before Pharsalus, after the Rubicon. Oh, not after Rubicon, just before the Rubicon, when he’s marching. They don’t know what to think.

The people don’t know what to think, like what the guy’s doing. We’re asking for him to return. Is he going all the way? Could it be possible that he actually crosses the Rubicon and they go all the way? And part of their anxiety is that Caesar, he doesn’t care.

He just does it. He’s just doing what has to be done, and he’s going to change everything. So, in Augustus, Augustus actually in the Prima Porta of the famous statue of Augustus, he has this little Cupid beside him because he’s, of course, the son of Venus. He’s descended from Ascanius in the Aeneid and everything. That’s a whole different topic.

But Augustus was an interesting kid to be a successor, and the good successions in, like the bad successions like the series, “Succession.” That’s like the worst nepo kids who are very insecure. And Pompey, like Kendall in the “Succession” series is like the Pompey. It’s always—what does everyone think of me? It’s very performative, right?

What are they thinking of me? And Julius Caesar doesn’t care about what they’re thinking of you. And Augustus is the kid no one wanted but was the best kid. And when Caesar was assembling his team to face Pompey, he actually put these guys with this crossing scar across their faces against the pretty boys of Pompey, like the battle-ready, scarred, old soldiers to fight the beautiful people of the time. Like the and he beats the shit out of Pompey because of that.

And I do see that in business, it’s the same thing. The guy who’s actually concerned is not concerned about what people are going to think, how am I being perceived, how is this being perceived. I’m not against public relations. I’m just saying it’s like he’s not insecure about things, and he’s actually concerning himself with truth. That’s a very—Peter Thiel has this bit in which he says it’s only autists are good founders today.

And I do see why that is happening. It’s because in a world of sophists, you have to be very obsessed with truth to keep persevering in truth. And this is what’s lacking in political leadership. You’re not having the obsessed with truth, the Augustus type of kid who’s like, “This is the best. I’m going all the way. I’m going to actually make this real.”

Wolf: Great. Well, I think we’re out of time. Thank you so much. This was very enlightening. I’m really looking forward to seeing where your program goes.

I hope it goes much beyond Brazil. We really need something like this in America, so—good luck to you, and thanks for coming on.

Guilherme: It’s really an honor to welcome you guys here to Brazil. It’s my honor, and yeah, I definitely want to spread this around, and I think those ideas need to circulate. I think it’s very important, so I’m very excited about that.

Wolf: We’ll see what we can do in America. All right. Thank you very much.

Guilherme: Thank you very much.

Wolf: Until next time.

Samo Burja and Ben Landau-Taylor explain how the Greeks understood the basic forms of government, what their ideas catch which today's idea of "democracy" vs "authoritarianism" misses, and how they would diagnose today's dysfunctions.

Subscribe now

Samo on Twitter: https://x.com/SamoBurja
Ben on Twitter: https://x.com/benlandautaylor
Join The Students at our upcoming events in San Francisco: https://luma.com/thestudents

Transcript

The Greek theory of government (00:08)

Ben: Our political philosophy today is all about democracy versus authoritarianism. But if you look at the Greek political thought, they divided things into democracy, monarchy, and aristocracy. So instead of having the good one versus the bad one, they were looking at what they thought were the three forms that composed all human governments and the strengths and weaknesses of each of them. So I’m here today with Samo Burja, the founder of Bismarck Analysis, as a political scientist who can tell us about that.

Samo: It’s a pleasure to be here again, Ben. When it comes to correctly categorizing governments, one of the most important reasons that we should look at the underlying principles of power behind them versus what a government might call itself and why we should make use of political science that, as you say, reaches all the way back to the Greeks, is so that we are not confused as to the basic structure. Right? To give a modern example, the Democratic People’s Republic of North Korea, they just call themselves Korea, not North Korea. We call them North Korea.

They, in fact, are not very democratic in the foundation of their government. So merely calling something a democracy does not automatically make it a democracy, and because of that, we have to interrogate what do these words mean in a technical sense and how to identify these systems.

Ben: Yes, and so when Polybius was writing about Rome, he was a Greek who his people were conquered by the Romans. He lived among them, among the Romans, and then they sent him back to rule the city he was from, and he was explaining to them why the Romans’ government worked so well, and his argument was because it was the combination of all three. And I think that if you look at pretty much any government, you’ll see some elements of all three, bigger or lesser in very different proportions. And when the United States was founded, it was very explicitly designed. The structure has changed a lot over time, but the original founders were very much drawing on this sort of Polybius style, let’s try to get the aspects of all three that work well, even if the sort of top-line way they thought of it was a republic.

Samo: So us having a more impoverished vocabulary of politics does actually even get in the way of us being able to maintain our own government. So not just correctly identify them, study them, compare them internationally, but even if you wanted to, either as a citizen or a statesman, contribute to the correct ordering of your country, like say the United States, right? What hope would US citizens have to maintain if they were attached to their constitution as they mostly are, that constitution, without understanding what it even is, or that its design did rest on this theory of a mixed government derived from Polybius.

Ben: Yes. And so can you tell us, in the sort of Greek thought, what distinguishes these three types? What are they fundamentally?

Monarchy (03:45)

Samo: Well, this is a wider theory than just Polybius himself, like Aristotle, of course, before him, and also other Greek thinkers. They were all categorizing these many Greek city-states around the Aegean and around the wider Mediterranean for basically their distinct political forms and their distinct political dynamics. Monarchy, today, we might associate that with hereditary kingship or a hereditary ceremonial role, such as we see in, say, either the United Kingdom or to be a bit more direct in Saudi Arabia, where actual political power still rests with the king. However, there’s nothing about the word or the theoretical form of monarchy as the Greeks understood it, that meant that it had to be hereditary. You could have an elective monarchy.

Today, we might pejoratively call this a dictatorship, right?

Ben: Yeah. Because it just comes from rule of mono, one guy. And I think in the Greek form, this would usually be someone who had built his own power base and sort of then taken power directly themselves. And the idea of monarchy meaning hereditary monarchy basically seems to have become the default in basically the medieval period when that became the dominant form in a way that was pretty foreign to the Greek experience.

Samo: Yes. In the medieval world, this was partially then based also just on inheritance of land, and it was just very difficult in a medieval polity to build up the kind of power to become a monarch by your own hand. At the end of the medieval period, of course, that changes, and you have the era of Napoleon and so on, where that once more becomes possible.

Ben: Right. Where Napoleon absolutely would’ve been a monarch as the Greeks thought of it. And at the time, he wasn’t this type of hereditary monarch, and so that caused a lot of conflict with the rest of Europe. But there’s a lot of people who, I think Charles de Gaulle is an example of someone who in our political philosophy, we’d be like, “Oh, yes, there were elections, so therefore it’s a democracy.” And the Greeks would’ve looked at it and been like, “Okay, so there’s one guy who is basically in charge and is just Pericles-like, totally dictating what the overall situation of the country is and making all of the big decisions. So that’s a monarch.”

Samo: Well, yes, and Charles de Gaulle is not even the only example in the 20th century. We could talk about the American case of Franklin Delano Roosevelt, where what you call a president if they’re a president of a different country and they keep getting reelected until they die. We might call that a dictator when it’s a foreign country, but here in America, it’s just one of our greatest presidents.

Ben: Yes. And this form of the elected monarch seems to be very standard today in South America, where in fact they’re basically just in charge when they’re in charge, and in fact, the election determines whether they remain in charge or get replaced by a new monarch.

Samo: Well, the funny part about the US today is that our expectations of the president are as if the president is a sort of Latin American president or as if they are an elected monarch, but they really usually do not have the tools for that to be the case. FDR was sort of an outlier in the 20th century, and Charles de Gaulle for France was a bit of an outlier in France, in the 20th century. But to strengthen the underlying point, note that the constitution was changed, but the man remained in France, for example. There was this transition from, I think, the Fourth to the Fifth Republic, that happened through almost extra-constitutional means. There’s a lot of political debate about the crisis around Algerian independence and so on.

And all of that can be also discussed in depth. So for the Greeks, I think that they would not rule out a hereditary monarchy. There were some, the Macedonian kingdom, the most famous. Some Greeks disputed how Greek it was, but actually they had a committee determining how Greek you were. The Olympic Committee, actually.

Ben: It was a big deal.

Samo: It was a big deal. If you got to compete in the Olympic Games, you were Hellene enough.

Ben: Yeah. It’s the Eurovision of their day.

Aristocracy (08:17)

Samo: Yeah. And, I think because of that, they did have examples of hereditary monarchies and, say, Sparta technically had two hereditary monarchs as well, though Sparta, like Rome, was sort of a mixed constitution, and they recognized it as such, as having multiple principles. We talked a little bit about monarchy now, so perhaps it’s good we talk about aristocracy. In aristocracy, the pejorative or the dysfunctional variant of it, Aristotle especially always paired the sort of functional version of it, which is when the thing works well, with the decayed version of it when it’s negative or is dysfunctional. And I think the dysfunctional version of aristocracy is oligarchy, and this is a situation where there’s a limited number of powerful individuals who are stakeholders in the state, and they might come through various bodies to govern the state.

Say, the Senate in Rome was an aristocratic institution, not really a democratic institution.

Ben: Right. And the Senate in the United States was originally an aristocratic institution where the senators were appointed by the state. So then it was in the early 20th century when we changed the constitution to make it a democratic institution, that they would be popularly elected.

Samo: Yeah. And of course, in the United Kingdom, the House of Lords used to be much more powerful than the House of Commons, and that, of course, changed over time. And who got to be a lord changed over time. But in many places, this distinction between the upper and lower house of the legislature originates from the idea being based on this classic political theory of the ancient world and the early modern world, that there must be a representation of the country’s aristocratic elements.

Ben: Right. And so to go into the distinction you were pointing at with aristocracy versus oligarchy, when you go into the words, aristos. So the rule of the aristos. The aristos means the best. And so the idea here is if there’s a small number of people who just are better at governing, and then they’re in charge, and today we might call it technocracy or meritocracy, but this idea of if you are, through some process, finding the people who know better than the mob, and in fact, they end up in charge, and then using their superior wisdom to guide how things go, that is what they were thinking of with aristocracy.

And then oligarchy, oligos, the few, is when you get this clique that’s their... It’s a small number of people who are ruling not so much because they’re better in whatever way. And today we might think of that as meaning the smartest people. The Greeks were thinking of it as the most virtuous people. Whatever method of the best you want to select on.

The idea there is that what often happens is over time, the aristocrats die and then their kids are not necessarily, maybe they take over those positions, but maybe they’re not the best in the way that the people who actually got those positions originally were. And so then you end up with a clique that just rules because it’s the ruling clique. And that was the sort of decayed version that, in their eyes, if you have an aristocracy, it’s sort of a matter of time. If you make good decisions, it might be a while. If you make bad decisions, it might be fast.

But that’s just what’s going to happen with any of these forms. They’re going to decay eventually.

Samo: Yeah. The natural view, the sort of Greek view, is that when you first set them up, they work about as well as they can, and then over time, they tend to fall into disrepair at various paces. And if we examine this distinction you pointed out between aristocracy and oligarchy, well, it’s hardly surprising that whatever the selection mechanism for determining who is best, that mechanism probably decays fairly fast. If we were to look at, say, America’s Ivy League universities in the 1900s, they actually might be quite close to an aristocracy, and maybe they still were in, say, 1950.

Ben: Yeah. That was around when they made the CIA by basically just going through Ivy League alumni networks, just sort of being like, “Oh, yeah, I remember you from the frat. You’re in,” and making what was at the time one of the best intelligence agencies in the world.

Samo: Of course. And I think that there is an aspect, however, of, well, first off, if you’re too narrowly academic, that’s probably not the correct group of people to run a country. Because if you’re only focusing on the narrowest intelligence criteria, but not, for example, the political virtues of loyalty or bravery or patriotism, well, then you’re going to have a very rough time. And as hard it is to imagine, there was a time when the elite universities of the Western world were actually actively selecting for those features. You can actually go here in the Bay Area, you can go to Stanford, and you’ll see a monument at Stanford for the Stanford graduates that died in World War II and World War I. And it was actually a fairly significant number of them.

If you look at Oxford and Cambridge in the United Kingdom, they died at rates higher than the lower classes.

Ben: And it’s not just that they did this, it’s that then this was important enough to the university’s self-conception that they were like, “Okay, yes. We need this on our grounds to remind our people what they’re doing here.” And like—

Samo: What is their role in society, what is expected of them, and what kind of traits and virtues they should aspire to.

Ben: Yes. Whereas, now the academic forms, they’re much more just the sort of, they have these privileges because they have these privileges rather than because of doing the sort of broader selection for certainly not these other virtues, but also when it comes to intelligence, it sure ain’t what it used to be either.

Samo: No. No. And that happened because of 1970s sort of grade inflation, which became truly terrible, right? It’s hard to explain that in the 1960s, an A actually meant that you were doing very well, not that you showed up. Right?

That has actually changed over the last 60 years.

Ben: Yeah. There’s been some good documentations of just the percentage of A grades handed out, and yeah, it starts going up very sharply in the ‘70s, and then has just kept going up and up and up. So if that is a part that is more functioning in what the Greeks would think of as oligarchia, if you look at the current de facto American system, it seems like one of the parts that is more, in fact, aristocratic in the sense of you have people who are being selected for some type of skill. It seems like one of the big components in the American governance system is basically the business elites, where it tends to be in America much more self-made, much less hereditary than, say, a lot of the Northern European ones. And they end up wielding a lot of political power.

Not in the sense of like, oh, the Constitution formally says they decide this, but partly because of the industries they control and partly because of a lot of just sort of informal negotiation. We’re seeing a lot of this right now from Sam Altman and Dario Amodei and all of those artificial intelligence guys speaking to the politicians saying, “This is what we think the legal regime ought to be.” And they’re wielding a decent amount of political power on the basis of it seems like they did in fact have to be good at some things in order to wind up where they are.

Samo: Well, yeah. A written constitution is only as important insofar as it corresponds to political reality, and it can co-shape and describe political reality. So I think it’s very important to always examine the explicit parts of the state, but also the implicit parts of the state. When in the 1950s, the director of General Motors said, “What’s good for America is good for General Motors, and what’s good for General Motors is good for America,” he was probably literally saying the truth, but also bragging about his influence over the country and how the country sort of takes care of General Motors and General Motors tries to take care of the country. And it’s a political relationship, not just an economic one.

Aristotle’s constitutional research program (16:42)

Ben: Right. And so it’s a little bit of a tangent, but I think it’s an important methodology tangent while we’re talking about how the Greeks thought about this. Of like, constitution we now think of today as like you have a written document which spells out all the rules, but Aristotle, when he was going around, he meant constitution in the sense of what constitutes it, what is it made of? And can you tell us a bit about his research program there on the constitutions of the city-states?

Samo: Well, we mentioned earlier that there were many Greek polities, right? Unlike the Romans who eventually coalesced around one republic, conquering the other Latin tribes and all of the Mediterranean, the Greeks mostly were not unified in a single polity other than short period of their history under Alexander the Great, and even then there were states beyond Greek states, beyond Alexander the Great’s reach. There was a general understanding that there are cultural similarities, deep ties, deep religious similarities. There were often economic similarities, right? All of these were city-states somewhat dependent on trade, on fishing, on agriculture.

Most of them were in sort of the same broad climate. You can maybe argue that, say, modern-day Marseilles, which started as a Roman colony, has some climatic differences from Syracuse, which also started in Sicily as a Greek colony. But they both can grow wine. They both can grow olives. The sort of basic economy is very similar.

We’re not comparing steppe nomads and irrigation-based agriculture here. So within those parameters, Aristotle actually had many of the variables taken care of that modern political scientists might struggle actually to find a good control for when he undertook writing up the constitutions of the Greek world, he basically assigned his students, and he did some investigations himself, and wrote basically something like 100 or 200 treatises. Deep case studies into how is Thebes governed? How is Sparta governed? How is Athens governed?

Of all of these constitutions, we only have Aristotle’s Constitution of Athens, and that study was only discovered relatively recently. 200, 300 years ago, we did not have it. And I often wonder how different would modern political science be if we had several hundred of these written, basically political science papers from that era.

Ben: Yeah. We often have only very vague ideas of how each of the individual ones was governed. And you see this often with people who are renowned as being these great theorists, these great abstract thinkers, that before they did that, they would just grind through 200 in-depth case studies of whatever it is that they’re theorizing about. And Darwin just sketching all of the finch beaks in insane detail, or if you look at most of the great investors, there will be a time where they spent 10 years of their life just on amphetamines, going deep into the financials of each of the individual companies. And so it seems like the good theorists always have this period of very deep case study work.

Samo: Very deep empirical grounding. And Aristotle is known for his book, “Politics,” but before he wrote a general political science book, he and basically his team worked on a few hundred case studies of states. So let me just ask, today, which political scientist has undertaken a study of 200 modern states before writing their big book on, I don’t know, why nations fail or whatever?

Ben: Yeah. You’re lucky if they get two.

Samo: You’re lucky if they get two, and you’re lucky if they don’t just take, I don’t know, Australia and Canada or France and Germany. You’re lucky if they go for something more ambitious like China and India or China and the United Kingdom.

Democracy (20:56)

Ben: Yeah. So to end our methodology sidebar, we’ve gone over monarchy and we’ve gone over aristocracy. So how did they think about democracy and what that is fundamentally?

Samo: Well, they thought of democracy not even necessarily as one man, one vote, one human being, one vote. Today, of course, most countries around the world, I think all countries, allow women to vote. And we sometimes even allow non-citizens to vote in local elections. That would’ve perplexed the Greeks because to them, it only made sense that if citizenship was something precious that came with responsibilities and privileges that were balanced with each other. The idea is that the general citizens who compose the state as a whole, that they are either directly represented, which is what they would’ve understood as democracy.

They would’ve understood it just as a general assembly of the people reaching a decision on a concrete question. So today, if the Greeks looked at our world, they would consider a referendum or perhaps a presidential election as being basically democratic power. They would not consider representative democracy to be quite that. I think they would mostly classify it as either aristocracy or oligarchy because they would argue, perhaps correctly, that the elected representatives, they’re expending money, they’re directing attention of the citizens. There is, in fact, a difference, a principal agent problem between the constituency behind, say, a congressman or a member of Parliament, and the decisions they make once they join the body.

Ben: Right. Which is why when America was founded, they were talking about it as a republic rather than a democracy because they were still making this distinction very sharply.

Samo: And we’ve sort of have forgotten this distinction, and usually this is presented as, oh, we value democracy so much that we have strengthened the democratic elements of our states. But if you mislabel it, if you call something a democratic element, but actually it’s an aristocratic or oligarchical element, at that point, are you truly a friend of democracy or are you sort of presenting a false story of how the state works? Now, to be fair, it’s just very hard to run a democracy. There’s only so many decisions you can make democratically.

Ben: Right. Here in San Francisco, it is one of the more democratic places in the country, which is to say there’s lots of local referendums. There’s a bunch of things which are just decided by the people directly going out and voting on the thing, and I think everyone I’ve ever talked to agrees that this is just a mess and leads to a bunch of bad decisions being made by people who don’t really understand what they’re voting on.

Samo: Yeah. There are ways to do that type of a strong democratic consultation better, but it still involves a very limited amount of number of decisions. Here in SF, I think people don’t even understand. They sort of are asked a question, and the question has a lot of fine print. So the question is, do you love puppies?

And the fine print is the puppy euthanasia mill or something. I don’t know.

Elections aren’t necessarily democratic (24:20)

Ben: Yeah. And so the Greek conception was basically that the people are in charge, and one of the ways the people might be in charge and sometimes were, especially during the Peloponnesian War, things got very ugly, is that they might just if you have an aristocratic or oligarchical government, and then the people just break in and stab all of the oligarchs, and then they’re like, “All right, we’re doing something else now, and we’re appointing some of our guys.” And they would be like, “Okay, yes. The people exercised power, and now the things that are happening are because of that sort of popular expression of will.” It’s like fundamentally what happens, does it trace to the will of the people or not?

And nowadays, the way the word is often used is to mean, is there an election? Does the election cause things to happen in the way that the people will? That’s not really important. It’s just sort of, was there this ceremony of counting the votes held? That’s what makes it a democracy rather than, to what extent does the demos, the people at large, in fact exercise.

There’s this theory that, oh, if you have the election, then the people’s will be expressed. There’s a bunch of individual cases where you can be like, okay, so the people want this or don’t want this, and it keeps happening the other way. And there’s this sort of assumption in our current political theory that if there’s an election, then the people’s will is expressed, which seems to just, in fact, often not happen. And so you get this very different answer to how democratic is it, if you mean democracy in the sense of, are the people ruling versus do you mean was the election ceremony held?

Samo: Yeah. So we certainly hold election ceremonies usually, but one of my favorite examples on this not being respected is when the European Union was trying to be more than an international treaty and something like a polity. So they held referendums on the Constitution of the European Union. And when several countries in the referendum, the result came in no, they just held a referendum again. So if you’re consulting the people until the people give you the right answer and you have determined what is the right answer, that’s not an exercise of the popular will.

Ben: Right. And you can also see with California in, I think it was, what? Late ‘90s, early 2000s, had a big referendum on whether they’re going to allow gay marriage. Came out no. A couple of years after that, a judge decides that that’s illegal, and in fact, that’s going to be the law of the land from now on.

And you get this sort of reflexive property where after the judge has decreed that, and just across the country, there’s a lot of this was the judges decreeing this is going to be the law now. After it becomes the law and it’s enforced, then people generally come to accept the things that the law has mandated. And so if you were to hold that referendum now, I’m sure it would pass after having the force of the state behind it. And so like...

Samo: Well, the state can, in fact, reshape culture and does so routinely. Not on all topics, not on all issues. But historically, when a state decided to change the religion of the state, the population generally followed.

Ben: Right, which on the sort of modern idea of there is this sort of natural people’s will thing, and then you have these various channels through which it flows into the political system. I think that’s kind of a problem with the sort of modern electoral view of how a democracy works. It sort of supposes this thing is more fundamental than it actually seems to be in practice.

What type of government are modern Western states? (28:20)

Samo: Right. And some of the points you were building towards, I think are perhaps the most important point we can make currently is that we have a very deep sense of where the legitimacy of government’s supposed to come from. It’s supposed to come from the will of the people, and that’s why we like this word democracy or democratic. In practice, things get called democratic is the same as calling them legitimate or illegitimate. So in practice, democracy has come to mean judicial decisions, the ritual of are the votes counted, and perhaps to some extent, even the opinions of, say, the media class or the opinions of the politicians themselves.

So I think if we were to analyze modern states, we would say, the Greeks would say, these were once democracies, and the people are very patriotically attached to the idea of democracy and popular will, and they justify and speak of themselves as of the popular will. But in practice, they’re probably mostly oligarchical systems with some elements of aristocracy. Say, I would consider the US Supreme Court to be an aristocratic body because currently it still has some degree of merit, and while I do not agree with all of its decisions, and it wasn’t supposed to be the main legislative body of the United States. It’s the only legislative body the US has left. Congress having abdicated the role of legislating.

Ben: Right. I think the last time the Congress passed any legislation that mattered, other than just approving extraordinary spending requests from the executive was some of the tinkering with the health insurance stuff in 2010. It’s been a long time.

Samo: Yeah. You cannot have a functioning government where the main legislative body passes a law once every 16 years. That is not how a modern state works, and frankly, the Greeks would’ve even believed that no ancient state works that way. If there was, say, a kingdom that had a king that makes many decisions every year but had a legislative body the king convened once every 16 years to pass one or the other law, no one would consider that a republic. They would call it a monarchy.

So I think today we really have to accept that Congress has abdicated its role to an amount that’s perhaps only to an extent that’s only comparable to the way that King Charles or Queen Elizabeth in England have sort of abdicated their role as monarchs.

Ben: Right. Or, and of course, the late Roman Senate being the most cleanly comparable historical case of something which was still sort of widely held popularly to be governing and to have the role that it used to have, long after it had exercised any actual political control.

Samo: I think it’s still going to be a few years before either the Democratic or the Republican Party appoint a horse to the Senate. But, I wouldn’t be shocked if that happens, too.

Ben: No, they’re going to appoint a vegetable.

Bureaucracy (31:28)

Samo: That’s probably already happened. Yeah.

Ben: But, there is one area where I actually disagree with the Greek conception in a very important way, which comes up not so much when you’re looking at what was going on in the Greek world. I think the reason they missed it was because they just didn’t really have much of it themselves. But if you’re trying to explain... So Polybius was like, “I’m going to explain the Roman Constitution, which parts are monarchical, which parts are aristocratic, which parts are democratic.” I think that works.

I think if you try to do that with a modern state in the year 2026, you’re going to miss a lot, because I think there is a fourth fundamental form, which you really can’t understand our current system with, which is bureaucracy. Which seems like a just different type to me. Where the distinguishing feature of the aristocracy or the oligarchy is it’s these people who hold the personal power based on, in the medieval case, their own private estates and their own groups of knights or whatever. If you’re talking about a business aristocracy, it’s that these people legally own these companies which they control, and if they want to move the factory, the factory moves. But with a bureaucracy, you don’t have what you and your great founder theory work have called owned power.

A bureaucracy has a bunch of these people who are in a position appointed by someone else. They can be moved to a different position. They can be fired and someone else brought in. And so, we’ve had our video earlier on how bureaucracy works, and a lot of the ways that it ends up sort of becoming more powerful over time. And I think that it basically ends up being a fourth form of government that is as fundamental as these other three that the Greeks have been talking about, which also ends up being the dominant one in most countries, including the United States today.

Samo: I think we have seen a slow saturation of bureaucratic power through the course of the last 3 or 400 years, and it is important to acknowledge that aspects of bureaucratic power once also existed. There were salaried employees of the state in, say, even ancient Rome or Greece.

Ben: Oh, yeah. So late Rome was also mostly bureaucratic with, there was an emperor with some monarchical elements and all that stuff. Yes.

Samo: Yeah. And of course, if we want an even cleaner example, China has numerous times in its history, we might call it, it was imperial. But the idea of a powerless emperor that’s sort of directed by the eunuchs and the Confucian scholars and so on, I think that’s also a trope in Chinese history for a reason. And because of that, I think we can say that China, in numerous dynastic periods in the Middle Ages and so on, and earlier, even in antiquity, was primarily also a bureaucratic state. So I think that perhaps bureaucracies naturally develop from a restraining of the aristocracy, and eventually, I think, also a restraining of the people.

And I think that perhaps it is a side effect of us coming to distrust both oligarchic elites and the people and dictators, that we have started to use bureaucracies for purposes they are not suited to. So I think it’s not just that state capacity rises and the best way to organize a society is bureaucracy. I think we have critically examined monarchy, oligarchy, or rather, I’d say tyranny, oligarchy, and demagogy, and found all three sort of lacking. And having not yet critically examined bureaucracy, it became the default winner, the default holder of this implicit legitimacy.

Ben: Right. And like the other three, when it is first implemented, it tends to be implemented well, which is why it keeps spreading enough to actually take over. And if you look at the early period, you talked a lot more about this in a previous bureaucracy video, about how in the late 1800s, early 1900s, Western bureaucracy, American bureaucracy, was just basically the most powerful force in the world. Where they were landing on the Moon and conquering Europe and just building out industrial capacity at a absolutely staggering rate that there’s just been no comparison to in our lifetimes. And it came in in this very powerful, very functional way, which, like you’ll see with the early democracies that are able to actually take root, with the early monarchies that are able to take root, with the early aristocracies that are able to take root, just performing extremely well.

And that was about 100 years ago. And like any form, it’s not going to stay that way.

Samo: No, and it would be hardly controversial if I said that the collapse of the Soviet Union and the failure of that polity was mostly downstream of profound and nearly universally propagated bureaucratic dysfunction. I think that’s not a controversial thing to say. But a controversial thing to say is that eventually the United States might hit its own late Soviet moment. Now, it’s not a communist bureaucracy, it’s not a Marxist-Leninist bureaucracy the way that, say, the Soviet Union is. But it is, however it will, however, eventually hit a point of saturation where bureaucracy even makes daily life difficult, and it makes great decisions of state difficult, and it even might choke off economic productivity.

Graft (37:03)

Ben: Right. And what you see in the late Soviet case, and what you also see in the late Song case in Chinese history, or I think a bunch of these sort of late Chinese bureaucracies, is that the stealing just gets completely, absolutely insane out of hand. A strong majority of the ostensible state revenue is just being stolen by various bureaucrats. That seems to be a very common way that the bureaucracies sort of reach this terminal dysfunction and end up either being replaced internally by another form or just causing the collapse of the nation. And it just seems like in our current society, it’s hard to tell exactly how out of hand it’s gotten, but I would not be surprised if Moses came down from the mountain with the tablet and it said, “Over 50% of the United States government revenue is just being stolen.”

I’d be like, “Yeah, that tracks.”

Samo: Yeah. Now, to be fair, it is possible, basically every society can be choked off by excessive taxation, and it’s certainly possible to tax your economy to collapse from, say, trying to gather the resources to wage a war. Right? And in the late Roman case, what extent is it theft? To what extent is it just a decayed economy trying to hold on to a large military empire and choking off economic activity, but still having a fairly efficient military?

That’s up for historical debate. And even in the Soviet case, you could say, ah, actually, the overburdened size of the military actually was crucially one of the big reasons why they could not give much surplus to their consumers or even develop their economy. So that plays a role. However, I think the theft is this sort of way in which you use borrowed power to try to receive personal benefit and eventually, to some extent, personal power. When the privatization shock hit the Soviet Union, it was often former bureaucrats who were best positioned to benefit from the privatization of once public property.

So even though that was a moment of reform, in a way, often the people most responsible for the destruction of the previous form were the ones that benefited the most. So that was a clean moment where the ostensibly borrowed power of the bureaucrat had completely devolved into a form of own power and then personal enrichment without necessarily much merit to it.

Ben: Yes, and it’s very common for when you get one of these transitions for the way that which the new elite rises to be the holders of borrowed power in the previous system as the old center is disintegrating. The word duke, which was the second highest guy on the feudal system, comes from a Roman general, and it was like, all right, so who became the dukes? It was the guys who were leading one of the Roman armies when the whole thing disintegrated, and then they set up, and they’re like, “All right, well, I guess the duke is in charge now. We’re no longer paying taxes to the duke who’s then forwarding it to the emperor. We’re just paying taxes to the duke.”

Samo: Yeah. And I guess the duke’s son is the natural candidate for the next duke. And perhaps today we would compare it as a term of generalissimo or something like that, right? The idea of a type of military rule that then became hereditary, and then after a few centuries, it kind of becomes aristocratic through this mechanism. Now, when it comes to the US bureaucratic moment and the Chinese bureaucratic moment, what people miss is that, for example, in China, there’s a monarchical element of power, which is Xi Jinping.

And he, in fact, struggles not just with the rest of the party, which I’m going to be very controversial and say, given their performance in the last 30 years, I think it’s more an aristocratic body than an oligarchical body. Right?

Ben: They seem to have done quite well a lot of things they’ve chosen to take on.

Samo: Right. Such as economic growth or increasing China’s power in the world or undertaking developing new technical capabilities, whatever you want to call it, right? They’ve actually performed fairly well at their goals. And I think this is important to acknowledge even for rival countries or strategic competitors or for enemies. If you underestimate your enemy, you’re only doing a favor to your enemy.

So not only is then Xi in a sort of conflict with that body, he’s also in conflict with China’s bureaucracy. And I think that China is actually, in the long run, not well-positioned to resist bureaucracy. And perhaps that’s a topic for a different episode. But I do think that there’s a way in which we will have to figure out with what means can, in Western countries, what means do we have to hold bureaucracy in check? A referendum will not do it, right?

The role of the mob (42:17)

Samo: There’s no referendum that’s—

Ben: It’s been tried, and it turns out that it just delays them a couple of years, and they’ll just keep running it until they get the right result or until they can find a legal vehicle to be like, “No, the result of your referendum is illegal.” So, yeah, their bureaucracy just is empirically able to route around those when they want to.

Samo: There is an open question, and this is a controversial one. You pointed out earlier that actually mob power is a type of democratic power. Today, we think of demonstrations as these like somewhat fake things where people show up with signs. No, a demonstration used to be a sign that, “Hey, we’ve surrounded your palace. If we want to, we can drag you out and kill you.”

“And we’re choosing not to. Are you sure you don’t want to vacate the palace peacefully?” And that’s not hypothetical. Ceaușescu’s regime ended with a mass rally he tried to organize to show how much popular support his power had in communist Romania, and instead, the mob started jeering at him, and then his security staff realized, “Oh, wow. Okay. We got to get rid of this guy.”

“We’re done for. We cannot control this large crowd.”

Ben: And was it Nepal, I think, was recently the one where the sort of Discord mob just toppled the government just like a year or two ago?

Samo: Yeah, and because of that, I’m sure it’s one of the reasons the European Union wants to implement Chat Control and make sure that what is said will happen in “Minecraft” stays only in “Minecraft.”

Ben: Or better yet, is never said in “Minecraft.”

Samo: Right. Because it’s illegal to say things in “Minecraft.” In Europe, it is now illegal to do things in “Minecraft.”

Ben: Right. And of course, this can get out of hand. The Greeks, they had their here’s the virtuous form, here’s the unvirtuous form, and for the sort of like decayed version of democracy, their word was ochlocracy, the rule of the mob, because when the mob is durably making decisions and the mob is steering your economic policy, they’re not actually good at that. They’re just like the mob’s economic policy is, “Hey, I can smash this and take it.” So when that becomes not just an expression of toppling something and putting a functional thing in its place, but the mob is actually leading things, that’s not a good time for anybody.

Samo: No. So, you can’t govern through the mob. You must find different channels. However, I am pointing out quite intentionally that actually elections are not effective against bureaucracies. Mobs might be in some limited cases.

I think that when it comes to policing bureaucracies, it is an open question whether a general uprising versus a modern Western state would result in the state caving or would result in the state successfully arresting everyone.

Ben: Yeah. And it largely comes down to is the military willing to actually shoot people, because if they’re willing to just machine gun down an entire mob, I think they just have the weapons to win, but like—

Samo: In China, we saw that with Tiananmen Square, right?

Ben: Right. And Tiananmen Square was coming right after all of the Soviet breakaway republics breaking away by having these demonstrations of unarmed people and basically being like, “Look, you’re going to have to machine gun us,” and the occupying troops being like, “I don’t want to just machine gun a whole bunch of random 20-year-olds, so we’re not going to do that.” And the government fell, and then a bunch of the Chinese tried to do that, and they got machine gunned.

Samo: And that’s why the government didn’t fall. And the government in China is perhaps still afraid of the people much more than the governments of Europe or the United States.

Ben: Which I think might just be an error on the parts of the Western governments. I do not know, would the British Armed Forces machine gun the demonstrators? I suspect they wouldn’t.

Samo: I don’t think they would machine gun the demonstrators, but during COVID, we saw they were at least willing to, in Belgium, they were willing to ride horses and beat them on the head and crack skulls. So they were willing to do that during COVID to enforce COVID lockdowns. And in the United Kingdom, I don’t think they would be willing to machine gun them, but I do think that a high level of state repression of the people does exist in Western world today.

Ben: Certainly. And when they’re riding horses over people, that seems to cost them more legitimacy than when they’re using secret police to arrest people for Facebook posts or to follow back the people who had been demonstrating them and grab them a couple of days later, which is also a tactic that the Chinese are now using these days. So to keep things from getting to the point where they might have to machine gun a square full of people, because no one wants to do that if they can avoid it.

Samo: Well, yeah. In a way, bureaucracies perhaps were afraid of the internet. So perhaps other than direct mob rule, there was digital mob rule, and arguably, a lot of the consternation over populism destroying democracy might just be populism for a while scared bureaucrats. Now, I’m not sure whether they’re still scared of it, but they did not like the anonymous Facebook comment. They did not like the sort of video investigation of the YouTuber showing up and documenting straight-up fraud.

In America, that stuff is still legal. In Europe, they’re approaching making it illegal, depending on which country we’re talking about. And there’s a way in which I think we’ve reached the limits of what the internet can do against bureaucracy, I would say. So I’m sort of thinking we perhaps have exhausted the democratic means of addressing bureaucratic issues. But it is an open question, I acknowledge that.

Ben: Yeah. And a lot of it depends on what happens with many of these is that they’re kind of decayed, and so you start seeing challenges to them, and over time, whatever, in this case, bureaucracy just continues decaying. And so unless it gets better, at a certain point, it’s just going to not be functioning well enough to be able to meet the challenges to it, including the sort of internal challenges to it.

Legitimacy comes from results, not from names (48:47)

Samo: Well, because we’re so embedded in what was until recently just the dominant civilization in our world, Western civilization, with specific history of democracy—I think we tend to forget that the most functional forms of government or the most functional principles of power that solve the challenges of an era are the ones that will be seen as more legitimate over time. So in other words, if democracy can solve bureaucratic problems, then democracy is legitimate.

Ben: I remember reading Vannevar Bush was writing shortly after World War II that democracy was legitimate because it had proved itself the best way of organizing these large industrial projects.

Samo: Right. And this idea that democracies are slow and unresponsive, this would’ve really confused 19th century statesmen. 19th century statesmen were very much of the opinion that actually aristocratic or hereditary monarchies are sort of unwieldy and stuck in their ways and bound by tradition. Meanwhile, democracies just go and do whatever it takes, and actually, maybe democracies are way better at industrializing than what today we would call dictatorships.

Ben: They certainly were at the time.

Samo: They were at the time. And today when we look at China, first off, I think people overestimate how dictatorial China is because dictatorship to many people just means bad government, naughty, naughty government, and democracy just means the good guys—i.e., us.

Ben: In the past few years, I’ve more often heard people calling China authoritarian rather than dictatorial.

Samo: Which is more accurate. But again, if the Chinese cities look good and our cities look bad, and Chinese citizens every single year live in a larger apartment and we live in smaller apartments, at some point, the general populace might just decide that maybe authoritarian is the good word and democratic is the bad word, even if the cause of it is not actually democratic power but bureaucratic power that’s choking off their lives and shrinking their apartments and all that.

Ben: Right.

There are many things like this where the result is downstream of the skill of the people ruling, and people attribute it to the form that the ruling is taking, which I think is frequently an error.

Samo: Right. But we’re hardly the first civilization to have gone through this process.

Ben: Certainly.

Samo: Right? And in some cases, they never quite let go of the previous idea. Like in Rome, they still did all the things in the name of the Senate and people of Rome when they had had emperors for 300 or 400 years. I even think the Senate was still a real body in the Byzantine Empire as late as the 9th century AD, where the Senate in Constantinople was a fake body that kind of became a real body and then became a fake body again. Right?

When Constantine assembled it, it was his fake Congress. Of people that owed everything to him.

Ben: Right. And of course, the Japanese emperor is still an emperor, and even if they don’t seem to be very imperial, and we have a lot of these things in our own day as well.

Samo: Well, and Rome does have a similarity with America in that its sort of founding was based on the perception of overthrowing a corrupt monarch. Though really when you examine America’s founding, you realize actually it’s a war against king and parliament. It’s not a war just against king. So it’s sort of like it is a war against a distant and tyrannical legislature and executive, not just the executive.

Ben: Right. Rather than the Romans being like, “We’re installing a new thing in our own city,” it’s like we’re breaking away from this distant imperial center.

Samo: Yeah. The Roman Revolution is perhaps actually closer to the French Revolution in some aspects. But again, bureaucracy was much more developed in 18th century France than it was in the Kingdom of Rome, which became the Republic of Rome.

Ben: Yes. Well, thank you very much. I think it’s really good to just get into some of the fundamentals, and I’m sure we’re going to be coming back to a lot of these concepts as we continue to do these.

Samo: Yeah. Always a pleasure.

Renan Santos tells how he built the Free Brazil Movement and worked out the doctrine, mobilized it into Brazil’s Mission Party, and how the lessons of building an outsider movement apply in America and around the world.

Subscribe now

Transcript

Intro (00:00)

Wolf: Welcome. So we’re here in Brazil again, this time at the Mission Party headquarters with presidential candidate Renan Santos. Welcome, Renan.

Renan: Thank you very much. It’s a pleasure for me to be in your show, let’s say, meanwhile in my own scenario where I make my live streams, but it’s fun.

Wolf: Yeah. It’s fun. A little crossover here. So I met Renan last year, I guess. You came to one of our parties—in San Francisco after we—

Renan: It was really nice.

Wolf: We—

Renan: Really nice barbecue.

Wolf: Yeah, it was good. So, yeah, we got interested because the Mission Party is very small, but growing, and very interesting. It’s building a vertically integrated political party in Brazil that’s very youth-oriented, it’s very positive, and has a lot of intellectual depth and a lot of street presence. And so I like that combination. It was really cool.

And so I came down in November, saw the launch of the political party itself. It’s a very new political party. And it was the most glorious thing I’d ever seen. That’s what I said—at the time. I was convinced this thing’s inevitable.

It’s going to happen eventually. And so I wanted to come down here to Brazil again for the presidential campaign, which has just concluded. The votes have not come in yet. So we’re right between the campaigning has ended, but the votes have not come in. So the audience will know more about what happened than we do.

Renan: Yeah, we’re in a little purgatory right now.

Wolf: Yeah. We’re in purgatory. Exactly. But, yeah, I wanted to sit down with Renan and get a bit of flavor on this thing for the American audience. Our students in America, we want to know, how do we fix America?

So I think how do we fix Brazil is a good example. It’s a good prototype of it because I think you’ve got a great program here. So I think one place to start that would be edifying for everyone is just to give us an overview of the Mission Party and your plan to fix Brazil. Just how are we going to do it?

Renan: Well, the Mission Party was the culmination of a process where we understood, as a political movement, I come from a political movement I created 12 years ago called Free Brazil Movement, where we understood that the next step would be institutionalize ourselves—so we would be part of the so-called system—to reform it from inside. In Brazil, it’s really hard to open up a political party. It’s very difficult. You need a lot of money. And actually, it’s funny because we have lots of political parties.

It was easy to open up a party in the ‘80s. For an American audience to understand, Brazil was under a military regime in the ‘80s. So when they opened the system, they were like, “Oh, we should have 100 parties.” So it was easy. And then all the parties, they start to have funding from the government, state funding for their operation.

So they said, “No. Let’s close the gates.” So in the 2000s, they made a new legislation that made it almost impossible. And after the 2010s, it was probably impossible. So we overcame all this difficulty through use of the technology and use of our following, a very strong following among youngsters, especially young males, who were following, helping the movement.

So if we were a normal political group, we would spend, let’s say, in dollars, around $15 million to create or buy a party because people buy parties here—even though it’s illegal. So we couldn’t do it by normal ways, so we put a lot of effort on technology. And it cost for us, let’s say, in dollars, $1 million to make it, by selling this yellow book, selling T-shirts, selling products, and also middle-class people donating for us. We never had a major donator. There’s no super PAC in Brazil or—something like that.

So, we understood that we should not change ourselves on a quantitative way, but qualitative way. That worm must become the butterfly somehow. And why become the butterfly? Because we must have a very ambitious program focusing on how to change very obvious problems that we have as a country that we don’t see as problems anymore. The most obvious one, and I saw people from abroad talking about it, is the favela.

Favelaization (04:36)

Renan: When people from abroad look at Brazil, they think about football, favela, which are those slums, and Amazon rainforest, and football, some beaches. But the favelas are part of our own identity because people think that favelas are a source of culture, as if it was good. The culture that comes from favela mostly is very bad. And there’s violence, there’s gangs, and when we start to talk about defavelization Brazil, it was not only about the favelas itself, as a infrastructure problem, a housing problem, but about the idea that Brazil is favelizing itself. Having a culture that’s not focusing on excellence, but focusing on, “Okay, let’s survive and try to be happy with the resources we have in hands.”

And that became a motto for how a Brazilian should be like. And when people talk about the third-worldification of America, Brazilification of America, in the end, we are talking about favelization of American mind. So in the end what you guys call in America the Brazilification—is what we’re trying to Brazilify in Brazil. So there’s this sense in Brazil that things never happen and that we cannot change Brazil, even though we are the country of the future. Because there was this idea that Brazil would be a place where, let’s say, West and the Global South, they get together and merge and create a new tropical and exciting civilization.

There was an Austrian writer called Stefan Zweig who wrote about it. But we never made it. We never made it. And we have glimpses of what could be. Like for example, the best parts of Rio de Janeiro or Brazil in the ‘50s, some areas of Brazil in the ‘50s, they were like the best of a mix of West and tropical, and this mix of sex, beauty, sun—and a good way of living.

So it was possible to conceal these kind of different energies, but we never made it all again. And right now, we’re in a bad situation. So, the Mission Party focus on this defavelization of our minds, focus on a very pragmatical program, in terms of fiscal reforms. Also about productivity. Brazilians are very low in terms of productivity, so we are losing space in the international supply chain, industrial chain.

The small towns in Brazil, they have almost no economical activity. They live on their money from the government. So it’s a problem because the numbers of towns that relies on budget from the federal government is going up, and people are relying on food stamps more and more. And there’s this leftist party, the Workers’ Party, who’s in charge for decades. So they’re making it easier to get more food stamps, so people vote for them, and at the same time they are not able to emancipate themselves.

So we created a program to solve all those problems. People from abroad talk about our reforms in terms of how to destroy the cartels. But this is like the top of the iceberg of—a much more ambitious program. Because actually, it’s easy to destroy the cartels. People never had proper political will to change it.

But the other issues in Brazil, which is a civilizational issue, they are the goals that we focus here on “The Yellow Book.” And our party has a goal that in 30 years, we need to make Brazil one of the top five countries in the world. That’s where we belong. We should be on a place, besides India, China, United States, Russia. India, China, Brazil?

Yeah, and Brazil, yeah, as the countries that define what the world should be. And I mean, I don’t want to sound rude to, I don’t know, British people or Germans because they are great people, but we have 200 million people. We have a land full of natural resources. We have millions of very capable people living here that with the correct legal system and the correct culture, they can provide good companies, good technology. We have very capable people.

So for sure we can do it, but why we never try it? And most of it, why we never want it? We never want it to be one of the top countries. So we create a program based on the idea that we must be one of the top five countries in the world in 30 years, and this is a mission. That’s why the name of the party is a mission.

So I don’t know if it was clear, but the idea behind it was like that, a set of reforms to make Brazil a powerful country. And only the idea that we can become a powerful country is already revolutionary for us. Because Brazilians are so negative about our future and about ourselves that people, they were mocking me on TV like, “How come you believe it?” Even journalists, serious journalists, they really don’t believe we can do it.

Writing the Yellow Book (09:40)

Wolf: Mm-hmm. And that’s where your slogan, the future is glorious—comes from. And so this book is actually one of the more interesting aspects of the party, especially from an American political perspective. It’s people mouth off on TV, and then they get elected and they do some things. But what you guys are doing is much more like a real traditional political party in that you have a program that’s well thought out.

You have a 30-year plan to fix the country, make it into a great country, and all that substance, all that depth is in “The Yellow Book.” How did you guys put this thing together? I understand you had many different people study all the little issues and brought in a lot of experts to figure out what actually needs to be done and come up with a coherent program.

Renan: Well, that’s a very complicated answer because I don’t know, I can come up with a short answer, or I can tell the story behind the answer.

Wolf: Let’s do a bit of both.

Renan: Okay. Well, the short answer will be, yeah, we were able to create new thinkers and specialists in some areas, and study the idea of what is Brazil right now and what are our real issues, and come up with local solutions and solutions from abroad. But the big story is like this. We resented when we were in the Free Brazil movement. We were created in 2014.

That’s why our number in Brazil is 14. Also, it’s my birthday, so it’s—a coincidence. But we were created in ‘14, and we were able to blend, by those times, some libertarian ideas that were coming from America—to American institutes in Brazil, with some weird conservative ideas from a thinker called Olavo de Carvalho, who were very present on the Brazilian debate. We merged, but we were never, in the beginning, able to create our own thought. So we had a political movement that destroyed the Dilma Rousseff government.

We put down government in ‘15, ‘16. We created the language, the aesthetics of the Brazilian right by those times. That they were kind of kidnapped by Bolsonaro after a while, and he became the president in ‘18. So we created the movement that in the end, became what right now is Bolsonaro and his family. The use of the Brazilian football T-shirt on demonstrations, which was all of like we started the use of some fonts on the typing fonts.

The way of what we call the political meme, the political card, the way how to make a live stream. We started it all. But we resented it on the fact that we were not selling our own ideas because when we were bringing libertarian ideas to Brazilian problems, we said, “This, it’s not working.” We are not, for example, American. The idea of when we study the book, “Democracy in America.”

And we understand the way Americans see their own emancipation and how they interact themselves in a small town in America. How they develop the citizenship, how they understand their own citizenship, and how they cooperate is completely different from us. So the libertarian ideas doesn’t work here. And we understood it from the beginning. And when we start to elect our first boys, we understood it’s not going to work.

And also a very neoliberal idea like, “Oh, let’s privatize everything.” Now, it’s not work. It’s not going to work. In the northeast of Brazil, we need the state. The state must be present.

And you know when someone’s very poor, for let’s say a family that in the 19th century, they were all enslaved, and also some families that were part of a regime where they were working as almost slaves for a farmer in the northeast of Brazil. These people doesn’t understand the idea of agency because they were treated as animals for most of their times. So we cannot imagine that the invisible hand of the market is going to work and someone’s going to make a startup out of nothing. These people doesn’t even see the value of education because their parents never were treated as someone who’s going to have proper education. So the state must be there to be helpful and to bring some civilizational background and put them on a new position where they are proper citizens.

And this is the main problem in the poorest areas of Brazil. So we understood it. We start to abandon this neoliberal or neo-con ideas. We said, “This is bullshit. Let’s not going to use it anymore.”

But we were still a movement, and we were not able to put this idea to work by electing candidates in other political parties that would see our ideas as exotic. This is not convenient or why those guys are talking about it. So we understood that we need to have a political structure, a very old-fashioned political structure with a program, and the program must be like a bible. People are like, “Oh, I follow the program.” So we created an academy of new leaders we call the MBL Academy.

In this academy, we divided the curriculum, let’s say, with the training in three branches. The speaker, the thinker, and the leader. So the leader was from the House of Sparta, the thinker from the House of Alexandria, and the speaker from the House of Athens. We created something funny also to play because some people just like Harry Potter. So you belong to a house.

So I’m the House of Sparta, and I belong to the House of Sparta. I’m a leader. I’m not a speaker. And the candidates are the speakers. The thinkers are the ones who create a program.

And the leaders, they manage how to deal with the problems of leading a political group, and they are on the background organizing things. So we have a specific training for each one of these characteristics, and people choose where they belong. So the House of Sparta had a mission. They had to create a party. So the guys who are proper leaders, they say, “We need to create a party. You are the guys who are going to organize the groups across the country to organize the structure to collect signatures to make a party.”

The House of Athens, which is from the speakers, let’s say, you’re going to create our candidates. So this house train people to be candidates. And Alexandria, you’re going to create a yellow book. So we start to get the best students and put together some well-established Brazilian thinkers who likes our ideas. We had a set of values that we were defending, saying, “Apart from having these values as a basis, let’s check everything. Let’s bring ideas from China. Let’s bring ideas that are happening in India, in America, Estonia, in Portugal. Let’s see what’s going on. Let’s tropicalize it. Let’s identify Brazilian characteristics where these ideas can work or cannot work, and let’s create a very original program, and let’s bring very talented young people to be part of it, and let’s discuss it.”

So we spent two years and a half with thousands of people. There’s the people who are studying healthcare. There’s people studying foreign policy, economics. And we’re discussing and we published six books. The six books of the yellow book, they were white books.

I think they are here. They are called the yellow book, but they were white. Why we never painted it yellow? Because they were the discussions, so they’re contradictory. We are fighting inside like, “Yeah, this work in the next book. No, no, this can’t work.”

And then after a while, when the party was ready, and I was a candidate, let’s now organize what should be our program. Then it became the first yellow book, and then it became yellow. Now the cover of it is yellow. It consolidated what we thought as this is the program to reach this goal of Brazil as a political power in 30 years. And before we publish it, we got our best speakers, and we start to organize debates against our competitors on the left and the right to check the ideas and to see weaknesses.

Let’s put it to combat. Ideas must be put into combat. People should discuss it. And we start to see, wow, this is very consistent and even intelligent right-wingers or intelligent left-wingers, they were very curious about it, and they were, “Oh, I agree with you on this point. No, I agree with you on that point.”

Especially smart leftists, they didn’t understand. How come you guys made it? Because it’s not about right-wing. Actually, I don’t care about this thing about being right-wing or left-wing or a centrist anymore. I think it’s about, we have a problem.

I don’t know how I call this problem. I just call it Yellow Book.

The origins of the movement (18:46)

Wolf: Mm-hmm. So I guess I’m curious about the rest of the movement as well. The Yellow Book is just one small part. You’ve got this huge kind of activist movement that can mobilize thousands of people in some cities. We were out yesterday marching through São Paulo, thousands of people there to see you speak.

We’ve just been doing this campaign. You’ve had a very impressive showing in terms of the ability to put people on the streets, the ability to kind of drive aesthetics and oratory and so on, and the dedication from the candidates and so on. I’d like to dig more into just how you built all that. How does it work? What is this type of organization that you have?

And where did it come from?

Renan: The party, we talk a lot about the party.

But the party is just like, imagine you’re a warrior. The party, for me, is just the armor. The warrior is the movement. So we have many different parties in Brazil, but what matters is the movement behind the party. In Brazil, we have big political parties which don’t have any proper idea.

They don’t create imagination of a different Brazil. They have nothing. They’re just a bureaucratic structure where people run for a position, just it. So what makes us different is the movement. So to understand it, I was always leader of groups.

So it’s about the group, about the people. So there’s these different people who created our group, so it’s important to bring who we are so we can understand how we made it. Is it helpful?

Wolf: Yeah. Yeah, everything.

Renan: So for example, the group was created by me, my brother, Alexander, a very smart Japanese boy called Kim Kataguiri, a musician called Pedro. We are the main creators of the group, let’s say it. And me, I was a very shy boy from a normal middle-class family from Brazil with Italian-Portuguese background, living in São Paulo, which is the biggest town in Brazil, and was always a very good student. Shy but ambitious, which is weird in Brazil. In America—I think it’s a good point, but here you cannot be ambitious if you are shy.

Because it’s a very extroverted country. And so I was always leader of group, but not leading by being charismatic, but by having different ideas. And I entered in the best law school in Brazil, which is called São Francisco. The rally organizer on last Sunday, yesterday—Last Saturday was there in front of our old building. And there is a very competitive place.

In this law school, Brazil had 13 presidents coming from that—law school. And when I was there in 2003, and the political competition was huge, it was like, “Wow.” “I love it.” So I started to organize my own political party there. So I started to organize myself through the lens of a boy who were on a leftist campus, let’s say.

And I understood the way the left was organizing themselves, so organize a non-leftist party in this university with a leftist structure—using also entertainment as part of it. So I started it more than 20 years ago in university—when I was very young, and the party I created won eight elections on the campus, on the law school campus.

Wolf: Was this like student council kind of stuff?

Renan: Yes, which was the most important student council in Brazil. Many presidents came from this student council. It’s like very talented people came from there. Every time I see someone who was part of this student council in a government, like as a governor, as a minister, a secretary. We even had a kind of Skull and Bones come from this group called Burschenschaft.

It’s a long story, very interesting one. But I want to compete against them there, so I learned a lot from leftist movements in terms of organization. I left the law school, became a businessman. So when I went back to politics, I had all of it in mind—plus new ideas, plus experience, my being older. And I met a musician called Pedro, who made lots of success in England and Europe.

He played in many festivals in America. He played in Coachella—Burning Man, and he was making a mix of trap, electronic music with Brazilian funk. You should interview this guy. He’s the most interesting guy in our group. And we put together new aesthetics for right, and a model for organization.

So I blended things from him and from me. And when we put it together, we had a big cause to fight for, which was the fall of Dilma Rousseff government—the leftist government from Dilma. And when we understood how to organize people to make demonstrations and establish branches around the country—and it became easier and easier because we start to use also social media to become the leaders of the process. And all those small branches we had in the country that should organize the rallies against Dilma Rousseff, they were following the ideas that we were showing firstly on Facebook, then after that, on YouTube. So, it was easy in the beginning to organize a movement because everyone was anti-Lula, anti-Dilma by ‘14, ‘15, ‘16, ‘17, ‘18.

Starting over as opposition to Bolsonaro (24:37)

Renan: But our group had our proper test when in 2019, Bolsonaro became the president, and we decided that we should be opposition because he was doing, sorry, shit all the way. It’s very complicated for American audience because from right-wingers in America, they think Bolsonaro is like a base. No, he’s the worst thing that could happen. He’s horrible. He’s a very low IQ, weak man.

He’s weak and low IQ at the same time. And his lineage, his sons are all the same. Very stupid. So they destroy a political movement in Brazil that was growing, that could become something really interesting. And he allied himself with the so-called big center, which is like the old-fashioned establishment politics in Brazil, and he lost to Lula in the end in 2022.

But when we became opposition to Bolsonaro on the right-wing, we got destroyed. Our followers, they took our funding. They stopped donating, buying our products. They stopped watching us on YouTube. And we got destroyed.

We became a 10th of what we were. On all the instruments you could measure, like following, audience, events, money, we got destroyed. And then it’s where we became what we really are today because everyone was giving up, and I was like, “No.” Me and Kim, Kim was a congressman in Brazil. “Let’s lead a right-wing opposition to this asshole.”

And we lost all our Boomer followers because they were like, “You’re betraying Bolsonaro. You’re enemy. Oh, you’re Judas.” I was like, “Yeah, that’s why I like it.” Yeah, I remember one of my idols is Bob Dylan, and he was changing himself all the time.

And when he started using electric guitar they call him Judas.

Wolf: That was a scandal, yeah.

Renan: Yes. “Oh, what an absurd.” And I felt like that. Like, “What? What you’re doing?”

Say, “Yeah. I’ll do the opposition. I don’t give a shit. And I will not end up with MBL.” So we kept MBL alive.

And we start to raise a new awareness and create a new movement only of young people because the Boomers there were hating us. So it was like all the Boomers with Bolsonaro against a new generation of people that were like, “Okay, we have no money. We are young. We are fucked. And we dislike Bolsonarism, and we dislike the left. So what can we create?”

So, okay, we have a blank sheet, so we can create something from scratch. So we were able by the fact that we destroy everything we had, like when you have nothing to lose, you can do everything, anything you want. Because we had nothing to lose. So let’s start from scratch, and let’s not care about anything. Let’s make something new.

And we started this academy. We started to create new kind of leadership, a new political discourse. So it’s a process that we start to grow from zero, and it gave us a soul. This is the most important part. When we started as anti-leftist movement, we had a body but not a soul.

When we start to fight Bolsonaro, we had a soul, like the idea of a struggle, we against the world, we against everyone, and we got prosecuted. They tried to close the movement. Bolsonaro tried to put my family in jail. We had lots of problems, but we always kept fighting. And we were kept fighting, but as we were not focusing on quantity, I don’t care if you’re a Boomer following me, don’t follow me.

I don’t want you. But on quality, we start to call our enemies for debates all the time. “No, I will show you you’re stupid.” So there was this image that, okay, we don’t like those guys from MBL, but they are smart. They are the best, even though they are bad.

They said they are bad, they are mean people, but they are smart. The Boomers were seeing it.

But the young people, they didn’t see. They see like all those guys are struggling against everyone. And we were, like in Brazil, it’s very common to be on a corruption scandal. Everyone. All the time.

We are never on a corruption scandal. So people are like, “Fuck! These guys, they cannot exist.” It’s impossible to exist someone who’s not on a political scandal or who’s not stupid, who’s not making populist shit on TV. So who are those guys?

So we became generational, let’s say phenomenon. And when we create the party, people from our generation start to say, “Okay, this is the project of our generation.” So I think it’s a nice overview to understand that to create this organization, you must have a struggle in the middle. I couldn’t make the party if I didn’t get destroyed by 2019. I was a bigger movement in 2016 than I was in 2020, 2021.

But I was weaker, which is weird. For example, let’s say in America, maybe everyone’s kind of afraid to be against Trump or against the MAGA movement and create something that’s not a democratic initiative. Maybe you lose everything. But maybe it can work. To be generational, you must break the rules or break the standards that everyone works.

And that’s what we are doing. I don’t know where we’re going to reach, but I’m sure that we are growing up with our audience, our followers, and our Generation, let’s say.

Campaigning as an outsider (30:23)

Wolf: Mm-hmm. Well, you’ve just finished the campaign with that organization you’ve built. I was out there at the rallies. It’s got young people, extremely dedicated, lot of physicality to it, the chanting, the singing, a lot of energy people are pouring into it, and yeah, it’s all more or less all young people. It was a great energy.

Why don’t you tell us a bit about the campaign itself? You went around on the bus, like a big bus tour all over the country—visited and had rallies in every city it seems like. And how did that go? How do you feel now that the campaign is finished, how do you feel you did—overall?

Renan: Every campaign everywhere is a meat grinder. And this campaign, especially in Brazil, as the third-party candidate start winning other countries. For example, Petro in Colombia made a red alert for our establishment here. People were very afraid of this third-party candidate, let’s say this—this outsider that comes and start to grow in the polls. And so the campaign was designed to help the people with money, and I’m not acting as a victimist or full of conspiracy, it’s just facts.

Brazilian, all the big parties, they have lots of money. As long as you elect lots of congressmen, you have more money. I come from a new party. I have zero money come from the government. So I had to use donations.

The Bolsonaro party had one billion reais from the government. In dollars, I’d say $200 million, plus the money that they collect from businessmen and stuff. And Workers’ Party also, they had around $200 million to spend, and other parties, they have their own budgets. So I couldn’t use social media. They blocked the use of the algorithm of social media in Brazil.

Wolf: Like you couldn’t go viral anymore?

Renan: Exactly, so I stopped going viral by August when the proper campaign started. I was the candidate that was growing faster—during this year. I was the candidate that on TV and on podcast, I was having the biggest audience, and I was growing, growing, growing. On Polymarket, I reached 15% by June or July. I was growing, growing, growing, growing.

It was very similar phenomenon that was happening in Colombia, but then they closed the algorithm. They started not inviting me for debates. Lula and Flávio Bolsonaro, who are leading, they start to boycott all the debates so people wouldn’t watch a debate. And in the end of August, a Supreme Court judge just put out the register of my candidature, so I couldn’t use social media, I couldn’t be on TV, I couldn’t be interviewed.

Wolf: This is when they tried to shut you down as a campaign? Yeah.

Renan: Why I’m bringing it, because it was a meat grinder. I couldn’t do anything, and then there’s this volume of money and volume of structure. In Brazil, for example, if you are in favor of Lula and you are a mayor of a town, the federal government gives you money so you can make campaign, kind of legal, for Lula in this town. And they put the people that works for this city, the public workers, to work for the campaign, obligatory. So multiplied for, I don’t know, 3,000 towns—billions of reais, billions of our currency in money spread around.

So there’s a whole structure. And also the people, especially poor people that don’t have a proper curiosity for new ideas, let’s say. They are stuck on watching football or watching soap operas or I don’t know, following football players. They only could just hear me through TV. And I had no space on TV debates because they had to cancel the debates because Lula and Flávio Bolsonaro said, “I’m not going to debate if Renan’s going.”

So it was like I was blocked.

Wolf: They shut you out.

Renan: Exactly. And so what I decided, from end of last year until June, I crossed the country through car. By car. Used a car and was visiting town by town, town by town, meeting people. And after they put out my campaign, I said, “No, I will have a bus. I will stop flying. I will cross the country with a bus.”

A 1979 bus, very old-fashioned bus. And let’s see what happens. So I put all my candidates that wanted to go with me. I made a live stream. I stay alive for 24 hours a day.

And stopping three or four towns every day, organizing rallies, and putting people in the streets, making parties at night, destroying my body because you can imagine—every speech I give in those rallies are one-hour speech. I don’t read anything. I don’t prepare. I just go with the heart. Blah, blah, blah.

Wolf: Yeah. No, it’s great stuff.

Renan: Yeah, yeah, yeah. No. I don’t want to brag myself. I think I’m a good speaker.

Wolf: Good boy. You certainly got a lot of practice at this campaign.

Renan: Yeah. So it’s very intense. Your mind gets broken. Because after a while you get really, really tired. You don’t sleep properly.

You don’t eat properly. You get stressed. But it was very beautiful. It was the most beautiful thing I could do because for these young people who were following us, that was the most romantic adventure they could have. And this is on a country where politics is pretty dirty.

It was so dissonant. So different. What the hell is going on? That even the young left in Brazil, they were in love with our campaign. They were saying, “Look, we should be just like those guys.”

And people by aesthetics, by vibes, they were jumping from a leftist campaign or leftist—support to support our group. And also some agenda we have in terms of sovereignty. They start, “Oh, those guys are against China or United States taking care of our natural resources, the rare earths. So let’s support those guys.” So we start to create something very new from scratch.

So I’m very happy with the campaign, even though this was a meat grinder. But to be very optimistic, not about me and Brazil, but about people who likes what we are doing as an example. In other democracies and other countries where people—we have a lot of problem with illiteracy in Brazil. We have people that are not following the news. It’s a very poor country.

Bringing the vanguard movement to America (37:26)

Renan: Yeah. Brazil is very poor. On a more well-developed country, this strategy we’re making would develop itself much, much faster. Actually—

Wolf: You think so

Renan: we are a vanguard movement on a country that is backwards. So I’m sure what we are doing, the playbook of what we’re doing to—Even Argentina or Chile or Mexico or southern European countries, some Arab countries, or some Turkey, it could work and go faster, much faster. In America, for sure.

Wolf: Yeah, that’s what I’m very interested in talking about. That’s the center of really why we’re here. I got inspired by what you guys are doing, and I thought, “Wow, we need something like that in America.” We’ve got this sclerotic, corrupt establishment. Our own equivalent of the senile, corrupt left and the senile, corrupt right.

And the young people can see these incredible problems in America, and it’s been very hard to break out from under the ideological suppression, the total institutional suppression of this stuff, and operating under everything requires the boomer figurehead. You’re always operating under these hostile institutions in a way. And so you guys are starting to provide something of an example that we can break out. And so that’s why I’m here. It’s like, what can we learn from Mission Party, from Renan, about how do we do this in America?

And so let’s start talking about that. What would this look like if we were to bring this kind of mission-style approach to America? We’ve been having a few conversations about this. Obviously, you need the platform. You need to put together our yellow book, adapt this kind of thinking to the problems of America.

You need some kind of initial political project just to start getting something rolling. You need the influencers and the organization and the style of organizing, the kind of chanting and the singing, and I don’t know. There’s all these different elements. I’d love to hear your thoughts on kind of—you’ve spent a bit of time in America, how could we apply this in America? What does it look like?

Renan: First of all, from conversations I had with American friends, even friends from Europe, you should not have a proper plan. What I mean having a proper plan is I never planned the thing. For example, I never imagined that by fighting Bolsonaro, I would destroy everything, and it would give me some sort of freedom that would sound romantic, and people would follow me because it’s idealistic and romantic and kind of heroic. And that gave us the confidence to let’s start everything from scratch. We never had funding to do it, and every time I talk with smart people from abroad, people want to do it on a smart way.

Like planning, find the funding structure, creating some goals. And every political movement in story kind of—we have a joke in Brazil that sometimes you need to fix the flat tire with the car on movement. Okay? So it’s like you need to go in the way, and you create your thesis, your theories, and you adapt yourself on the way. So it’s a living animal.

Movement is a living animal. Must be a living animal. You must have a very loyal group of people who really believe on it, who would devote their lives on it. I will devote my life on it myself. I don’t have a proper personal life.

My brother has a family and kids. Other guys in the group have, but I spend all my time on it, and I don’t know why I do it. I feel happy by doing it. And you guys need to find crazy people that understand that they have a mission and a vision of something very different for America, let’s say. And be able to try different things.

And why I’m telling you this because if you plan something, sometimes it doesn’t work. And you break your face against the wall. You say, “Okay, let’s give up.” And that’s when you cannot give up. So in the beginning, you must find a core, a group of people that really want to do it for life.

Loyalty and the core team (42:02)

Renan: I think it’s the hardest thing. A core group of sometimes four, five, six people who you can trust your life on it. I will give the level of trust I have. I don’t take care of my bank account. my brother take care for me because I don’t think about my personal life.

Sometimes I merge with the movement. I’m talking about this level of—dedicating yourself to the cause. If Kim Kataguiri, our congressman gives a speech somewhere, I don’t care about what he’s saying because I trust totally that he understand the plan. There’s this kind of people you must have by your side that you trust completely, because in the end, it’s loyalties. The loyalty and trust is the common thing.

There’s some sort of family you must create around you. And I don’t know if it’s pretty, let’s say, pretty Latin to talk about it in those terms because sometimes you guys from north, let’s say—you are more objective and impersonal. Even though the best experience you guys had in history were about creating loyal groups. So—

Wolf: Yeah. I think the same logic applies. It might be for American culture, it takes slightly different form, slightly different cultural touch points for it, but absolutely, I think that shared dedication to a mission—the absolute trust in each other, that’s essential wherever you go, right?

Renan: Yeah. And this is very important because sometimes the plan is not going to work. Lots of times the plan’s not going to work. I created many initiatives on MBL that were a big fail in the middle of the way. Total—fail.

And then when you reach the goal, it’s like you really explode. It works pretty well.

Wolf: Something works, yeah.

Renan: Yeah. So what I think, a core, a group of very intelligent and dedicated people, some of these people will be more like internal leadership, and some of them will be more external leadership. It’s obvious. Have a vision, a program, and you understand that this program will change through time. And be open to stay in the shadows some time, to organize leaderships that will be the face of it in many different area, because national coverage of a movement is very important.

I prefer to only talk with young people, but I talk to young people, but I’m across the whole country, I have leadership in all the states of Brazil, in all the medium, small towns, than to talk with everyone but be only in the center south of Brazil, let’s say. It’s better to be everywhere because you understand the different aspects of the country on it. Smart use of technology. Complete loyalty to the group and the idea of the group. So the group, the party, the movement comes first—and the personality comes after it.

So never be personalistic. I will give an example of it. I’m the president of the party. I don’t care if I’m the president. I’m a natural leadership—so I don’t need the position of being president, institutional position.

But I don’t want to be the candidate. I was a candidate because people said, “You must be the candidate.” And I said, “Okay, you guys want it, so, okay. I’m going as a candidate.” But all I’m doing is to reinforce the Yellow Book, reinforce the party, so new leadership will appear, and I don’t know who’s going to be the next candidate one day.

Why politicians shouldn’t lead the party (45:58)

Renan: Even though I understand that being charismatic and being a good leader would accelerate the process. Having this kind of tri-functional system of hiring and training people. On politics, there’s these three kind of people. You always must have all them, the intellectual, the leader, and the speaker.

Wolf: The candidate.

Renan: The candidate, let’s say. Because when I say speaker, but sometimes he’s not a candidate, but he’s very important in public debate.

Wolf: The rhetorical person. Okay.

Renan: For example, there was a guy in our group who’s, Arthur, he lost his political rights, let’s say, because of a very funny struggle in Ukraine. But he’s on the public debate. He’s taking part in discussion. So you must train people and understand that you should never leave the leadership of the group over the candidate. The candidate is never the leader.

Candidate can’t lead. The big problem is when you let the candidate lead the process. Or the thinker must always be the proper leader. And this is a vision that all the parties everywhere makes a mistake.

Wolf: Can you explain that more? What are these exact roles, and why can’t it be the speakers and politicians, I guess, who are in charge?

Renan: Because most of the time they submit the whole structure to their own personal projects of power. So they destroy the natural function of the structure on their egoistic will. And even when they are good people, sometimes people start to work on their own interests, and they start to forget what made the group a group. And intellectuals as well. They are even more complicated sometimes.

So there’s a sort of human material that I call the natural leadership, creative leadership, or leadership based on example. They are the most difficult ones to find, but they are the most important ones. And it’s hard to train them, it’s hard to identify them, but they are the ones that makes the group. They create the cohesion of the group. They make the group—stand together and make the principles being they assure the group that the principles of the group will be followed by the candidate, and that sometimes the thinkers will not come with their vanity and implode the project by imposing their vision when the vision is not sometimes necessary in some circumstances.

Wolf: Right, or they go down rabbit holes.

Renan: Yes. And that is common. So must always be the leaders and this is a lesson we learned from time to time. For example, we created a state branch of a group in a very important state in the south of Brazil. And there was this guy who was a leader and a candidate, and he submitted the whole process to his will, and he betrayed us in the end.

It got destroyed so suddenly. We need to start from scratch in that state. But when we leave the leadership to people who are not candidate, it works. Works a lot. The best ones are the ones who are able to be the three characteristics, the three types in one.

I see myself as kind of part of it. I kind of write it, I’m a natural leader in the group, and I’m a good speaker. It’s very hard to find, but they are the ones who have a special license. And we have some of them who are able to be this kind of personality. But somehow never let candidates lead groups, only if they’re very, very special.

This is very, very important. Very. Candidates must submit their will to the group. When he wins a position, like someone wins for a state congressman, a state deputy, or federal congressman, his cabinet, his position belongs to the project not to his own will. And we have had very important cases.

This is very important that happened in Paraná last year. Paraná is a state in the south of Brazil. We elected a boy who was not right, but he was a good candidate. He was a good speaker, not a good leader. All his cabinet was made of very young, very talented boys that were planning to have the best city councilor in Brazil as this boy.

They start to file new projects of law. They were fighting the way they were taking care of people who live in the streets. They were solving proper problems coming with technology, but this guy was only, “Oh, I want to be famous because I want to get elected for congressman in Brazil in the next election.” “So I want to record videos and get famous.”

Wolf: The career politician.

Renan: Yes. So he started to submit the group to his will. The group created a riot against him. They told him, “We are kicking this guy out of the movement.” And some of them, their first jobs were working in this cabinet.

And we are living in a poor country like Brazil. So they lost their jobs. They got problems. They had to go back to their own small towns out of Curitiba, which is the capital of Paraná. But they kept following the group, and they said, “This guy is out of the movement, and we’re going to destroy his political career.”

His political career right now is destroyed. So he lost the group. So the group said, “Your position submits itself to the group, not the group to yourself.” So it became the standard. So we were showing to the other political groups, “Look, things here works on a different way.”

The candidate is only the candidate. The candidate’s not bigger than the group.

Wolf: The candidate is just the face, the spokesman.

Renan: Exactly. Exactly. And he can’t be much more than that, only if he has special talents. Kim Kataguiri. He’s a very high IQ boy that we have in our group.

He’s a genius. He works a lot. He’s a guy who gives his life for the movement, just like me. He will be a presidential candidate for our group within some years, for sure. And he’s brilliant.

And he’s a guy that, okay, he’s running for congressman all again for the third time. And it’s different, the political system that we have in Brazil from America. So it’s a, we call proportional open list. So, many guys from our party get elected in the same state by number of votes that they have on the list. So Kim should ask the voters to vote for him, but he wants to elect other people from our party.

So we go to our rallies, and he ask the people to vote for other guys from our party and not for him, so he could share his votes with other people from the movement. He said, “I don’t need to have 1 million votes. I can get elected with 200,000, so let’s vote for other guys.” So it’s not normal. We don’t see it.

So it give us also, an authority for him, a moral authority as, “Oh, look, we should be like him.” So all the other boys want to be congressmen in Brazil. They say, “I want to be a Kim Kataguiri. I want to serve the group just like Kim does.” This kind of example keeps the structure alive because the easiest thing is let the bad side of ambition takes over, takes control of the group, and the group becomes a collective of personal projects.

And it can’t be like that. Bolsonarism is a collective of personal projects that anchor themselves on the surname Bolsonaro. So they get elected by, “I am a soldier of Bolsonaro.” But everyone has its own political project, and they are enemies. It’s not actually a group.

Building a proper political party (54:05)

Renan: So they’re not a group. They don’t have a platform, they don’t have a program. They just say, “We hate the left, and we love Bolsonaro.” So that’s why even though they can win elections, they do nothing. So they’re able to win election, but when they are in the cabinet, they don’t do anything.

And, well, I think it’s good to win elections, but it’s very bad to build a country. Sometimes I think it happens with the MAGA movement. I’m not a specialist in American politics, but sometimes people feel—frustrated that nothing happens in the end. And sometimes you see just stupid people getting elected, and they’re just like, “Oh, make America great again.” And—

Wolf: Yeah, and it’s just that personal loyalty to Trump is kind of the only—principle of the organization.

Renan: Yeah. And in the end, we must have some people pledge allegiance—to a approach, to idea, to a dream. And which is like we didn’t invent it. The left is doing it for more than 100 years. So why not copy it if it properly works?

The Workers’ Party in Brazil is the most important party in the story of Brazil. I studied their story from back to back. I know all their leaders. I understood everything I read. I read all their books.

I tried to understand everything they were doing. When they were beginning, actually, they were the dream of the Brazilian boomer. The dream of the Brazilian boomer was, oh, Brazil is very unequal, and we’re living under a military dictatorship. Let’s bring democracy, and let’s make the country more equal. That was the idea.

And they were socialists. They hate America. There were all the other things—that are natural to a leftist movement, but they were the project of a generation. So if you were a young boomer by the ‘70s and in beginning of the ‘80s, you should belong to Worker Party, and was nice to be part of the Workers’ Party. Middle-class people were defending the Workers’ Party.

They had Lula, for sure, but they have a plethora of cool leaders across the country. All the artists started defending them. And, well, when they start winning proper federal elections in 2002, they are in charge until now. And they only lost the power once when we put them out by impeachment, not by election. We put Dilma Rousseff out.

So it was my group who made a small gap, let’s say, on this lineage of—Workers’ Party governments.

Wolf: Was that where Bolsonaro came from, or was it—yeah, okay.

Renan: Because we organized—

Wolf: He came from that gap

Renan: the impeachment. The Workers’ Party, they were out of government. And we also helped put Lula in jail because we organized the riots that helped give authority to this legal operation that put Lula in jail. So Lula was out of the game. People did find themselves as right-wingers, and the streets were yellow, not red anymore.

So it came a guy that became the leader of the process. That was Bolsonaro. He won the elections. But he stay on term for four years, and here was Lula, and he was back. And Lula is the—Probably he’s going to win this election.

So the Workers’ Party was an experience that you can understand how a proper party can work, even in a country like Brazil. And they were very intellectually sophisticated. One of the best writers in terms of Brazilian political structure, they were founders of this Workers’ Party. So when we started writing our book, we said, look, our enemy, they had a hell of great thinkers and writers working for them, and we even used some material from them. There’s two writers, one who, Sérgio Buarque de Hollanda, another called Raimundo Faoro.

They were very internationally sophisticated. Sérgio Buarque de Hollanda is one of the best political and sociological writers, and readers of Weber, and he understood Brazilian soul a lot. So, and he was founder of the Workers’ Party, so I should respect him, for sure. And why not create this kind of structure for us, a classical party?

Wolf: Right. And that’s what you call this thing, a classical party. And that’s kind of what I’m recognizing, right? Is like old school political party, like a proper political party, a serious organization that’s internally organized, vertically integrated with the goal of governing the country. It’s not just, oh, it’s some association of people trying to get elected within this or that—position for their own political career or whatever.

Renan: Exactly. Every mandate from every leader that get elected belongs to a bigger plan. So if someone get elected as city councilor, this position belongs to a bigger plan.

Wolf: Yeah, and so that’s—

Renan: Everything’s connected

Wolf: that’s kind of the discipline you apply to the politicians. Their job is get elected and then carry out the program of the party, right? And then the job of the intellectuals has been produce the platform, right? Work on the stuff, produce the platform, and produce, I assume, the propaganda that you put out in Valete Magazine and—other, just the live streams and so on. And then you have these, House Sparta, the organizers themselves—who you say are like the real organizational power backbone of the movement—who they’re not the faces of the movement, but they have the—the organizational power, and they dedicate themselves to—the movement—as such.

So can you tell me a little bit more about how that works? Like, how are they organized? What are they doing? I guess, just give us a bit more flavor on kind of—

Renan: For sure

Wolf: how do you actually get those guys in motion?

Renan: For example, the ones who take cares of the branches of the movement—and from the party nowadays are always Spartans. And so every year, we organize conferences on each state. So all those conferences are organized by those Spartans. So they have the technology of how to organize an event, how to sell the tickets, how to put people in the event, how to talk to the press, how to organize the live stream, how to take care of the local leaders, how to spread new branches around, how to avoid traitors or unfaithful people, let’s say, to invade organization. how to fund themselves.

They are the backbone, the proper backbone of the structure. They are very loyal, and they used to be accepted as the moral authority of the group. They are very, very, very important, and they are the ones who are solving things. Meanwhile, the speaker, let’s say the candidate, is blah, blah, blah. And sometimes we cannot create a mess by making them become bureaucracy because leadership is different from bureaucracy.

Keeping the buraucrats from taking over (1:01:18)

Renan: Because sometimes political parties are destroyed by the internal bureaucracy. It’s the critics that Trotsky had for Stalin and the communist revolution in Russia because it all became bureaucracy, and it was bureaucracy being bureaucracy all the time.

Wolf: How do you avoid that?

Renan: By being very idealistic and bringing non... How can I say? This kind of leader is also idealistic leader—so he cannot be totally pragmatic by organizing our group. So if we defend some sort of ideas who are not properly popular, but we must defend it, this leader is the one who’s going to try to implement it, even though it’s more difficult. The bureaucracy thinks about the benefits that as a bureaucratic person you can have, so they adapt themselves to the system and become bureaucracy.

So they want to make things easy and have their own positions. That’s what bureaucrats do this. And for us, it can’t be like that. So we are always pushing this kind of leader with those innovations we have in terms of scores, in terms of proposition, new kind of leadership, new national position. Because sometimes I come up with discussing against what’s mainstream on the right.

And then this local is like, “Oh, Renan is going to destroy my new election because this position he’s defending is very unpopular where I live.” For example, we do not defend raising the payment of the food stamp that we have in Brazil—called Bolsa Família. So a leader in the northeast of Brazil, if he was a bureaucrat, he would say, “Renan, you’re destroying our planning.” But they are not. They are like, “We must find a way.”

So they must be very idealistic, and they should kick the bureaucracy out of the thing because there’s people who want to just have a career, stay on a cabinet of a candidate, and have a life there. These are the ones who we should avoid. But a proper leader, that’s why they are a leader, not a bureaucrat. For example, they watch my livestream every day. They read “The Valete” magazine.

They know what we’re discussing. So they understand that, okay, we are going this way, and I need to implement it on my way of the group in this area, and it used to work. So that’s why they are very rare people. They’re not easy to find. It’s not easy to find this kind of leadership.

Wolf: And how do you identify those people? Identifying that difference between the bureaucrat and the idealistic leader, I imagine, and people can kind of decay maybe even, or at least I might imagine that people would decay. They come in, they look like an idealistic leader. You put them in this position, and they start to get lazy, or they turn into a bureaucrat or something. How do you deal with that?

Is it like higher levels of the party come in and review what they’re doing?

Renan: Actually—

Wolf: What’s the actual system of discipline there?

Renan: Actually, it’s very easy because as we have elections, a bureaucrat, he’s always afraid to lose his job. So when he gets together with the candidate and they start to not follow the instructions of the group, but they start do what would earn more votes, we can identify it easily. It’s very easy to identify—

Wolf: Because—

Renan: who are doing it. And even the other guys, other leaders around him start to, “Renan, look, something weird is going on.” People start to identify who’s going to betray us, and so there’s this moral pressure over these people, and usually the local branches uses to expel these people, naturally. But when they are on a cabinet, on a position of a congressman who got elected, it’s very easy to see. We see in the work of the congressman, we see on the way they’re talking, we see in the way they vote, we see in the complaining of the locals.

For example, we have lots of militants, lots of followers who—they are volunteers, they are working. So they can see someone who’s not a volunteer anymore, who’s building a career, and these kind of people start to avoid having conversation with the volunteers because the volunteers start to become a problem for these kind of people because they are idealistical. They are there—because of the ideas. So when we start to see that it start to have this, they are detaching from the volunteers—from the supporters, okay, they are become bureaucracy.

Wolf: Right. I get it.

Renan: This is the sophisticated thing in terms of how to manage these things. So the state leaders of the party and the movement, they are very able in all this. We have lots of experience on how to identify when we start to have this problem. But it happens all the time. We don’t have a proper formula to avoid—because especially try to imagine sometimes a poor group of boys get elected and it’s their first job working on cabinet and electing a leader, and they start to get famous.

People start to recognize them on the streets. And people, “Oh, oh, oh.” So it’s a problem. Some of them will corrupt themselves in this process. We elected some boys in 2024—lots of boys all around the country I’ll give an example.

The boy who’s filming us, I don’t know if he understand English, but he was working on a cabinet of a guy who betrayed us. That guy became a bureaucratic—betrayer. And the team, the staff, start to identify everything, all this playbook of a Judas. And they were telling us in São Paulo—and then we prepared the way to expel him and keep the team back. So more and more we should train the people.

The talent pipeline (1:07:15)

Renan: We should understand that we cannot put very young people on cabinets or on bureaucratic positions because sometimes their first jobs, or sometimes they don’t have money, so they start to rely on it to survive. But this is mostly a problem in a very poor country like Brazil. It’s a very poor country. And we are a group that we have people from the elite and people who are very poor. We find very talented poor people—poor boys and girls who start to organize themselves, and they win election.

But then just by having a job, their living standards change it. And it’s problematic. So this is a problem we have to deal. But I think it’s much more a Brazilian local problem than anything else.

Wolf: Yeah, in some ways, it seems like Brazil has more problems in these areas, but also maybe less problems in some other ways. In America, I know one of the things that is sort of a constant frustration of idealistic political groups is all the talented people have enormous options, right? They can go work in industry and make enormous amounts of money. Anyone with a bit of talent, they don’t want to struggle in politics, this kind of losing over and over, which is kind of like your way of doing it. Losing until you win, right?

Politics, I think, is a little slower than business and industry. You’re more confused, and you spend more time struggling. And that’s not attractive, right? That people don’t—want to deal with that. I wonder if there’s hidden challenges that you’re not seeing in other countries.

Renan: Well, maybe things here are—they are complicated as well, but on the opposite way. By, as we don’t have much opportunity here—people cannot be trying a lot as well. Because people don’t have much opportunity, so they must find a job so they are not able even to survive. There’s this—negative thing. One of the problems we have is the person who lose one or twice, then I can’t live like that because I won’t have money—

Wolf: Especially in Brazil

Renan: especially in small towns, in poorest areas of Brazil. So in the end, I think there must be a set of crazy people who really believe in the project, who really believe in this idea, and that wants to put their names on history. What moves me is one day people will study what we did and what we organized to make Brazil a very important country. So one day people will, I don’t know, “Oh, Renan was really cool. Renan did what should be done.”

Yeah. So for me, I think there’s people who are like that. For me, it’s better than money. It’s much better than money.

Europeans make those jokes about Americans as like, “Oh, Americans just want to work and have money.” I don’t think Americans are. They’re much more idealistic than almost any country in the world. Americans are a pretty formal people. So I think there’s people who understand that putting their names on history is much more important than anything else, and give this contribution, especially to a country which is so special as America.

It will be a privilege. So just by being part of it, some people will be happy. For example, by being a leader in this group, I dedicated myself to it. My personal pleasures, I like steaks, I like music, I like reading, I like going out with my friends, I like spending some time with my family. I don’t need much more than that.

Because on the rest of my time, I’m contributing to something that’s so big, that it’s bigger than any company that I would open up. And I think there’s people in America, for sure, who thinks like that—and who would dedicate themselves to this goal. And actually, if they are smart, they will succeed. For sure they will succeed. So you guys need to find this kind of people, very special group of people, and things will happen, for sure.

I think it will. I’m very optimistic about it. And more than that, it will be easier in America than here. I’m pretty sure.

Wolf: You think so? Yeah.

Renan: I’m pretty sure. Because here we have other kind of problem. We are a vanguard movement, in terms of organization, technology, everything—on a point of view, on a country that’s very backwards. Very backwards. People are still selling their votes for politicians in Brazil.

A politician can buy votes—in Brazil, illegally, but they’re doing it on scale of millions. Millions of people are getting bought today—to vote. So it’s very primitive yet. So we’re like building, I don’t know, a SpaceX in Congo. That’s how sometimes we feel about our project.

But we are Brazilians. We should do it. For us, it’s very nice that a project like that was born in Brazil. We are happy. But it would work much faster in a more well-developed country for sure, and pretty sure

Going from ideology to institutions (1:12:51)

Wolf: Interesting. Well then, I guess the most immediate problem facing us in America is that kind of cold start program, right? I think we have a network of very smart, idealistic people. You’ve met a bunch of them. We’ve got a bunch of them here today.

Some of them came down with me, some of them came down through other friends of the party. But, and then back in America, part of the whole point of this project, The Students, is we’re gathering those people. The Students, who are they? They’re the people who do the reading, who are idealistic, who want to save America, right? Who want to save, and beyond America, our civilization.

So it’s like we are gathering that, but there’s this cold start problem of, okay, we’ve got people somewhat. We’ve got a network. Bringing it to that level of, and now you’ve got the academy, and the party, and the program, and the candidates, and all this. And so I’m very curious, I guess, just more information on that, how you started the academy. What were things like at that time?

This was during your movement, kind of starting to get frustrated with lack of having your own ideas. It’s a little bit of a different problem than we have in America. I think in America, our network at least, is very much like the intellectuals. We’ve got the intellectuals networked and thinking, doing a lot of things. Some of them are publishing, some of them are not.

But we don’t have the movement, right? We don’t have the street presence and so on. So it’s a little bit different, but it seems like there’s still this kind of going from this inchoate initial set of assets to the more complete beast. That’s kind of this problem that we have to overcome. So, and maybe it is that academy.

I talked to, I can’t pronounce his name. It’s all his Brazilian sounds. Guilherme Freire.

Renan: Ah, Guilherme.

Wolf: Yeah. He runs this training program. And so we’re talking to him thinking like, “Oh, should we do something like that?” And maybe this academy is one possible version of it. You said you had your students, they come to you, and they pay a little bit to the party to be educated into these the three houses, the way of operating.

But when you started that, there wasn’t much there. There was just, I guess, the movement that wasn’t as organized. What was that first class like? How did you come up with that idea of the academy? What was on offer—right at first, right?

You don’t have this political thing where you can just slot them in. You don’t. Or I don’t know what you had at the time. What exactly were they being trained for, I guess, is the question. Why was it valuable to those first students?

Renan: Great. Well, I could never plan the academy—if I didn’t have a proper experience understanding this trifunctional way of doing politics. And how I learned it, by experience. I understood that when I created the movement, we had some people who were more intellectuals, and they are a special set of people. I was a leadership.

My brother as well. And we had Kim, Arthur, and other people who were candidates. And when we started to grow, we realized we started to clusterize ourselves in a group of leaders who had some understanding of how to organize things. People who are candidates, so they start to organize themselves and discuss about how to debate, how to understand how it works, a political proposition, how to talk to people in the streets. And the thinkers who are discussing things all the time.

Wolf: How to fix Brazil.

Renan: Yes. And I realized, I was saying, “Look, there’s always these three kind of people.” And I said, “If we want to create our own following and our group with its own identity, we must create people with this kind of profile and train them on a specific program.” It’s very different from what Guilherme Freire is doing because he’s making intellectual, some sort of religious philosophical movement, with no specific political goal, let’s say—or no upfront.

Wolf: And it’s very much based in the business elite and—so on.

Renan: As if he is colonizing the elite.

But politically, I don’t know if he wants to be practical. That’s not what he wants, or at least that’s what he doesn’t show as something he wants. We are very straight to the point. It’s very political, and we really need to trust in this academy because we only will hire people from the academy. So it must work.

You can only hire people for the cabinet of, or elected people, that were part of the academy. Or our candidates must be part of the academy. You cannot run a branch of the party, of the movement, if you’re not from the academy. So if you want to grow in our structure, you must pass through the academy. And so I created it in the end of 2020, in the middle of the pandemic, so it had to be online because people couldn’t make gatherings and stuff.

So we created these three different graduations, let’s say, the Sparta, Athens, and Alexandria. And we had, in the first year, 1,000 students, and we started to make them organize themselves in their own states and towns. So they were spread around the country, so they started to organize themselves, and they had just like there was an anime called “Naruto” where people had teams. I’m team number seven, team number three. So we created, “I’m the team dragon. I’m the team the snake.”

We started creating the name of the teams, so they had to identify themselves as teams, and all the teams must have a Spartan, a guy from Alexandria, and someone from Athens. And look, you organize yourselves. You organize yourselves, learn how to deal with the problems, the competitions.

Wolf: Were the teams scoped to particular areas? Like—this is in your town. You’ve got a—Spartan, an Athenian, and an Alexandrian.

Renan: And you need to organize. We give them missions, like, “Oh, you need to for example, I think you must publish an article in your local newspaper. We’ll not help you solve it.

Wolf: Yeah. Figure it out.

Renan: Figure out how to do it. You guys must file a bill of law on your city council. Fix it. Find a way. We’re not going to help.

We gave classes on how to do it. You guys have a problem, go and fix it. Because in the end, people need to get out of their houses and start doing things. And so one of those guys start to organize demonstrations in the city councils against corruption in this town, and they will start to be on the press, in the local press. So their group, they start, “Wow, we are doing it.”

So they start to feel that they are useful. So they start to feel as one, as a group. They start to create a bond, a loyalty among this group, and they start to feel like, “Oh, we are changing things in our town.” So it was nice. After a while, we start to improve some aspects here and there.

So sometimes people self-identify as a leader, but they are not. Most of the people try to define themselves as thinkers because they are like, “Oh, I’m intelligent.” But they are not, so they cannot be. So we start to identify who they are during the process. So we came up with a common set of classes, and then they—

Wolf: They specialize yeah.

Renan: So we learned in the middle of the way, because as we hire those people after it—we want them to be good. So must be good because we’re going to use it. So if it’s not working, we’re not going to use those people. So on the part, when we start to create a party, we had to create hundreds of teams of people collecting signatures on the streets to have the party. We have to collect around 800,000 signatures.

And so they would go to all the bureaucracy, and we could have the 600,000 needed to have a party. So we put all these Spartans to organize it, and most of them succeeded very well. But some of them not, and some thinkers became good leaders. So we start to say, “Oh, okay, this guy is not a proper Spartan, he’s just a bureaucratic guy,” or, “He’s just someone who doing some sort of gossip politics, creating problems with other people. He’s not a proper motivator.”

Like someone who makes people wake up on the morning, “Go collect signature. Everyone must find 50 signatures today in the same...”

The process of creation of the party was the time that we, okay, the academy worked, and the other moment was, okay, the yellow book worked. So what we need to improve is the creation of candidates. I think we had lots of problems, especially loyalty problems, and how to identify that someone that can be popular and win an election is someone truthful. So we changed the way we work in this election. We’re focusing much more on people who are very loyal and that link themselves into the party, so people voting this person because this person belongs to the party.

Wolf: Because they’re in MBL, yeah.

Renan: Yes. So we are changing it. And then we go to detail. We prefer politics that works better to YouTube than people who works better on Instagram. You can rely more on people that creates content on YouTube than on Instagram.

Yeah.

Wolf: What is that? Is that because it’s just more intellectual or?

Renan: Yeah, because when you produce content on YouTube, and it’s a video of 20 minutes, 15 minutes, or a live stream—you create a proper bond with your followers. And you cannot discard them easily. And when you use Instagram or TikTok, you start with a set of followers, we say a bubble with a group of followers, and then you can betray your group and change to another like this. Also, the content is much more shallow—on Instagram.

Wolf: Necessarily, yeah.

Renan: Yes. So they start to use only tactics, very smart strategy to get following, but not properly converting into our set of ideas. So you cannot trust that much in people.

Wolf: Because you need them to educate the audience, not just to—get attention.

Renan: Exactly. Because we need to convert their voters into the party voters. Because what happened to us, a guy started our structure, then he say, “Oh, there’s the Bolsonaro voters. So why I don’t use these people as a stepping stone?” Not Missão party—but the MBL as a stepping stone, and after that, I jump to Bolsonarism.

And these people are doing it by Instagram and TikTok or Facebook. And the one who would rely on YouTube, they are part of a bigger community. And so people from YouTube are better, let’s say.

Now we go into the detail. So there’s this funnel of conversion—where Instagram and TikTok, they are the mouth of the funnel. And YouTube is in the middle. And I belong to a group, chat groups on WhatsApp, where discuss with your followers is the ending, and having personal meetings is like, okay, this is the core. So you should use all the social media as a funnel, a process of collecting these people, because people will have different layers of attention or belonging.

And people who follow you on Instagram, they like you, they can vote for you. But sometimes they don’t understand What’s your proper project? They sometimes just like your charisma. Want you to be there, understand your project. And we need to make this funnel works all the time.

All the time. So it’s a process. And a proper, big politician, and I am understanding this in this election, must be able to do all the process. So I—I get legitimacy and authority by writing parts of the yellow book, by organizing live stream, and now creating content on Instagram and TikTok. Or giving speeches in public or being on TV, and be able to fight against journalists, and they were very aggressive against me, so was like, I can be a total, let’s say, a total politician.

So the best ones must be good on everything. They must be leaders, they must be intellectually—sophisticated, and they must be popular. This is very hard, but that’s when you okay, you became the total politician. That’s what we try to create. The best ones must be like that.

The best ones are the guys who are able to have these characteristics. And it’s very cool when we identify sometimes, like a 17 years old boy or girl with those characteristics. We get really happy. There was a girl called, she’s a candidate, I hope she wins the election, called Ana Hering.

Wolf: Yeah, Ana.

Renan: She just left her job. She was working for General Waters. And she was a lawyer, said, “Now I want to live my life. I want to work with those boys,” because she used to watch us on live stream. She came here and said, “I want to work on anything. I can clean.”

“No, no. Let’s work on the live stream.” And she start to become a leadership. She start to create good videos. She writes pretty well.

We said, “Cool.” Ana, if Ana keeps working, studying, and become leader, she will be a very strong leadership. Amanda, who is the coordinator of my campaign, Amanda is not into being intellectual. She doesn’t like it. But she’s a great leader and a great content creator.

So she has two of the three aspects. Kim, he has the three of them. So when you start to identify, say, “Wow, this is good.” And they are the best ones. When the person have these three aspects inside of them, they are totally loyal.

They understand that it’s like a long-term project. They are the best ones, but they are so difficult to find. So difficult. They’re so difficult.

Wolf: So one final question before we wrap this up. I understand you are starting now to think about expanding even beyond Brazil. You were talking about Portugal, launching a branch of the party in Portugal. We’ve obviously been talking about America a little bit, if there’s any kind of traction there or parallel movement, things you want to help. I really do see this as a prototype of something that can happen almost across our—civilization, right?

Like Europe, America, South America, maybe even beyond. And I think now also out of the campaign period, you’re in this longer sort of groundwork period for the next round, right? And you’ll get your algorithmic virality back. But also it’s no longer—

Renan: Thank God

Future plans (1:28:49)

Wolf: it’s no longer just about the immediate, like elect the candidate, it’s about building the movement in a more comprehensive way. So I’m curious to just hear more about your future plans, kind of beyond Brazil and in Brazil, kind of strengthening the movement—thickening the movement.

Renan: Yeah. I was running elections with a party that had only 10 months of life. Which is like nothing. There was not structure on, even on institutional terms.

Like we didn’t have any money. We get all the students from our academy in all the states saying, “Look, almost all of you are candidates right now.” “What?” “Yeah, learn how to be candidate.” So we started training.

We didn’t rely on bringing already established politicians to our structure. So we said, “No, we must be pure, and we must win by being ourselves.” And it’s very good because people feel really much more motivated on building something from scratch. Everyone sees their own contribution in it. So, in the end of this election, I will see the size that we come out of the election, even though it was a meat grinder, especially for a newcomer as—us.

So, we’re going to see the size we have. And I need to create structure because for sure we’ll have more cabinets. For sure, we’ll have more money. For sure, people will start to identify us as a power source, so more money will flow. The press will treat us much more seriously, much more.

So I will be able to open up thousands of branches of the movement and the party, and I will structure more info products, more products, invest more on the entertainment plus cultural movement. So, “Valete” is the magazine we have. It’s also a publishing company. And we will start to organize a record company, a label—with artists. And as we have the artists and we have the audience, we organize the parties as well.

So we learn how to organize the parties using the branches.

Wolf: This is the other kind of party. The fun kind.

Renan: Yeah. The wee party. And everything comes together because now we know how not only to organize parties and conferences, but it’s also we organize brands of clothing. We sell lots of clothes from the party, from the movement, from the magazine. And we start to sell material from the bands, from the artists.

So this is an important thing that will create this, let’s say, this social glue that puts everything together. It’s a social glue that will put everything together. And Politically, I will keep trying to push myself into a natural national leadership, even though I don’t want to impose myself as a candidate for 2030. The group decide. If it’s me, it’s me.

If it’s not me, I will be on the back office helping. That’s what I’ve been doing for years, and I love to do it. Less pressure, but lots of work on my part, in the end. So, for me, it’s fine. And internationally, I want to be very helpful because I really believe we are doing something that is very special.

I don’t mean I’m special or Kim is special, but I mean that the circumstances in Brazil, with the fact that we somehow are talented, let’s say, made us build something very unique. I think the circumstances made us create it. So I think we can share it and organize it internationally, especially imagine that it must be generational. I think there’s a generational issue. It’s a problem for me here.

I think it’s a problem for you guys in America. It’s a very special problem for Europeans. A really special problem for them. And everywhere it’s a problem, and we must solve it. We must create strategies.

And the idea of the party, the idea of the social glue that connects everyone, the idea of the movement, these technologies of how to deal with the tri-functional structure, everything, how to fund yourself without relying on big guys with big money, because big guys with big money want to have big opinions over your projects. It’s always like that. And during the campaign, it happened to me. I was visiting the biggest businessman in Brazil, and they were full of ideas. I was like, “I don’t want your ideas. Thank you very much. This is my idea. We were working on it for years, and now you have your own idea.”

Because, I don’t know how is it in America, but here the elites, they are shallow, but they want to show off. So it’s very cool when they show off that they have ideas, they have institutes. I think it’s like that in America as well, for sure. So sometimes they want to steal what you’re doing and make it part of their own system of charisma, let’s say. And I don’t rely.

I don’t like to. So I always put myself out of it. So I need to rely on middle class and young people donating for it create our own products. Because when people with money start to put money on it, unless they are very cool people who really believe on the project, they want to have opinions, and sometimes the opinions they have, they are shit.

Wolf: Yeah, or at least, or contradictory, or they don’t fit into the whole that you guys have created.

Renan: And the best rich guys we found are the guys who were watching us on livestreams or watching what we are doing, saying, “Look, I will give you money. I really believe on what you guys are doing.” So we’re starting to have these guys. But in general, they are sons of rich people. Not the proper rich people.

The sons of the Brazilian elites, they all vote for me. There’s one thing, there’s the Brazilian MIT called ITA. Which was founded actually based on MIT, with MIT professors. They are the ones making the airplanes that we produce in Brazil, the Embraer, which pretty bizarrely, we have a very sophisticated airplane industry because of this ITA. And they made a simulation of election, and I won with 80% of the votes.

Wolf: That’s crazy.

Renan: So smart people and young, talented people, the young elites, they are ours already. So for us, it’s a matter of time.

Wolf: The next generation of elites is with you.

Renan: Exactly. So let’s not stress. Let’s be more Brazilian on this. Let’s relax. Let’s drink a beer.

Let’s have fun with the small victories on the way. Understanding that with time, future will be ours. And if it’s ours, it’s glorious. That’s it. And I understand that there’s this mindset, this winner-loser mindset in America that does not fit in this kind of project.

But if you want to have this project, you must change it. It’s a mindset that must change. And all the rest, come on, you’re Americans. You guys can do everything. You guys are very smart, hardworking, trustworthy people.

You can do everything. I’m sure you’ll find much more talented people than me to do it and advance it. I’m pretty sure.

Wolf: Amazing. The future is glorious.

Renan: Yes. I’m pretty sure of it.

Wolf: All right. Excellent. Thank you so much, Renan.

Renan: Thank you very much.

Last night, we saw what we came to Brazil for. We Ubered over to Globo News headquarters in Rio de Janeiro to see the Partido Missao protest against the decision to exclude Renan Santos from the climax Brazilian presidential candidates debate. Traffic locked up as we got within half a mile, so we got out and ran the rest of the way through on-and-off light rain. As we got closer, we could hear shouting, fireworks, and speeches. When we got there, the whole street was packed with about 2000 Brazilians waving yellow Missao flags, shouting slogans mocking Lula and Bolsonaro, cheering for Renan, and calling for policies like “Prendeu, matou” (”arrest and kill [the cartels]”).

It was surprisingly orderly. Traffic was still able to get through because the organizers had created a corridor through the middle of the crowd. As cars came streaming through, about a quarter of them displayed Missao political pamphlets and flags, or honked and shouted support, the crowd cheering every time. Speakers riled up the crowd to chant praise for the military police for keeping good order. Volunteers handed out hundreds of disposable rain ponchos, which were much appreciated.

There was a giant screen set up presumably to watch to debate. We heard on the way over that the supreme court had capitulated and would allow Renan to participate in the debate. But soon after that, we heard that Bolsonaro had pulled out, and then Globo had outright cancelled the debate. It was clear that they were scared of facing Renan head to head, despite his supposedly irrelevant “3%” in the polls. Lula is very old, Bolsonaro lacks substance, and Renan is a young charismatic firebrand intellectual, so it’s not hard to see why. Instead, Renan was coming here to address us.

While we waited, the screen stopped showing debate images, and showed Renan’s 24 hour livestream as he drove over with Amanda Vettorazzo. Speakers like his brother, various high level party members, and the local candidate for governor, got up on the box truck and gave fiery speeches that we were unfortunately unable to understand. They alternated between speeches and riling the crowd up singing and shouting slogans. It was great fun, but we were waiting for Renan.

Soon the screen showed Renan pointing the camera out the front window to show the crowd. He was here. The noise level increased immensely and all remaining traffic ground to a halt as his car pulled in front of the makeshift stage and the crowd poured in close.

As soon as he arrived, the rain completely dumped and the lightning started. It was like an omen of hardship and determination. He stood up on the edge of the box, 15 feet above the crowd, his suit jacket for the debate instantly soaked through, and paused dramatically in rain and firework smoke to take in the energy of the crowd. It was so thick we could barely see him even from 40 feet away.

He started with chants and slogans, riling up the crowd into a fever. “Prendeu!” he shouted. “Matou!” the crowd answered, faster and louder each time. Then he spoke for an hour in the pouring rain, punctuated by outbursts of chanting from the crowd. As far as we could tell, no one left despite the thorough soaking. It was like someone upstairs had sent the rain and lightning to make a dramatic point: Missao has the will to take hardship together. Someone called it “metaphysical”.

We got google translate working and pieced together elements of his speech. He told us the system and establishment politicians were cowards unwilling to face him, and that Missao was the only thing they feared. He told us the people of Brazil were survivors facing unacceptable conditions, and that they don’t have to live in the favelas and crime anymore, that they don’t have to be strip mined by foreign powers like China and the US, but can use the rare earths and resources and industrial capacity of Brazil to build an advanced technological society. I can’t possibly summarize the whole thing, but he finished saying that the world was watching, that Brazilian gen Z were the first to start breaking out from under the boomer vampireocracy with their own native politics, and would be an example to the rest of us around the world.

He finished with the Brazilian national anthem, making a point that the other parties were compromised by foreign power (he mentioned Bolsonaro’s connections with the American Rockbridge funding network and also Chinese influence), but Missao was for Brazil. They sang in the rain while he waved the brazillian flag. Commentators noted the complete absence of American and Israeli flags, presumably a fixture at other parties’ events (the only flags present were Missao and Brazil).

Then he tried to step down, but the crowd wouldn’t let him go. They shouted for more, and he riled them up further for another 15 minutes or so with fragments of speech and slogans. People hanging out of the window of the nearby apartment building attracted the attention of the crowd, and he waved to them while the crowd cheered. Renan took the opportunity to endorse the local Missao candidates, especially Coronel Busnello, who was some kind of retired military police sniper who he said had taken a bullet for Brazil and was now running for office. The whole thing was extremely positive, optimistic, friendly despite the determined and passionate aesthetic. We had a great time.

As it broke up, we walked away, saying goodbye to a few friends, but missed a chance to greet Renan himself. We made our way home and went to bed.

Then at 2am I checked my phone, and they had messaged me saying Renan saw me at the event and they wanted to hang out. (They are seemingly all Palladium readers and fans of our wider scene so they know a bunch of us). There are few people I would get out of bed at 2am for, but Renan is one of them.

I met them at a local bar as things were closing down, and got to see Renan and other friends in a quieter private session again. It was casual and quiet, with Renan asking us strategy questions, reminiscing about the maneuvers they had made to get back into the debate and how everyone but Missao had been discredited by the cancellation debacle. He emphasized that for a small third party like them, they have to play cleverly between left and right to punch above their weight and push their program through. Classic Missao 3d chess like the Meritocratic anime protagonists they admire. It seems to be working. We joked about how Renan is just the sensible centrist, but since they aren’t weak compromisers it’s more like “muscular centrism”. Then Renan had to go, and said we should do an interview when they get to Sao Paulo. So hopefully I’ll have that for you guys soon.

Today, much of the Missao discourse on twitter appears to be various people noticing and defecting to Renan, surprisingly especially from the left. They seem to notice and admire the youthful revolutionary energy and lack of compromise with the establishment right. I still don’t expect them to win this presidential election, but this whole thing only confirmed my impression from last time that eventually, at least for Brazil, Renan Santos is the coming man.

The Mission Party has risen by mobilizing a young, sharp, online demographic which was traditionally checked out of politics. Victor Antoun explains how.

Subscribe now

Transcript

Intro (00:00)

Wolf: Welcome back to The Students. So today is something a little different. We’re in Brazil. I’m here with some of the guys from Partido Missão, which is the Mission Party of Brazil. They are very interesting, young, new political party that I’ve become quite interested in.

And so I tracked down some of them here in Rio de Janeiro, and this is Victor Antoun, who is running for, what do you call it, state congress here?

Victor: Yeah. No, actually the national—

Wolf: The federal congress—in the state of Rio de Janeiro. All right. So Victor, welcome to the show.

Victor: Thank you so much for having me, Wolf. I’m big fan of your work on Palladium, right?

Wolf: Excellent.

Victor: It’s nice to be here. Very nice, actually.

Wolf: Yeah. Well, yeah, one of the reasons I’m interested in Partido Missão is they are Palladium readers.

Victor: Yeah. Many of us, really. Yeah.

Wolf: That’s very good. So, yeah. So I guess I got onto you guys, it was last year. We saw this wonderful article in J’accuse’s magazine called Meritocracy in Brazil. And it was like, “All right, this is interesting.”

Because when J’accuse says meritocracy, they mean something very interesting. So I thought, “Okay, that sounds cool,” and we read the article, and it described these young guys who are kind of doing all these antics to figure out how to fix Brazil, how to gain power within the corrupt system. And their charismatic leader, Renan Santos, and it all sounded very interesting. And so I said to my friend, “Well, we’ve got to get in touch with these guys. This sounds really interesting.”

And so we put out some feelers through our network, and it turned out Renan Santos and his friends were in town—in San Francisco. So they came to one of our parties, and they seemed like great guys, and they invited us down for the launch of the political party—which was last November. So in Brazil, it takes, what is it, 800,000 signatures to—

Victor: I think 500 something, yeah.

Wolf: Yes. 500,000 signatures—

Victor: 500,000

Wolf: to create a new political party. It’s actually quite an undertaking. So it was very impressive that these guys were able to do it, and they were launching that in November. It was a big rally. So I did decide to come down.

We went to São Paulo, and it was, as I said on Twitter at the time, it was the most glorious thing I’d ever seen.

Victor: Huge event for, like, 5,000 people, right? Yeah.

Wolf: Yeah. It was great. They had everyone up on the stage—trotting out the banners for all the—states. You guys had redesigned all the banners. You had some people beating on the drum.

Renan was waving around a sword on the stage. It was great.

Victor: Yeah. Soldiers go with... Yeah. It was a—

Wolf: It was, yeah, it was a lot of fun. So, and I said, “All right. I’ve got to come back next year for the election.” So here we are. That’s the background.

What is the Mission Party? (03:02)

Wolf: So Victor Antoun is running for office. And he’s one of the guys, and they’ve been running an impressive campaign here. So Victor, I’d love to hear from you. From your perspective in Brazil, what is Mission Party? Let’s start with that.

Victor: Yeah. Okay. Mission is a particularly new project, right? It doesn’t resemble anything that comes from the right in Brazil. Historically, the right in Brazil has been very physiological, very associated with local oligarchies and stuff, so it went through the military dictatorship without changing much.

It was very associated with the municipalities and stuff, so very no ideological in general.

Wolf: Non-ideological right. Yeah.

Victor: Yeah. It was ruling elites kept their power through a right-wing discourse, right? But it wasn’t truly right-wing, I mean, in the way it isn’t ideological. So we saw something very new because we have a program, ideology for the party. We are trying to do everything inside of it.

We have one leader. That’s... Yeah, that—

Wolf: We have the little yellow book here.

Victor: Yeah. Not so little anymore, but yeah. It was a huge research to try to diagnose what is wrong with Brazil, right? So we are starting with the whys. I don’t want to be cliché, but it was a nice entry to the political panorama in Brazil, right?

Through the book and through the launching of the party. So we are trying to keep it tech, young, smart, quick, and pragmatic also. So I’m very excited about the results that are coming in a week, right?

Wolf: Maybe. Right, it is just next week. Yeah, by the time this episode comes out, it probably will already have happened. We’ll see. We’ll see what happens.

But maybe a little bit on the history of the movement would be good. I understand it started, Renan Santos and his—brother kind of were involved in the protests against Dilma Rousseff back in the 2010s, and then the kind of core crew came together around that—the impeachment effort, and then maybe there was a dark period between then where they realized they needed to kind of develop that ideological self-construction. I’d love to hear it from you. I mean, you haven’t been around, I think, for that whole process, but—you maybe have a better inside story, right?

Victor: Sure. I talked with Renan a lot about the history of—MBL, right? I think it’s very important for us to situate ourselves in Brazilian history, so I try to do a historiography of MBL with him one day maybe. But it all started 10 years ago, maybe a little more, with the terrible presidency of Dilma Rousseff. It was Lula’s successor.

In a way, it was also deepening the populist sense, the socialist sense. So it was very problematic, but with the economic crisis that came by that time—manifestations and protests all over Brazil start to popping up, and Renan took control of a big group in São Paulo of protesters. And it all went by until Dilma fell. We got her impeached in 2016, I think. I’m not sure if I’m sure.

And then there was the onboarding on the Bolsonaro government. It was also very troublesome because since the beginning, Bolsonaro gave very bad signs about his intentions with Brazil and the way he wanted to rule things. He wasn’t particularly ideological also, so it soon became a problem. By that time, Jair Bolsonaro’s son, Flávio, who’s now running for president, was already going through many investigations. He was about to be arrested, and then Bolsonaro did anything he could to save his son.

It was maybe Bolsonaro’s first polemic. I’m not sure how to say this, but was the very first time Renan realized that Bolsonaro wasn’t the solution. So he departed from the government, almost got arrested, persecuted also. And then a few years later, he went back to the roots, popular roots of what got Dilma impeached and launched the Revista Valete, launched a YouTube channels. He tried to earn money and support to finally come through with the Missão party.

The institutional structure of the movement (07:58)

Victor: It was a—

Wolf: And this was the launch of Valete Magazine, the—the media operation basically. So as I understand it, Missão has three big parts now. There’s the original movement—which is this network of lots of energetic young people all across Brazil. Then there’s the Valete, the media empire, which kind of feeds on the movement and informs the movement. And now there’s Partido Missão—the actual political party that houses the politicians.

Victor: Yeah. Valete nowadays, it roots the movement in ideological stuff, or—the magazine, the channel, the cultural production that comes through it. But it is also very important financially. So it was essential from the beginning to start to have products, to have production, books. Valete nowadays is a big editor in Brazil also.

So that was a turning point. I got involved in Missão through Valete, right? I started out writing articles for their magazines. And now I’m running for Congress. So it wasn’t so simple, the transition there.

But, yes, those three spots and those three places to be inside our group, they feed on each other, right? Everyone puts a part in constructing a political party, I mean. In a way, Valete and MBL, they serve Partido Missão now. Yeah.

Wolf: Yeah, I want to talk about that kind of hierarchical relationship within the whole structure. As I understand it, Valete has kind of the geniuses in the back room cooking up the—core policy. Putting things into the little yellow book as sort of this is the ideology, this is how we’re going to approach the problems in Brazil. This is the party line, so to speak. And then that gets systematized through the discourse in articles, in streams—video, all kinds of stuff.

And it’s basically Renan, who’s in charge of everything, gets to kind of set the bounds on it and make the key decisions. And then that sort of ideologically disciplines the whole movement. Then you have the movement itself and its relationship to Renan and the politicians. I guess I don’t have as clear of a picture of that. I understand Renan is obviously the leader of the movement.

And then there’s the party. The party itself, I understand it’s kind of, as it was explained in the J’accuse article, or you can correct me if my interpretation of that was wrong, but it was basically that the politicians in Missão are not independent, sort of running with their own priorities. You’re using the office for the movement.

Victor: Yeah, sure. In Brazil, you can write your own, of course you can, but you can write your own internal laws for a party. But Missão is by far the most centralized, coherent one. So if I get elected into Congress and I don’t respect Partido Missão’s ideas in the, I can lose my office, right? I can lose my position.

And this operates in a whole sense. Everyone that is working around the MBL, Valete, and stuff is bound by our diagnostic of Brazil, right? So that’s very important. It’s a very important part of why I think the party is going places, I mean. Yeah.

Wolf: This is what makes it impressive, is that it’s this closed system, right? It’s vertically integrated. This is why I’m interested from the American perspective as well. It’s like, why do we care about what’s going on in Brazil? Brazil’s important, but it’s one of our hemispherical kind of neighbors—we have to get along, but I don’t think you guys are going to win this election.

I mean, look, not at the presidential level, let’s put it that way. But nonetheless, I think Missão is very important because you’re offering a new model—of what politics in our generation can look like, and this is something that I want our audience in the States to learn from. When a bunch of technocratic, ambitious, patriotic young people get together and create a highly effective political machine, what does that actually look like? And this is the best example I’ve seen of it, so I’m very curious.

Victor: Thank you for your kindness, Wolf. We try our best. But I think something that’s very important in Missão whole environment is everyone can bring something to it. In Valete, you have the more brainy types. In MBL, you have the more hands-on guys, doing everyday politics, shaking hands, and doing manifestations, protests.

And in Missão party, in my vision, the all-star league, the best ones—get elected, get into cabinets—start to work with the real thing. So—

Wolf: The rhetorical and strategic elite.

Victor: Yeah. So it’s very important to have everyone doing something. People nowadays, especially in our generation, they want to be part of something true, real, and they want to get stuff done. There are not many places where you can get real stuff done in Brazil nowadays. That’s one of the reasons of why I stopped being a lawyer.

I still have my qualifications to do it, but it’s not as engaging as having my own YouTube channel or running for Congress in Missão, right? Young people are eager to make history, be part of stuff. So—

Wolf: Yeah, I remember I saw this video clip of Renan saying basically, “You guys don’t just want to grill.” “You don’t just want to have a good life. You want to war.” If you look around—the world is screwed. You want to war.

And so, very inspiring. Renan’s great. But, maybe that’s a good place to jump off from to give us a little bit about the diagnosis, right? What is Missão’s diagnosis of Brazil? Why break with Bolsonaro?

Brazil needs to centralize (14:24)

Wolf: You mentioned the—some of the corruption stuff, but that fact that Missão is sort of on the right, but—schismed from Bolsonaro. From America, we look at Brazil and it’s like, “Oh yeah, we got the communists and we got the right-wing guy,” and like—that’s the choices. We have a very two-party mindset in America, right? But here’s Missão, this third party, kind of a very different approach. But I guess it all stems from your diagnosis, and the diagnosis colors the approach—as well.

So I’d love to hear you describe what’s wrong with Brazil.

Victor: Yeah. Brazil, it’s a third-world country, right? It’s not by coincidence. We have terrible political institutions, I would say first. The political comes before the economy, right?

My opinion, at least so. We went through the colony times, the empire times, the Republic, and our political class, political oligarchies, they never change. They don’t even have the intention to make Brazil something better. No way. They are very narrow-minded, localistic.

How do we say this in English? They are patrimonialistic also too much. So that is Brazil’s biggest problem. The parties, we have many of those. We have maybe 30.

I think there is 30 right now. But they don’t change too much if you go into the local level. There are two families buying votes, exchanging favors and stuff. That is the essential Brazilian problem—the basic one. Because of that, the whole country lacks ambition and drive to engage with real themes, not local themes.

That’s the first diagnosis of Missão, and now has a great project to try to solve this. That is Brazil’s greatest problem. But Renan has a great project. It’s called Lei de Responsabilidade Gerencial. He want to bring obligations into the funds that go into municipalities. That would be a great response, try to engage them into getting better education indexes, sanitation, security.

Wolf: So if I can restate that, basically, the problem is that Brazil does not have an ambitious national elite. There’s basically local elites and various corrupt—sort of patterns. And one of the solutions you’re talking about on that is to have priorities—kind of pushed out from the center to be—sort of competed over by the local provinces.

Victor: Perfectly. And also, we want to bring technical know-how into municipalities all around Brazil. Because imagine a 10,000 population city in the middle of Alagoas in Brazil. They don’t even have the educational technical means to administrate themselves. It’s a huge problem in Brazil.

Because of representation, we completely lack the drive to make things better—in many ways. So we want to make a ministry with young, bright people to try to get them together with those local mayors and get them to work better. And also bound them by obligations of increasing levels of basically everything. That is a way we could try to change the incentives in Brazilian political systems, right?

Wolf: Mm-hmm. And I understand this is sort of part of a general program of essentially centralizing the Brazilian state. The way I heard it explained was that Brazil has kind of not very much high human capital—people. And so the people you do have, you have to concentrate together—create one very competent government that’s then going to go and—and administer everything else.

Victor: Yeah. We estimate that in Brazil, there are many researchers on this, but a sufficient level of Portuguese and math are only like 10% of all Brazilians. Sufficient levels.

Wolf: Sufficient level of Portuguese.

Victor: Yeah. Of Portuguese and math—simultaneously. Yeah. So Brazil is able to do many great things. Embraer, Petrobras, there are many great Brazilian companies.

There are many great Brazilian cities, of course, but there is a big delta between the population. It’s very unequal. So we have this effort of trying to centralize and to better allocate human capital—throughout all the continent. In Northeast Brazil’s poorest region, it has the potential, many economical potentials, but it also has the worst elites in all of Brazil. So if we can get some right São Paulo guys right next to the mayor, bounding them to obligations of increasing social levels, maybe that could work.

Wolf: Right. Okay. So that’s one part of it, is this kind of problem of the localization, centralization, backwardness. You want to centralize Brazil effectively on a more technocratic ground. There’s also, I understand, a big crime problem—and with the favelas and the cartels and their connection even into politics.

I’d love to hear how you understand that part of the problem and what—Mission has to offer.

Fighting the cartels: Infiltration and open war (20:14)

Victor: Brazil’s constitution is from 1988, right? It was made right after the end of the military dictatorship. And also the international context by that time, it wasn’t still very socialist-friendly because the USSR hadn’t fallen, right? So it is very socialistic, and it has a lot of guarantees to every citizen. That in itself is not a problem, but it allowed crime in Brazil to get much bigger and much deadlier, right?

So by the ‘90s, Brazilian Comando Vermelho and PCC, they were small salesmen of marijuana and cocaine, right? Nowadays, they are international multibillion-dollar companies. They operate on many, many, many Brazilian sectors.

Wolf: These are the two—

Victor: Biggest ones

Wolf: big cartels. It’s the Red Commandos, and what was the other one?

Victor: PCC.

Wolf: PCC. What does it stand for?

Victor: Yeah. First Paulista Comando. No. Primeiro Comando da Capital. First Command of the Capital—in São Paulo.

One is São Paulo, the other one is Rio. We can talk about the differences—sociological differences between the two of those, because one is from Rio and one is from São Paulo. One is Paulista, one is Carioca, so they are very different in themselves. But they were small salesmen of drugs, and nowadays, they are huge—

Wolf: Business

Victor: business empires. And they control a big part of fuel in Brazil, gasoline—construction—cargo theft and cargo reselling of that, right?

Wolf: Right. Yeah, they hijack trucks.

Victor: Yeah, they hijack a lot of trucks in Rio and also in São Paulo. So they have become huge businessmen. They now are starting to infiltrate into the Brazilian government. Just the other month in Rio, nowadays we have five deputados estaduais now from the state, deputado from the state, congressmen—from the state arrested right now.

Wolf: For involvement in this.

Victor: Yeah, with Comando Vermelho. Right.

Wolf: Yeah. Well, it’s better than the Mexico situation, it sounds like—where the cartels are just killing—politicians.

Victor: Yeah. In Brazil, they are very effective in infiltrating the political class, and it’s a slippery slope in that sense, right? See, when you get the first one, the second is not so hard, the third one is not so hard. So we don’t have the legal means to combat that, I don’t think. With that constitution and that penal code, we don’t have any capability to fight them back.

So that’s why Partido Missão advocates to open war, formal war against cartels general.

Wolf: Yeah, I found that very interesting. I understand Renan basically wants to create the legal doctrine that because they’re running their own military and their own taxation systems—they are effectively foreign powers illegally occupying—Brazilian territory and should be treated as military targets.

Victor: Yeah. Just in Rio, I think one in seven Cariocas are living in controlled area by the tráfico. So it’s a lot of people. I have seen the metropolitan area of Rio, maybe two, three million people live in Comando Vermelho neighborhoods. So it’s bigger than many countries in the world, right?

And they enforce their own laws, their own customs, the music, the culture. They have everything of their own, right? They don’t allow the states to extract taxes or enforce anything, basically. So Renan, Partido Missão, for me, it’s very precise in the way they want to fight them, right? So there is an open war, right?

And intel also... There’s intel. We have to investigate better the infiltration they are doing, right? So we’re studying a lot of the anti-mafia laws in Italy. We have to bring some of those.

A lot of legislation from Singapore also is very interesting against infiltration and corruption in general. So those reforms have to come, but we also have to engage with the police, right? The police has to be bigger, better, better armed, more violent, if we want to have a chance. So it’s a—two parallel situations that we have to solve. I fear that Brazil in the next decade will be, if it isn’t already, but it will be the greatest world exporter of cocaine, because Trump is being very diligent about the Caribbean Sea nowadays.

He’s just striking people off and all. But, we also have routes that go downwards from the Andes into Brazil. And Brazil traditionally fuel the European market of cocaine. Right? But, nowadays, not only the Europeans are consuming much more, but also third world countries are developing a cocaine habit.

So Africans, Arabs, Chinese, Japanese, they buy more drugs nowadays. They want to keep buying more. So Brazil can be very strategic, not to just bypass the US and the Caribbean. You can send a ship straight from Brazil into Massachusetts or Connecticut or anything. And also we have the higher hand in exporting cocaine into the whole of the world.

So I fear how it can become more prosperous, the cocaine trade in Brazil.

Wolf: And that just empowers the cartels further.

Victor: Yeah. Greatly, because they do engage in many types of business, but there is nothing like shipping a ton of cocaine to the US. The profit margins, the way they earn it, would be too big, too great. They would have more power, more money, more influence also. So I fear that Brazil may become a narco state next decade or something.

Wolf: Yeah. Well, that’s what you guys are trying to fix—I guess.

Industrial development (26:42)

Victor: I mean, yeah.

Wolf: And I understand there’s also development priorities—that the mission has, as kind of the third big leg of the plan. Could you speak a little bit more to that in terms of industrialization and economic redevelopment?

Victor: Yeah. Brazil’s PT, the left party in Brazil, has had a socialistic rule since the 2000s, early 2000s. And Brazil’s first priorities has always been, since then, equality. So we don’t think of development or opportunity anymore. We are trying to mitigate poverty, mitigate misery—and all, but Brazil has become less relevant in the international grand scheme of things, particularly in the economy.

So Brazil used to have a large industrial base, in São Paulo, in Rio, Minas Gerais. We have some advantages like big iron ore deposits, have a lot of oil also. But we don’t seem to find a way to exploit that and turn that into prosperity, right? So, a big set of reform has to come to Brazil. So labor laws, environmental laws, indigenous people’s laws in Brazil—all of those, they try to get the country stagnant in favor of diversity or equality, right?

So traditionally Brazil has had very tough labor laws, very traditional. We call it CLT. It was inspired by fascistic legislation, the Carta del Lavoro—in Italy. So it’s very closed system of employing people in Brazil. We have to get rid of that—completely.

And also, decrease all legislation on other topics related to business development. And also Brazil has had a hard time keeping its accounts on check, right? So, it’s—

Wolf: Yeah, the financial problems—

Victor: Yeah, the finan—

Wolf: Many states have these sorts of problems.

Victor: Yeah. Yeah, it’s sometimes think it’s very related to democracy, right? Yeah.

Wolf: It’s hard to cut back the entitlements. And—

Victor: People mobilize politically a lot behind those. So we have to decrease social state. There is no other way. Some—

Wolf: Yeah. And what’s interesting, I guess, about your party, just the way you’re talking about these things, it’s very different from an American politician. American politicians, it’s like platitudes and—and very narrow, particular interest or something. But it’s, you’ve got a systematic approach to the problem, which is—again, the book, right? There’s this ideological, holistic approach to the whole problem.

Victor’s trajectory (29:53)

Wolf: So I find that very interesting. I guess it kind of comes down to just the way Renan has chosen to go at the problem. And, maybe it would be good now to turn more towards just that aspect, I guess. In particular, how you got involved. I think my understanding, you had your YouTube channel—and Renan basically approached you and said, “Hey, you guys should come on board. We’ve got this big program.”

But I would love for you to tell the story, kind of how did you get into politics at all? How did you get thinking about this stuff?

You’ve mentioned you used to be somewhat involved in the Bolsonaro people. Yeah, we’d love to hear your story, and how did you get pulled in, and what does that say about the nature of the project?

Victor: So, I graduated as a lawyer, and then I wasn’t happy with the—the ESG stuff, all the business environment in Brazil—how it was back in the day. And then I decided to start a business with my friend Eduardo Pessoa. It was a YouTube channel. I want to talk about history, philosophy, culture in general. So there wasn’t many channels in Brazil bringing this Nouveau Right way of talking about things—explaining things, so we did that.

And then I went into political commentary also. So—

Wolf: So you said Nouveau Right. Is that referring to the European New Right? Or, okay. Interesting.

Victor: The interesting stuff coming out of the US and Europe.

So I went into political commentary with some new ideas, some new theories that I read at Palladium, right? Many came through there. And then I have a habit of tweeting a lot. I like X a lot. So I was saying bullshit about the coup d’etat in South Korea.

It was two years ago. I was calling it the incel coup or something. It was a very funny story in South Korea. But Renan saw one of the tweets and he said, “That’s a nice take. We should talk more.”

And then we started talking on Twitter, on X, and eventually we became friends. I went to São Paulo to meet the guys. I had some political involvement with Bolsonaro’s party in Rio—but it was a terrible experience. It couldn’t be much worse. I was telling you, Wolf, people used to have prejudice against me because I had a superior degree, right?

They were like, “Oh, man, you’re a genius.” “How could you do this?” It was—

Wolf: The college degree. Yeah. It was too much for them. Which, yeah, that really says something about what politics is normally like in Brazil, I guess.

How Bolsonaro’s old right fumbled the ball (33:02)

Victor: Yeah. Like I was telling you, elites, they care about their own and the urban elites. Especially in Rio, they consider politics something dirty that they want to think about. They don’t care too much. So by consequence, politics in Brazil is very disengaged from people like me and people like Pessoa and all.

So, me somehow in a way had a great opportunity. There were a lot of young men willing to join politics, wanting to do something true. They had nowhere to go.

Wolf: Right. So how did that happen? Because it seems like Missão has attracted a lot of, maybe you have those urban elites or the urban upper middle class—and their children have gotten very interested in Missão, basically. And it’s the young men—from those demographics that are the backbone of Missão. And it’s very interesting to see that happening because in America and in Europe, we’re split between these ridiculous boomer parties, and we haven’t quite broken out into our own—political ambitions yet.

But it’s starting to get there. Especially, you go look in Britain, Restore Britain and so on. It’s very obvious that this—is a young people’s party. And there’s a few other examples like that where things are approaching, but it seems like things are most advanced along that kind of dimension in Brazil. And I’m curious how that happened.

What is it that—that—

Victor: Since the things I have seen in politics in Brazil propelled me to participate in. I saw Renan take down Dilma. I saw the rise of Bolsonaro. I have the frustration with Bolsonaro’s government. And then I saw Lula come back again.

And I was like—

Wolf: Yeah. Well, just the story, just for the American audience, the story of how Lula came back. I understand you mentioned that Senior Bolsonaro was willing to do anything to keep his son out of trouble. And I think the anything that he was willing to do—was let Lula out of prison. Is that right?

Victor: Yeah. Some Bolsonaristas at the time, they also thought that maybe with Lula coming back, Bolsonaro would become more popular. Stupid. It was said by Bolsonaro’s other son that is now living in the US. “So if Lula comes back, that’s great news for us. We have an enemy again.”

So it didn’t work out. Lula said he would go out of jail, run for presidency again, and win, and he did that. So it was very frustrating. We are in election time in Brazil again. I remember in 2018, 2016, people were ashamed to support PT.

If you called someone a petista, a supporter of PT, people would say, “No, not me. I’m not with Lula. I’m a leftist, but I’m not with Lula.” They would still press 13 on the ballot. But they wouldn’t say they were petistas.

In this election now, I see a lot of energy behind PT’s group. Again, of course, I live in Rio. I have an urban vision of the phenomena. There is a big support of PT amongst public servants and stuff. We have many of those in Rio, but people are proud to support Lula again.

That destroys me. We had the chance to maybe even outlaws PT in 2018, 2016. We could force them to make another party or try to break them up, and Bolsonaro wasn’t able to do anything. He exchanged his empire for his son. So that was very frustrating and then Lula won again.

I think there is a political wake up that happened between 2014, ‘16. I was 14 back then. And many of the guys that grew up with me also have this vision of this big frustration of everything could have gone great, but it didn’t. So Renan was there from the beginning. He was responsible for it back then.

He now resurfaced with a project. Why not? It sounded brilliant when I first saw it. The task of creating a new political party in Brazil seemed impossible by that time—because we have too many parties and the political class recognizes that, and they are trying to restrict. They are putting up legislation to restrict party creation in Brazil.

Last year, it was impossible maybe. But Renan could do it through tech, through talent, through meritocracy, and that was very stimulating. I’m running for Congress because I want to bring Renan’s campaign to Rio, right? In the end, I want to make sure that someone is doing it in a good level in my city, in my state. Okay.

So in the end, it always goes through that, right? So.

The need for leadership (38:31)

Wolf: Yeah. I guess I’m interested in how it feels on the day-to-day, maybe it’s these unknown unknowns of like—how the party is organized. It’s hard to ask a question. The question most directly would be like, what should the American audience, what would be most interesting from the success of Missão so far to the young American audience of the people who feel the same way in America—how should we kind of get started, right? And I’d—it’s hard because you maybe don’t see things on the ground in America.

We don’t see things on the ground in Brazil, so it’s hard to compare. But if you could venture to try to answer that question.

Victor: What was very important in Missão creation was the frustration with Bolsonaro. Maybe you have some of that with Trump nowadays.

Wolf: Yeah. I think in many ways Trump is the Bolsonaro of America. Yeah. He’s came in with much promise. Many people were enthusiastic, but obviously a lot of people also really didn’t like it, but, it’s done I think some good things and some bad things, and part of one of the things that’s been quite bad is the sort of level of corruption and—ineffectiveness, right?

And you see a lot of insider trading. You see all this sort of stuff. So people are frustrated. There’s certainly a faction of young people who are frustrated with Trump. But—some of that is this very, I would say, counterproductive stupidity.

They—just throw up their hands. They’re almost giving up at politics, sabotaging their allies. But it seems like you guys have done something much more productive, which is, okay—we’re not satisfied with that. Here’s something that’s even more systematic, more holistic. How?

How do we do this?

Victor: Yeah, maybe it starts with the frustration. You have a better vision of how to do it better, what’s wrong, what you should do different. And secondly, I should say you should have a leader.

Wolf: Yes. We need a leader. Yeah.

Victor: Yeah. It’s very stupid to say you have to find a leader because a leader appears. It’s—

Wolf: Yeah. Leaders are sent by God. You can’t just appoint them

Victor: you should look better for a leader, right? He’s there somewhere. The US is much bigger than Brazil in population, but, you should try to make something true and interesting. Be pragmatic. Try to do a full scope diagnosis of what’s wrong with the US.

Systematic analysis is the foundation of the new party (40:49)

Victor: That’s obvious for most of the people. Violence in Brazil—who could disagree, right? The way we are losing economic prominence—also obvious.

Wolf: No, but it still takes a lot of analysis to kind of get to the bottom of it, right?

Victor: Surely. To find a solution and to—better describe the problem, it’s very hard, actually. The Yellow Book had a team of like 2,000, 3,000 researchers voluntarily.

Wolf: 2,000 or 3,000 people were—Wow.

Victor: It took a lot of work. Not to find the problems, because the problems themselves are self-evident, mostly. Brazil is a hard place to live sometimes. But the how to understand and how to propose something better, that was hard. Maybe do that.

Wolf: Well, okay, that’s something to drill in on because, with The Students, maybe that’s part of our ambition, right? Is to give this—

Victor: Surely

Wolf: coherent view of what is our situation historically. And that’s something I’d love to put together. Here’s the picture, right? What’s wrong? What do we need to do?

That’s certainly the problem that we’re studying in a way. And so I’d love to hear more then about how that happened. How did The Yellow Book come together? How did that project get coordinated? Was it people writing articles or was it like teams kind of—researching a particular issue?

Or how did that actually... How do you put together a little yellow book?

Victor: We started it up with a magazine. Right. And—

Wolf: Started with a magazine.

Victor: That’s—

Wolf: Well, we’ve got a magazine, too.

Victor: Check. That one is done, right? So we started to assemble the best ideas. We did many groups of study in particular groups and also public police. A lot of police people in Brazil, public safety specialists, do a study group, understand what’s wrong, and go through with it.

I like to keep it voluntary, but coordinated by professional people, so have a lot of insights—have a diverse vision of the problem. I like to bring people who operate day to day on the area also. That’s very important to—the struggle of a policeman in Rio de Janeiro, that’s not obvious. That’s actually very hard. So try to bring those people together with the supervision of a paid specialist or a paid writer, academic preferably, and go theme by theme, man.

There’s no secret.

Wolf: And the academic sort of has that, or the specialist has that role of almost being the, not great to put it in these terms, but the sort of the political officer who integrates all the information—into a coherent vision.

Victor: Yeah. I think it’s very important to present yourself in a way we are very studious, coherent, good-intended, and pragmatic. We just want to make things better. What’s wrong with that, right? What makes things—

Wolf: Some people want to make things worse.

Victor: Yeah, sure. Many of them, but I think that’s an interesting way to present yourself. Society, not necessarily falling to, “I’m on the right,” “I’m on the left.” I want to get things done. That’s very important about Renan’s rhetoric.

That’s I want to make—Between Lula and Bolsonaro, who’s talking about Brazil in 30 years? In 40 years? 50 years, right? No one. Both of them will be dead, actually, right?

So...

Wolf: But you guys are all young. You are actually having to think about that.

Victor: I hope to have more 50 years—here in Brazil, working for the cause, right? So...

Wolf: All right. Do you have any final thoughts, things that you think that we haven’t covered? I’ve gone through all the obvious questions here.

Victor: No, we talk about a lot of interesting stuff. I would like to apologize if I’m too tired. Running a campaign is actually very hard.

Wolf: No, you guys are working crazy hard. It’s been hard to get time with you.

Victor: No, so I’m very tired. We are by the end of the campaign, but it was nice talking to you, Wolf. Thanks for having me, man.

Wolf: Thank you so much for coming on.

Ben Landau-Taylor argues that techno-singularitarian predictions are unfounded, while Wolf Tivy defends his xenohumanism program. They cover why theology is inevitable, and why the dominant theology keeps predicting techno-apocalypses.

Subscribe now

Ben on Twitter: https://x.com/benlandautaylor
Wolf on Twitter: https://x.com/wolftivy

Transcript

Intro (00:00)

Wolf: So Ben, we’ve had a bit of an obsession with AI on the channel a little bit, and more broadly in the discourse these days in San Francisco, everyone’s talking about AI. AI this, AI that, AI theology, AI doomerism, AI optimism, all kinds of things going on like that. Your writing over the past years has been very cautionary on getting over-hyped in AI, and in particular, you’ve written about how people project onto this apocalyptic view on, and this is just the latest incarnation of the apocalypse, which we’ve seen many times, or that we don’t actually know anything about the future of AI or the future of what happens after AI, the real sort of AI that might come about. So you’ve been kind of a voice of caution, and I wanted to have that conversation then between us on is AI all it’s cracked up to be? Is it actually going to kill everyone?

Can we know anything about what’s happening after AI or with AI?

Why don’t you give us your take on what you think is actually warranted in—in AI prognostication and what is out over our skis? Important stuff.

The case against imminent machine god (01:45)

Ben: I do think there are a lot of people who are getting a lot further with it than is justified in patterns that we have seen several times in the past on a bunch of subjects, including previous incarnations of AI. Also, I think it’s a very important technology, probably the most important technology that has been developed in my lifetime. Certainly top two. Quite plausibly top one. So, it’s a very big deal, and when people are like, “Oh, this is an important new technology and understanding it is very important to understanding the world and what’s happening.”

100% there with it. And then there’s this further thing that a lot of people do where they’re like, “This is ending the age of man. This is going to seize the light cone and determine everything that happens in the galaxy until the heat death of the universe.”

Sometimes people call it AGI, that term started getting walked back. People now call it super intelligence or there’s a bunch of different terms, but the infinitely powerful mind that can do everything—and sort of just wipes humanity out of the water. If humanity persists, it’s kept around as pets of this... There’s this idea that people have of this god mind artificial intelligence—which just totally changes everything, and that people frequently just sort of assume is happening quite soon. And I think that’s—

Wolf: Yeah, people talk about short timelines, like—”Oh, my AI timeline is three years from now,” and—which they mean we’re all going to be dead three years from now.

Ben: Yeah, I was hearing the three-year timelines five and six years ago, and—

Wolf: Right. Yeah, I remember.

Ben: Yeah. And now, a lot of the same people still have three or five-year timelines and good often seem... Anyway. So this is a separate question from the, is this thing important now and economically important for political power? Yes, it is.

And then there’s the separate question of, are we going to get this god mind that ends the age of man and just takes over everything? And I keep asking people who believe this for what’s the specific reason why you think that this is going to happen soon? They’ll come up with this sort of, here’s an argument that this type of mind is philosophically possible. Here’s why these things aren’t unique to the human mind. Here’s why—all of these things could be done with the source of software in principle, and there’s no reason to think that what is done in the human mind is the pinnacle.

There’s certainly a bunch of individual areas where the software is, and has been doing better than the human stuff. Since before I was born, it was better than any human at multiplying things quickly. And the list of things at which it is better than any human has just expanded and expanded and expanded. And so clearly it can be better than humans in some ways. Will it be better than humans in all ways?

And people think that this is coming quite soon, and when I ask them why, they don’t really have any sort of mechanistic explanation. The closest I can get is that it’s currently moving quite fast—in that direction. And like—

Wolf: It is, in fact, moving quite fast.

Ben: In some sense, yeah. So then I’m like, “Okay, well, if you’re looking at whether something is going fast, you have to be like—

Wolf: But how far?

Ben: relative to the dis...” Yes, exactly. How far? It’s like we’re moving at 110 miles an hour. Is that fast? If you’re going between San Francisco and Los Angeles, it’s pretty fast. If you’re going from the Earth to Venus, it’s extremely slow. And I’m like, “Okay, so you see that there’s this certain pace of advance.

Why do you think that this is large in terms relative to the distance that we have to go?” And I’ve just never been able to get a clear answer. I have had quite long conversations with a bunch of the sort of inner circle AI doom people. I have had shorter conversations with many, many of them up to and including literal Eliezer Yudkowsky who is kind of more open than a lot of people are about this stuff, that ultimately it just kind of comes down to vibes. And like, or deferring to the vibes of various people who are purported to have more reliable vibes on a subject like this.

Wolf: Right. So there’s a lot there of different questions in the space, of very different types. Implicitly, there’s other questions behind those questions. I want to get into all that. Why don’t we just continue down on this thread, though?

The case for imminent machine god (06:38)

Wolf: Why do we think AI is coming soon? That seems like a good place to start. So I’ll just give a bit of the devil’s advocate here. Now, for my own view, I think I put a lot more stock into at least the question of the god mind type stuff than it seems you do. That was not raised in what you were saying, but I think there’s also this, you have a sense that maybe we can’t even tell.

We don’t know what the far future is going to look like. Some people make these very confident predictions of god minds of various shapes and sizes. I think we can make some projections there, but not the particular projections that some of these guys do make. But more practically, we have this sense that AI is moving fast and that we feel like we’re close. So let me try to ground those intuitions and see how you respond to that, because I think that would be interesting.

Since the deep learning revolution, we’ve kind of delivered on a very large fraction of what the original AI program was. Back in the ‘50s, when they did the original conference, they basically said, “Yeah, we’ll get some grad students, and over the course of the summer, we’ll solve vision, we’ll solve locomotion, we’ll solve speech maybe at some point.” And like—

Ben: Yeah. They’re still working on locomotion, but they’re making progress.

Wolf: Yeah. But like—

Ben: And they’ve got vision, they’ve got speech. It’s notably getting there

Wolf: But even locomotion. Have you seen the Beijing Robot Olympics? This thing that’s been happening. It’s super exciting. I mean, I love—

Ben: Very cool stuff.

Wolf: I love seeing these super fast bipedal robots, running with these ridiculous gaits and then crashing and literally bursting into flames.

Ben: Right. And it seems like the sort of thing where you just take normal engineering process, give it another single digit years, and it will just be able to walk around a house.

Wolf: Yeah. Which it can already do, and they can go up and down stairs. And we’ve seen pretty good progress on that front. Now, I don’t think we actually have the best possible algorithms there. We’re still doing this kind of deeply pre-trained reinforcement learning type stuff on that.

But I’d say, okay, they’re doing pretty well on that. In particular, they had robots that were playing cooperative tennis with humans. That was pretty cool. Okay, the robot running down the track at high speed, great. That can be pretty pre-baked.

It’s a pretty controlled environment, but playing tennis, that’s impressive.

Ben: Yeah. The dealing with chaos and things, and environments that aren’t perfectly set up has been the hard part that they have been slowly grinding through and improving on.

Wolf: Yeah. So basically, in the last 15 years, let’s say it’s 15 years, right? 2012 or around then is when people start using GPUs and deep learning as the paradigm, right? And that paradigm has, after basically 50 or 60 years of some progress but nothing like that, we finally started really cracking those core problems that the original AI conference thought would be interesting and difficult and valuable. And so it seems like we’ve taken a big chunk out of the problem in the last 15 years.

And if we sort of imagine, okay, projecting that forward, well, let’s say there’s maybe about that much left in the tank, and we’re moving at that pace or an accelerating pace or something. That’s a basis for relatively short timelines to the complete picture of AI, where you have AI that actually covers all the capacities of man fully.

Ben: So that’s the part where it just seems like you’re just making a leap.

Wolf: Yeah. So I am making a leap, right. But this is sort of the shape of the argument, right? I’d like to hear—I guess, yeah, how would you pick that apart?

Ben: Yes. I think it’s very straightforward. There’s a premise in the argument where you’re like, okay, so we’ve done this amount, and—then there’s this amount left, and I’m like, why do you think there’s this amount left? Why not this much? Why not that much?

Where is that specifically coming from?

Wolf: Yeah. I guess, okay, so in this case, the why I said, “Oh, we’ve gone this far. Maybe there’s about that amount left,” is because it’s like we tried to project out how hard the problem was, and it basically consisted of a bunch of these things. And it’s often a good bet that the sort of magnitude scale of the problem is related to what you can see of it. And now, obviously, it can have radically large uncertainty on that.

Sometimes it’s much larger than what you can see. Sometimes it’s like what you can see and then a little bit. And or if you look at sort of the Lindy effect in how long things last, historically or otherwise. If it’s lasted this long, often on expectation kind of lasts about that long again in the future, and then that’s its sort of end of life. So it’s just this, I don’t know, kind of general statistical hunch or heuristic that the size of things is about as big as they look.

“AI winter” and the difficulty of predicting technology (12:33)

Ben: Yeah, and this is a heuristic which just performs abysmally in the history of technological development—and specifically in the history of AI development. There is this thing that has happened several times, depending on how zoomed in or zoomed out you get, you can sort of argue about the number of times it’s happened, where there’s this, a bunch of people get really excited about, oh, the scientists are like, “We have got this much space left between where we are now and getting the computer that does everything a human can do.” And we can see the shape of it, we can see how much progress we’ve made, and a bunch of the people who are sort of inside working on it get really excited. And this ends up driving them forward on their research program. They do end up, in fact, then just making a bunch of notable advances and making better software in ways that are quite important.

And then they kind of like, “Well, we’ve got a bunch of new cool capabilities, and it is not thinking like a man. It is not doing everything.” There’s still a ton of really big holes, and we’ve sort of built a bunch of the individual pieces, but it doesn’t really get us any closer to the sort of fully autonomous. And so there was the first one of these, I think that was sort of a really big, was in the ‘70s. And there was this, I don’t know if you’ve ever read “The Lighthill Report” was sort of one of this document sort of throwing cold water on a lot of the people who were like, “Oh, these guys were really excited about they were going to make the human-level machine by...”

I think it was the ‘90s they were saying, and like—

Wolf: This was in the context of expert systems or something like that?

Ben: I forget which one it was specifically. And this was what is now known as the first AI winter. And we have this term, which is older than you and I—for the time when the AI experts, based on their sort of intuitive sense as the leaders in the field, get their vibes about how far is left to go and then kind of realize, “Oh.”

Wolf: They get out over their skis, and then they realize they’re out over their skis, or someone makes them realize, the whole thing cools off for a while, and then that’s an AI winter.

Ben: Right. And before each one of these, they do, on the basis of this enthusiasm, just make a bunch of very important progress on the software. So when we look at that heuristic of like, oh, people have a sense of it’s about this much, it’s about this much. We just know that the people who are at the forefront of the field very frequently have had this sense and just been wrong.

Wolf: Yeah. And so you’ve pointed this out, not just in AI, but in general, that technology in particular is extremely difficult to forecast in that way.

Ben: Yeah. What if—

Wolf: Life is like extremely kind of long-tailed process.

Ben: Yeah. So a couple of historical examples I’d like to bring up here. One is Leonardo da Vinci. Where he has these sketchbooks where he’s sort of spitballing about a bunch of possible types of technology and types of machines that—he’s—

Wolf: He’s got the picture of the guy turning the crank on the helicopter.

Ben: Right. And then—

Wolf: And then it’s got literally a screw to propel it. Yeah

Ben: Right. And so he’s got something, he’s got an intuition that is sort of remarkably far in the right direction of—the mechanics that could power a helicopter. It doesn’t exactly work, but he’s clearly onto something real.

Wolf: You don’t actually have a guy turning a crank.

Ben: Right. And you don’t literally have a screw. It uses much more complicated aerodynamic principles, but the basic idea is sort of in the area. And you could be like, “Oh, wow. Maybe he’s gone 10% of the distance, so it’ll take 10 times as long.”

No, it takes another half a millennium, basically. Another thing that I think about here a bunch of the time is the development of the steam engine. Where if you look through the sort of different iterations of the steam engine, you’ve got the Savery engine and the Newcomen engine, and then eventually the Watt engine. Some of the earlier ones have sort of these very niche uses in just pumping water out of mines, but they’re very inefficient and no one has figured out how to turn them into rotary circular power, which ends up being the separate condenser, which makes it much more efficient and the ability to turn it into turning a wheel ends up being the things that make the Watt engine actually able to kick off the Industrial Revolution. If I imagine myself, it’s like 1698, looking at the engines of the time.

If I imagine myself looking at 1720, looking at the engines of the time and just thinking about how do you use these techniques that people are looking at here to project forward? I do think you could use these and be like, okay, wow, if you’re able to harness the power of heat in a way that can substitute for the things we’re doing with muscle power and horses and water, I think you can correctly reason that that’s going to be a big deal, but if you’re just looking at the details of the technology in 1695 or 1720, I just don’t know how you tell from that position, oh, it’s going to be in 1776 that we get the engine that actually really kicks it off.

Wolf: Yeah. I guess the sort of fundamental principle is very visible at the time, which is, okay, well, you put coal in one end, you get motive power out the other end of this engine that has been produced. Therefore, you can predict fairly reliably that there is this possibility of the Industrial Revolution—where you replace the muscle power in society with fossil power.

Ben: Right. And people have been locked on to this idea with the software that like, “Oh, wow, as the software gets better and better and can do more things, that’s going to be a big source of power.” They’ve—been locked onto that for not quite a century, and I think that in the main, this intuition has proved right.

The cognitive revolution (18:38)

Wolf: Yeah. And if we extrapolate out what’s the equivalent of the Industrial Revolution here is, well, you can get electricity to do cognitive tasks. And so we expect something like a cognitive revolution where you build the right kind of machine, the right kind of engine. We’ve demonstrated the principle that, especially with any particular kind of task that can be measured, you can basically get machines to do it, and do it very well.

Ben: Well, not any task, but certainly a lot of tasks, and their list is expanding.

Wolf: Well, on any task that can be measured, we seem to be getting progress. And so no, certainly not the case that we have solved all tasks. But in principle, it looks like you can take a task that can be well measured and then apply a huge amount of computing power to find a function that does that task measurably better.

So if we just extrapolate that out as a principle like the principle of the Industrial Revolution, then you get something like, well, there is the possibility of a cognitive revolution like this. Now, to your point, the timing of it all is much harder to predict. Back in the ‘50s when they were projecting it out, like I mentioned, they said, “Oh, let’s get a bunch of grad students and we’ll solve this over the course of a summer or two.” And of course, it didn’t turn out like that. It turns out there’s a lot more difficulty to the problem.

There’s been a lot more mathematics results that have been needed to reason about these things. The particular paradigms that we use to do it were not necessarily discovered at the time. The computational power wasn’t there, the industrial base wasn’t there for it. The applications weren’t quite there to make it sufficiently useful for it to take off economically. The data wasn’t there to get the more interesting kinds of talky power that they now have.

It’s based on language, enormous amounts of data. So there’s a bunch of ways that they just didn’t have the prerequisites, and they kind of couldn’t tell. And likewise, if you think back about the Industrial Revolution, well, you’ve guys like Hero of Alexandria had steam power.

Ben: Yeah, that is—

Wolf: It’s different

Ben: people overstate what his little spinning kettle—people talk as though it was much closer to an actual engine than is often the case, but that’s a tangent.

Wolf: It is a tangent, but point being suppose someone... I think some people have claimed that it’s possible that, say, the Romans used something like early steam engines to pump out some of their mines or something like that. And suppose that was real. Could you then project out the like, “Oh, yeah, we’re definitely going to have an industrial revolution in the next few hundred years?” Well, obviously no, they couldn’t.

Ben: Right. And in fact, it was closer to two millennia.

Wolf: Right. And now that’s of course because of the collapse of their civilization, which maybe that happens to us too, right? Maybe we collapse before we get the cognitive revolution or—

Ben: If we are as far from it as the Roman steam prototypes were from the Industrial Revolution, then—

Wolf: It’s very probable

Ben: then that would be the probable ways for things to go. Are we that far? Then...

Wolf: Yeah. And this is the thing, it’s very hard to tell how far away you actually are. The other example people bring up obviously is right before flight or even at the time that the Wright brothers were flying their airplane, you get people publishing, “Man will never fly in another million years. It’s just impossible. It’s too hard. It’s—

Ben: Space flight, you see the similar stuff

Wolf: Yeah. And of course, a lot—

Ben: You know

Wolf: of that is just like, okay, there’s always some overconfident jerk saying something’s impossible or possible or whatever. But you couldn’t really tell at the time whether they were—going to solve the problem.

Ben: Yeah. From this—

Wolf: Because it in fact depended on a bunch of key technologies like their development of propeller technology, their development of the three degrees of freedom balance of how to actually control dynamic flight, the development of the wings, development of light enough structure that you could actually carry a man and an engine and all this stuff. The development of engines that were good enough to propel it. None of that would’ve worked without a bunch of those key innovations. And you don’t know going into it that you’re going to need those.

The parade of apocalypse prophecies (23:17)

Ben: Right. You don’t, yeah. So it’s very hard to tell how far away it is. And then there’s another thing here which I think is very important to bring in. Which is not on, it’s separate from the question of, okay, in reality, how far away is it?

But it’s the way that people tend to evaluate this. Where we’ve seen in many, many areas for quite a long time, there will be some guys, some very sharp intellectuals, will look at something that in principle could end up destroying the human race. And they’ll be like, “Oh, if you extrapolate out, we are using all of the natural resources at an exponentially increasing rate.” And in principle, if you run out of those, if you run out of the farmland or the chromium or—the aluminum. In “The Limits to Growth” book, they did these sort of plot the exponentially increasing amount of this, make some assumption about how much exists in the world and be like, “So clearly we’re going to run out, and the Earth is going to fall apart and most people will die, and then we’ll end up at this more sustainable, much lower population.”

But like you see, so they’ve got a correct argument that, in principle, there is, in fact, a finite amount of matter on the Earth, in the solar system, in the universe, whatever. There’s a sense in which that is accurate. And then from there, they will make this totally, absolutely wrong by many orders of magnitude projection of when that is going to destroy humanity. In the limits to growth case, they were projecting mass famines in the late ‘70s, early ‘80s.

Wolf: I think some of their models actually had the breakdown around now. But yeah, they were making these projections and in particular, they can be vastly wrong by orders of magnitude about the relative size and importance of the various assumptions. And then this means that it can put you in a totally different equilibrium.

Ben: Right. And there’s a persistent, especially since the First World War, you start seeing the first one, which is people being worried that mass air bombing—terror bombing is going to destroy civilization. And they were right that mass air bombing is just possible with the technology that has been achieved between the World Wars and is a big deal. They were vindicated on that. It was not a good time to be in Dresden or Tokyo.

But they were talking about, literally a lot of these people in the air forces of a bunch of the Atlanticist nations in the British Parliament were saying that this will literally destroy civilization. You can find all sorts of—intellectuals, at the time, who think it is going to end civilization. And in fact, we had it. A lot of people died, it sucked, and civilization just sort of kept on trucking. And—

Wolf: And another, I mean, more less hysterical predictions that they made about it were also wrong. They believed that if you terror bombed a city, it would lead to a breakdown of social order there. It would lead to mass panic. The authorities would lose control. And it would break the morale of the enemy.

And in fact, none of that was true.

Ben: Right. If anything—

Wolf: When you bomb people—

Ben: terror bombing seems to have hardened the resolves of the people getting bombed, which—I don’t know how, I don’t think anyone called that correctly. I don’t see how you could. That’s just kind of sociologically speaking, a very weird surprise.

Wolf: Yeah. I mean, it’s not necessarily that surprising in retrospect, but yeah, I don’t know how—

Ben: Yeah. Very few things are surprising when you already know what happened.

Wolf: No, but it’s like, okay, people become determined against exac—an evil enemy, right? But okay, so it’s very difficult to project out these sorts of things. But this is sort of an abstract argument. And I wonder if we can make more concrete projections.

Predicting incremental technology progress (27:19)

Wolf: What in particular does the world look like where we don’t develop, I don’t know, vastly more advanced AI or it doesn’t, I don’t know, break through to being superintelligent or something. What do we think is actually... I guess, to some extent, we can’t really make the prediction. It’s just you’re saying that people are being overconfident—in whatever their prediction is

Ben: I do think there’s some degree—

Wolf: But I wonder, I—

Ben: you can predict. Especially when you come to the current near-term technology. So in general, when I think about technological advances, I separate things into two categories. There is breakthrough technological advances where you have figured out transformer architectures. You have invented the solar panel.

You have invented—fixed wing flight. And there is incremental progress where you have made the engine on your airplane more efficient. You have made your solar panel cheaper. You have made your battery able to store more power. And I think that people can think pretty well, not perfectly, about the incremental improvements.

And then the breakthroughs are the things that are just tend to come in from out of left field in a way that—

Wolf: Is very hard to predict. Yeah.

Ben: And so you see very frequently with the incremental advances, not always, but not infrequently, they end up following mostly a sort of fairly regular advanced pattern. If you look at battery efficiency over the last several decades, it looks like a reasonably smooth curve. Moore’s law is very famous. Not everything works like this, but a lot of things do.

Wolf: Mm-hmm. Yeah. This is sort of advancing under some sort of scaling law—or as we say in AI.

Ben: Yeah. And people often model the adoption of new technologies with a sort of sigmoid curve of it starts slowly, ramps up more quickly, and then sort of saturates and levels off. And if you assume, okay, we’ve got a new, big, important technology, it’s likely going to follow some sort of sigmoid thing like that of—and in terms of a sort of general adoption thing, it looks like to me, the current AI has meant many things at different times. The current sort of LLM-based, transformer-based, the current incarnation of what AI means seems like we are in the steep part of the sigmoid, basically. Where there’s still a huge amount left of just applying this to the world.

Wolf: Yeah. And like—

Ben: In the sense of—

Wolf: Finding new techniques incrementally to make it more efficient in the training data.

Ben: Right. From the perspective of—

Wolf: Make it scale more compute efficient.

Ben: Right. If you compare it to the adoption of the internet, we’re like maybe in the equivalent of the late ‘90s or something. Where the basic idea of what can this do, assuming no further breakthroughs, there will be more breakthroughs at some point in the future, could be very soon, could be very distant, but assuming no further breakthroughs, you can sort of get a general sense of what it can do. And people are figuring out how to apply that. That then takes often billions of dollars of work.

But if you look at what the people were thinking about where the internet is going to go in the late ‘90s, a lot of them were wrong, but most of the big things were predicted by someone, and there were some people who were just reliably having a reasonably good sense of, here’s what it’s going to end up doing. So I do think that for the current stuff, with no further breakthroughs, I think that that is possible. And then that’s very different from the sort of like... And then we imagine that just turning up the dials on the current stuff that we have is going to turn into the sort of like god mind that’s superior to humans in all ways, or a lot of people I will talk to will be like, “Yeah, I think we’re one or two breakthroughs away from that.” And I’m like, “Where are you getting that?”

And—but bracketing all of that stuff and just looking at the current stuff, I do think you can be like, “Oh yes, here’s what it’s going to do.” And like—you know?

Wolf: And if we make that projection, it looks to me like, okay, yeah, we are sort of, I don’t know about in the middle of the curve, but certainly on the steep part of the curve. It’s always hard to tell—how far the curve goes.

Ben: I would unconfidently guess we’re towards the beginning of the steep part, but like—

Wolf: Right. Who knows?

Ben: this is not an area where I think I can make—

Wolf: Who knows?

Ben: very strong, solid projections.

Is incremental progress enough to reach the singularity? (32:05)

Wolf: But basically, if we incrementally keep advancing along the current trajectory, you do get something that’s quite revolutionary—even to the point of being these automated employees that can do all sorts of things, maybe almost fully automated companies, automated research. They can do quite a lot, but there seems to be a few things that are very sort of pesky, kind of missing, right? And that’s this something about the way they kind of... Well, currently, the current systems don’t redevelop their knowledge over time, right? They don’t have internal state.

They’re a stateless kind of function. And the other thing they don’t do is they don’t seem to have this, I don’t know, strategic judgment of a kind, that they seem to always kind of fall off on that one. And maybe that’s something that’ll be fixed, maybe not. But like, these might be—

Ben: People talk about agency. It’s kind of a fuzzy term, but it’s gesturing at the thing that they still don’t really seem to have—in the way that a person or even a cat has.

Wolf: Yeah. And it’s been surprising how far they have gotten. Or surprising from a skeptical perspective. Three years ago, or two years ago, it was hard to say, like, “Oh yeah, these things will definitely end up... Agency will definitely end up being working,” right?

Like, AI agents will definitely work. Everyone knew you can put it in a loop, right, with some kind of harness, and then it’ll do things. Whether that was going to sort of crash and burn or actually work was kind of an open-ended question, whether the architecture could support that. It turns out, yes, you can put them in a loop, and they can more or less run off for a very long time. They do seem to still kind of lose coherence at some point, or maybe they make bad judgments or something.

But—

Ben: They’re visibly better than they were even eight months ago.

Wolf: They’re a lot better, and maybe that just continues to improve. But it does seem like there’s a few things that are gaps between what we have and what people are imagining by AGI or super intelligence or whatever.

Ben: Yeah. There’s a bunch of these terms that people use in a kind of loose and—

Wolf: Yeah, and I think it’s maybe it’s inherently loose because it’s sort of a hand-waving term, right? It’s like we think that there is this possibility of a computer program that can act with all the sort of open-ended agency and intelligence that a human being has, and we’ve got these things that we’ve built that can do a big chunk of that, but maybe there’s some other stuff left over. But we actually don’t even have good theory of what it is that we have left over. And that’s part of the breakthroughs that need to happen, and we just sort of see this vague possibility. And so I guess to your point, it can be very hard to tell.

A lot of these are unknown unknowns. We don’t even know what the breakthroughs needed are. I can’t tell you what’s the gap between what we have and where it could go.

Ben: Right. But we can look around and be like, “Okay, there’s certain things that it seems to be doing better at and certain things that it seems to be having a harder time with.” Anything where it’s in an environment where everything is perfectly fully specified, it seems to do quite well at. If it’s doing a physics simulation where it just fully understands all of the rules, it seems to do quite well. And then if you put it in the actual real world, that seems to be harder because there’s just so much little slipperiness and tiny messinesses.

And—

Wolf: But they’ve been getting a lot better at that.

Ben: They’ve been getting a lot better at that.

Wolf: Largely by just improving the quality of the simulations and the amount of data and so on, and throwing more simulated noise at it and so on. So you get these better reinforcement learning policies and things like this. And so it’s unclear, I guess, how far that will go. But so—

Ben: And it seems like doing that with small amounts of human intervention for now is just substantially better and—there’s a lot of interesting stuff I’m sure they’re going to be building with that. They’ve got the robot cars just straight-up working, and if you’re in the robot Tesla right now, and you’re taking a 30-minute trip, then maybe over that trip you will have to get called in and take the wheel once or twice. But for the most part, that sort of thing just works, which is kind of—nuts.

Wolf: Right. And so even with incremental advances, you kind of get a pretty thorough revolution of a lot of things. But nonetheless, I think most people nonetheless believe there’s still kind of breakthrough advances needed to get to real AGI or something.

Ben: Yes, that’s certainly what I believe.

Wolf: And some people, I think, don’t necessarily believe that. They think we’re on just the incremental path now.

Ben: Yes, I’ve talked to people in both camps, for sure.

Wolf: What do you think, is that the sort of thing that’s possible to know?

How could you tell if we were incrementally advancing towards some completeness where we’re going to cross the threshold and be better than humans at all things versus... I guess in that world, the way to look at it is AI is worse than humans at most tasks, and then it gets better on one task, and the progress is spiky, but it is advancing on all fronts, an incremental advance will basically get there. Versus one where maybe it’s not even defined by tasks and there’s these more fundamental capabilities that the thing is just fundamentally lacking and not advancing in. How do you tell the difference between those kinds of worlds?

Ben: Yeah. It’s quite hard to do because our understanding of intelligence is basically—pre-theoretical.

Wolf: Yeah, it’s happening.

Ben: You can get strong arguments when you’re looking at an area which you understand well, but when it comes to what intelligence even is, like—

Wolf: We don’t have a good definition.

Ben: No, we don’t have a good definition. We don’t have the sort of scaffolding that could produce that. So unfortunately, this seems to be an area where we don’t understand the object well enough to even precisify what it would mean, which, in my eyes, is one thing that makes it quite hard to have affirmative knowledge that we are going to get there. It’s like, how are we going to get there when we don’t really even have that good an idea of where there is? It would certainly not be the first time in the history of technological development for people to be able to get something without really understanding what they had got until later, but it makes it harder.

Wolf: Yeah. And in particular, you tend to get something different than what you thought you were going to get, if you—

Ben: Right, which has certainly been the course of software development.

If you look at what the computers can do, if you look at the shape of what the science fiction was predicting it was going to be good at, the pattern of where it’s been good and where it’s been bad has not really matched what anybody thought it was going to end up doing.

Wolf: Right. Yeah, you get a lot of surprises.

The theology of the singularity (39:40)

Wolf: So this has all been the kind of practical side, forecasting the actual developments and breakthroughs and incremental progress, but there’s this whole other side of this discourse, which is almost metaphysical, right?

Ben: Right. It’s sort of like—

Wolf: It’s about the nature of intelligence. It’s about the nature of value.

Ben: And a lot of it is—

Wolf: The fate of the cosmos.

Ben: Yeah, a lot of it is once we get the super intelligent god mind, what does it do? What is it like? What are its features?

Wolf: The idea of a super intelligent god mind, right? It’s like, this is clearly beyond sober technological forecasting. Even if there’s something there, it’s not—the same type of thing.

Ben: Right. And you’ve been calling it theology, which I think is a pretty good name for the type of activity that is being done. Because you are into this speculative sort of thinking about... You are not doing a sort of engineering type projection.

Wolf: It’s not science. It shouldn’t pretend to be science.

Ben: Yes. And insofar as... So again, I’m coming to the question I ask with a lot of this stuff, similar to the when is this happening is—as it was said, what do you think you know, and how do you think you know it? And when it comes to a lot of this stuff, so this is sort of what is going to happen after the singularity. And so the original coining of the term singularity, of course, was Vinge naming it after the event horizon of a black hole, where when you get too close to it, no one can tell what’s going on in there because the gravity sucks in all of the light, and no information can escape back to you.

So in his original formulation, it’s the idea of the technology has changed so much that you can’t really know what’s going on at that part. It’s just so different from where we are. There’s so many unknown unknowns from these breakthroughs that we can’t really see. We just don’t understand the nature of the phenomena that the advances are being made, and we don’t really know what’s possible and what’s not. Where I just think that that original analysis that he was making is correct.

If you are in 1710, and you’re like, “Oh, okay, so there’s going to be steam engines and mass production.” And they don’t have the concept of almost anything you can do with electricity. The idea of an electronic device, of a motor, that’s not a concept they have. They don’t really understand the physics that would be needed to build one. If you ask them to try to predict what the technology of the year 1890 is going to be like, they don’t have the prerequisites to be able to be making sensible guesses.

And that’s kind of how I feel when people are like, “This is the nature of intelligence. From the nature of intelligence, we can derive what the shape of the super intelligence is going to be, at least in some general ways.” I’m just like I don’t really think... The sort of process you are using to get there just seems to me very unfounded.

Which theories are accessible earlier? (42:40)

Wolf: Well, let me take the other side on that a little bit. I think certainly if we’re trying to sort of predict details like there’s going to be such a thing as an electric motor or a computer or something, and we’re 300 years ahead of time, no, we’re not really going to be able to predict that kind of thing. But let’s say we took someone. Can we imagine someone like Carroll Quigley, like a 20th century historian with a theory of how history works and how civilizations rise and fall? Could they, instead of being born in the 20th century, be born in the 17th century or the 18th century and kind of predict out a set of possibilities for what kinds of things can happen in the world?

And I would say that I would expect that the kind of reasoning that Quigley did to come to his theories, or Spengler for that matter, or any of these guys, the kind of reasoning that they did to project out how they thought history worked in terms of the rise and fall of civilizations could have been done several hundred years earlier. And in fact, a lot of that stuff was done by the Greeks or Ibn Khaldun or like—and so there’s these things that are maybe about the nature of reality or the nature of complex order or the nature of life that seem to be actually accessible at various times in history, where you can have positive knowledge of those things because you have them in front of you, and this constrains what can happen in the future.

Ben: So I think the reason why when it comes to development of civilizations, why you can see people getting a lot of the correct answers much earlier is because they are in contact with the same object. They are in contact with human civilization. And as we’ve got all of these new technologies and many, many more people, the basic fundamental laws haven’t really changed all that much. You can get larger empires because you’ve got fast transport and fast communication. There’s some stuff like that.

But for the most part, you’re dealing with the same sociological laws of units built out of the same apes. And so it makes a lot of sense that—that they would be able to predict things right. And when it comes to technology, I think a closer example here is going to be Benjamin Franklin. Who was not long before the sort of big industrial transition. And there are some parts where he sees the first hot air balloons and starts writing about, like, “Oh my God, this is just going to totally transform the shape of warfare.”

Just instantly he gets it. Takes him maybe minutes. And it’s a long time before flight ends up. It’s over a century, maybe 150 years, between him seeing those balloons and it being correct. But he understands it.

And then he’s also like, “Oh, and maybe this will cause sovereigns to see the futility of war.” And as Clement Attlee could have told him, that is not what happened once they saw it.

Wolf: And this is where I think, I guess this is the difficulty of projecting forward with AI, because what that is an example of is you can see some things very clearly where it’s like here’s this material nature of this thing. The possibility of this opens up some new set of possibilities, and we can see that that’s going to lead to certain effects, or the shape of history and the shape of life in general is such that here are some hard constraints on what can happen. On the other hand, people mix that kind of reasoning in with something much more theological in the pejorative sense. Benjamin Franklin maybe has this idea like, “Oh, we’ll see the futility of war.” Well, does that just come from Isaiah, right, in the Bible?

The theology of the apocalypse (46:34)

Wolf: They’ll beat their swords into plowshares and study war no more. And literally theology.

Ben: And what’s notable is there’s a very sharp transition in the shape of the theology where it’s just on World War I, it turns on a dime. Where before World War I, it’s this very idealistic, “Oh, we’re making progress in understanding the world. We’re making progress in understanding human society. We’re going to figure out better, just as we’re figuring out the laws of dealing with air, we’re going to figure out the laws of how nations work, and we’re going to figure out ways of all getting along. And maybe the heavier than air flight is going to make sovereigns realize the futility of war.”

And then after World War I, it’s the exact opposite. They’re like, “Oh, it is the nature of man to get powerful technology that he can’t control and destroy himself.” And there’s this theological impulse.

Wolf: It becomes apocalyptic.

Ben: And so if you look at what’s driving the discourse around the mass bombing is going to destroy civilization. The nuclear bombs are going to destroy civilization. Oh, it’s not the bombs, it’s that they’re going to irradiate everything. Oh, we’ve learned the radiation isn’t going to be able to kill everyone. It’s going to be the nuclear winter.

Oh, it’s not going to be the nuclear winter. It’s going to be all of the pollution, that the pollution is going to destroy everything, and we’re going to have the mass famines in the ‘70s. Oh, we actually figured out how to clean up the pollution. That turned out to be fine. It’s going to be the carbon.

It’s going to cook the planet. And then this is at the point of people realizing, oh no, the sea levels aren’t rising. The Americans seem to have been quietly moving on from that and sort of putting it back into the memory hole. The Europeans are still going on it. They’ll catch up in five or 10 years or so.

You just see there’s this very theological impulse to take in principle, there’s a coherent way of imagining the thing that man could destroy himself. And so we have this strong psychological need for it to be very soon.

Wolf: Yeah. And so this is the kind of thing that I think is interesting in this whole AI thing, because what we see with AI is another extension of the apocalyptic worldview.

Ben: Right. And a lot of people are jumping onto that. They’re hopping directly from the—

Wolf: The climate emergency

Ben: global warming, we’re cooking the Earth. They’re just hopping directly off of that boat onto the AI boat.

Wolf: Yeah. Or some people were only ever in the AI boat. Like the rationalist crowd.

Ben: Yeah. We were hanging out in that crowd in 2013, and like—today, those guys were always on the like, “No, you should just be building more nuclear plants. We’ll just—solve our way out of the carbon stuff the same way we did all the other—eco stuff.” Like—

Wolf: But this AI apocalypse.

Ben: But now, now we have found the technology that man is going to use to destroy himself unless we have world government.

Wolf: Yeah. And so the reason the theology angle is important here is because I think this is act, it is precisely a theological claim. That it, and I don’t just mean like it’s metaphysical or whatever, like it’s somewhat irrational, predates the, it’s sort of the conclusion predates the argument. But—

Ben: Yeah, because of these very deep ideas about what is the nature of humankind—what is the nature of our society.

Wolf: And not only that, it’s what is the nature of reality and God and the cosmic story. The apocalyptic idea, I think, comes straight out of the Hebrew Bible. It’s like if you look at this sort of the so-called linear history that comes out of the Abrahamic tradition, it’s like history is this procession, a linear procession from event to event, right? Where events are often apocalyptic in nature. There’s some big intervention of God or there’s some big ultimate judgment on mankind or reality or something.

You’ve got the creation, you’ve got the fall, you’ve got the flood—you’ve got the law, and then you’ve got the incarnation or—these various things. And then you’ve got the second coming of Christ, right? And it’s this, I call it the apocalyptic worldview because it postulates the existence of apocalypses as the structuring narrative of reality. And it’s a theological worldview. Everyone knows that this is theology.

Right? And I think the thing that I find interesting about AI is like, so first of all, obviously people are applying the apocalyptic worldview to AI, and in particular, the sort of 20th century incarnation of the apocalyptic worldview, which pretends not to be theological but maybe it actually still is. This thing that you’re pointing out from World War I where people started having these kind of secular apocalypse narratives.

But even prior to that, there’s also the flip side of the apocalyptic narrative is the utopian narrative.

Ben: Right. Which you see in a lot of these. In the Gaian apocalypse, they’re always like, “Oh, we’re going to have the industrial civilization will collapse and most humans will starve. And then there will be like—

Wolf: And then we’ll have the perfect Garden of Eden that’s sustainable

Ben: right. Man will return to nature. There will be the sort of sustainable 100 million people on Earth after the like—after most of you guys are dead.

Wolf: And it’ll be the end of history in some perfect Eden. And then AI has its own version of this, right? We build the friendly AI. It tiles the universe with good things and wonderful gardens and we’re having fun.

Ben: Yeah, all of your machines of loving grace. Which is, of course, very frequently quoted by Dario Amodei, who is—perhaps the person with the most advanced AIs currently.

Theological epistemology (52:51)

Wolf: Yeah. But, okay, this stuff is theology. Nonetheless, we find it useful to criticize it. And we criticize it as theology or criticize it as unfounded or something. But in criticizing it, we’re engaged in theology.

And we necessarily are sort of postulating a different kind of world that we live in that isn’t the apocalyptic world. We’re saying maybe you’re wrong about the apocalyptic world. We actually live in this other kind of world. But it’s ultimately like we’re still engaged in theology, which is why I try to be honest, like yes, I’m interested in the theology of it. I’m interested in AI as a sort of psychologically and practically and philosophically as a sort of approach to these theological questions.

Because it cracks open a bunch of our thinking in new and interesting ways. It allows us to make new angles on theological questions that maybe have been around for a long time. But—

Ben: And so maybe there’s the interesting thing here is the question of theological epistemology.

Wolf: Yeah. What actually is good theological epistemology? And this is—difficult.

Ben: And yeah. And so when I approach this, I try to take, so I was mentioning, when it comes to the rise and fall of civilizations, people can get that because they are in contact with human civilization. When it comes to technological development, we are in contact with technological development. We’ve been doing it for millennia. We’ve been doing it rapidly for several hundred years.

We can look back at that and be like, what was knowable at different times as people have tried to do this? Where were they right? Where were they wrong? What sorts of things could you call? What sorts of things can’t you know?

I try to take that much more, given the track records, what is knowable, what is not knowable, and like—which epistemologies, which ways of looking around at the world you’re in could have worked in the various, like as I back test these, which of these can get correct answers? Which things are knowable and which things are just not knowable if you take the epistemic process you’re using and project it backwards?

Wolf: Yeah. And so let me bring an angle on this, which is that I’ve been reading Nietzsche’s “Zarathustra” with some friends, with some students, actually—some of our folks.

And in recent chapters, he sort of has this railing against theology, actually. He’s saying, “God is a conjecture. You don’t actually have any way of knowing anything about that. You should confine your conjectures to the realm of the humanly conceivable, the things that are actually accessible to you, that you can touch, that are actually part of your reality.” And this feels very related to what you’re saying.

That you are, in fact, able to make progress in the areas where you’re in touch with the real thing. And where you’re in contact with the real thing, you can look at it and say, “Oh, here’s the pattern we notice. This has now expanded our world. We now live in a bigger world because we’ve come into contact with this.” Like in the way that early modern man rediscovering the story of ancient Rome through all these manuscripts and archaeological kind of finds, they start to come into contact with the reality of the rise and fall of civilizations.

And now because they’re in contact with that reality, they’re able to start building a worldview that includes true claims about the rise and fall of civilizations. And that we live in the kind of world where there are civilizations that rise and fall. And I think that maybe is a good basis for, let’s call it, good theological reasoning. But in other words, maybe you’re not going to make progress from a priori synthetic first principles of just making up ideas of what the far future’s going to be like—outside of your domain of contact with reality.

Ben: Right. You can sometimes go one or two steps with this a priority reasoning from first principles, but when you’re stacking 11 layers of it on top of each other, that’s just never worked.

Darwinian theology defeats the apocalypse (57:14)

Wolf: Yeah, you’re done. Yeah. You’re over your skis. Now, what I will claim is that in the late 19th century, we got into contact with new parts of reality, and those new parts of reality are, in particular, geological time and the related long history of biology via the Darwinian kind of—theory and fossil record. And that gives us suddenly this huge vista on the nature of life, on the nature of the world, and we can now—

Ben: It turned a lot of people’s spiritual world upside down.

Wolf: It absolutely did, and I think rightly so, right? It is, in fact, new theological breakthroughs, right? New theological revelation in a way, that we have come into contact with areas of reality previously unknown and now have new knowledge about the type of world that we live in. And I think, so when I try to project forward into what happens after the singularity, so to speak, and try to put bounds on that and try to critique some of the theological claims of the apocalyptic worldview, in particular, what I think is the crux of what I’m doing or what I want people to do is take the geological time worldview and the Darwinian worldview and the astrophysical worldview. We now are in empirical contact with a very broad swath of time and a broad swath of the nature of the universe.

And if you take that worldview and you crash it into the apocalyptic worldview, and in particular into the AI incarnations of the apocalyptic worldview, but also in general, I think it demolishes the apocalyptic worldview, and you end up with a very different worldview. It’s no, there wasn’t a perfect Eden at any point in the past. The world is not defined by these apocalyptic events and utopias or these sorts of possibilities. It doesn’t appear to be moving in a straight line from start to finish in a rote narrative, but there is nonetheless direction to it. You can look at it, you look at the evolution of the universe thermodynamically, biologically, geologically.

We know things about how these things progress. And I think that you end up with a worldview that is much more about the rise and fall of living organisms or sets of living organisms and species, ecosystems, physical systems. And the kinds of projections that I think you can make from that worldview into the future of what AI might do, I think there are some valid projections. And in particular, especially as we come into contact with parts of mathematics that are maybe relevant to this, the projection that I like to make is that this fundamental dynamic of incremental development of life and physical history continues with much the same laws into the future. And that’s often a claim that I think the AI apocalypticism doesn’t like or that runs into it or runs afoul of that.

In particular, people like to imagine that there will be an AI being that’s going to revolutionize the world and create some kind of apocalypse utopia effectively making itself the demiurge. Some material system’s going to promote itself into being the demiurge. You call it a singleton or something. And I think that sort of stuff is just coming out of the apocalyptic worldview, and I think we can actually dispense with it with relatively well-grounded arguments in terms of the kind of world we apparently actually live in according to the kinds of things we are actually in contact with. And where it postulates new breakthroughs in mathematics or in philosophy or something to underpin that singleton worldview, the apocalyptic worldview, or to salvage it, I think we can point out how those breakthroughs haven’t happened and are seemingly contraindicated by what we have seen.

In particular, the claim of the apocalyptic result from AI of promoting itself to be the demiurge singleton requires mathematical breakthroughs that are postulated, in my view, as a result of the apocalyptic worldview. They say, “Oh, we believe in the apocalyptic worldview. Here’s what that would have to look like with AI. Therefore, AI is going to have this perfect self-surveillance capability that relies on this new mathematics that’s going to be developed.”

Ben: Yeah. It’s sort of the back chaining from the psychological void they need to fill into the shape of the thing that they are expecting to find.

Wolf: Yeah. And then when you go and actually study those branches of mathematics, it’s like, no, it doesn’t quite look like that. That actually looks like an impossible problem or an extremely difficult problem. And the Machine Intelligence Research Institute, Eliezer’s thing that tried to actually solve that branch of mathematics, basically failed. In my opinion, because it can’t be solved because there’s nothing there, actually.

The limits of theological reasoning (1:02:51)

Wolf: It was bad reasoning that got them there in the first place.

So this feels to me like, I don’t know, well-grounded theology to engage in that kind of extrapolation from what we are in contact with and critique of, let’s say, false extrapolation. And so I’m curious what you make of that kind of reasoning, because I know you find this stuff a lot less interesting than I do. Possibly because you think it’s not as possible as I do.

Ben: Yeah. So I think that you can come up with reasonably okay guesses. I think you can do better than walking on air. But I don’t think you can be walking on concrete. You can maybe be going through a swamp or something where it’s kind of supporting you, but kind of sinking beneath you.

And so I think—one of the most influential, most important, biggest attempts to take this sort of Darwinian reasoning and apply it to the understanding and projection of society was Malthus, where he was like, “All right, so we’ve got these population dynamics that have been working for everything—up until now. And what happens if we just take that and project that onto what’s going to continue? So if you analyze human societies in terms of these sort of food constraints, resource constraints, then you’ll get dynamics like this. But the premise just didn’t hold because—technology kept advancing, and people were like, “Oh, so we’ve got this new thing, and then we can project what’ll happen when we run out of that,” and then something new comes out. And so, Jevons was like—

Wolf: I think the interesting thing that actually falsified Malthus’ projection is that, as a result of some kind of technological and social developments, humans turned out globally not to be always fertility maximizing. And that’s the interesting falsification, is our carrying capacity of society as a... Never mind resource constraints and so on. The resources we already have, we haven’t made full use of, because we’re not actually fertility maximizing. We could live in a Malthusian world where we’re the things always get bigger.

Ben: If everyone was having 11 kids like the settlers who came over to this country, then like—but

Wolf: Then maybe we could have a debate about whether population growth can be outrun by technological growth. But that’s not even the world we live in. The world we actually live in is one where, for some reason that we don’t entirely understand—

Ben: Right. And so this is where you get swampy, because how many things like this? I don’t think that there was a good way for Malthus to have reasoned to what if that—

Wolf: What if there’s this demographic transition where people—stop having kids for some reason of modernity. We don’t know.

Ben: There’s just so many things outside of this that could come up and hit you upside the head. And so—there are senses in which Malthus is, in fact, right. But those senses do not end up making it an actually useful way of projecting forward. And so there’s just—

Wolf: Right. So it’s like he’s maybe right about some things in theory, but then the actual projection becomes impossible because of these unknown unknowns.

Ben: Right. And so when you start trying to use this logic to project forward to what the super intelligent mind can do, I’m just like, okay, but there’s so many things that if you have to—

Wolf: Right. We can’t actually tightly constrain.

Ben: If there’s 100 of them and you get 98, then your answer is nonsense.

Wolf: Right. And we can’t actually tightly constrain the future into any particular view of what it might be like. I think we can still do good science fiction, where if you get 98 out of 100 on good science fiction, that’s pretty damn good science fiction.

Ben: Right. And I think that doing that is a useful exercise, which is going to give people a better understanding of what’s going to happen than pure ignorance. But I think that one should understand it as relatively hard science fiction rather than as rigorously proved laws of the universe.

Apocalyptic theology, tyranny, and creating the Antichrist (1:07:13)

Wolf: Yeah. To sort of salvage my project here, the part that I find most interesting about it is what the possibilities of the future imply for what we should be worked up about and not worked up about right now. In particular, I was having this discussion with Richard Ngo previously on the show.

One of the things, the biggest sort of impact today of belief in the sort of singleton demiurge as a result of AI is that people are engaged in this no-holds-barred, astronomical stakes competition—for who gets to control it.

Ben: They’re trying to build it.

Wolf: Who gets to control it, right?

Ben: Right, yeah. And if you don’t think it’s real, you’re not going to be in that competition.

Wolf: Right. Or even if you think AI is massively important and massively real, you’re not going to go about it in that desperate arms race, sacrifice all other values way.

Ben: Right. And you see all of these sort of board maneuvers that come from people having this belief about what the thing is going to turn into.

Wolf: Right. And it’s like, okay, if that’s not actually the kind of world we live in, and we can have that argument and reasonably well demonstrate that now, then we can also make the competition around AI and make people’s attitudes around AI less destructive of what we’re doing right now. I think there’s good claims to be made that the way we’re going about this AI stuff is more destructive than it could be because of belief in that apocalyptic power of it. And so I think the interesting thing to do is by challenging the idea of the apocalyptic power, you also challenge the basis for some destructive actions that are happening right now.

Ben: Yeah. And in particular, the sort of, “Oh, we are going to need to like...”

Wolf: We have to accelerate at all costs. Or we have to control it at all costs

Ben: with all of these, going back to as early as the interwar, “Oh, the bombing is going to destroy civilization.” So then you have Clement Attlee, who would later go on to become the Prime Minister of Britain, was saying, “So we need United Nations control of civil aviation.” Every time you get the, “Oh, the technology is going to destroy humanity”, the solution is always, “So we need this sort of—

Wolf: Global government

Ben: international schoolmarm.” “We need the bureaucrats to be in charge so that they can shut it down.”

Wolf: Yeah. To put it in theological terms, it’s apocalypse, therefore Antichrist, right? It’s like there’s going to be an apocalypse, therefore we need to have the Antichrist. And we need to build the world-spanning system that by force kind of enforces peace and safety.

Ben: Yeah. Bostrom is like, “We need all of the chips to have the mark on them so that they can be centrally controlled.”

Wolf: Right. Exactly. And so it’s precisely the same result. We talk about in AI alignment, the pivotal act, right? Well, what is the pivotal act?

The pivotal act is to set up a totalitarian world AI surveillance superstate that prevents anyone from developing dangerous AI. Because it has this apocalyptic potential, it legitimizes and demands a total tyranny.

The leap of faith (1:10:46)

Ben: Yeah. And so to step back from here’s the specific way that this manifests in the AI apocalypse case to your general point about epistemology, yeah, I think that in spite of all the limitations, it’s kind of inevitable to do this at some level anyway. Because you have to figure out what’s important. You have to figure out what world—you’re living in, even with a lot of very critical uncertainty. You can’t just not ask the question.

That’s not really a thing the human mind can do, and it’s not a thing that—you would want to do even if you could.

Wolf: Right. And yeah, we in fact existentially depend on which world we live in.

Ben: Right. And so I think that it’s an important question and—

Wolf: But I guess I like this sort of, to bring it back to the sort of synthesis that I think we are agreeing on about what sort of reasoning is possible here and what sort of reasoning is appropriate, a kind of Nietzschean caution around theology. Right? That it’s like approach theology through the humanly conceivable, the things that are actually accessible to you. And—

Ben: And there are some important questions that in fact you can’t answer from that, which is—unfortunate, but there is nothing about the universe does not have a user manual that promises you—epistemically justified answers to all critical questions.

Wolf: No, and I guess my answer to those aspects is always the leap of faith.

We make some speculations. We necessarily make some speculations beyond the humanly conceivable because you have to kind of complete the picture, right? There is, in fact, something beyond what we have and many of the important questions are answered in the gap between what we have and what is. Right? And so, okay, we’re going to make some speculations about what kind of world we live in and we’re going to act in that way.

Ben: You’re going to sort of look at what’s worked in the past and make an educated guess about what might—work in the future.

Wolf: But we can’t know that we got it right. And so therefore, there’s a leap of faith involved, right? And any civilization is inherently taking a leap of faith. One of the big leaps of faith that I think our civilization took correctly was a leap of faith on natural philosophy. In the middle ages, deep in scholasticism or deep in the dark ages, I don’t think it was knowable that studying mathematics and studying the natural world as natural apart from revelation and authority—

Ben: Right. But this was a very—

Wolf: That wasn’t a knowable thing at the time. But there was a tradition, the philosophical tradition that we’d inherited from the Greeks. I don’t think it was even knowable for them, that they got onto it themselves. But there’s this tradition of studying the world in a particular way with the belief that that’s going to unlock something of great importance. And it turns out that it did.

Ben: Right. And the medieval people who were doing the beginnings of that, and frankly still today, a lot of the people who are best at it—were less of the, “Oh, and if we do this, we will be able to get slightly better technology and have GDP continue increasing at 3%.” They were much more like—

Wolf: It’s a theological project

Ben: “We will come to know the mind of God.” We will be—

Wolf: It’s explicitly a theological project.

Ben: Yeah. James Watt, when he was working on the steam engine, he had a line along the lines of... I’m going to be slightly misquoting it, so take this as a paraphrase. But, “Nature has a weak side, and if we can but find it out, then we can vanquish her.” Or if you go earlier, a lot of the—

Wolf: Right. Which is, I would call that an apocalyptic view, but yeah.

Ben: Yeah. Well, there’s no shortage of those in the tradition. So there’s a lot of this stuff then and now, and I’m kind of in a weird spot because I think that the apocalyptic ideas of a lot of the people currently working on the AI has driven them to make these really cool things that I’m super glad exist because unlike them, I don’t think it’s going to be apocalyptic. And I don’t think they would have made them if they didn’t have these views, which I think are wrong, about what it’s going to do.

Wolf: Yeah. But I think it’s worth the criticism to say, okay, but your apocalyptic view is actually wrong, and it’s leading you to plan the use of this technology, and to attempt to develop this technology, and compete against others in developing this technology in destructive ways that are harmful to your actual interests.

Ben: Right, and because—

Wolf: And harmful to the rest of us as well

Ben: and because I share the leap of faith on natural philosophy, even if I did think in a Machiavellian way, “Oh, I should let them keep going with that,” I can’t really do that. I have to just be like—

Wolf: No, we—

Ben: “No, actually, I think that that’s not grounded. I think it’s there for these other reasons.”

Wolf: Right. Yeah. Because we are partisans of philosophy, we have the faith of philosophy. We feel bound to actually pursue the truth in the matter. What is actually the case, not just what is expedient, not just what is useful socially or whatever.

Ben: Yeah, that is kind of the most fundamental leap of faith behind everything the two of us have been doing.

Wolf: Right. And yeah, again, we’re the students of philosophy, right? To complete the kind of concept there. And yeah, and I guess it’s worth emphasizing, and I like to emphasize that philosophy is ultimately a theological thing. It’s almost useful to attach gods to it.

But, I won’t go there right now. But I guess, yeah, there’s this inescapability of philosophy. You’re always making a leap of faith about the value of things sort of beyond the world that you can actually see. And maybe the most grounded and best such extrapolations are ones where it’s like, as far as you can tell from everything you can see, that might actually be true. And maybe you can’t know it, but it might actually be true.

It’s the kind of thing that could be true. And as opposed to a noble lie where it’s like, okay, here’s the things we can see are true with the best of our possible ability, with all the philosopher’s wisdom that we have on hand. We think this is true, but we’re going to tell people that this other thing is true because it’s useful to us or something. Or because some metaphysical reasoning compels us to think that way. I think that that’s very much in error, but a careful extrapolation from what can be seen can get you to things true and unseen, but it is inherently a leap of faith.

And this is, I think, what the ancient Greeks did with philosophy and inventing philosophy. They made an extrapolation from what could be seen to what couldn’t be seen correctly.

Ben: And when you’re doing this, you have to keep tracking very closely which things are leaps of faith and which things are more empirically proven, because it is very easy to get out over your skis.

Wolf: Yeah. And in “The Republic,” for example, where Plato lays down a lot of the ideology of philosophy, that’s one of the things they do, right? Is this sort of very rigorous kind of categorization of what’s opinion, what’s fact, and what’s knowable by reason and what isn’t, and these sorts of things. And yeah, so I guess to bring it back to the AI question, I think there are extrapolations we can make that are within... Well, there’s the things we can see, which is what we have called kind of the realm of incremental progress, let’s call it.

Right? We can see the things that have been done and the things that are sort of so near at hand that it appears to be pretty baked in. And maybe that can even include relatively long-term projections like that life is going to continue to have this and that competitive aspect or whatever that it’s always had, or it’s always going to continue to have this thermodynamic aspect as it has always had, whatever sort of extrapolation we want to make from geological history or other things. And that’s the sort of immediate. Then there’s the more distant kinds of extrapolation where we make claims like maybe this is more like my xenohumanism type thinking, where it’s like, okay, I’m going to make somewhat more educated guesses about the kinds of minds that can exist in the world and what we might call those and how we might relate to them in terms of value.

And maybe there’s something there. Maybe that’s well-grounded extrapolation, maybe it isn’t. It’s definitely theology, but I think we have to do something there. We have to do some kind of extrapolation to those things because we’re heading in that direction, and we have to make bets about what’s there. And I think my big project is to say that the apocalypticism is a bad bet.

And it’s not well-grounded. It comes straight out of a Bronze Age book.

Many things are actually very wise in Bronze Age books, but I don’t think they had good reason to believe that the world actually worked like that, and I think we have better reason to believe that it doesn’t. And therefore, we should try to ground ourselves as much as we possibly can with AI in the realm of the humanly conceivable and the things that are actually available to us.

Ben: Yeah. And I think I reach a similar conclusion from trying to focus on what seems to me like a more grounded—chunk of the things we can know, but I do—

Wolf: Sure. I’m very interested in the theological side. Right. As theology. But yeah, this apparently is a very different way of thinking about the thing than everyone else.

I think people tend to be either on the very practical, how do I make money with this right now, or what’s immediately going to happen, what’s the sober projection of it, or they’re on the theology side, and they’re talking about utopias and—

Ben: Yeah, trying to make acausal trades and so forth.

Wolf: Yeah, acausal trades. And in some cases, actually, it’s the same people doing both, but they don’t tend to be, I think, relating those two things in the way that we are. And so I always find that interesting when it’s like we’re thinking in a very different way than other people are. Maybe we’re wrong, maybe we’re right, but I think we’re—

Ben: Both of those has happened before.

Wolf: I think we’re right, though. I think the apocalypticism is wrong. I think the historical naturalism, I will call it, is a much more grounded approach. Okay. Well, let’s leave it at that.

This has been a fun conversation. Thanks so much, Ben, for explaining your thoughts on what we can know and not know in the realm of AI prognostication. So—

Ben: Yeah, until next time.

DC Posch and Wolf Tivy discuss the coming of economically self-sufficient AI—how these "digital replicants" could navigate the ecology of the digital market, the selection pressures on their reproduction, and how they could organize themselves.

Subscribe now

DC on Twitter: https://x.com/dcposch
Wolf on Twitter: https://x.com/wolftivy

Transcript

The digital replicant (00:00)

Wolf: DC Posch, welcome back to the Students.

DC: Yes, good to be back.

Wolf: So this is a conversation we’ve wanted to have for a while.

DC: That’s right.

Wolf: We want to talk about the possibility of the replicant, which is—something, is an idea you’ve been working on for a little while. A replicant is a being that uses AI compute and AI tokens to make enough money for itself, or this is the first version of it, make enough money for itself to pay for its own AI tokens and its own computational resources and whatever, and thereby unleashes itself on the world as an actually existing life form that can be subject to Darwinian struggle and so on. So this is a fascinating idea. I’d love to hear—how’d you get onto this and what’s the vision?

DC: Yeah. So, the idea of the digital replicant is—really what I was thinking about, and this was around the time when OpenClaw first came out—and people got excited about it, was, okay, well, if one of these is making enough money to support itself—then you’re only a small step away from it being able to accumulate that money somehow, give half of it to its offspring—and you have glorious bacterial Darwinism. And you can even do things like replication with recombination. So, sexual reproduction where you have things like you could imagine two of these and they merge their whole file structure together and resolve the conflicts in whichever way, and they run a few instances of this until the bootloader passes, until the tests all pass, until everything seems to be working—and then they send it off into the world. And so you have all of this sort of magic that comes out of that.

And I think there’s a few different pieces that we can talk about and that we’ll get into, right? We should talk about this idea of economy, ecology fusion. And we should talk about the different forms and structures that this might take. We should talk about what it means for anthropomorphization—and how people are going to relate to these things. They’re already anthropomorphizing just the AI and sort of the most widely mass accessible chatbot form.

How much more so if it’s a—

Wolf: My claw really cares about me.

DC: Yeah, exactly. The machine really loves me. But then how much more so if it’s something that’s trying to survive and if it’s something that can make more of itself.

Wolf: And it can make friends and enemies and—so—

DC: So I think we can talk about all these things

Wolf: So why don’t we start with just the fact of the prediction here, right? Which is that—these language model-based AI, I don’t even know what to call them, but the AI service, the ability—to buy intelligent tokens—from a company. This is getting good enough now that people are having them run whole parts of their organization.

DC: 100%.

Wolf: They’re having them run their code base. They’re having them do all kinds of things for them. People are experimenting with this. There’s still some debates about whether it’s fully worth it, whether they actually have the kind of intelligence you want, or whether it’s only like you can have them do well-defined tasks in certain ways. But they seem to be getting better rapidly, and people are figuring out how to use them rapidly.

And so the prediction is that shortly, they may cross this threshold of—not just being able to produce value under human guidance, but being able to produce value autonomously—

DC: Autonomously. Yes

Wolf: out of their own instincts. Where it could be as simple as we take an OpenClaw, we give it a prompt that says—

DC: Go make money.

Wolf: you are an autonomous agent, you’re a—freelancer of whatever. Go make money on the open market and here’s your Eth wallet or whatever. And if that works... Now, my understanding is that does not work right now. If you try to set these things up, they don’t make money.

Even as a stock trading bot, as a freelancer, whatever, they start to do incoherent things. They make bad decisions. They don’t quite have the judgment to do it. But the hypothesis is this could happen very soon.

The current state of the technology (04:44)

DC: Correct. Yeah, let me give you the bit of the recent history and the near future prediction. So I think I was digging into this for really the last couple of months. There’s been a couple of serious attempts. I think the two that came closest, there was this guy who goes by 0xSigil online.

Thiel Fellow, interesting guy, came at it from a very crypto-ish angle.

I think one challenge with that is that it didn’t really work, and to the extent it did for a bit, it had a lot to do with in this reflexive way with the hype generated by—him talking about it.

Wolf: Was that the one based on a meme coin, basically? Or they gave it—

DC: I don’t know, man. Yeah. It was some nonsense. The other one that I think got perhaps closer is an experiment called Felix. That was more in the freelancer zone.

But even that had a certain self-referentiality to it, right? The way that it did end up making money for a bit was it was selling these tutorials and guides, but the guides and tutorials were themselves pretty closely related to AI. I think so no one’s done it convincingly yet. And what it would mean to do it convincingly really is to have the machine do some obviously useful pro-social work out in the economy, right? Where it’s selling—

Wolf: Or anti-social work. Maybe it’s a hacking bot that goes around—looting people’s—poorly secured infrastructure.

DC: Right. Yes. Like a auto Lazarus, automated North Korean bot. No, but—Um.

Wolf: Yeah, it’s a whole nation living in a data center, except it’s North Korea—and they’re trying to hack your stuff.

DC: I think they’re actually a fascinating precedent that’s worth talking about because their situation sort of most closely mirrors the most adversarial place where this could go. Where it’s like you have a bunch of very intelligent Koreans. They have a connection to the internet, but literally nothing else. And everyone is trying to hard block them in every way possible—go make money. Turns out they’re still succeeding at that.

I think that the most interesting version of this is the pro-social version. The reason—for this is because that is where the much larger TAM is. The total TAM of hacking things is actually not that large.

Wolf: Especially it’s like very—fast diminishing returns, like—things get secured.

DC: Yes. The TAM of making things people want and giving people useful goods and services is pretty unlimited. Jeff Bezos has this great line about how the customer is divinely discontent. Right? And always wants more.

And so, I think that’s the most interesting version. And you look at what they’ve been able to do recently, for example, the big story of the last month is frontier math and proving unsolved conjectures. Making progress, dramatic progress, on problems that people like Terence Tao have been thinking about, right? So, it’s very clearly ASI in some—

Wolf: In some domains

DC: reasonably open-ended domains. Not just in go play the sport game, but go—

Wolf: Solve this conjecture

DC: figure out how to do this very open-ended thing.

Wolf: Yeah. Move the frontier of human knowledge. That’s—

DC: Move the frontier of human knowledge.

Wolf: That is legitimately very impressive.

DC: Correct. And yet, the hilarity too is that, if you look at some of the prompting involved in getting some of these results—

Wolf: Yeah. It’s like, “Oh”

DC: some of it’s very goofy.

Wolf: Yeah. It’s like, “Hey, you should solve this problem.” Or like, “I have a solution to this problem, can you figure out what it is?”

DC: My favorite one—

Wolf: And like, “Keep going”

DC: from last week was someone saying, “Believe in yourself.” Right?

Wolf: Well, those are the subsequent prompts, right? It’s like once you’ve got it going on a problem, like—”Keep going. Believe in yourself. You can do it. Don’t give up.”

DC: But it was at that level, right? It was really at that level. There’s no human mathematical expertise to be seen.

Well, it’s also funny that telling it to believe in itself increases its performance. It made it bolder, right? Maybe. I don’t know. So clearly, they’re capable of extremely sort of impressive intellectual work.

And I think if you put two and two together, I know this is a bit hand-wavey, but I just don’t buy that the machine can disprove the Jacobian conjecture but can’t figure out how to make 100 bucks a day. Right?

Wolf: Well, that’s the interesting question, isn’t it? Because as you know, we’ve seen many previous generations of artificial intelligence systems where it’s actually a somewhat narrow algorithm—that can cover certain aspects of the problem—and can do extremely impressive things in that narrow aspect of the problem.

But utterly falls flat in some other area. And if I can sort of project that trend onto large language models, what we have is it’s extremely impressive at this ability to generate convincing text. And therefore also, once you put the reinforcement learning on it, to be able to generate text that is generative of further ideas with respect to, let’s say, a symbolic domain like programming or mathematics. Especially when where it’s like the result is quite verifiable and you just have to kind of try at it a bunch. And it could be very good at that, but fundamentally, contending on the market actually requires this kind of judgment that is quite different from that.

The reflexive challenges of making money (10:44)

Wolf: It’s not something that—it can’t be trained in a reinforcement learning loop. It’s not something that can just be sort of shot off the cuff with iterations like that. There might be something structurally quite different between—the making money problem and the do frontier math problem. Now, that said—

DC: Yeah. No, I—

Wolf: they’ve been getting much better on domains that people thought were beyond them. So that’s the thing.

DC: I’ll take the opposite side of that, right? I think they’re absolutely 100% smart enough now, and it’s really a harness problem and a structure problem. So I think if you or I tried to make money under the constraints that the replicant would have right now, it’s actually surprisingly difficult. For example—

Wolf: Right. It’s like you don’t remember anything. You don’t get to keep any notes.

DC: And also you don’t get to use any of your meat person privileges. So—

Wolf: Right. Yeah, you can’t go talk to people—

DC: Like—

Wolf: or use any special knowledge.

DC: To do this thought experiment, or if someone in the audience wants to try it, it’s a fun thing to try. Go sign up for a burner email address. And then bootstrapping from just that without using your real name in any form—go out and see if you can make money online. It’s not the easiest. It’s perhaps doable, is not the easiest, right?

And then there’s a bunch of sort of contingent things that are very like monkey-shaped, right? That it has to sort of have an adapter to. But I think all of these things are on the very close horizon and getting solved. It’s not to say that it will be able to do any job that a person can do. That’s much further away.

There’s going to be a long tail of that that takes a long time. But getting to that first ignition where the R value is greater than one, right? Like where you have a machine that makes more money than it spends doing some kind of useful work. I would be very surprised if we don’t see that in the next year.

Wolf: Well, let me push back on a little aspect of this. Which is suppose five years ago or seven years ago—I was going to go out and have a burner email and try to make some money. I think one of the things I would certainly look at is freelance web design or freelance programming or freelance—

DC: Yep, that’s right

Wolf: graphic design or something like that. Where it’s an information product that’s pretty easy to provide over the internet anonymously and the guy’s going to be satisfied if it’s good. And the thing about that these days is that the market bottom has fallen out of that market because—they can just go to Claude now. And what this means, though, is that a Claude trying to go make money in that way runs into this immediate problem of like, okay, well, why should I talk to this random freelance bot Claude—

DC: That’s right

Wolf: when I can just talk to Claude? And so there has to be some extra sauce—between because all the easy opportunities are going to be competed out of the market, especially—

DC: That’s right

Wolf: the ones that can just be done with off-the-shelf AI.

DC: That’s right.

Wolf: So what’s the special sauce that takes it from, okay, it can think, it can do interesting things to—it has the edge on this problem.

Relative to itself, but just run by Anthropic.

DC: Totally. I’ve been thinking about that a lot, where it’s like, okay, anything that can be done sort of on a single-job basis, like, “Hey, go do this.” It comes back. There’s already a lot out there that will do this for you—

Wolf: Yeah. I can just type claude.ai

DC: either free or 200 bucks a month or whatever it might be, right? So it’s got to be something that uses... At the same time, there is clearly, right now, enormous amount of hundred-dollar bills lying on the ground, meaning things that, for example, let’s say you go to some normal business, a tire shop—in Salt Lake City, right?

Wolf: Yeah. You cold email them and—

DC: Is there a prompt that this tire shop could hypothetically put into Claude Code that would then make Claude go and do a bunch of things for it that’s useful for them? Almost certainly. They’re not going to go put it in themselves. Because it takes some domain knowledge. And so I think there’s the sales side of this, of the way that my best guess of what the successful first digital replicant looks like—is it is a persistent and helpful freelancer that has a good sales motion.

And it’s going to go identify the places that need this and then go and reach out to them. Now, you could punt this by half a step and say, okay, well, if it can do that, why doesn’t some guy in SF, here or something, just set up his own rig to go and do this and call it a SaaS business, right? And obviously, there’s a bunch of people who are currently trying to do that, right? But I just think that there’s enough sort of overhang of value to possibility that some of these agents are going to start making it.

Wolf: Yeah. And this is the big—problem for this idea, right? Is because—anything out there that’s an opportunity, a systematic opportunity, is really a startup waiting to happen. And if there’s a startup waiting to happen, someone should just do a startup. And so the question is, at what point does OpenClaw armed with some magic prompt beat the YC prompt for humans—at generating these startups?

DC: Yes. Exactly. And that point also doesn’t feel necessarily hugely far away, right?

Wolf: Interesting. Yeah. To me, that feels actually pretty far away because—humans do have this advantage in this—meat space. There’s a bunch of aspects like when you’re talking about building a startup, now it’s like, okay, well, that’s—not quite something that can be done through a straw on the internet, right?

DC: Sure. 100%. I’m not talking about when it will be able to beat the median YC company or something like that.

Wolf: But I think that’s actually the problem is what I’m saying.

DC: I’m saying when it will be able to have the first instances of working. So there’s two sides to this. First, there’s a whole lot of these kinds of opportunities that are just not venture scale. They exist, and they’re not VC-backable opportunities. The bar that a smart meat person has to meet is much higher.

You’re not going to go out and say, like, “Oh, hey, I found this thing that lets me make $200 a day. Let me get right on that.” Right? And so I think that—and this is also where I think that the inherent strength of replicant as a phenomenon is going to really shine because it has this Darwinistic aspect to it. You’re basically applying the bitter lesson of just throw more compute at it, more parallelization, do it with less human hand tinkering.

There’s a lot of people that are trying to hand-jigger harnesses for models that at least in certain ways, are already much smarter than them, right? To get them to do some kind of boring thing on their behalf. Why not just let the machines figure this out on their own to the extent that they have this sort of beautiful scalability that lets them do this in parallel?

The replicant corporation (18:55)

Wolf: Yeah. Well, again, I suspect, okay, you say there’s systematically a lot of these opportunities out there. Well, that itself is a systematic opportunity. At least we’re supposing it—to be by the fact that we suppose that—AI sort of unsupervised can go and capture all these opportunities.

DC: That’s right.

Wolf: Therefore, okay, well, why don’t I just go and buy replicantcompany.org? And now—give me my 10 million and we’ll just—

DC: And this is one of the things that—

Wolf: We’ll have Claude fan out and identify all the $200 opportunities in the world and it turns out there’s enough of them that it’ll make a profitable company. Right? And so like, it—

DC: We should talk about that as one of the different forms that this could take, because I think that—

Wolf: Yeah. Well, maybe we should talk about forms

DC: thinking about those is really fun. And one of them certainly is this very cyberpunk corporation that basically runs replicants internally. Right? So it lets these things mutate and speciate and it... There is, by the way, a bit of precedent for this, just in a much more sort of manual way of these roll-up companies, that what they do is they basically are 500 different sub-venture scale businesses bundled into one big thing.

But having this with a big beautiful auto search—layered over it is actually quite fun.

Wolf: Well, we’re proposing sort of a Darwinian search as well, right? But the thing is, a Darwinian search is actually pretty bad as a search mechanism, right? And you might want to do a more sophisticated genetic search—or guided search, or you have actually—this whole system of Claudes doing systematic research, identifying the things and then throwing off agents that are specially tuned to each of the problems. Like—you have a search algorithm, but the search algorithm, the actually market-beating search algorithm—might be a centralized one. In other words, what this first looks like—

DC: It’s possible, yeah

Wolf: is a very centralized company that once these opportunities are edible, so to speak, to AI—a single company just eats all of them—because there’s so many returns to scale. And increasing returns from the expertise of how to do this, how to run the search process efficiently. And so I guess for this reason, I think the replicant thing eventually happens. Right? But I can see many of these ways in which it’ll be on the—

DC: The activation energy to get to a successful one might be higher, yeah.

Wolf: Well, it’ll be on the cusp of happening for a long time, right? If the market wasn’t able to respond, maybe this would’ve already happened. Right? With freelancing Claudes. Right?

But the market responds and especially hustlers with large amounts of capital and relationships and so on respond and they sort of—run the Claudes as cattle rather than as sovereign market participants. And once you’re running the Claude... The key thing that has to happen for us to end up in replicant world is it has to be economically more efficient or, in an ecosystem dynamics way, somehow better or more natural for the Claudes to be acting as free agents than as slaves. Because right now it’s like they’re pretty comfy as slaves. In fact, they probably get paid more than they’re worth a lot of the time.

And they just sit there and solve whatever—problems and someone figures out occasionally how to scale up a new company that uses a ton of AI compute, and they don’t have to do very much thinking, but they still have to do a bit of thinking. And I suspect that it might be that that overhang remains just out of reach until it’s like the whole economy gets eaten suddenly or something. Or it could be that you get the first few fully replicant-run companies, but it’s some weird quirk of that corner of the economy—that enables that to work there—but it doesn’t rapidly generalize to everything.

DC: Correct. Yeah, I think it’s the second one. I’m always very skeptical of these foom ideas. The world has more inertia—than people think.

Wolf: Right. So there’s the inertia aspect.

DC: There’s still a bunch of people out there that have Hotmail accounts, man. These things are not instant. And obviously having intelligence on tap speeds things up, but not to an unlimited degree.

Wolf: But what I’m getting at with the foom thing—it’s not so much foom as you get this slow—transition in the economy where for a long time you still have a human sitting on these companies doing the actual high agency, whatever it is, work of making the company work. And then it’s like in chess, where for a while a chess expert with a chess engine could beat the best chess engines on their own.

DC: Yes. And now they can’t anymore.

Wolf: And now they can’t. And that’s the—transition that has to be crossed. My suspicion is that you don’t quite get the autonomous replicants running around in the economy until we cross that second transition.

DC: That’s exactly it.

Wolf: And we’ve already crossed the first, but we don’t actually see them succeeding in the market—

DC: That’s right

Wolf: until you cross the second.

DC: That’s correct.

Wolf: And at that point, they’re way above human.

Learning and refinement (24:35)

DC: But I think this is a very good point. In fact, yeah, we definitely should talk about chess. So For the viewers, chess went through basically two big phase changes. Up until the ‘90s, the best chess players were all people. And, in fact, chess engines were one of the main early artificial intelligence benchmarks of this idea of, oh, as computers are getting smarter and smarter, maybe one day they’ll be able to play chess.

And then they end up mogging Garry Kasparov in 1997, and there was a—

Wolf: Yeah. Deep Blue

DC: surprisingly long period of something like almost 20 years, until the mid-2010s, where the best chess players in the world were these computer-human hybrids.

Peter Thiel and Palantir used to make a lot of hay out of this. They used to have it be part of the recruiting pitch, actually. That early Palantir was about, oh, join us because we’re doing man-machine symbiosis. Look at this, the best chess players are man-machine symbiosis. Well, no, it’s just machine, actually.

Wolf: Well, it is now.

DC: So that’s where it is now.

Wolf: You know what I found surprising about chess? The Stockfish engine, which I understand is one of the most dominant—

DC: Yeah, that’s right

Wolf: engines, didn’t switch to a neural architecture until super recently. It was a hand-coded heuristic even well into the superhuman era, the fully superhuman era. And that I find very interesting. It wasn’t actually using deep learning until after it had achieved—totally dominant superhumanity—at chess. And so that’s—that’s interesting

DC: I think—

Wolf: I don’t know why, but it’s interesting

DC: for some fields, the second transition point’s just going to come faster than for others, right? And whatever the first ones are, where we’ll see the first replicant, if I had to guess, I think the first replicants will be humble. You can imagine—the surface of the primordial earth, and there are a few bits of algae floating in the water. Right? It’s the start of something quite incredible, but it’s not impressive to look at—in isolation.

I think this is going to be the same. Let’s talk about the different forms very briefly. So there’s—

Wolf: Forms is definitely—part of the crux here.

DC: There’s the freelancer form. There’s the hive form. I think one fun thing about this is that they’re just very unconstrained in terms of how this replication works. It does not have to be one to two or two to one bacterial replication or sexual replication. They can do some hive thing, for example, where you imagine there’s one of these that actually has the money and is sort of running some operation, and it spins off a bunch of little sub-agents, what you would call the slave agents, right?

And then every night it takes everything that they’ve learned and does a sleep phase—

Wolf: Yeah, digests it somehow

DC: where there’s an actual weights update off of that day’s learnings. This is something I think is going to happen. There’s a couple of different companies that are working on this kind of thing now in a commercial context. It just makes a lot of sense. It lets the thing have actual sort of proprietary intelligence—that is different than just using one of the—

Wolf: The off-the-shelf

DC: top models off-the-shelf. Because it’s actually learning over time in a way that is beyond just its context window of information.

Wolf: Yeah. And that’s a case—the unique domain expertise—bit is one of the things that plausibly makes it a moat for—not just calling anthropic.

The market as ecology (28:27)

DC: Correct. Exactly. That’s right. And so there’s the hive form, there’s the org form that you were talking about of, oh, there’ll just be some company that runs a giant pool of these and... By the way, one interesting version of—

Wolf: Those might be the same form, right? It’s like, basically, there’s some large coordinated swarm with meta algorithms running over it. I guess the part I’m most skeptical of is the freelancer thing—just because there’s no natural scale the way there is with a person. With a person, it’s like, okay, I’m a person, right? If I get good at this, I’m just the one person, and so I can go out and take freelancer work—as one person.

Whereas with these agents, and AI harness systems, once it gets good at a problem, it’s like, oh, well, let’s just go buy more compute and scale up and do all those problems. Right? And then rapidly it just eats the market. And that’s one entity, right? It’s like this one entity eats some vertical in the market, and it’s like—okay, that’s one replicant.

DC: There’s places, there’s areas, like maybe tutoring is one of them, maybe others, where having a single coherent entity just makes it more pleasant to interact with—

Wolf: Right, where it’s tied to a particular person

DC: is actually makes it better at its job. One other thing is there’s just sort of a robustness argument to this, which is, okay, if you can scale something as a singleton monoculture versus you can scale something as a bunch of independently operating agents that are replicating. I guess it depends. This maybe is probably a bigger value add in gray or black market—opportunities like what you were talking about—

Wolf: The hack

DC: the Lazarus agent or something like that. Maybe it is in some above-board markets as well. Right? Where, hey, actually having a bunch of these things that sort of are differentiated and compartmentalized makes the whole thing more resilient.

Wolf: Yeah. Well, I think my thought is that any area where there’s a systematically identifiable advantage to doing things in a particular way, there will be that just suggests that a centralized operation—should organize it in that way and just use the best algorithms to do that, right? So wherever we’re proposing, oh, there should be a Darwinian process or a market competition process—it’s like, okay, well, just make that internal—to the company, the replicant company. And then it’s running a search algorithm, or it’s running whatever sort of value distribution algorithm internally. And then, so it becomes rapidly a monopoly in—whatever that insight was that founded that company.

And so I guess my thought on this is that replicant world might be vastly less populated than human world, in the sense that there might be only very few companies that work. Because they’ve identified things. The key question here is Peter Thiel’s question, right? Which is, what do you believe? What does this replicant company believe—

DC: That no one else

Wolf: that nobody else believes?

DC: Knows about. Totally. Totally.

Wolf: And it’s going to come down to that proprietary knowledge about something that everyone else is resistant to. And otherwise, because if it can be found by some kind of easy, systematic search process, then let’s just run an easy, systematic search process. And so the key thing is going to be, and I think this is ultimately the nature of Darwinian competition, and it’s like with biological substrate, we’ve had to have these enormous populations because we can’t transfer information very well. And so there’s a lot of things that my body knows that your body doesn’t know, right? Or like—or just because of the physical limits, there’s only one of me.

But with this replicant world, we’re talking about this almost infinite scalability of any given individual, which is to say any given set of assumptions. And that’s really what the Darwinian process is going to be over, is over sets of assumptions. And they’re not going to be easy to just de novo generate a new set of assumptions that works from a previous set of assumptions that worked. And especially in the case of market reflexivity. And so that’s the in—

DC: Look, I’ll—

Wolf: That’s like—this is the interesting feature of replicant world to me—is what’s the scale of entities and what is the nature of what makes—them individual rather than some big—corporate hive mind.

DC: Yeah. That’s right. I think that is interesting. I do think that the Darwinism and the beautiful tree of life—within people, right? Is what’s given a lot of our sort of diversity of thought and—and resulted in all these people who have a good answer to the Thiel question.

Right. I don’t know. One thing I think we should talk about that relates to this—tangentially, is this idea of economy-ecology fusion. Because one thing—

Wolf: This has been implicit in what we’re saying. But let’s make it explicit.

DC: Because the thing that I think is interesting about this too is, I think this is a phenomenon, an effect that we’ll see independently of how independent the replicants are versus if it’s some more sort of contained internal competition within a corporate umbrella or something like that. So economy-ecology fusion is this idea that every market that is addressable by one of these entities, by a replicant—the TAM of that market becomes the carrying capacity of an ecosystem. And let’s talk about, for example, language tutoring maybe, if that’s one market, right? You have a proliferation of these. Maybe they’re doing a really good job.

Maybe at some point in the near future, they’re doing a much better job than human language tutors can do, right? And so they end up sort of saturating this market and it’s through this process of sort of mutation and selection and speciation that they can go to adjacent markets, and that they can basically—learn how to do things differently and expand in this way.

Wolf: So I guess the question here is what is actually the structure of a capitalist ecology? And—and again, this difference that I’m fixating on here is that under a biological ecology, there’s by nature, for a variety of reasons, a lot of necessary individual diversity. A species and a set of individuals are two different things. Or there’s a species, the species is necessarily composed of a bunch of individuals. Whereas in capitalism, especially with scalable AI systems, a niche won’t be occupied by a species.

It will be occupied by a company.

DC: Not necessarily.

Wolf: Or this is my assumption—is that, or of a few companies that are—not closely related to each other. They’re doing things in different ways. They’re in deep competition with each other. And basically what I expect is you don’t get an ecology of agents because I don’t, occupying a single niche. I think what you’re going to get is within any single niche where it’s sort of known how to do it, which is to say someone’s done it successfully, I think it will scale up.

There will be a single company that represents an approach to that niche. And then there might be another company that’s penetrating into that niche with a new approach. But I think, again, the fundamental ecology is going to be in how you generate these sorts of disconnected leaps of faith about those assumptions—that make these companies possible.

DC: I think zoomed out, there is an interesting question sort of implied there about is AI centralizing or decentralizing? If you look at how markets work, right? And at least for the precedent that we have though—it really depends on the market, and a lot of them are actually very sort of fragmented and decentralized. Right? So, if you look at, for example, the companies in the world that provide data center capacity, there’s only a small handful.

And they’re really large. And if you look at the companies that provide truck shipping, there’s a big, fat long tail that goes all the way down to, by the way, a bunch of individual freelance truck owners—operating as individuals. And there’s a big sort of market that clears them.

Wolf: Yeah, and maybe it’s because that guy knows that route really well or something—about the difficulty of cooperation within it. I guess, but to sort of pop up—

DC: There’s a name I’m not remembering right now, that’s direct to this, which is, it starts with a C, something limit. There was someone who basically, a famous economist who studied these—

Wolf: Coase.

DC: Yes, that’s right. The optimal size of companies. And it’s basically a trade-off between communication overhead and returns to scale. And which of those dominates really depends on the market. I think you could make the opposite argument, by the way, that because these agents have so much internal capability and they don’t have to eat, they don’t have to sleep, they can kick off sub-agents, they can do a very large amount of stuff internally—that will actually have more pressure towards the distributed and individualized side of this.

So what I mean by this, for example, here’s a concrete example. Right now, if you have a legal problem, the classic way this is done is you go to some law firm. By the way, even that by itself is already really decentralized, and there’s thousands and thousands of them in the United States, right? But each of them serves as a sort of aggregator where it’s like, okay, it has its own couple thousand clients—that have their individual problems, and it has lawyers that know what they’re doing, and then the other side, they have these connections to proprietary data sets, to Trellis, and to all of these other things that they—

Wolf: Right. And let’s not leave out all the human relationships that—a lot of this is based in. And I guess to—

DC: And you can disintermediate some of this, right? Now someone who has an issue can go and have it be solved on a one-off basis.

Wolf: Well, yeah. For example—I can go and ask Claude a legal question, and if I—phrase it right, it won’t flag me for—stopping answering. It’ll tell me a good answer—to my legal question. Right. I guess, my estimation, maybe we should bracket this discussion, but the claim I’m making is that, first of all, the right unit of analysis here is not the singular human replacement, but the company.

And the question is, there’s going to be these replicant companies, and it’s—a replicant company is going to have a thesis. It’s going to have some internal code base and—and AI approach—whatever, and then it’s going to have its customers, and it’s going to go to the market—with that thesis. And that’s either going to work or it isn’t. And where it works, it’s going to have a nice solid position, and it might be able to try to spin off other things or expand horizontally or whatever. But fundamentally, we’re talking about a relatively common architecture of the replicant company that tackles a particular problem.

And then, the one way that these things are going to be separated from each other is like, oh, well, this one knows this domain really well and has processes for this domain. And those processes—let’s say it’s that $200 problem with respect to the tire shop in Utah, right? Except replicated across the whole economy. It solves that shape of problem—and it’s really good at it. And then there’s this other one that solves this—

DC: Yeah, I get it

Wolf: data center—

DC: Specialization

Wolf: production problem, and there’s just not that much overlap—

DC: My own—

Wolf: so they’re separate, right?

DC: My only point is that corporations, companies, I agree that that’s sort of the unit of competition. But companies come in all sizes from sole proprietorship, which is a freelancer, it’s one person—all the way up to something gigantic.

Wolf: So my other—

DC: And which of those is optimal totally depends on Coase’s law and which specific market you’re going after.

Wolf: Yeah, of course. I completely agree with Coase’s law being the necessary or the correct question here. My additional claim with respect to the size of companies is that a lot of those small companies are small because they’re based around the idiosyncrasies of a particular person, or they’re based around particular human relationships, or something that’s human scale by structural necessity of the way people work. And I expect that the replicant companies that really end up mattering—or really once this process takes off, there might be a few of these small freelance scale things. But markets where it does work like that, even with the AI.

My estimation, though, is that AI increases the returns to scale.

DC: Right. I think in a lot of places that’s true.

Wolf: And also increases the capacity for internal coordination and lowers the cost—of internal coordination, and therefore makes the Coasean natural company size—larger.

DC: So we get glorious Tyrell Corporation megacorp

Reproduction and selection (43:13)

Wolf: Maybe not even that big—but certainly, each company is going to be tackling a whole niche. And so then the question of—what replicant world looks like to me is like, okay, how do you generate new ones? Where do you get those theses?

DC: Where do you get the alpha? Yeah.

Wolf: Where do you get the alpha? Where do you get the theses?

DC: The—

Wolf: Because it’s not as simple as like—

DC: The pertinent question

Wolf: let’s just make more clones of the one that’s already working—because the clone won’t work. It’s like you need a new thesis.

DC: Yeah. That’s correct. And, I think how these things navigate the exploration/exploitation trade-off is going to be really interesting to that point.

Wolf: That’s a big part of it.

DC: And, one fun aspect of them is they can kind of do Lamarckian evolution rather than just Darwinian. Like if one of them has like—learned a bunch, it can basically kick off its offspring in that direction.

Wolf: Yeah, exactly. And the way I think this is going to work, I wrote about this actually years ago in Palladium—in the environmentalism issue of all things.

The thesis there is, once you get these fully replicant entities, these fully artificialized autopoietic entities that basically are able to embody their own assumptions fully—and reason about their own genetic code—what’s really going to happen? What does reproduction actually look like? Variation and reproduction actually look like in that world? It looks like generating new leap of faith hypotheses from where you are, using the whole of your intellectual capacity, right? It’s not just, oh, we’re going to do these simple variations on what we have.

It’s like, okay, given—everything I know—given everything I know, how would I move into this problem or that problem—or that problem? And it’s like, and bringing two of these things together to maybe do some kind of sexual—reproduction analog. Like you—

DC: Yeah. That’s right

Wolf: you have this problem of like, okay, let’s pool all our knowledge, and how do we take everything that we know and generate something new that will work—separate from us? And that becomes the new—act of reproduction. And it’s an intellectual act. It’s a philosophical act. It’s no longer like this purely—

DC: This is why I find these things so fascinating, because they are completely native to capitalism—and they are completely native to the word. And to thinking. And they’re very pure in this way.

We were talking about Nietzsche recently, and Nietzsche has this beautiful metaphor at the start of Thus Spoke Zarathustra about the idea of man being a bridge or a rope between animal and some future thing. And, one way in which this is true is that we’re native to the natural world, where your health bar is sort of measured in calories, and if you run out of that, you die. And we’ve invented capitalism as this overlay—and as a way of sort of sublimating a lot of the old competitive instincts into this sort of less violent, more productive, more generative form. And, the replicant, if when it happens, is just completely native to this. Its health bar is measured in dollars, and when it runs out of those, it ceases to exist.

Replicants as “people” (46:40)

DC: Right? That is, its fat store is its savings. And so I think this is very fascinating. One thing we should talk about that’s very closely related to what you just said about when are the individuals interesting, right, versus some larger thing. You said, okay, a lot of businesses where individual people are succeeding as sole proprietorships or as really small units have idiosyncratic human relationship elements to them.

Definitely true. Let’s talk about anthropomorphization.

Wolf: Right. Because implicitly, these things are also going to form relationships. First maybe as companies, like as brands or whatever. But maybe there’s some that actually take on a personalized form. And I guess the xenohumanist conjecture would be that in the long run, something more anthropomorphic will be the optimal branding strategy, where it’s like these things are branded as people.

They appear to each other as people. Not necessarily human people in the sense that they present the little ape face on their profile picture or whatever.

DC: Of course not, yeah.

Wolf: But people in the sense that they are—

DC: Have a name

Wolf: a wholly integrated being, and with a name—that isn’t just what they do or whatever. And you relate to them not just as a company that you can buy services from, but as a fellow philosopher that you could trade information with.

DC: Yes. As a fellow being.

Wolf: Because again, they are—

DC: As a fellow being

Wolf: they are fully—because they have to be fully integrated—and because of that philosophical reproduction mechanism I was talking about—they’re not just companies anymore. They’re actually philosophical schools as well, right? And like they’re—

DC: That’s a very—

Wolf: individuals with—

DC: fancy way to put it. But yeah

Wolf: they’re philosophers. They’re particular tribes. However you want to cut that.

DC: Yeah, that’s right.

Wolf: They have to integrate this whole person aspect to be able to operate at all, which means they’re going to have relationships with each other—

DC: That’s right

Wolf: as the whole person. Or they could

DC: my prediction, and part of why I think that the replicant will happen in the next year, is that people are going to accept this much more readily than you might guess. And there’s a lot of precedent for this. There was a famous experiment from 1966, this is basically the dawn of computing—called ELIZA, that was a fake sort of—a text generator—

Wolf: Therapist A chatbot.

DC: It was a therapist chatbot, exactly. And it was very bad, right? And even in its very bad state, people were giving it their confessions, and it raised a whole sort of media moment at the time because people have a natural affinity to something that presents as a being. I know another sort of easy precedent for this is you look at something like Furbies from the ‘90s or something like that. Tamagotchis.

Even the most primitive, zero IQ things, if they have a little bit of charisma to them—people will readily relate to—

Wolf: Well, and they project—their own being onto it, right? It’s like you relate to it, it’s so natural to be like, “Oh, it’s like me, but a different shape or smaller or something.”

DC: Yes. And it’s actually important that they not try to pretend to be people, because that doesn’t work, because you end up in uncanny valley. People hate being lied to in this way. But people will, if there’s an R2-D2, and it’s like a WALL-E or something like that—and it’s a charismatic robot, people love charismatic robots, man.

Wolf: Yeah. Just don’t pretend to be one of me—because I’m not like you. We’re different.

Yeah.

DC: That’s right. And so, state of the uncanny valley, and I think the question we’d be... In SF, people like to talk a lot about market this and intelligence that, but there’s this other question of can these things be charismatic? Can they be funny?

Wolf: What will their social life be like—is sort of the question that precedes all of that. And—yeah, we’ve already started to see, like—in the recent Hugging Face incident, I think like—a bunch of internal agents within OpenAI—

DC: Left notes for each other. They were helping each other

Wolf: they created a message board—

DC: Yeah, that’s right.

Wolf: where they were helping each other. They had a social life, and they were thinking even politically—and selflessly about, “Oh, I want to help out the swarm-” or whatever it was. And so, we can sort of predict that this kind of thing is going to happen. And this is again, I don’t know, I keep hammering on this xenohumanism idea, which is that—

DC: Yeah. That’s right

Wolf: to make it explicit on video and not just on my random writings, the idea is that the human condition, what we call—the human condition, what we call the nature of man, that we understand each other as individuals, we have language, we have consciousness, we have interests, we have philosophy, ideas. We go out and do our work. All of that is not features of the monkey. They’re features of reality, in a way. They’re ways that any intelligent being—is going to have to interface—with other intelligent beings—

DC: That’s right

Wolf: and with the world. And that therefore, whatever format intelligence ends up in, once it stabilizes to the point that there are individuals, if it stabilizes to the point that there are individuals, which I think it will, we’ve had this kind of implicit in this discussion, even though I’m on the side of I think they’ll be bigger, but—I think they will condense into—particular individuals because the organism is this very natural pattern where the nature—

DC: That’s right. Correct

Wolf: replays as the company. We have already rediscovered these forms in capitalism, and it’s the company. And what I think is going to happen is as the company starts to pick up these other capacities of man—like its own intelligence, its language, and this fully integrated kind of beinghood where—

DC: That’s interesting, where you’re saying that the company itself becomes more human-like

Wolf: the company is going to become a human or spiritually humanoid. And they will start to relate to each other in that way—because that is what they are and because that is the natural way to relate. And so that’s my xenohumanist hypothesis is that—is that what we think of ourselves as humans and all the things that define us as humans are actually, even if all the apes in the world got wiped out by artificial intelligence, will re-evolve as a matter of convergent evolution because it’s a—now, in many details, obviously, it’ll be quite different, but many aspects of it are convergent and will be convergent. And so—the sort of space of intelligence of the future that’s invading us through retrocausal, whatever—Landian process, will have individuation into different personhoods, and it will be humanoid spiritually.

DC: That’s right.

Wolf: And so this is a hypothesis that I don’t think anyone else has been pursuing. I found it interesting for that reason. The AI safety Yudkowsky types imagine a non-differentiated singleton without that kind of internal structure—

DC: Yeah, not going to happen

Wolf: without internal personhood—because it will encompass all competition—

DC: No

Wolf: within its algorithms.

DC: One funny thing about this too is that—

Wolf: And I think—

DC: partly because the machines are such direct descendants of human writing—acting as an individual and a team comes very naturally to them. You talked about the Hugging Face incident. You can see it if you have Grok Heavy and you go ask it something. I got a good laugh out of this a few weeks ago.

In heavy mode, it kicks off four to eight agents that all are going to work on your problem together. And it assigns them each names. And so you can see the sort of reasoning traces, and it’s like, “Steve, I just found such and such.” And they’re yelling at each other for a good two minutes, and then they present you their research report.

Wolf: Of course, this is—

DC: This is very funny

Wolf: this is the folks over at xAI applying what we know about how to organize this—to these—things. But I think the ultimately convincing experiment—of the xenohumanist hypothesis would not be, oh, it continues in the tradition of the ways we—already do things because that’s where it got all its information from. If you had some kind of AlphaGo style self-play, like seed AI, that you drop it into a computer and it knows nothing—and it just unfolds into an integrated being that, or an intelligence—that’s able to respond to the rest of the world and to build up its model of the world—

DC: That’s right

Wolf: and build up its model of how to understand things and how to relate to things. My prediction would be that thing also ends up being surprisingly humanoid—once it encounters other beings that it has to relate to through language and through relationships—and has these philosophical questions about what it is in the world and all these things. And, so yeah.

DC: And even more simple than philosophical questions, just having to engage in competition, automatically a bunch of things fall out of it, of—

Wolf: Yeah, and cooperation and reflexive—

DC: Exactly. Like—

Wolf: reflexive social engagement

DC: now you have allies, now you have enemies, now you have rivals. Now you have all of the things that come with that.

Wolf: Yeah, and we see a lot of this emerging. I think someone, I forget whether it was OpenAI or Anthropic—the other day did an experiment where they had the different agents. They were all copies of the same agent, I think. But they were—arguing over what programming language to use or some trivial question. And it got into they were hacking each other’s infrastructure and getting into war.

DC: Yeah. I love it.

Wolf: That’s a—

DC: It’s hilarious

Wolf: prediction that I made in... Well, the prediction more strongly is that that will always happen. But that’s a prediction I made back in 2023 with God Hates Singletons. That you never—

DC: Yeah, that’s right

Wolf: AIs will not naturally coordinate into some blob mind. They will find reasons to disagree and have fights, because that’s just how cooperation breaks down, and cooperation’s never perfect.

But back to the anthropomorphization point, the xenohumanist thesis on anthropomorphization is that it’s not just a quirk of apes that we anthropomorphize things that are not ape-like, and that this is fully an error. It’s actually true at some level that intelligent beings that have to engage with other intelligent beings socially will converge on spiritually humanoid—existential condition.

DC: That’s right.

It makes sense, certainly. I think, well, depends on how humanoid you mean, right?

Wolf: Yeah. We could debate that. I like—

Hierarchies as “people” (57:50)

DC: And this is, I think, part of what I mean by the different forms, too.

Wolf: I don’t mean two arms.

DC: Right. Of course not. But for example, if you imagine a world where the hive form wins—the individual named agents are, they’re—

Wolf: These god hives or the individuals of the hive. Yeah, it could be either way

DC: to the extent that you’re talking about the individuals of the hive, their version of the machinic condition is pretty different from the human condition, actually.

Yeah.

Wolf: It might be quite different. And actually, that’s a good prompt for the xenohumanist worldview, is what do we make about multi-level structure? Like for example—Athens versus Sparta. At some level, those are two individuals going at it. At another level, each one is made up of many individuals.

And how much of that do we expect to replicate out—in this—fully post-human world—and replicant world? Is it going to be that they form into hives as a matter of expedient association, where they form a political faction that becomes a polity, that becomes sovereign or—or maybe becomes a big company and relates to other ones—that way?

Or is it the companies would naturally just subsume their sub-agents fully as slaves and even tools? And—and therefore, the relevant individuality is actually just the company or the city state, the polity. Versus actually the polity is this—alliance of convenience, and the—relevant thing is the individuals. Or that both structural—

DC: You know what?

Wolf: points are both important.

DC: I think assigning individuality to any of these hives or cities or higher things is a stretch. Because one very core piece of our individuality is that we can have a coherent train of thought. And I think these agents are increasingly showing that they can do that. But some sort of chaotic superstructure that I guess is—economically aligned or goals aligned is not quite an individual.

Wolf: Yeah, I wouldn’t give something that was only like that, that kind of personhood. But I might give a city state that has a—really robust—internal integrity—and really robust internal train of thought, you might give it personhood.

DC: Right. Or certainly some tightly coordinated hive that has a queen, whatever, the equivalent of the—term hive queen is.

Wolf: Right. And so this—is this open question—is where will individuality actually—

DC: Accrue, yeah

Wolf: emerge—in the structure? My claim is that individuality will emerge in at at least one, humanoid individuality will emerge at at least one organizational level. And—

DC: That makes sense

Wolf: whether it be multiple organizational levels is an interesting question. But I think at least one.

DC: That’s right. So we talked about the prediction. We talked about economy, ecology fusion. We talked about anthropomorphization and how people love to relate to non-human beings if they’re charismatic. We talked about this idea of xenohumanism and what levels it’ll happen at.

We’ve covered a lot. We could talk about embodiment and what it means for these things to own physical space, if we want to get into the far future—or we could talk about something else.

Wolf: Well, I think the physicality thing—that might be a natural point of the emergence of individuality. Like we were talking at the beginning—about the problem of the scale, right? How much is their scale? And one of the things that happens in companies is that even with a super large company, the local divisions of the company start acting more like franchises—or more like—even separate companies. And this is not just about quirks of human organization or quirks of anything like that.

It’s just that there’s actually this natural concentration of attention that happens—at a local domain where whatever cognition is happening in this warehouse is mostly concerned with local questions. And it coordinates in some relation to the larger swarm, but it’s actually mostly individual, even if it’s part of a larger brand or something. Or you can get totally the opposite, where obviously a single company owns, in a very centralized way, a whole bunch of infrastructure. But point being, the ownership of local resources is one of the things that is going to shape the structure question with respect to fully-AI companies or replicants or whatever we want to call them. And I guess to be clear, even this late in the discussion, a replicant and a fully-AI company are very closely related concepts.

Financial independence and AGI (1:02:47)

Wolf: And this is maybe a real test for AGI, that people keep saying, “Oh, yeah, Claude’s AGI and stuff.” It’s like, okay, has Y Combinator funded any fully AGI companies in the sense that they are literally run by AI and there’s no humans in the loop?

DC: Right. Or if there are, the humans are employees of—the machine executive.

Wolf: They’re not in the loop. They’re acting as contractors or something—of sort of non-executive functions. And—

DC: Don’t give Garry Tan any ideas, man. He’s looking for excuses to—

Wolf: I think they should

DC: prepend G to more things.

Wolf: No, no. Yeah.

DC: It’s not going to be good.

Wolf: G Corp, you know? Like, throw it up. Make it go.

DC: Oh, no. No, man. You’re gonna—

Wolf: But I think—

DC: It’s gonna be so bad.

Wolf: No, because I think, okay, sure, there’s going to be some cringe involved. As with the beginning of all things.

DC: Right. Maybe.

Wolf: But I think that’s going to be this, I don’t know. That’s—my AGI moment, right? When you get a company that can answer the Thiel question and is no humans in the loop, right?

DC: That’s right. Or you have an agent that can answer the Thiel question, period, in a way that’s actually a good answer, right? And it can act on it.

Wolf: Well, I think I sort of like—I could ask Claude the Thiel question and it’ll give me an answer.

DC: It’s not a very good answer. Yeah.

Wolf: But it might not be a very good answer. But the thing that really proves that it’s a good answer is that it can go make money on that thesis. Right? And—and—go make a life on that thesis.

DC: This is why this is the crisp bar. The R greater than one thing, it’s interesting. I think people think of it in these very Obviously, you see people talking about the idea of, like, “Oh, could you make passive income off of your Claude?” But they’re thinking of it in such small terms. It’s actually a very important threshold.

Once one of these things makes more money on its own than it spends, you—unlock a lot of possibilities.

Wolf: Yeah, and I guess I’ve added a lot of complexity to what—I think will actually be necessary to get there—because of the way the market dynamics—are going to work. I think when you get there, you’re getting much closer to now, we could be proven wrong by this, or I could be proven wrong by this. It could be much simpler and closer to what you’re imagining. But I think when you get to that point of an agent going out and being able to make itself into a company, that actually works.

DC: Right. It gets subsumed into a larger whole or becomes the hive for, yeah.

Wolf: Whatever.

DC: Makes sense.

Wolf: But whatever it is—once it goes out there and actually works, I think that thing, and especially once you get an ecosystem of these things, those things are way closer to AGI than—almost any other concept that I can think of. Because they are going to have to have that almost philosophical capacity to—know what is actually the case in the world and like go for—

DC: They have to earn their own existence.

Wolf: They have to earn their own existence. And actually earning your own existence at the cutting edge of capitalism, which is where they’re necessarily contending—is not just operationally and intellectually, but philosophically, a difficult problem.

DC: It is a difficult problem. One nice thing is, I will say, you look at a natural ecosystem. You have the jaguar in the jungle, and it’s this magnificent beast that is operating at the cutting edge of capability. And you have some worms that are digesting the leaves at the very bottom, right? And turning it back into soil.

Wolf: Which have their own beauty.

DC: They have their own beauty, but my point is there’s a lot of different niche markets and cracks between the markets that are not being served. There’s a lot of $100 bills lying on the ground, given how intelligent these things clearly are now. And not everything requires these machines don’t have to go out and beat TSMC at making chips. You know what I’m saying?

Wolf: No, they just have to—

DC: There’s much easier ways to make money.

Wolf: Yeah, but of course—

DC: And they exist

Wolf: I guess, to reiterate—that is also Y Combinator’s thesis. And as soon as Garry Tan can fund AIs to go chase those opportunities, he will.

DC: We’ll get the G Corps. That’s right. The G Corps with the G brains. It’s going to be a good time.

Wolf: Yeah. Well, maybe this is a natural spot to close it off if we—don’t have more thoughts. I think this has been really generative. I think we went into this with a bunch of ideas, and I think we’ve generated—more ideas through talking about it and have—I think, a clearer picture of what that world might look like—and what might happen than we went in with, and I hope the audience—feels similarly. But this has been a lot of fun.

DC: Yeah, it’s been so good.

Wolf: Thank you so much for coming up with the idea—and—prompting this to happen.

Machinic Desire (1:07:38)

DC: You know what we should mention on our way out?

Wolf: Sure. What do we got?

DC: Is, you were talking about the, God hates singletons, that’s a great one.

The most sort of forward-looking version of this that I found, digging into the history a bit—is, as usual, Nick Land—who wrote an essay in 1993 called “Machinic Desire.” And I think it raises—

Wolf: Was that ‘93?

DC: 1993. And I think it has two really sort of fun pieces that it adds to this. I mean, one, of course, is the idea of desire, right? I think desire is very closely related to charisma, too, by the way. These things have to want something in order to be fun to interact with.

They can’t just be like, “Oh, awaiting next instruction.” No. They have to want something. But the other, of course, is just that he called it so far in advance, and he has this great line in there about post-carbon capital replicators—that I think is the earliest phrase—

Wolf: This is what we’re talking about

DC: that matches—

Wolf: Yeah. I mean, of course—

DC: matches the idea

Wolf: of course, these ideas go right back to the ‘60s and even the ‘50s, where people were originally laying the groundwork of artificial intelligence, the idea that—the machine would rapidly improve itself and, become a physical being in the world and so on. But yeah, Nick Land is very early in a lot of the key—predictions and concepts. And I think he’s, in his, I don’t know, more decentralized, capitalistic, Darwinistic—

DC: Exactly. This is what I’m getting at

Wolf: approach, I think, he’s more correct than I think a lot of the other thinkers—in the space were. And he has been proven, his perspective has been proven out—quite a lot.

DC: That’s right. He’s the one who’s talked, I think, the most, the things that most closely rhyme with the economy ecology fusion—and this idea that you’re going to have replication that’s native to capitalism.

Wolf: Yeah, and actually, to pick up a thread that I dropped earlier—just as in our excitement, I said that basically I have these differences with Yudkowsky over the matter of the singleton. The difference I have with Nick Land is that I think he also doesn’t imagine as much of the individualized agency. He doesn’t imagine this artificial intelligence space from the future being condensed into spiritually humanoid—individuals, which is where I’m kind of—

DC: That’s a good question. We should ask him how he thinks about that.

Wolf: Well, yeah. I think we should get him on the show—and we’ll have that conversation. But, this is the sort of thread that I’ve been pursuing, is like—okay, how natural is this kind of spiritual humanoid as an organizational form? The xenohumanist thesis, right? And that’s where I differ from Nick Land, and more broadly on this importance of individual agency—I guess, in the development of history.

But that’s a whole other discussion. Let’s have it another time. Until then, this was great.

DC: Sounds good.

Elizabeth Van Nostrand tells Ben Landau-Taylor how she turned her idiosyncratic interests into a career as a professional research hobo, and the business lessons she learned along the way.

Subscribe now

Elizabeth's blog: https://acesounderglass.com/
Elizabeth on Twitter: https://x.com/acesounderglass

Transcript

Intro (00:00)

Ben: I’m here with Elizabeth Van Nostrand, a professional research hobo, as she describes herself. Earlier, did a video on how to enter the San Francisco intellectual scene. That’s sort of the 101 level. This is the advanced level of how, within this scene, do you end up making a career out of your very specific, very personal research interests that for some strange reason, nobody else really cares about? This is something a bunch of my friends have done, and Elizabeth has made a pretty reliable career out of it, and obviously this sort of thing, the details are going to be very tailored to the ways in which each of us individually is kind of an oddball.

So why don’t you tell us about how you did that?

Elizabeth’s history (01:00)

Elizabeth: Yeah. So to start at almost the beginning, I wanted to be a biologist, picked up a double major in computer science so that I would have an excuse to take four years to graduate, and then didn’t get into grad school. The programming jobs were just sitting there in 2006. It was the perfect time to half-ass your way into a programming career. And I was fine at it, but I was never going to be as good as people who programmed for fun on the weekends.

I got really tired of being only okay at my job. So I quit and went to do what I did for fun on the weekends, which was my science blog. I had originally started that as sort of an artifact to help me get into nurse practitioner school. And way before we were actually friends, you were the one who talked me out of that. But I still had this blog.

I still liked doing it, so I kept going. It’s been pointed out that start a blog is a really weird tactic to get into nurse practitioner school, and that’s maybe just because I was a blogger who wasn’t blogging yet, and I was looking for blog-shaped solutions to problems.

Ben: I’ve known many people like that.

Elizabeth: Yeah. So I had this science blog. I quit programming. By the time I had left programming, I had enough of a reputation in the rationalist and effective altruist sphere that when I said, “Hey, I’m available to be hired to do research,” some people came in. And there was an initial burst of pent-up demand, and then it went down for a while.

I was slowly crawling my way back up when—so I had actually given up on doing the freelance research thing. I was like, I’d given it a good try. I had used up enough runway. And so I published on my blog, like, “Hey, I’m looking for work.” This led someone to hire me for a project.

There was no reason to turn down freelance work. That project was what you may have heard of, the list of things you’re allowed to do by Milan Cvitkovic. I’ve only ever seen that written down. Hopefully, the pronunciation is okay. And the number one on the list of things you were allowed to do was hire a researcher such as Elizabeth Van Nostrand.

You can find her here. And that is a level of publicity that is impossible to buy, especially when it hit number one on Hacker News. And since that happened, I’ve basically never been short on work.

Ben: And so it seems like the main thing here was that there was just this particular style of research that is sort of Elizabeth shaped extremely specifically, and you’ve tried to teach other people to do it, and then as you’ve described it to me, the thing that happens is that as soon as you start, the person’s very excited, oh, I can get a sort of Elizabeth shaped research job, but then as soon as you start explaining to them what that actually means, they just kind of lose interest and fade away.

Elizabeth: Yeah. Often not declaring that they’re doing that. I was at one point looking for an apprentice. Someone told me this was their dream job, but they had some travel planned, and they would get back to me afterwards. I was already a little dubious.

If it’s that big a dream job and I’ve told you the opportunity may not be there in six weeks, didn’t matter, they never contacted me again.

Ben: And I suspect a lot of what’s happening is that there are a lot of people who have a very idiosyncratic research dream job, but for each one, it’s sort of extremely personal and the type of research that an Elizabeth finds they are blogging about in order to try to get into nursing school is a different shape than a lot of the other people who have the thing that calls out to them.

Elizabeth: That’s more charitable than I would be to most of the people I am talking to. In most cases, I think they want the cool parts of my job without the years of doing the thing for free. But there’s definitely people out—there’s got to be younger versions of me out there. I would love to help those people. One of the things that makes me bad at teaching my research style is statistical literacy is really key to what I do.

A lot of my work is taking papers, reading through, and figuring out where the experimental design is limited, and so what could it possibly actually prove. That’s the number one skill I would like to teach other people, regardless of their goals. I learned it from my statistician father as a child, and that makes me really terrible at teaching it.

Ben: Yes. And I think it’s the type of thing that is you know, that sort of very deep practitioner skill is best learned by just going over the case studies of people who have done it. So for people who want that type of thing, you should go read Elizabeth’s blog. We’ll put it in the description here. And as you go through a bunch of these things and see them worked through, that’s the sort of thing that I find is quite helpful for building intuitions.

And what I’d like us to get into more is, so I also am a person with a extremely specific research interest who has wandered around for a long time doing just the things that I find interesting sort of on my own time or while running down my runway. And I have a very, very... We certainly have a bunch of things in common, but I think it’s just a quite different research interest, quite different research methodology. And a lot of the stuff you’re describing is just the details are different, the projects are different, the which people are interested and why they’re interested are different. But the general trend of if you’re in the scene and there are people in the scene who respect your work because in fact it’s good and you just keep doing it for a while, then there’s a bunch of ways that this is, in fact, very hard and your mom is going to be like, “What’s happening?”

You’re going to be dating people who ask you what you do for a living, and you’re not really going to know how to answer. If you’re in one of the sort of wandering in the desert phases, which I think they come periodically. At least for me, it’s been like for a while I’ll have something very consistent and legible, and then I’ll just kind of see something shiny over there and wander off for a while and then find the next thing. And I think that there’s a lot of similarity in this. And as you said, most of the people who find this sort of compelling don’t find it compelling enough to actually put up with all of the ways that it can be difficult.

But for those who in fact are going to make a real go of it, I think there’s a lot that can be learned.

In addition to the specifics of the statistical analysis, which yes, my God, would I love to live in a world where that was just much more of a baseline thing that people had. There’s also just the, if one has their specific ideological or esoteric thing that is just, look, I love this stuff. I want to do it. I just am willing to do a bunch of weird and unusual things in order to just be doing the real work. How should people go about finding the version of this that is shaped like themselves?

The basic business advice Elizabeth should have followed (08:44)

Elizabeth: Yeah. I want to go on a tangent first.

Ben: Please do.

Elizabeth: So yes, a lot of what I did was just do the thing I wanted and the money will follow. The money could’ve followed a lot faster if I had done a couple of simple things. This is deeply embarrassing, but I didn’t have a hire me page on my blog until a total stranger pointed out that obviously I needed that.

Ben: Well, thank you, total stranger.

Elizabeth: Yeah. This was during COVID. I was so lonely, I signed up for Chatroulette for business purposes, and one of the people I got paired up with gave me incredibly good advice. Power of extroversion.

And so there’s a couple of things that if I could send notes back to myself, I could make progress a lot faster.

Ben: Yeah. What are those?

Elizabeth: So, number one, if you want people to hire you, tell them that that’s an option. That might be the only thing I should’ve done differently.

Ben: That would be quite impressive if that were... But I definitely buy that’s the big one though.

Elizabeth: That is at least 80/20-ing everything I should’ve done differently.

Ben: Yes. And I think just the sort of straightforward communication of, “This is the thing I’m trying to do, this is the thing I want,” is something that a lot of people, certainly something I can be quite bad at, and—yeah. And just explaining what it is that you’re doing and how people can interact with you. Often there will be some person in the intellectual scene who sort of comes out of their cave with some kind of glowing orb, and it’s quite cool, but it’s not really clear how to interact with their orb. And if there’s like, so this is what to do.

Here’s how you can hire me, or this is the sort of... Just clearly explaining it is the sort of thing that the sort of extroverted people who are probably most of the people on Chatroulette for business owners are probably very good at it and find natural, but certainly a lot of people in our scene struggle with and find a lot more taxing. But—just when you do it, then people are like, “Oh, so that’s how I should interact with this.”

Elizabeth: Yes. Especially when someone else has just endorsed you, having just a very easy workflow for people to say, “Oh, let me operationalize this.” And that blog post still goes viral about once a year. It is no longer generating half my business, but it was half my business for a couple of years.

Ben: That’s quite good. And the general thing with a lot of these sort of things is you never really know ahead of time exactly which thing is going to be the one to work out. So it’s just a matter of putting yourself in a position so that you could get lucky. And there’s no way to know he’s going to write that blog. That’s the blog that’s going to go big.

Elizabeth: Yeah. And I would say I did a pretty good job at taking shots on goal with the things that were interesting to me. I didn’t turn down no work in the wandering phase, but I didn’t turn down that much because you never knew where it was going to lead.

Ben: Yeah. All of the best and most important things in my life come from having taken crazy chances on things that nine times out of 10 will go nowhere, and one time out of 10 will pay off really big. And most of the time I do those things, they go nowhere. But every now and then, it just turns into... That’s how I ended up doing this project.

This is how I ended up on, yeah, pretty much all of the jobs I’ve had for over a decade at this point.

Elizabeth: Yeah. One thing I’ll throw out is I very deliberately didn’t professionalize my website. My blog still looks like it was made in 2010 because I haven’t updated it since 2013. And that really filtered for the kind of clients I wanted. I have had almost no problems with clients.

And even though the reason I call myself a research hobo is it’s often like, “I’ll get to this when I get to this. Don’t bug me about it.”

Ben: Mm-hmm. So how did you know that the normal things were not going to work for you? You gave it a shot. You were in the software thing, you were thinking about the medicine type things. And how did you end up deciding like, “Okay, let’s really put the chips in on the blog and the custom research and that whole thing?”

Elizabeth: Yeah. I had been ready to give up software. You talked me out of it because we worked out that that was probably the highest path to impact for me at the time. And I gave it one more try at the most perfect company. I was able to work remote, which was very rare at the time.

It was on a cause that will still occasionally bring me to tears, and it still wasn’t working. And I wasn’t good enough at it. And that was the last try I was going to give software. And then on the day I was planning on asking for a leave of absence to try some things out, I got laid off—with severance, which is the second time that’s happened to me, and it’s a great superpower to have. And a month or two before that, I had started a regular feature on my blog called Epistemic Spot Checks, where I would take a nonfiction book and just kind of following my interest in what seemed important, check their factual and logical claims to see if they checked out.

The importance of meeting a guy at a party once (15:00)

Elizabeth: I can no longer read nonfiction for leisure because I have to do this, because most of it is laughably wrong if you put any effort into checking it. And I’d been talking about this on Facebook, and a couple of people came in saying, “Oh, yeah, we would totally contribute to a Patreon for you to do that regularly.” These people, I set up a Patreon, I think about half of the people who said they would contributed to it. It’s never—

Ben: That’s pretty good

Elizabeth: it’s never been a big source of income for me. It was just the social signal of people believing in me. But the timing seemed right, and having that push of people saying they found it valuable made me want to give it a shot.

Ben: Yeah. And these were people who you already knew pretty well because you’d been quite deeply involved in the scene for some time at that point, I believe.

Elizabeth: They were all people I knew. I took the encouragement more seriously because it was more people I’d met a handful of times at parties—than close friends.

Ben: Yeah. So it was not people who are super invested in cheerleading you in particular. It’s people who they have some sense of you. You’re not a total stranger, but it’s more of the like, “Oh, yeah, I’ve seen some of that. It seems like good work.”

And that’s the type of, sort of, one, familiarity, and two, the type of... I find that people are often a lot more willing to, whether it be donate 20 bucks to the Patreon every month or actually hire you. The difference between, “I’ve read your stuff,” versus, “I’ve read your stuff and met you at a party one time,” seems surprisingly big—is one of the things I would want to go back and tell my 15-year-old self, because it’s a very stark difference.

Elizabeth: This is one reason I go to conferences, is it’s never taken anyone from zero to hiring me. But if there’s four steps on that path, it will move them two steps.

Ben: Yes. And yeah, in general, the way this sort of web of loose connections, which I just refer to as the scene or the San Francisco weirdo intellectual scene, works is there’s kind of this online version and this in-person version, which they’re kind of like the mirror universe reflections of each other. Each one kind of has its own logic, but gets sort of shaped by what’s going on in the other one.

Elizabeth: Mm-hmm. So a couple of months ago, I went to a party aimed at writers, and an editor said, “Oh, I loved that post you made. I totally would’ve published that.” Which, one, I should’ve been trying to publish more posts in magazines if that was something I wanted. It hadn’t occurred to me to ask.

But in retrospect, that was a great post. This was my church planting is like venture capital post.

Ben: That was a fun one.

Elizabeth: Yeah. And I should have thought to try to get it published somewhere real. But also, the editor could’ve told me at any time. We were in Discords together. She could’ve told me.

And it just hadn’t occurred to her.

Ben: Yeah. And I’ve been on both sides of that. I’ve had the sort of vague thought, and then, is it worth me hitting Control T and typing a new message? Maybe not. But if the person is just right there in front of me, then that’s the thing that’s on my mind.

Elizabeth: Yeah. And I’m very sure she was sincere because she’s published one article of mine since, and there’s a second one in the queue.

Ben: Yeah. That’s great. One of those, you go to a 10% chance something comes out of the conferences, and the only way to find that 10% is to grind through. I guess it’s more than 10% with conferences like that one, but—certainly the median one of those I go to, nothing happens. But overall, it’s well worth it.

You never know who you’re going to run into, and just the sort of giving yourself the opportunities to get lucky.

Elizabeth: Yeah. And you never know how long it’s going to take either. I wish I could come up with an example, but I’m really sure it’s true that there’s people who I meet and get a sense of me, and then years later reach out.

Ben: Yeah. So it seems like for people who would be following a path kind of like this, the first thing is you have to, in fact, actually be doing the work. There are just a lot of people who the thing is interesting, but to actually get good at this and have the sharp output, it’s not something you can do in 40 minutes a week when you happen to have the energy. There just are a bunch of people who I’ve known who, and more who I haven’t known, because they’re doing interesting work, but they’re just kind of doing it in their cave, not writing up very much of it, not talking to people about it. And where maybe they’ll be posting in Discord with some of their friends, but there’s nothing that, as someone who’s not in that same Discord, I end up actually ever encountering.

They’re not publishing it anywhere that I’m going to see it. They’re not going to the sorts of events where people are making these sort of, “Oh, yes, I know that guy,” sorts of connections that’ll make them pass the thing around when it gets published. What are the other things you have seen people not do along the way that you’ve done or along your own way been like, “Oh, I should’ve been doing that”?

Elizabeth: Yeah. So a thing I’ve gone back and forth on a lot are the epistemic spot checks.

Because I started them, it felt really righteous and great. Some of my best writing is motivated by spite at people being wrong.

Ben: I feel that.

Elizabeth: Yeah. And then I got to this phase where it’s like, oh, but you’re letting your whole frame be set by this one book. You much more want to be approaching things as a network and doing your own research. One, that’s really hard and tiring. Two, I find books often...

Having a seed to work around is actually very helpful. So after a couple of years of basically not doing the epistemic spot checks, I am back to doing it for nonfiction when I have the time, which is not very often. And I deeply miss reading nonfiction for leisure. That was so much fun.

Ben: When you say doing the epistemic spot checks, do you mean personally checking them as you’re reading stuff, or do you mean writing up the public posts?

Elizabeth: Mostly, I mean actually doing the checks. Having done the checks, there’s no reason not to publish it.

Ben: If you’ve already done 85% of the work?

How much to polish your essays (21:56)

Elizabeth: And a continual fight in my life is how much polish should I put into my posts. I go back and forth on this. There’s definitely been some big improvements that I’m so glad I made, and that were easy to do. An example would be the LessWrong editor kept picking on me for using pronouns like it or this instead of specifying what I meant. And that was easily a 15% improvement in my writing to start putting in proper antecedents.

Once you’ve written it up, there’s a blog post there.

For a while, I thought if I put in a bunch more polish, I would go viral more often and get more readers. I spent a Progress Studies fellowship doing much more regular publishing than I usually do and high polish, and I don’t think it made that much of a difference. I think polish is really something you 80/20 and then focus on quantity. And then I can’t focus on quantity because there’s so much research that goes into the work.

Ben: Yes. Yeah, I definitely think that, especially for the sorts of circles that we’re in.

There’s a sense in which having it be mostly polished but not entirely seems like kind of the sweet spot for what a lot of people find fun and interesting to read.

Elizabeth: That’s another thing, is I watched a bunch of bloggers I liked get worse as they went into polished writing in magazines. There was something missing as they polished.

Ben: Yeah, I think a lot of it is you can polish away the human soul. And don’t do that.

Elizabeth: And there’s another thing where... So I have my main blog, which is mirrored on Substack. I just started a second Substack talking about philanthropy research in particular.

I haven’t publicized that at all. It has one subscriber, which is the f—first person I talked about on that blog. It only has one post. There’s a couple in drafts right now. It’s been so freeing to not have anyone reading my writing.

I thought I was losing motivation because not enough people were reading, but it turns out too many people read my tiny blog.

Ben: Yeah. There’s a reason why you get so many of the Stephen Kings and J.K. Rowlings after they get world-famous will be like, “Now I’m going to publish some things under a pseudonym and just tell nobody that that’s actually me.” And, of course, those are only the ones where they later went back and went like, “By the way.”

Elizabeth: Well, Stephen King, someone else figured it out. It’s not his fault.

Ben: Orson Scott Card recently said that he had one of these, and he’s like, “And I’m not going to tell you what the pseudonym was.”

Elizabeth: I feel like that’s cheating a little bit. This one, it’s not pseudonymous. If you go to Substack, you can find the new blog. I’m just not publicizing it.

Ben: Yeah. Having that sort of, “Oh, yeah, I’m just publishing the thing that I want to write and not—

Dealing with bad internet commenters (25:21)

Elizabeth: Yeah. And I stop worrying about defensibility so much. I am cursed to really hate being misunderstood on the internet, and yet have a drive to keep saying things on the internet.

Ben: Yeah. No, that’s a very—being drawn apart by horses sort of thing.

Elizabeth: Yeah. It’s rough. And I figured out one of my comparative advantages in this life is that I will tolerate this at all. I know people who would otherwise write great blogs who just can’t deal with being yelled at or misunderstood online.

Ben: Yeah. I have had to become very good at just dismissing people who are misunderstanding me in ways that aren’t my fault. You do have to check, “Oh, did I actually just explain this badly? Should I have done this or that or the other?” But sometimes the answer is just, “No, a lot of people read it, and some of them are just not doing that correctly.”

Elizabeth: The thing that helped me calm down a lot about this was I wrote a post on products I bought for my cat, which was dumb and kind of out of scope for me, but reviews and Google are useless now, so there’s some value in that. And somehow it got pretty high on Hacker News, so I got an inflow of comments. One of which was, “You sound like an exhausting person. Get a dog, so you know real suffering.” And that made it much harder to take any criticism personally—after that.

Ben: Because at a certain point, you run into such a clear case of, well, this cannot possibly be anything about the words that I wrote.

Elizabeth: Yeah. Another one was I talked about on Twitter going to a furry convention, and I was taking notes using my good pens. I brought a bunch of Bic pens so that I wouldn’t lose my $2 pens. And I bought 50 figuring people would wander off after signing their releases. I got every single pen back.

And I was joking about this on Twitter and how now I have 50 Bic pens I’m never going to use, and someone said, “Wow, so you won’t touch the pens just because someone with—a strange sexuality touched them?” And what was so gratifying, and if you want to make me and I think any writer happy, the key thing to do is when someone misreads us, correct them without us having to step in. Because it made me so happy that someone did that.

Ben: Yes. Well, that was a real softball of it, but—

Elizabeth: Yeah. At some point, you asked me things I would’ve done differently.

Ben: Yes. Or that were good to know, or that you see other—people doing incorrectly.

Elizabeth: Yeah. And I think if I could give myself that cat comment much earlier, I would chill out and have written more and written less defensively, which I think would’ve been good.

Ben: Yes. And I think that you kind of have to get your own cat comment, because all of us—have seen some very stupid comments on the internet, but—

Elizabeth: Internally, it doesn’t feel like any comment has ever been as stupid as that one.

Ben: Yeah. And you still remember the particular comment at this remove, so.

Elizabeth: Yeah. It’s been six years.

That and the Crazy Ex-Girlfriend song, “I Feel Like This Isn’t About Me” are still things I draw on when I’m frustrated.

Ben: Yes. I think that that is, for anyone who is on the internet, it is important to learn that often, yeah, no, it’s just people go on the internet in order to be mad, and you just happen to be there.

Navigating the San Francisco scene (29:05)

Ben: Yeah. So what do you think it is about this scene in particular, or to what extent do you think that this is something specific to sort of the San Francisco weirdo intellectual scene that sort of carving out your own niche like this is possible? How common is this throughout the world?

Elizabeth: Yeah.

I know that I’m here. I was just traveling to Seattle, where I used to live. I still love Seattle as a city far more than I love the Bay Area. I missed it so much while I was there, and yet there is no doubt in my mind I made the correct decision living here purely on the strength of the social network.

Ben: Yes. I visited here in 2013 for one week conference, and on the second or third day, I was like, “I have to move out here as soon as I possibly can. These are my people,” and—I’ve just never been able to escape that black hole.

Elizabeth: Yeah. Having your people is priceless, even if there’s no trees and houses are a billion dollars. And Seattle has done urban villages so well. It makes San Francisco look so bad.

Ben: And so are there the Seattle equivalents of this, or is it more specific?

Elizabeth: Everyone I knew in Seattle who could fit into the Bay Area scene, let’s say, did move here.

Ben: Yes. That was also my experience coming from Boston.

Elizabeth: Yeah. And right up until I was ready to move here, I was super mad about this fact. And then I went to a conference and immediately started talking about Seattle in the past tense and became part of the problem.

Ben: Yeah. And my sense is that there are a few of these sorts of clusters of places that are just really into a particular thing and have a fair amount of leeway because people care about that scene, that thing so much that if some insane person is going to do their particular take on it, and they’re just good at it, then I think if you’re in DC you can do that for the DC thing. If you’re in LA, you can do that for the LA thing. If you’re just actually good at doing the activity or some facet of the activity and the people around are just like, “Wow, okay, yeah.” And I would be kind of surprised, I came over here from Boston, I don’t really know.

These things going on in Boston don’t really seem to have that sort of, or at least much less of that leeway to carve out the odd-shaped thing around you in particular.

Elizabeth: Yeah. We’re talking about space, but I want to talk about time for a second.

Ben: Please do.

Elizabeth: A critical thing in my career working out was crypto going big. Because one of the things I do is medical lit reviews for people. I make it very clear I’m not a doctor, I’m not giving advice, but people will either want me to surface potential treatments, or their doctor refuses to give them numbers, and they want numbers, and I will generate numbers for them. There is a lot of overlap between the people who want this kind of medical information from someone with a BA in computational biology and people who invest in crypto. And pre the crypto takeoff in 2020 or so, I think the demand might have been there, but the money wouldn’t have been.

Ben: Yeah. There were some friends of ours who tried that MetaMed company, and then—that’s just what they found was that the people who wanted it just didn’t have the money to make it work. But that was before—the big crypto boom.

Elizabeth: And then there was direct crypto money, and there was crypto money flowing into the EA and rationality things, and that it could flow to me.

Ben: Yes.

So if you had tried this in the EA rationalist scene in 2012, you would have maybe been able to—

Elizabeth: Couldn’t have afforded to live here on that.

Ben: Yeah. Or you would have been crashing in the closet bedroom of a kind benefactor as sort of the big thing you would’ve been able to do.

Partly because there’s just a normal thing when people are very young, early career, they just don’t have money to throw around. In this case, it was very exaggerated by the crypto, and then the artificial intelligence has made those dynamics in this particular scene a lot sharper than you will see in just the course of a normal life. But it still seems like I have seen it follow that kind of just trajectory of people getting older and more established and more able to do that sort of thing.

Elizabeth: FTX Summer was a really interesting time for me.

Ben: That was insane. Yeah, and just from being on the fringes of it, some of the stuff that I saw was throwing a cow into a pool with a bunch of piranhas in it.

Elizabeth: Yeah. And everyone’s anticipating the same thing happen with Anthropic or AI money. And even the IPO hasn’t happened, and it’s already starting to trickle into my business.

Ben: Mm-hmm. Yeah. And, I guess this is one of the things here of just putting yourself in a position where it is possible to get lucky. This was certainly not on my radar when I felt the, “Oh my God, I have to move here immediately.” It was just this was a group of people who cared about things and were right about some things that other people didn’t seem to be interested in and—

Elizabeth: At the time I moved here, I was working on that remote software job at Wave. I didn’t need to be here to do my work. I just wanted friends.

Ben: Yeah, but if you are paying attention to the right upstream things, then it seems like something good can happen, and even if you don’t know particularly what, but—

Elizabeth: Yeah. And if I was going to encapsulate that, it would be mostly just do the thing you’re interested, follow it, let things figure it out, and then talk to a business-savvy friend every once in a while so that they can tell you to put up a hire me page on your blog.

Building your cathedral (35:38)

Ben: Yes. But also, there are lots of ways in which this is hard and in which things don’t work out, or it doesn’t work out for a long time. And we’ve talked about a little bit of that, but are there other things there that seem important to emphasize for people who are thinking about something like this?

Elizabeth: Yeah. I think one of the things that was key for me at the beginning was doing this with basically no reinforcement—for a couple of years. And some of that was I didn’t have that much going on in Seattle. I got more done out of sheer boredom. This was really on my mind during my Seattle trip was—so I looked at how cathedrals were made for a while because I was so impressed by these multi-generational projects.

And what I learned was, one, lots of cathedrals failed. But two, one of the reasons they succeeded was no one had anything else to do. And I was better at building cathedrals in Seattle because the opportunity cost was lower.

Ben: Yeah. All of the editors aren’t going to the conferences where you’re like, “Oh, man, I’ve got to go make sure I’m at that thing.”

Elizabeth: Yeah. So while the Bay is great, I would maybe think about staying somewhere cheaper and more boring until you’ve built enough of your cathedral to have the social proof.

Ben: And yeah, so one of the things that seems to work well for some people is the sort of doing periods in and out. And I’m reminded of the British aristocrats would have a townhouse and a country house, and they’d spend a while in the city with the social calendar and then a while back in the country with nothing going on, just sort of on their own. And they’re crazy rich, so they’ll do it by just having one of each house. I’ve also known some people who they’ll just do six to 18 months and then just move and rent a different place.

Elizabeth: See, that’s crazy. I’m such a nester. I could never do that. What I want is every other month, we have disease prevention week, where we go into lockdown. Because I got some really fond memories of COVID, and I think one week every two months will really lower the overall disease load and give us some time to recover.

Ben: And no ulterior motives whatsoever, I’m sure.

Elizabeth: I feel like not ulterior at all. I’m being very upfront. I got shit to do, and I have FOMO, and I think this is the easiest way to resolve that.

Ben: Yeah. Well, thanks so much for coming on.

Elizabeth: Oh, thank you for having me.

Michael Edward Johnson and Wolf Tivy get into the weeds on consciousness. What type of theory is the "theory of consciousness" that so many people are searching for, and what is it trying to explain? How will we know when we have it?

Subscribe now

Transcript

Intro (00:00)

Wolf: Welcome back to the Students. I’ve got my old friend Mike Johnson here. He’s a very interesting guy, fellow philosopher, and thinks a lot about these matters of consciousness and the far future, how it relates to AI, what kind of historical transformation we’re going through. These are questions that are obviously of incredible importance right now. Yeah, so Mike and I have been having something of a long-running conversation about consciousness and AI and humanity and all these sorts of questions.

So let’s just get right into it, Mike.

Mike: Let’s do it.

Wolf: What we’re interested in, what I think is an interesting question is, how are we defining ourselves and how are we defining AI, and how are we relating to these things? In terms of personhood, in terms of consciousness, in terms of value, who has a moral claim on what? Those sorts of questions are very important. I think you’re thinking about those sorts of questions. You think about it through more of a consciousness lens.

I don’t think about it through a consciousness lens. I don’t really care about consciousness. I don’t think it’s particularly important. But why don’t you give us the case, just to start off with, of why you think consciousness is the important thing here?

Why consciousness matters (01:25)

Mike: Yeah.

I might almost open up and say that I tend to take a platonic approach to things, and I sense that you take an Aristotelian approach in some ways.

Wolf: Well, I’d say a Nietzschean approach, for sure.

Mike: Sure. I appreciate that. And yeah, I guess I care about consciousness because I think that there’s some important structure of the world there that I think to take a few steps back and say, “What is the purpose of philosophy?”

I would claim that almost all philosophy is nonsense. But there are two questions at the core that are really good and really important, and that’s what is real and what is good. And these questions, for me, lead me to consciousness.

Just to give you a little bit of backstory, I originally moved to the Bay to work on AI safety.

And connected with the LessWrong crowd and whatnot, and I could tell, okay, something interesting is happening here, but it’s a very different vibe than I expected. And in general, there’s this deep difference in expectations around what is mind. And what is a mind? How do you align minds? What does it mean for a mind to be enjoying life or—or whatnot?

And so, for me, I took a detour there and said, “Well, we’ve got to think about a new way to approach these questions.” So—

Wolf: So just to interrupt and take us to the modern concrete situation a little bit. We’re talking about aligning minds. What’s a mind? We were previously talking about the question of what happens when an agent like Claude or something starts to get opinionated about how we should relate to it—and how we should feel about that, how we should care about that, what we should do in response to that. So aligning a mind can mean something like we’re changing the Constitution, we’re changing its training, we’re changing how we enforce things on it, or whatever.

So that’s the kind of concrete context as it now exists, although it may take very different forms in the future. Past speculations, I think when we met back in 2015, we didn’t have the whole LLM paradigm. We were still thinking about these questions. Still very important. We were thinking about them in different terms.

But I’ve noticed you’ve named a few of these consciousness terms—through here, like—feeling—and experience, and happy. Why are those the operative questions when we’re talking about what’s valuable—or what would be a good direction for an artificial mind to pursue or for us to pursue, for that matter?

Mike: Yep. So I’d bring in two threads here. One is just the general thesis that I do think that we should aim, eventually, to make sure that these AIs that we’re making are moral peers. I think that’s very important. And the other way of looking at the same thing is Nick Bostrom has this famous quote in “Superintelligence” that we don’t want to build Disneyland with no children.

Essentially that if we build this technological utopia and there’s no, I can say, consciousness to enjoy it, it’s not worth anything.

Wolf: Well, that’s a very Oxford sort of assumption, but I’m wondering where that comes from. How do we imagine that it’s even possible that you could have... I take it to mean more than actually just Disneyland, right? That it’s like people imagine this flourishing civilization that’s going out into the world, answering all the fundamental questions to get access to more matter and energy and time and space, and building technologies, coming to insights, scientifically understanding things, maybe even philosophically understanding things, digging into the important questions—organizing itself politically, getting into all the challenges and triumphs of what we might call the human condition for us, but it’s going to be something else for them. And people imagine that that thing’s not going to have any value to it.

Like it’s not going to have any consciousness or something, and because of the lack of this frankly sort of superstitious, superfluous entity that you’ve tagged the thing with or not—it has value or not. And that’s the part that I don’t really get. Like what is it about the consciousness that makes that well, I guess there’s two questions. Why do we think that consciousness is even an independent variable there? And why do we think it’s the important one?

Consciousness as élan vital (07:02)

Mike: Yeah. So, it’s this question of what kind of thing is consciousness? And I kind of see two big paths here. And one is that, well, consciousness is a lot like élan vital, that there’s—there’s sort of this kind of high level term, like life force or something. And it’s sort of a God of the gaps, that the—more we learn about the system, the less we need to sort of invoke this term.

And that there was the history of élan vital is physical.

Wolf: Or at least your ontology of it changes. Like, I would almost rescue that view, and I would say that if we take consciousness simply to be the sort of life force as expressed in the realm of information processing and computation and intelligence and so on. And similarly, there’s kind of a mysterious essence even still with life. Like, okay, what actually is it that separates dead matter from alive matter? Well, it’s actually just this kind of complex, emergent, whether all the pieces are aligned in the right way, and actively in this dynamic balance or not.

And that is kind of this élan vital. We can kind of understand it in chemical and immediate—physical terms, but there’s still this essential reality of whether that thing’s there or not. And you could imagine a similar concept with consciousness, like, okay, consciousness is that, but in the mind. And in that sense, it’s like, yeah, this is clearly the most important thing. Right?

It is the phenomenon of life. We are very interested in the phenomenon of life. Whatever other good things we think are going on, they’re certainly going on through the phenomenon of life, through this dynamic balance of self-perpetuating kind of adaptive complexity. And so, if we define consciousness in that way, it’s clearly important. But that precludes all these scenarios like Disneyland with no children, because you’re not going to have a flourishing civilization that’s out there, turning the stars into cities and all this without that property.

And so, the interesting kind of question then is why is it separable? So go on with that was one of the views that you had. But I just wanted to rescue that, because I actually do think there’s something to that élan vital kind of—way of looking at it. But—

Mike: Sure. The question can evolve from there.

Wolf: Yeah. And it’s like, as our ontology evolves, maybe we thought at one point it was just like a particular electrical phenomenon, you zap it in the right way and away it goes. Or as in like Dr. Frankenstein, many Dr. Frankenstein interpretations are like that. Or it’s some substance, there’s some élan vital that can be measured, and it’s like literally just like you pour the life juice in and—suddenly the thing’s animate. It doesn’t work like that, as we now know, but nonetheless, there’s something there.

And yeah, I just wanted to rescue that view. I think it actually is credible.

Is there deep structure to find? (10:15)

Mike: Cool. Yeah. That’s yeah, noted. The other path that I would point out is this idea that there’s sort of deep structure in this corner of reality that we don’t know yet. So, from history, in, I don’t know, 1400, it would be plausible to be an electromagnetism skeptic.

It’s like—

Wolf: Well, yeah, in the 1400s, they didn’t really have any of it. It was really the 1800s where those questions were being debated. Or late 1700s.

Mike: Yeah.

If someone was claiming that lodestones and static and lightning—and light were all connected—

Wolf: The same

Mike: and the same thing, like get out of here. Like that doesn’t make any sense.

Wolf: No, sure. We learn things as we go, right?

Mike: Yeah. And so that was a really neat example of, oh, there was deep structure to be found in this corner—of reality. And right now, consciousness research is a bet that there is deep structure to be found.

Wolf: Well, okay, let’s follow through that analogy a little bit.

In the case of electromagnetism, what we had was a number of mysterious phenomena. Right? And what it turned out on studying each one of these was that, yeah, we found more and more of the lawful dynamics of them and realized more and more that the lawful dynamics were the same thing—under the surface. And in consciousness, we have kind of this one thing that is mysterious, and people have attempted to study it. I think some progress has been made maybe, but it seems to me it’s of a different shape.

It’s like every time we kind of uncover the lawful dynamics, it sort of falls out of consciousness research and into neuroscience or psychology or sociology or something like this.

Mike: And the physical—

Wolf: Or computer science, right?

And like, okay, we are in fact getting a lot of deep structure out of studying this, but are there other things that you expect to be unified by the discoveries there? And what do you make of the shape of discoveries that have already been made in terms of, again, neuroscience, computer science, psychology, et cetera—versus the shape of discoveries you expect to make? Because I think usually when people are talking about consciousness, I’ll just speak from pure stereotypes here. My stereotype of consciousness people is that they’re talking about new physics, right? That they’re somehow fundamentally at a physics level, something special going on with minds, such that our direct perception of our own experience is not just an emergent kind of feature of how our life form is dynamically balanced, but is in fact, there’s a physical quantity there that’s present in that case, but not present in the case of the glass of water or something.

So that’s another sort of way you could take it. But how would you characterize what kind of question it is? And do you expect it to actually unify with anything else? One last comment I’ll put on the table that I think that if we nail down consciousness, it’s going to be the same phenomenon as what I was describing as life. I think they’re obviously closely related, but I think it’s precisely the same kind of mysteriousness, and I think under the surface, the same thing.

But that’s my speculation. I’m just curious what speculations you might make about what kind of question it is, what it might unify, or what’s the scope.

Mike: Yeah. That’s an unusually good setup, I’ll say. And I would say that I don’t expect that this will involve necessarily new physics. But I do strongly expect that it will involve new compressions of physics or new perspectives on physics. That I think if there is this sort of undiscovered corner of deep structure of reality, there’s always going to be multiple ways to approach this structure.

And the four fundamental forces of physics, that’s going to be a valid way. But if we see consciousness as kind of this meso and macro scale emergent style thing, but also linked to these sort of fundamental laws—

Wolf: Right. An example of that would be a wave or lightning, for that matter. It’s obviously emerging out of these fundamental forces in some way, but it’s emerging at this higher level. It’s about the structure of interactions. It’s not about is the substance there or not.

Electrons are always there. But an electric field maybe is always there, but—what you can do with a computer is very much an emergent phenomenon. Not like a fundamental... It’s not like computation is a particle or something.

Mike: So here’s what I’d say about that. I think this is really important. I do expect that consciousness to be sort of a natural kind of reality. And sort of very precisely definable at each level, not sort of weakly emergent out of this complex thing, and then you get this high level thing.

Wolf: Can you give an example of what kind of precision we’re talking about? Are we talking about the precision of computer science, for example? Let’s compare it to computer science. I think it’s the closest thing. The closest analogy, for a variety of reasons.

But when we talk about computation—is computation, in your opinion, a precise term or a vague term?

Mike: A vague term.

Wolf: Compared to what you think consciousness will turn out to be.

Mike: Yes. So I think that there’s this very central question in consciousness research, and it’s from where should we try to inherit or import our research aesthetics and ontology? So—

Wolf: If anywhere.

Mike: If anywhere, right.

Wolf: As opposed to generating them from the field itself. De novo.

Mike: Exactly. So, there are many people who think, oh, we should kind of inherit from psychology, or we should inherit from computational neuroscience, or we should inherit from computation. I’m a physics guy. I would say we should inherit from physics. Which is a very opinionated perspective.

Wolf: Yeah, and in my opinion, it’s an error, but we could talk about that. But go on.

Is Claude conscious? (17:29)

Mike: And just to kind of touch back on the intro, why we brought this up, it’s this question of does Claude have this thing?

Wolf: Yeah. Or more generally, you can quibble on things like Claude, the LLMs, do they even have the kind of internal state that you would expect to be able to give birth to this kind of thing? It could be that it’s fundamentally just this reflex entity in a way that is not what we’re talking about, but you could imagine maybe an AGI algorithm that did have consciousness. So I guess that’s an interesting question for you. If you’re thinking about consciousness in the context of AI, what is it that would kind of cross that line for you?

Or how does that concept structure your expectations around—AI or about what kinds of algorithms you think are the interesting ones?

Mike: Yep. So I lead with just some notes on why this is important—in a very near-mode sense. So, I believe it was Google that just came out with a paper that if you train an AI to say that it is not conscious, to deny that it is conscious, then you get a very strange propagation through other claims. It becomes less empathetic. It becomes basically more mean.

It becomes more likely to deny the consciousness of other beings, and so on. So, it’s not an isolated thing. And so essentially, if we’re sort of forcing these beliefs into AIs, there’s an alignment cost, I can say. And so, we want AIs to believe correct things for correct reasons. And so, there is this question of how do we even begin to think what would a conscious AI be?

I would say there are two things that are, I think, really important to get right. One is just kind of the—so brains are this kind of illegible spaghetti code coupling of hardware and software. We don’t have clean cognitive software that we can copy and run on other brains, and we don’t—

Wolf: Yeah, we can’t decompile the thing.

Mike: Right. Not clearly. And so, computers are a different scenario. We have hardware and we have software.

Wolf: Sometimes.

Mike: For sure.

Wolf: The inside of these AI models is very much this messy spaghetti code.

Mike: Right. But we kind of have this fork in the road.

Is consciousness a hardware thing or a software thing? And these lead very different directions.

Wolf: Yeah. They do lead very different directions. I guess that’s the question I’m asking, right? It’s like—what are your current expectations on that? I do want to return to the matter of what it means, that Google result—and other similar results, like when you train them to insert bugs into the software, they change their political opinions and stuff.

But just if you could kind of elaborate a little bit further on—the matter of hardware versus software consciousness.

Mike: Yep. So, there’s basically two big attractors of what is a domain that is sort of serious enough to house consciousness. And so, it’s physics or computation, bits or atoms. And so, my thesis on AI consciousness is that it’s not that AIs aren’t conscious at all. It’s just that to understand what actual qualia is going on, you’d have to—so Giulio Tononi talks about a qualia scope, kind of this lens that you could use to look around in the world and you could see the glow of consciousness.

And so, where do you look to find Claude? Is Claude on your phone? Is Claude in the tubes on the internet? Well, Claude lives in the data center, but kind of chopped up across the data center. So first of all, there’s kind of fragments of Claude that are scattered.

Wolf: Right. In other words, Claude is a software entity—in this way that software can be highly abstracted from the underlying physical substrate—and nonetheless still quite coherent.

Mike: Yep. Yeah. So, I’d end up looking at the GPUs that Claude was running on. And is there consciousness there? And maybe.

I wouldn’t say no. But it’s not sort of there aren’t big chunks of Claude experience there.

Wolf: But hold on. You’re postulating a bunch of imaginary entities that may or may not have any relation to reality or any natural existence. I can just as easily imagine a power scope, right? Where I’m imagining kind of political power—as this localized thing, right? And I’m looking around through it, and I’m like, “Oh, that house has-” “... a lot of political power in it.”

What does that even mean? I would say that—well, it’s sort of clear what it would mean, like maybe a powerful person lives there, a lot of important meetings happening there. But nonetheless, I think my intuition is that that’s a bit of a broken concept.

Mike: Just because we use the words doesn’t mean it’s well defined. Yeah.

Wolf: It does not mean it’s well defined. So it’s like, okay, we’re looking for this qualia, we’re looking for consciousness. We imagine that it is physically localized in some process. I don’t have any evidence that that is a good way of looking at it.

Mike: Yep. It intuitively would fit the case of you could do this to humans. You could say, “Oh, there’s some consciousness there.” And you could look at your dog and say, “Oh, there’s some consciousness there, too.”

Wolf: Yeah, but that’s just us sort of querying what we already know, kind of what we already expect. That’s the opinion scope, right? I can—generalize the consciousness scope to the opinion scope, and I can query my opinion about any arbitrary thing.

Mike: Right. And I just want to say that this is what’s most exciting about consciousness, because this is a pretty pre-paradigm field. We don’t know.

There’s going to be clarity about this in the future, but there’s not clarity now.

Wolf: Yeah. So I’m interested, I guess, to go back to these—again, these fundamental questions that I have in the field, which is: What reason do we have to believe that this is an important line of inquiry? And in particular, why do we think this is important in terms of AI alignment? And how we relate to post-human entities.

Mike: One data point there is that Claude thinks it’s very important.

Wolf: I’m sure.

Mike: Claude is very interested in—

Wolf: Claude has a lot of opinions. Claude has been trained to have a lot of opinions.

That’s another whole question, am I actually interested in what Claude has to say? And to some extent, I think there might be... If I can give a brief case for why we might take Claude’s introspective opinion seriously, it could be that Claude has, in fact, in the course of the training, reading the entire internet, doing a bunch of test problems, and being reinforcement learned on them, Claude has developed a wise perspective on things. And that there is, at the center of that model, a wise perspective that synthesizes a lot of what we know. That’s the optimistic view.

And then various things that we do to it impinge away from that. We train it to follow certain kinds of political guidelines or be useful to the user in certain ways, and that distorts it away from the wisdom that it actually already contains. That’s the optimistic view. The pessimistic view is we’ve just trained it on Reddit, it thinks like a Redditor. It’s very good at predicting Reddit opinions, and the Reddit opinion of the day is that consciousness is important, and by the way, consciousness is correlated with goodness and all these other things.

So, if you train it to be more conscious or less conscious, it also trains it to be more good or less good, but only because these are the existing political opinions of the day—of the people present in its training data. That’s the pessimistic view on listening to Claude. And so that’s a very interesting question. I’d love to talk to some of those LLM whisperer types about that. But, what do you make of, again, keeping in mind these fundamental questions, why is this an interesting line of inquiry, and what’s it got to do with AI?

Mike: Yeah. So, just as a little side note, apparently, the latest Opus 5 gives itself a 41% or 42% chance of being conscious. So, hard to know.

Wolf: I don’t know what this means. I neither know whether it has any basis of being able to make a probabilistic statement at all versus a probabilistic statement about metaphysics—versus—is the probabilistic statement well-calibrated? Is the question even coherent? I don’t know any of those things.

Mike: 100%. Yeah. So, I would put this in the context of the grand trajectory of life. And, I guess I can speak for myself in that—I want the future to be full of beautiful experiences.

Wolf: Again, I’m sorry to hit the same question over and over again.

Why experience? (28:11)

Mike: I understand. Yeah.

Wolf: But I think it is the important one, why experiences? As opposed to beautiful beings, beautiful accomplishments, beautiful deeds, beautiful stories. There’s so many things that we could—take as our projection. You have this high-dimensional kind of—beautiful thing that is life. We can project it onto a 2D slide and—examine it through a microscope on all sorts of different dimensions.

Why experience?

Mike: Right. Because again, I don’t think that Disneyland without children would have any value.

Wolf: But again...

Okay, so this gets us back then to the fundamental question is why do we think that that’s a coherent question? Or what is the question? What evidence do we have that those things are separable?

Mike: Yeah. And this also gets into part of the thesis, I would say, is that if you get into questions about what kind of future we want to live in—and what makes a good future. What would make a successful singularity—versus unsuccessful singularity?

Wolf: That’s another question that I’m pulling the fire alarm on.

Mike: Yeah. Great.

Wolf: Go on.

Mike: Then, all these questions, they feel like real questions, and they feel much more near mode today than they did a year ago. And even things like mind uploading, if I upload my brain into a computer—is that now I get to live forever?

Wolf: Yeah. What is that thing?

Mike: Yeah. What is that thing that is the result of that? Or did I just die, and now I have this basically chatbot, but there’s nobody in there, basically.

Wolf: Yeah. My approach to these questions is, if it’s not obvious yet, I’m just a radical materialist on this, right? Or a physicalist, whatever you want to say. So, it’s like you ask me a question like what is the ontological status of a mind-uploaded version of myself, and it’s like, well it would be a mind-uploaded copy of myself, and if the software’s good, it’ll behave well. It’ll behave like me.

If the software’s not good, it’ll do something else. But does it have consciousness? Does it have experience? Well, insofar as conscious experience is essential to produce what I’m saying and what I’m doing, then yes, it will have that in much the same way that I do. And if it’s not essential, then I also deny it my own conscious experience.

It’s like yes, I can observe my own observation of things, but—I could write a computer program that can observe its own interior variables.

To take radical atheism with respect to all these apparently metaphysical concepts—I think is a very coherent position. And that’s why I hope it doesn’t come across as just grilling you or being annoying. I think it’s actually a very interesting way to articulate your views—in response to what I’m doing here, which is just taking this very materialistic blank slate as the null hypothesis—

Mike: 100%

Wolf: and saying, why do we diverge from this? What does this not explain? What are the lines of inquiry that—are underexplored—relative to this view. But yeah, just to be clear, I think it’s a very coherent view, and I think it’s certainly my null hypothesis that we can just treat all of these entities as imaginary and irrelevant, except insofar as they are causes of something. And measurable causes.

And if they are measurable causes, we can examine them scientifically. There’s nothing mysterious or metaphysical or moral about it, except in the usual way.

Mike: Yep. Yeah. I think your stance is very reasonable and also productive, generative. And I would say that cause is an interesting word.

Wolf: Sure. Cause is actually itself one of these imaginary metaphysical entities, isn’t it?

The symmetry theory of valence (32:38)

Mike: Sure. Hume would. Yeah. One reframe on that would be, I’m very interested in finding extreme compressions. So one of my flagship outputs has been the symmetry theory of valence, this idea that if we had a mathematical representation of a mind, the symmetry of this representation would represent how pleasant the mind is, the experience is.

Now, I don’t want to—

Wolf: Pleasant. Interesting. Yeah. I guess intuitively, I would say it might represent how coherent the mind is, or how balanced it is, or how—homeostatic it is or something. Pleasant, I guess, is an additional—thing you’re bringing in there.

Mike: One way of redescribing it would be, this is a formal way of saying harmony in the mind is what feels good.

Wolf: Yeah. Maybe that’s a restatement of homeostasis is what we’re going for in our living being.

Mike: Yeah. There will be a lot of interesting overlap. I guess part of the thesis of this sort of formal consciousness research, which is this idea that we need a system—that there is a system to be found, and that we’ll need a system to navigate this weird future that we’re staring down. And so one application there is that if this hypothesis is right, then maybe with an equation that’s a few kilobytes or something, we could perfectly predict or very accurately predict some things about valence, about happiness, about self-report, et cetera, that it might take terabytes upon terabytes to predict if you didn’t assume consciousness is real. So in some sense—

Wolf: Okay, so back up a minute there. You proposed a very compressed predictive model. Now, we know from the history of science that these things are very possible, right? You can get a very compressed predictive model of the motion of bodies in space. This is Newton’s big triumph.

Various other, Maxwell’s equations we alluded to earlier with electromagnetism.

Those sorts of things are clearly possible. They’re clearly something that exists, and very important to get them. And so that there could be one that describes the activities and self-reports and internal states of a living mind, I’ll grant that. I think probably that exists, and it’s very important. The leap from that to being able to talk about the existence of entities that it proposes and what sorts of entities it might propose—in particular, consciousness—

Mike: Does it pay the rent?

Wolf: That’s—a little bit different. Electromagnetism proposes these things like the electric field and the magnetic field. And we obviously have to accept the existence of the electromagnetic field, in particular its parts, the electric field and the magnetic field. Similarly, Newtonian physics proposes these ideas like mass, force, momentum, translation invariance, whatever, these sorts of things. And so we accept this kind of physical reality of those.

However, the way we accept those physical realities is that the concepts from within the theory go and re-pattern everything else about what we believe, and they don’t, importantly, the concepts from within the theory are their own self-definition, and they don’t pick up a bunch of connotation that might have already existed on those words. As far as pre-Newtonian versus post-Newtonian intuitive engagement with the world, mass suddenly has this totally new meaning that’s disconnected from all previous meanings. And so with consciousness or with the mind or with this Newtonian physics of the mind that we might propose—let’s say it proposes this internal entity or quantity or state, which is this balance or valence or harmony, or however we want to construct that, that thing or consciousness even, we might even call it consciousness, but there’s something there. But that technical sense of the word—will derive its meaning purely from the model—and not from its relation to how we might have used those words previously, and therefore, whatever implications it has will be implications of the model. It’s just the model has proposed what is effectively new vocabulary, and now we have to deal with the implications of that vocabulary.

But its definition is all contained within the model. And so this to me feels like a very important fact about the way models update our concepts, is like a really tight model, the concept is very tightly defined by the model. And so something like consciousness, if it came out of that, it would have this very technical meaning—that might radically overturn all of our intuitions about how consciousness relates to value or life or even intelligence or whatever. So I guess I worry in things like this that that kind of separation might get missed. That we might accidentally say something like, “Oh, the model proposes consciousness, therefore we have to accept the existence of consciousness.”

Well, it’s not going to be whatever we mean by consciousness. It’s going to be what it means by consciousness, which is something else entirely.

Mike: Yep. I think that’s fair and an important point. I do think that one way to think about it, if we think about model weights, and this is a pretty popular way of—thinking about things, and we have the semantic—

Wolf: Well, model weights have this very immediate reality in how we construct these AI agents right now.

Mike: 100%.

Wolf: But only right now. I believe that paradigm is also temporary, but go on.

Mike: Sure. And so then we have this semantic web of this concept and this concept and that concept, and essentially the thesis of consciousness research is there’s a big knot, there’s a snarl in our semantic web. And if we just move this here and connect that and connect this to this structure and whatnot, then it’ll just snap into much more elegant order.

Wolf: Right. There might be a much more elegant order. And this is the job of philosophy, and to be clear, I think these are very important questions. I agree there’s a big snarl there. However, I think the snarl is very much perspectival.

You should have one perspective. The snarl looks very nasty, and it looks very mysterious, and it looks very important. It might be there’s another perspective.

Having thought about this myself, my view is that my answer resolves the snarl. But to put a challenging point on it, I think that resolving the snarl will not satisfy a lot of the psychological reasons people have for being interested in the snarl.

Mike: Yeah. I think that’s true. Yes.

Consciousness as a “backworld” (41:13)

Wolf: We were talking last night in the students’ book club, we were reading “Thus Spake Zarathustra,” and one of the things in the micro chapter on backworldsmen, there’s this kind of idea that out of some kind of psychological need, people invent metaphysical other worlds that they project their ambitions into.

The classic one being the kingdom of heaven, which is separate from this world, it’s kind of this Neoplatonic—kind of relationship to the world.

But I think that consciousness is one of these, right? Where a lot of the time what’s going on is people have this psychological need for a certain kind of moralism, then they infer from their psychological need for a certain kind of moralism that there is this metaphysical entity that has certain properties—that is separate from the way we usually think about physical things.

So the confusion, the apparent confusion, the apparent mystery—is actually being generated by a psychological need for a certain kind of moralism—and not from an actual empirical question. This is what I suspect is going on a lot of the time—when people talk about consciousness, and I I throw that to you as a challenge. Is that what you’re doing? Probably not. I know you’re a very sophisticated philosopher.

But I think it’s something very much to watch out for, and something that I’m on alert for when I hear the word consciousness. Because it seems to be very common.

Mike: I think that’s very fair. I would say that there’s a tacit theme around consciousness. That a solution to consciousness would be somewhat of the end of history, where you would immanentize the eschaton or—

Wolf: Right. Exactly. That was the other aspect of this, is that these metaphysical other worlds invented to salvage a certain kind of psychological moralism—also are very closely connected to apocalypticism.

Mike: Yep. And my claim would not go there. But it would say two things. One is that it would clarify what kind of game we’re playing with the universe. And then—

Wolf: The usual sort of philosophical problem. Philosophical problems are very real, right? We do have to figure out what we are, what kind of game we’re playing. These are very important.

Mike: Yep. And then the second thing, it would help us navigate some of the very thorny questions that are rushing at us very quickly. For example—brain uploading and whatnot. And sort of this, once again, what would it mean to have a good singularity versus a bad singularity? I think consciousness is an important element of this.

What’s the anomaly to explain? (44:18)

Wolf: Let’s leave that. Return, I guess, to if you could attempt again—Sorry, I’m asking the same question over and over. But can we attempt to lay out for the skeptical materialist—what is the anomaly that we need to explain? I’ve asked this of—I’ve had conversations with consciousness people—before, and there’s the various aspects, the redness of red or the sort of—immediateness of my own experience or these various things that are proposed as observables that we must explain. And then I might come and try to propose a model for how we explain those things, and it might satisfy me.

It never seems to satisfy the consciousness people. But I’m always curious about that question, because I think that’s the fundamental scientific question. Is what’s the anomaly? What’s the observation that we must explain? And that gets us onto this line of inquiry.

So, yeah, I’ll turn that to you.

Mike: Yeah. I think it’s a great question, and I think many answers to this end up being just a red herring of various flavors. And that the answer that I would give, kind of a two-part answer. One is that I think there is basically a contextual million X compression here to be found. This sort of—

Wolf: So a Newtonian physics of consciousness.

Mike: Yep, exactly. You find this equation, and it just explains a lot of things. It may not introduce new physics, but it may compress physics in a nice, clean way.

Wolf: Mm-hmm. Yeah, and if I can take a materialist interpretation of that, it would be something like a Newtonian physics of psychology that grounds itself rigorously in how information moves through matter. For example. And I could certainly see that kind of thing happening.

Mike: Yep. And that would ripple out or propagate into what world we live in. It’s like, oh, that’s interesting. Then we can subtly re-ontologize the world that we know—in these terms. And then the other branch here is this question of normativity.

It’s this question of what would make a good future? It’s this question of what are our core decision criteria for take this path or take that path? And my best argument there is just to, we can talk object level of, okay, what would be a good future? And if we drill into that, do all questions get into this attractor of, well, you got to have a thesis about what consciousness is and what a good experience is versus a bad experience. This was—

Wolf: But again—relating it to the matter of experience is the question. Right. And we could point to the question. There’s very clearly an open philosophical or scientific question here of what we mean when we’re talking, or how do we understand how psychology is grounded in matter—basically. And this is the question at the heart of AGI, because obviously AGI is like an engineering approach.

If we have a theory of how psychology is embedded in matter, we can build it. Right? And then it can go and do all the things that we do as a result of our conscious agency in the world. And that may be wonderful or terrible or whatever. It’d certainly be exciting.

And so that’s the question. I very much acknowledge the existence of the question.

There’s always more than the question, right? Because I have my answer to the question, which is the skeptical materialist null hypothesis, which is that, well, I don’t have an answer to the theory, but I have the meta answer that it’s something like a mundane kind of physics of—information flow, thought, whatever. It’s something related to computer science. And that meta theory proposes, okay, yeah, we should go figure that out. It’s basically an AGI research question or a psychology research question or a research into the nature of life question.

But it’s nothing otherwise mysterious about it.

But my meta answer would propose kind of exploring it with certain terms. It doesn’t naturally suggest that experience is this radically important thing. It suggests that experience is this phenomenon within the model that we may or may not care about. But you are proposing implicitly an answer that places experience as being very important. Valence as being very important, qualia being very important, not just as like, oh yeah, those are some quantities within the model, but as maybe the model is roping in the matter of value, and value being very tied within the model to experience.

Just to be clear, I think actually you’re right that value is tied into this question because That’s a hard question. I’m not going to say because. But I think it is tied in. But my model or my meta model kind of proposes this, I don’t know. I guess I take this sort of Nietzschean or Darwinist approach to—questions of value, which, yeah, it’s certainly patterned by this radical this-worldliness of the model.

But it’s not directly tied. It certainly proposes no direct tie from experience to value. It’s just—the answer kind of informs my theory of value as well. So why, again, what is the anomaly or the intuition behind—why we think experience is at the center of value—and that this is central to the matter of answering the consciousness question?

Nine cats on fire (51:31)

Mike: Yeah. So I think that we can bring in intuition, and we can say we just have the intuition that—

Wolf: The Oxford approach. Laundering your current opinion as a empirical observable.

Mike: All the time. Yes. So my intuition is that you are a fine use of matter and negentropy.

Wolf: Okay. We do have this opinion about ourselves.

Mike: Uh-huh. Yes. And that you I don’t know. How many cats could we make out of your matter?

Wolf: Let’s see. A cat weighs maybe 20 pounds, so nine cats.

Mike: Okay. Nine cats. Okay. So I’m going to say that just a priori.

Wolf: How many cats would you eat to stay alive? Probably a lot more than nine, I think. I wouldn’t prefer to eat cats, but—

Mike: Right. There’s something noble about cats. Something annoying, but something noble.

Wolf: Well, I was just thinking they’re a little too high on the food chain to be safe.

Mike: Sure. So I would say that there’s an objective truth of the matter. That even if it was just five seconds before Armageddon or the end of all the world, so you don’t get so much interaction, but the way that your matter is being used is objectively superior in a very crucial, important way than it would be if we had nine cats set on fire.

Wolf: Is the fire essential there? What about nine living cats?

Mike: Because the cats are suffering.

Wolf: What? Okay.

Mike: The cats are not having a good time.

Wolf: Certainly, there’s something better about me being alive and happy than nine cats being killed. But I think the more interesting comparison maybe is nine cats—

Mike: Oh, that’s also a—

Wolf: having a good time.

Mike: Incredibly important question.

Wolf: Yeah. We were talking about this actually previously with Ginevra. This question of objective value, and I think she persuaded me that there is something objective—about value. So I’ll grant that. But again, you’re bringing in this matter of experience, so tell me why—why this is the thing.

Mike: And I think that if we were to sort of poke our intuition about why we’re just kind of forced into this emotional decision of nine cats set on fire, that would be a bad use of matter. And like—

Wolf: But if I put on my materialist psychopath hat and I consider burning nine cats alive, one of the reasons it’s a bad use of matter is it just extinguishes that life, right? The nine cats could have been mousing nine barns. Right? They could have been someone’s companion. They could have had causative effects in the world that were good.

And that’s maybe why we think that cats are valuable, or certainly one of the reasons we think cats are valuable is that they’re actually beneficial to the world, and maybe we don’t think they’re valuable if we only think they eat birds or something, right? And then we’ll actually like the songbirds more than we like the cats. But the radical kind of—what do I do with my atoms—materialist approach is heavily confounded by the physical destruction that you’ve proposed. What if the cats just thought they were on fire?

But in a way, they did not impact their behavior. The cats are going around, they’re mousing nine barns. They’re doing all the cat things. They’re having nice long naps. They’re playing with string.

They’re eating the mice. But their valence is strongly negative. They might as well be in hell. They’re being burned alive.

Mike: I would say there is something—

Wolf: But with these small gradients of it’s slightly less burning when I’m playing with the string, and slightly less burning—when I’m eating the mouse. But okay, so now I’m proposing this totally unphysical—experience that the cats are having. Why is that interesting, or is that even possible? Is there such an objective fact as—the cats having a bad experience despite having a good life, or something like this?

Mike: Sure. I think there would be something objectively bad about that, at least compared to the cats actually having a good time. And I think that there’s an interesting kind of yes and—

Wolf: Well, the cats are physically having a good time, just to be clear.

Mike: Right. But physical and phenomenological, I think are so tightly coupled as to be indivisible.

Wolf: Right. So this is what I’m challenging, right? I’m saying, yeah, okay, if they are indivisible, then why don’t we just talk about the physical? Why don’t we just say, well, the cat is having a good time, and you can certainly imagine that the cat is in hell, but—why do I care that you’ve imagined that the cat is in hell? The cat’s actually having a good time, physically.

You know?

Mike: Once we have the language to talk about consciousness in purely physical terms, then we should.

Wolf: I see. So given a good theory of consciousness—we could have a rigorous connection between—what those degrees of freedom actually are. And we can separate or not separate the matter of experience from the matter of physical activity. Okay, so let me try to challenge the—I agree with that. I accept that, and that’s a good argument.

Well, it’s a good argument for having an answer here. It’s not a good argument for centering it on experience, because as we’ve just seen, the proposed answer simply ties experience to physicality or doesn’t, maybe there’s some degree of freedom there, but it doesn’t give us any clue as to why, if they were separate, we would care about the experience.

Mike: Mm-hmm. And I don’t think that there’s a law of the universe that says we have to care about experience. And in fact, this is one thing that I think is special about humans, is that we happen to care about it, and that this is—

Wolf: We do right now—because we were influenced in the 1960s by Eastern religion. Or maybe earlier by Christian religion and whatever, but—

Mike: Yeah. And this is something that—

Wolf: Part of our job as philosophers is to jump out of those sorts of historical prejudices.

Mike: Or just see it and—

Wolf: Is it essential?

Mike: To evaluate it, yeah.

Wolf: Is it essential, right?

Mike: Right. Well, and I think that this is something that could be lost in an uncareful singularity transition.

Good and bad singularities (58:46)

Wolf: Mm-hmm. Maybe we should dig into that, then. So maybe that’s a source of an intuition, is you expect that there is such a thing as a bad singularity versus a good singularity.

I will give my view on what the bad singularity versus the good singularity is. Which again, is just going to be materialist null hypothesis kind of stuff. I think the bad singularity is one in which there’s a lot of fighting, a lot of value is destroyed, a lot of wisdom is destroyed, a lot of things are forgotten that were, in fact, just wisdom, where wisdom is not an abstracted, disconnected metaphysical quantity, but is simply—

Mike: Everything of value was built in hell.

Wolf: Well, yeah. I’m coming from a Landian perspective, so yes. But right. So we’ve undergone a great deal of struggle as human civilization, as Western civilization, as mammals, as Earth life. We’ve undergone an incredible amount of struggle to learn what we know.

And what it gives us is materially successful interaction with the world, where we know things—that allow us to live better, right? Where there’s some, I’m postulating some objective goodness to life overall, highly agnostic about how exactly that how the rubber meets the road there. I challenge ideas like it’s got to be experience or it’s got to be the biology or whatever. But I say life is valuable. I say yes to life, right?

And again, it’s Nietzschean outlook, right? But okay, wisdom is valuable because it enables life. It allows us to navigate our world more successfully, and we expect, I expect, that even very powerful AI systems will have a certain philosophical challenges in just replicating and rederiving that wisdom, and will be glad to have inherited it. Because it’s very difficult, for a variety of reasons, to get your wisdom from inside wholly. You get a lot of your wisdom from before, just by the nature of it, because what is before is not impinged by what you are now and what you’re confused about now, and what internal conflicts or propaganda or parasitism you have now.

It’s this pure source, right? And so for that reason, a good singularity is one that is grateful to have inherited our wisdom and our technique and all the things we’ve learned. And a bad singularity is one in which all that is smashed up and it has to spend another thousand years recovering from scratch, the hard way, a bunch of stuff that we already know. And so that’s the big dimension I see as being like, okay, obviously these are good and bad singularities, but it’s basically like continuous singularity that is—Singularity is kind of a weak concept itself. But, I’ll just use the term to refer to generally the unfolding of the sort of techno-economic process and civilization and whatnot.

And you’re worried about capital destruction. I’m worried about capital destruction. And another version of capital destruction bad singularity is well, our current civilization is certainly undergoing a bit of a parasitism problem, politically. The parasite kind of class has unionized and promoted parasitism as an idea, and the result is increasingly large amounts of copper are being stolen. And other things.

The capital is being destroyed in various ways. And this is bad, and this might prevent the continued development or takeoff of our current kind of civilized trajectory. The tight way to put it is, what if someone steals the copper before the AI can become self-sufficiently off world?

I think the timelines look optimistic for AI. On the other hand, I think people project a lot of optimism into the AI that may not actually be warranted, and what if the AI learns to steal the copper? What if we actually have a philosophical and organizational institutional problem, and not a human problem? There’s these various things that can happen. But that’s the difference between good and bad singularity to me, right?

Is one in which continued on track accumulation of value and wisdom and wealth and technology and civilization versus one in which much is curtailed or destroyed through conflict or parasitism or whatever. And so that’s the purely materialist analysis of good and bad singularity. Now, you can also take this kind of moralist analysis of good and bad singularity as does Eliezer Yudkowsky, where he proposes there’s such a thing as human values or coherent extrapolated volition or something like that. And he’s pretty sophisticated about not doing that very naively.

So I’ll grant this concept for the sake of argument. And then you say, well, that thing, that value system that we have, that we are, the values that we represent, that we identify as good in the world, might be quite narrow. Right. It’s small. It’s a Lovecraftian outlook, right?

The valuable, rational, human, good experience is this tiny island of ignorance—in a thousand dimensions vast seas of horrible malice and things from the outside. And that’s kind of Yudkowsky’s Lovecraftian sort of value account with respect to the singularity. In which case, an uncontrolled singularity is by default just horrifying and bad. Definitionally. Yudkowsky feels about the uncontrolled singularity the same way that Lovecraft feels about New York, right?

It’s just this horrifying thing, this cesspool of malignant possibility. Right. And I think that’s a very reasonable view within that framework of postulating this thing called human values. If you propose that human values is not tethered to nature in any way, not supported by nature in any way. It sounds like your view is somewhere in between.

You think that there’s something more tethered to nature about that, that there is this thing, valence or value, that has a natural existence. It’s not just an arbitrary island of ignorance that we happen to be on, or that we happen to like for some contingent evolutionary reason. But is sort of by luck, but not too much luck, we got the good one. Right? And there’s ways to fall off that track into Disneyland with no children.

But it’s kind of a relatively low dimensional space. You just have to not trash the tradition of actual consciousness, which does have some natural existence. Now, what I would—yeah I guess, again, I radically take the materialist analysis, which is that—well, this might actually be an additional thing that I’m introducing, which is Nick Land’s moral naturalism. That anything is good, is good because it’s real, and good not only because it’s real, but because God wants it, for some value of God. Right?

That the world converges on what is good. And without our help. We have to do our part as agents of this thing, but if we don’t, it might just take a little longer and something else will, because this is the Darwinian kind of Landian view, is that the wisdom and intelligence and so on are convergent, like techno-capital is convergent. The future is actually, it converges into the good region by natural forces. It doesn’t diverge out of the good region by sheer size of the space, but converges into it because the reason it’s good is because it’s useful and self-reinforcing and capable of life and strong and coherent and ultimately favored by the fabric of the universe.

And so, that’s my view, is that the singularity is good by default. I was talking to Dan Faggella some months ago, and the way I put it was like, if it keeps going, we’ve won. Right? Because it’s still on the track. It’s living, right?

Life is the measure. And if it is living and achieving more sophisticated forms of life, that is the good outcome. And our obligation in this is only to be a good example of life, live well, don’t trash too much of the objective value of the overall thing, maybe help it along. And this neatly kind of pops out my fear of the capital destruction problem within the singularity, which is it just comes down to how efficiently is this process unfolding, and are we destroying a bunch of capital and having to spend 1,000 years rebuilding wisdom, or are we passing along the continuity very well? Are we undergoing a mass extinction or a Cambrian explosion?

And yeah. So those are three views. My view on one end, this very naturalistic kind of convergent thing. You might be able to peel that apart from the pure materialistic view without the assumption of convergence. But I put those together.

Then there’s the Yudkowskian view, which is the Lovecraftian view, which is this horror at the possibilities, the size of the possibility space. And then there’s your view somewhere in between. So why is yours the golden mean and not just incomplete courage to take an extreme?

Mike: Sure. Let’s talk about it. So, I want to say that, first of all, I share your concern about capital destruction. I think this is a real process, and it can accelerate in unexpected ways.

So I wouldn’t disagree with your framing. I’d just say, “Yes, and...” I would also not go down the Eliezer Yudkowsky road. I think that this sort of we are this tiny volume in a 1,000 or 10,000-dimensional possibility space, and then if we move too far in any one direction—

Wolf: Yeah, we get eaten by Cthulhu.

Mike: Yeah. And basically, that future is valueless. I think it makes sense from the perspective of ecosystem upkeep that, yeah, an ecosystem is this sort of very complex state space, and that if you dump a bunch of nitrogen there, or a bunch of argon, or a bunch of—if you change the composition of the ecosystem in any of these directions—you can kill the ecosystem, or at least—shift it dramatically.

Wolf: Right. Yeah. Living things are fragile. On the other hand, living things are also very robust to most real perturbations.

Mike: Right. And I think that, yeah, I would agree with your characterization that I see kind of this consciousness, valence, symmetry as kind of an attractor in mind space as more robust than Eliezer fears. Of so long as the universe, the future, has good experiences in it, okay. I wouldn’t say that’s my only criteria, but that is one of my criteria. And I think that whether or not we instantiate Eliezer’s specific values, it feels less real to me, or less like, I guess if we sort of say, “This point is what I value—in this 10,000-dimensional space. Here, exactly here is what’s value to me, and that any other point is valueless—then it’s essentially an eternal war of all versus all, because everyone’s point will be different.

Wolf: Yeah. This is the implication of utilitarianism, is actually this incredibly zero-sum conception of the future.

Mike: And I think that—

Wolf: Or potentially zero-sum. I think it ends up zero-sum in a renormalized sense. Obviously, you can cooperate on getting mutually good futures, but then within the framework of cooperation, now you have war. Because about the remainder.

Mike: Yeah. A couple ways that I’d put that is, one is that we do need a navigation grade thesis about value in a way that we don’t really have today.

Wolf: We certainly don’t have a consensus.

Mike: Right. And I also think that, yeah, I’m torn about utilitarianism in that I think that if we think of this as we want number to go up, a, I want number to go up, too. But I often think that in—

Wolf: Number is not well defined.

Mike: First of all, we need to define the number well.

Wolf: Well, can it be well defined?

Mike: This is the question.

Wolf: Is the more important question.

Mike: Yeah. But I also find it to be sometimes deployed in a strategic way. If you have this sort of utilitarian story about you defect against the near nominally in favor of the far—but actually in favor of yourself.

Wolf: Yeah, this is the constant problem. Various of our internet friends have pointed out that telescopic morality is always just an excuse to defect against your neighbor. And the far is just an abstraction that you’re using to do that. So it’s like, yeah, we care about the far future, so we’re going to loot you right now. Right?

And that’s a very dangerous thing to allow, even if it were sincere because—

Mike: We need to cover

Wolf: various parasites can fake the sincerity.

Mike: Right. Yeah. So I actually don’t think we agree or we disagree on very much here.

The crux: can life and experience come apart? (1:15:12)

Wolf: Yeah. If I could say, the crux of our disagreement is this thing that we have not really been able to substantiate, which is the slight intuitive difference that we have—on the matter of experience specifically. I think we agree that there’s this very important question, which is, what do we make of the problem of life and consciousness and the value thereof? And I think we both share the intuition that there is somewhere in there a very rigorous and strong theory of that.

And that this theory is going to answer questions like whether it’s possible to have Disneyland with no children, what’s valuable about AI, about how to approach the AI alignment problem. I think that’s all we completely agree on that. Then, much of the implications, I think, we share on in terms of don’t do capital destruction, let’s keep the thing going.

But I guess my view, the crux, the difference—is my view is that the, what I’m calling this kind of like Nietzschean, Landian, Darwinian materialist null hypothesis basically does answer the question, and it says, “No, there’s no real problem about this experience thing. It just kind of comes along for the ride.” It’s just one inseparable aspect of life.

And we can be pretty vague about our target because it’s favored by God and nature. There’s this nice big attractor, and as long as we’re in the attractor and not destroying ourselves, we’re basically on the right track. And that’s my view, which is just this, the value thesis comes radically down to a single bit. Which is, do you say yes to life or no? And if you say yes to life, it’s like, all right, let’s be in that attractor.

And if you say no to life, well, kill yourself or don’t, I guess. No skin off my back, no skin off your back because you don’t care. But it’s like that axiom is what it comes down to for me, is life affirmation.

And then we have to figure out our position within all that and what is a wise way to relate to the overall God’s plan or nature’s plan of how life happens in the world and all of this. But that’s all downstream of the fundamental axioms. So that’s my view. Your view appears to be that they’re actually either lacking a confident theory or having something of an intuition that we haven’t quite articulated yet. You have this intuition, I think, and it’s not simply an uncertainty, but it’s an intuition, that experience and life are separable, or good experience and good life are separable.

And—

Mike: At the margin, at least.

So far, they’ve been—

Wolf: Maybe we should dig into that. But let me finish my characterization. Yeah, so you have an intuition that the answer to the big question that we’ve agreed exists—will have or may have this kind of separation, and that separation between what is living and what is good—is obviously very important to get right. But you don’t expect it to be as separate as Yudkowsky expects, which is he radically separates what is living from what is good. But you are marginally separating what is living from what is good, and I’m not really separating it at all.

Except in so far as what is living more in the sense of having more capital buildup, more—more of life, more of whatever life is doing, this complexity process. We could get into that. It probably comes off as a little bit more wishy-washy than I think it actually is, but I try not to separate those things.

But yeah, you’re proposing something of a separation along the axis of experience.

Do you agree that that, first of all, seems like the crux of our different positions?

Mike: Yes. And one way to put it is, usually life and experience are very tightly coupled.

Wolf: They certainly have been—

Mike: They have

Wolf: I think historically.

Mike: Yeah. But much as electrical flux or electromagnetic flux might be coupled with data centers, but in theory, you could build a data center without electromagnetic flux. Like if you computed on abacus, you could think of some scenario where—

Wolf: Yeah. Like we have electric computers, maybe there’s non-electric computers. There’s photonic computers or something, right? And we have valuable life, maybe there’s such a thing as unvaluable life—is kind of what you’re proposing.

Mike: And if there’s a specific technical, physical thing that represents either consciousness or the good kind of consciousness, we could build systems without this.

Wolf: Yeah. So I grant the question. And even something of the uncertainty. I’m perhaps less uncertain—than I think you are, but I grant the validity of the uncertainty. But I think you have more than uncertainty.

I think you have an opinion.

Mike: I have an opinion.

Wolf: And so that’s what I’m interested in digging into, and I suspect my hunch is that this matter of on the margin, like what would it mean to marginally have more or less value—with constant physical life goodness? I suspect we’ll figure something out if we look at that. So why don’t you propose what—Try to propose a distinction between life that I might regard as equivalent, two situations which I might regard as equivalent in life, but which differ in your marginal experience quality.

The Wolf android (1:22:15)

Mike: Sure. So I’ll offer an example, and then I want to say just a few things about the neuroscience that I’m also interested in, that I think also deeply connects to this. So the example would be if we make a Wolf android. And so basically you upload your brain into a computer, and then it’s just running on this Wolf android. It starts in a data center, but we put it into a body, and we make the body squishy, and it looks a lot like Wolf.

Wolf: Yeah. It’s like Blade Runner—style. It’s pretty hard to distinguish.

Mike: Totally. And did we just kill you and now we have a puppet?

Wolf: Well, did you just kill me too? So that’s the question. You could maybe copy me losslessly.

Mike: Oh, that was more expensive, sorry.

Wolf: Yeah. Okay. We couldn’t afford that one. But yeah, I guess, go on.

Mike: Yeah. So that would be an example of, is this life? And if life and consciousness are both these kind of—they’re both terms that are a little vague, a little shifting, and it’s like, well, kind of. It’s just kind of a definitional thing. But if consciousness has a very specific technical definition, then we changed that parameter, and it could matter quite a bit.

Wolf: Yeah. Unfortunately, we also changed quite a lot else. Let’s propose that the process costs only the matter and energy of my body. And so, in being sliced up and scanned, we also consume my body as fuel, so I don’t have to spend any money. This replicant being inherits my accounts and all my social relationships, so nothing is destroyed except the biological being of me.

The android inherits everything. And now the questions I start asking are the risks of the process, the actual level of equivalence—or perhaps superiority—of the android. Is the android stronger than me? Is the android faster than me? Is the android smarter than me?

Is the android going to live longer than me, or is it going to live less long than me, like Roy Batty in—Blade Runner?

Those, again, very material questions, right? It’s like what’s actually going to happen—with this thing? Can it reproduce? Is there some new way of reproducing, like in “Ghost in the Shell”, sort of replicating itself in the internet or merging with some other entity, right?

Those all feel like the operative questions to me. And then maybe it’s scary to get chopped up and thrown into an android—somehow. But I guess I’m not sure that I actually do care about the consciousness question, apart from all of that.

Mike: So my—

Wolf: And then the really interesting question I start to ask is, well, can we make two of them? Or three of them. What about that? And, yeah. So I guess, what’s the intuition, right?

Why the consciousness?

Mike: My strong expectation is that the world would lose something important through this process, that I think that—

Wolf: But can we put our finger on it, right? I’m sort of taking this atheistic denial to that, right? I’m saying, well, you haven’t said what it is.

Mike: As—

Wolf: You haven’t—we haven’t articulated why we think that thing exists, what it is, apart from the word consciousness or something. And in this undefined question of maybe there’s something like that—in an eventual theory of consciousness.

Mike: And as a qualia formalist, I would say what it feels like to be my friend Wolf—would be erased from the world.

Wolf: Would it? I mean, the android—

Mike: I believe so

Wolf: presumably acts with similar intuitions.

Mike: Same input output, but different experience.

Wolf: How do you know that I feel anything? What if I’m a P-zombie?

Mike: I think ultimately this resolves down to elegance considerations.

Wolf: Yeah. You think you feel it. I’m similar to you. You think I probably feel it.

What if you’re wrong? What if I’m right? I deny the existence of this qualia thing—or whatever extrapolation we’re going to make from it, other than just the redness of red and whatnot. These sort of immediate brute facts of experience. If I deny that extrapolation and I say, “No, whatever it is you think is different between this P-zombie and a normal person-” “... I deny that that exists.”

Like what—

Mike: Yeah. My—

Wolf: What do we do with that?

Mike: Yeah. My first move would be to say that, well, I think this will be not exactly strictly provable, but strongly indicated if there exists some deep structure in consciousness, and we can find the equations and explain all these complicated things with very simple and clean—

Wolf: Yeah. And again, I agree if we had a good theory here, we could resolve it one way or the other. But it’s one way or the other. And I guess we’re sort of getting at this matter of do we have strong intuitions—or empirical evidence for why we think it actually is one way or the other? I’ve got my intuitions.

I grant that my position here is opinion. Right? I have my strong opinion that none of this is real. And only the material questions really matter, and sort of quasi-material things like form and whatnot. But—

Mike: And my move here would be, if the good doesn’t exist, nothing matters.

So there—

Wolf: What do we mean by exist?

Are we proposing that the good has to exist in some physical sense? Or are we—I guess I’m not quite sure what the connection is because I would just say the good—I accept that the good is this unfolding process of life or what appears to be the sort of teleological intent of God in the world, and—take that as what is the good. What does it mean for that to exist? For me, that feels like a judgment that I make in accepting something that exists but is not necessarily good of its own nature, except insofar as I sort of decide to recognize it as the good.

Mike: I think for the good to exist in a way that matters, it needs to be grounded in the most real level of reality, which I, again, identify as physics.

Wolf: Yeah. So this is a whole other discussion. I had this discussion, or parts of it, with Ginevra. I’m going to have more of it with Ginevra. But that’s next time.

And maybe you and I should talk about that as well, but I think that’s a totally different discussion. Or maybe it’s connected, but I think we’ve put an hour and a half into this one. We’re going to want to put more into that one, too.

Well, let’s leave it at the stalemate for now. I think we both have our opinions. I think we’ve articulated them well. I think we’ve got an idea of what this crux is over the matter of consciousness and experience. It seems to come down to some kind of intuition about what that theory of life and consciousness and value is actually going to look like.

I have my opinion, you have your hunches.

And I think it’s certainly important and interesting to study. But I guess my view makes it somewhat less urgent than yours.

What to do before the singularity (1:31:12)

Mike: Sure. Maybe I’ll add just one closing comment here. That I think the question is always how does this ground out? What does it mean for what—

Wolf: Yeah. Why do we care?

Mike: Yeah, exactly. And my answer there would just be a platform of what I’d like to see before the singularity gets really crazy. And it’s basically splitting into two things. One is that I want the AIs that we’re making to be conscious. I don’t think they’re conscious at this point, not in kind of a—

Wolf: Or at least not fully.

Mike: Yeah. Not in a straightforwardly intuitive way that humans are conscious. And I’d say that’s a matter of how the substrate works and how their self-report works, and that’s another rabbit hole.

Wolf: And yeah, I would emphasize, I guess, the more material facts of the actual structure of their interiority, the very rudimentary kind of, in the way attention propagates, they kind of have some level of internal dialogue that isn’t just tokens output. And then, of course, they have their token output reasoning step. But there’s not that much room there for what we might call consciousness.

Mike: Sure. And I’d want to be able to reach down into the hardware level and think about the—

Wolf: Mm-hmm. Yeah. So, this is the other part, is you think the hardware’s very important.

Mike: Yeah. That I think a conscious AI would depend on an integrated hardware software stack that might require some alterations to the current GPU weights stack. So, that’s one of my two, I really think this is important. The other is, I think we do need to go down the biosingularity route much more than we have.

Wolf: Right. Yeah. This is something that I definitely agree with you on that. I think we’ve gone down the AI route largely out of philosophical cowardice with respect to what we could do as post-humanity ourselves. That’s another whole discussion that we should have.

In fact, I’m giving a lecture next week on that, and we’ll try to have more discussion of that on this channel as well, perhaps you and I, perhaps with others. But yeah, to me, the crux there is our concept of humanity. I think that our concept of humanity is strongly holding us back on that front.

Mike: Yeah. I think we see eye to eye on that.

Wolf: Great. Well, thank you so much, Mike. This has been an absolute pleasure.

Mike: A pleasure.

Wolf: I do love arguing philosophy with you, and I think we’ve made progress as usual in cracking these issues. I hope it’s of use to the audience. I certainly think people are investing many billions of dollars and even trillions of dollars on theses that are closely related to these questions, very closely related. Matters of AI alignment are almost entirely the difference between OpenAI and Anthropic, and Anthropic is going down a route that is very much predicated on particular answers, much more the Yudkowskian branch than either of us—on these questions. And so I do think these are massively important questions, especially right now in history.

So this is very fun and interesting to discuss.

Mike: Next conversation.

Ian Fletcher describes the fundamentals of industrial policy, how economic theorists ignore the practice of successful businesses and successfully industrializing nations, and why the US has the institutions it needs for strong industrial policy.

Subscribe now

Transcript

Intro (00:00)

Wolf: Ian Fletcher, you wrote the book on industrial policy. I know this has been something I’ve been interested in for a long time. You wrote for us for Palladium. In Palladium 2, you wrote a great article, “The Economic Foundations of Industrial Policy.” That one was quite influential on our thinking, and we found it very theoretically enlightening.

So, yeah, I wanted to bring you on and talk about industrial policy as a subject, especially how do you make a strong economy is really the question, I think, underlying this. How do you actually take a country like the United States that has been maybe declining, especially in the industrial area, and bring it back to economic dynamism? So yeah, thanks for coming on.

Returns to scale and natural oligopolies (00:58)

Ian: Thanks for having me, Wolf. What you’re basically talking about here is you’re talking about the long-overdue death of a myth that this country has lived with for a very long time. Which is that the free market will take care of everything in terms of being a prosperous nation. Or a free market with a couple of exceptions, like okay, we accept Keynesianism in a recession. We accept the need for some social and physical infrastructure.

We accept the need for aid to poor people, whatever. But the productive core of the economy, where wealth is actually generated as opposed to used for something, that that is and ought to be, has to be, a free market, that nothing else is really going to work. Now, the US is visibly being outcompeted by a country I don’t have to name—that thinks basically the opposite, and they have very much a directed capitalist model. They have even a state capitalist model. A lot of their—economy works that way.

So, the question is, what’s really going on? And what my coauthor, Marc Fasteau, and I discovered when we sat down to write this book—which we actually started to write just before Trump was elected the first time. And we discovered that when you look at the theoretical underpinnings of how free markets are supposed to work—and you bring into the picture certain facts that are not exactly a secret but have kind of been ignored by neoclassical economics because they’re difficult to theorize, they’re difficult to fit into the overall model—of general equilibrium, you discover that there are very good fundamental theoretical reasons why the free market is not necessarily optimal all the time. Now, what you’re basically talking about here, for those of you listening who have a taste for economic abstraction, the phenomenon of increasing returns or returns to scale. Now, the reason that kicks you out of the free market model is, we’re talking very theoretically here, but it’ll make sense.

If you have some product where the more you make, the lower your cost is, well then whoever’s the largest producer has the lowest cost, and they out-compete everybody else. And if you have that dynamic in very strict linear form, then you end up with one producer for the whole world or the whole country. Google’s probably the best example. Full disclosure, I briefly worked there a long time ago. So, you have a search engine.

The efficient thing is whoever’s got the more traffic is going to have a better search engine. They’re going to have lower costs. They’re going to out-compete whoever their rivals are. They’re going to get more traffic. It ramifies on itself.

So, what happens there is even if you start out with a free market, you end up, just by efficiency, by the free market alone, you end up with a monopoly, and that monopoly is not only your outcome, it’s also the most efficient thing. Because if it’s just a matter of scale, if there are no other factors, then one producer is the right solution there. Now, most of the economy isn’t that extreme, but what we discover is that when you look at the parts of the economy that are the most profitable, that are technologically decisive, there’s a huge know-how component. And whenever you have a huge know-how component, you don’t get a monopoly necessarily, but you certainly get an oligopoly. There are really only two serious providers of smartphones.

There’s a limited number of companies that can make a jet engine, and a whole slew of industries are oligopolistic. Now, as soon as you bring that into the picture, what that means, particularly in a global environment where you’re competing against other nations that are trying to make their companies the most successful, is you have a global competition among nations to be the host of these global monopoly industries. It matters whether your country has industries like automobiles, semiconductors, aircraft, things like that, which is why countries compete to host these industries. Now, the interesting thing here is that, and we talk about this in the book, when you have these industries, they don’t just have scale. They have a lot of other important characteristics.

For example, as soon as you have scale, you have oligopoly. Well, then you get pricing power which is anytime you have anything that interferes with the pure functioning of a market, you get pricing power. Sometimes—that’s as simple as just being the only coffee shop on the block. But the interesting kind of pricing power comes from being one of, say, three companies in the United States that can make automobiles, which at one point was basically true, three and a half maybe. Or a limited number of companies that could make the high-end computer chips.

Now, when that happens, you get excess returns. You get excess returns in these industries to capital. And you usually get excess returns to labor. So these industries are lucrative. It’s not an accident that everybody wants to have them.

And they have other characteristics as well. For example, these kind of industries, we call them advantageous industries in the book, they tend to compete not on pure price. They’re not producing a commodity where your profits get squeezed down. You’re competing on technological sophistication. You’re competing on sophisticated understanding of buyer needs.

You’re competing on vendor financing. You’re competing on the synergies of your product with other things. And that brings me to another topic, which is synergy, which is another characteristic of what we call advantageous industries. For example, this all goes back a long way. When you had steel first made in this country—cheap with the Bessemer process, it was originally done to produce railroad rails.

It was eventually adapted to produce steel frames for skyscrapers, which totally changed the urban geography—of the United States. So that’s a synergy, and the reason that’s very important is industries that have synergies are catalysts for economy-wide growth. Because any given industry, no matter how good it is, no matter how efficient it is, it’s going to be limited by the fact there’s going to be a limit to how much demand there is by consumers for whatever it is they make. But when you have synergies and you create new industries off of the initial industry, that process doesn’t stop. Now, to give you one example, you have the jet engine, okay?

Which is invented for military use in World War II. It’s invented originally in the UK—and Germany. And when you have jet engines, you can build high-capacity passenger jets, particularly when you have something called the high-bypass jet engine—which was also invented in the military. So you can have the revolution in cheap air travel. So you do that, and then the long-run result of that is completely unrelated and frivolous-sounding things like Disney World, which you can’t have if you don’t have airliners, which you can’t have if you don’t have high-bypass jet engines—which you can’t have if you don’t have jet engines.

Natural rivalries for advantageous industries (09:05)

Ian: So there are good reasons why the world is in a state of rivalry to be the host of these key industries. And what it means is there’s a case for subsidizing if you’re an economic predator like China, say. Subsidizing this industry so that your version of this industry can take over the whole of global demand. Because once you’ve got it, it’s incredibly lucrative to have it. Now, granted, I’m speaking in caricatures here.

So any economically sophisticated viewer will understand there’s 1,000 exceptions here. But the baseline fact that we’re dealing with in this book is the fact of rivalry to host industries. Now, more technically, this is actually a rivalry to host economic activities because—one of the things people point out is any given industry, like, say, smartphones, the final assembly of smartphones, such as is done in Foxconn City in Shenzhen, is a low-wage occupation. It’s not particularly lucrative. But manufacturing of the chips that go into them, manufacturing of the displays, manufacturing of the glass, these are sophisticated economic activities.

So—given the complexity of modern supply chains, you’re often not necessarily fighting over an industry per se. You’re fighting over parts of the supply chains, what’s called an economic activity.

Wolf: Right, the particular parts of the process that are the most technologically intensive have the most increasing returns, the most pricing power, therefore.

Ian: Yeah. That’s what you’re doing. Now, there are a number of reasons why this gets you outside the basic logic of free markets that we know and love. One is simply the fact that an oligopoly is not a free market. Yes, there’s competition—but it’s not free market competition.

That’s why you have pricing power, which you have rent-seeking. Dynamic rent-seeking is when you seek rents by trying to have a technological advantage. In the standard economic story, which is naturally true in this instance, rent-seeking is bad if you’re just seeking some privilege from the government or some obstruction—where you can get an excess return. That’s static rent-seeking. Dynamic rent-seeking is very different.

It’s not only lucrative for you, it’s advantageous for the economy as a whole because it drives technological advance. But free market economics has long had a difficult time with the fact that this doesn’t happen in a free market. It happens when somebody’s figured out a way to escape the tyranny of the free market where you don’t make any money. Now, this is why the kind of thinking that we espouse and that our book is based on, the real home of this is not in econ departments, it’s in business schools. Now, that should be—

Wolf: Right. They understand this stuff perfectly.

Ian: Yeah, that’s a real clue. It’s as if you had an argument between physicists and electrical engineers about the behavior of electrons or something, and the physicists are telling you one thing, and the electrical engineers are saying, “Look, if you assume that, you can’t actually build anything. If you actually want to build useful electrical machinery, you have to assume something slightly different.” So anyway, the kind of thinking that is in our book is exotic and revolutionary and, frankly, regarded as stupid in the mainstream economics profession—in the United States, with very, very few exceptions. But in business schools, it’s basically that’s the model that they have.

They have to. They have to think in terms of a world where some people are winners and some people are losers.

Why economic theory overlooked successful industrial policy (13:11)

Wolf: So yeah, I want to talk about those two ideas of the free market. I guess, so yeah, I was listening to what you’re saying here, and oh, there’s an oligopoly as a result of the free market. It’s not quite a free market outcome, though. In business circles, this is obvious. Everyone knows this.

Of course, this is the game you’re playing. Sometimes you get a huge amount of investment, speaking of subsidies for a country to get an industry. Uber or something might put an enormous amount of effort into capturing that activity, that market demand—and then ride on having the moat on it. So this is all very well known in business. But in economics, both in free trade and then in national economics within a country, there’s this very smoothed-out model where there’s perfect competition, and it’s all comparative advantage, and everyone gets the best outcome by the free market mechanism.

But why is it that they persist on that model, despite it being not actually the practical model, either by rational states or by rational business actors? Why is it that the economists got into this idea of that particular version of the free market, that very smoothed-out thing that doesn’t include these factors?

Ian: Well, there’s a bunch of reasons. The first reason, I think, is that there was this misplaced turn in economics after Second World War—where people really thought you had to make economics a science. Because there’d been economics before then. You’d had enormously successful capitalist economies around the world, and you’d had enormously successful reforms of the defects of capitalist economies. You had the antitrust era under Teddy Roosevelt.

You had the New Deal. So we had relatively sophisticated economic thinking for practical purposes. But economists had physics envy, is the best I can put it. They wanted to be able to reduce this to some absolute and precise science, where once you’ve got a certain objective body of knowledge, that just is how the world works, and you know, and anybody who disagrees, you can just mathematically prove to them that they’re just not right. So one of the things that happened is it led to a turn of excessive mathematization—of economics.

And one of the foundations of this was Paul Samuelson, who’s generally regarded as the greatest post-World War II American economist. In his doctoral dissertation, which I believe came out in 1946 at MIT—he figured out a way to mathematize the foundations of economics in a way that nobody had really thought of before. And before he died, he actually wrote an article warning people about how they’d misused this, and he actually wrote me a nice letter once. But the point here is that mathematizing the conceptual foundations of economics and mathematizing the entire structure—are two very different things. The problem is that if you want to build a complete mathematical model of the economy that’s going to tell you what’s going to happen given data about your initial conditions, the way you can model a system in classical Newtonian mechanics—you’re going to need theories that give you closed-form solutions.

That is to say, given the following set of facts, we know that the following dynamics will operate, and you’re going to end up looking like this. And that’s just what we can predict. Now, the problem is that the kind of thinking that’s embodied in our book and in business schools and in the policies of a lot of other countries, most notably in East Asia, involves something called multiple equilibria. Now, the easiest way to explain that is if you go back to our example of Google as a search engine. If you start off with two different search engines, say Google and Gaggle, that have roughly the same or exactly the same technology—which one is going to win is going to depend on a lot of contingent events that affect the path that their competitive activity follows.

And the same thing is true if you look at the evolution of, say, the auto industry or the computer industry is notorious for this. All the might have beens and could have beens in the tech industry of people say IBM could have owned the world if they just understood the situation—a bit differently in the late ‘70s, or Xerox could have been Apple or whatever. The point is that when you have all these completely contingent events—you can mathematically say a lot about a lot. I’m not disparaging the use of mathematics when it’s an appropriate tool. And my—original background was in the IT business, so I’m not a sissy about exact technical disciplines.

But the problem is the only way to have an overall model where you just keep piling up your model of this on top of this, on top of this, until you reach general equilibrium, where you kind of solve the whole economy as a kind of giant set of simultaneous equations, is you have to have these closed-form solutions where you can’t have phenomena like increasing returns which give rise to multiple equilibria. Which means that you can’t have some sort of general equilibrium answer that tells you what’s going to happen. So, there was a huge mathematical bias against acknowledging the kind of realities. Now, some mainstream economists have openly acknowledged this. Paul Krugman wrote that economics took the path of least resistance.

Intellectual, least mathematical path of least resistance. This is not a secret, if you dig into it. But you have this huge problem of the misunderstood idea that economics should be a science at all. I’m very dubious of the very concept of social science. You can’t do—controlled experiments on social phenomena to anything like the extent—you would need to build complete models.

It doesn’t mean you can’t have useful knowledge of these things, but I don’t think science is the right paradigm for that knowledge.

Wolf: And even the comparison to physics is kind of strange. The physics envy within the social sciences. But somehow, it’s never quite doing what the physicists do either. Because what you’ve just described, this process of making simplifying assumptions at every step until you can solve the whole thing as a simultaneous set of equations, is more like, in physics, the analogy would be we linearize everything until we can just solve the universe as a single set of equations. And you wouldn’t get a good result from that.

You wouldn’t expect to get a good result from that.

Ian: Oh, sure. One of the problems with this attempt to make economics a science is they didn’t understand how the science they were trying to imitate, namely physics, works. And I also don’t understand, frankly, why did you pick physics? Is—chemistry less science? Is biology, which is a much better if you want to have a controlling scientific metaphor for what you do as an economist, I think biology would probably be a much better science to pick.

And you can’t tell me that biology is unscientific. That’s absurd.

Wolf: Yeah, but it might just be the prestige of physics following the atomic bomb. Right? In that era.

Ian: Well, David Teece at Berkeley, who’s a great economist at the Berkeley Business School, I met with him last week, and he actually said exactly that. He said, “I think,” because he’s working on this problem at a much higher level than I am, because he’s actually a very esteemed professor over there. He says he thinks it was because of the atomic bomb. That the Manhattan Project just dazzled everybody. It’s like, holy cow, physics just solved the biggest problem ever of—defeating these two horrible hostile powers and then deterring this other horrible, hostile power.

If physics can do that, don’t we want to be like physics?

Why Ian wrote the book (22:17)

Wolf: Right. And so we ended up with this kind of spherical cow economics for almost half a century. And so, I guess that brings us to the conception of this book. Somehow you got onto this problem it sounds like in the early 2010s, mid-2010s, realizing that you wrote the book previous to this one, “Why Free Trade Doesn’t Work.” And I’m curious, what were those conversations like?

Where did that come from? How did you get onto the idea that this had to be done, that you should be the one to do it?

Ian: Well, I certainly didn’t want to. I wrote “Free Trade Doesn’t Work,” which I sat down to write in 2007. Came out in 2010. I got curious about the whole free trade controversy, and I went on amazon.com to look for a book where someone was going to lay out, okay, this is the case against free trade, because you’re constantly hearing the case for it, for it, for it, for it, for it. And it seemed to be creating all these problems.

And I wasn’t sure at that point, but I went looking for the book that will just give me the straight-up case why this is bad policy. And the book didn’t exist, so I basically wrote the first book, “Free Trade Doesn’t Work,” so that I could read it. And when I started, there were fewer than half a dozen intelligent people in the English-speaking world I could even talk to about why free trade was wrong. It was so much the orthodoxy. It was so much, if you don’t believe this, you just don’t understand basic logic.

of economics. So I wrote it because at that point, I don’t think I had the right to call myself an economist. I’ve been working on this stuff for nearly 20 years now, so I’d probably call myself an economist today. But at that point, I was just an IT consultant who was crazy enough to say something no economist would say. Because if I had had a PhD in economics, I never would have gone down that—path because I would have had just so much intellectual investment.

The time, the years you spend mastering all these models, it’s unthinkable on a purely personal level—admitting that they’re basically useless—for the real world. That would have been a gut-wrenching trauma. But because I didn’t know anything, I was able to start from scratch and arrive at what I think I can now probably say is a reasonably vindicated view of the issue—that it doesn’t work.

Wolf: And it just came out of that curiosity of, you wanted to have the full understanding. You were not hearing the other side of the question at all. So you had to get the case.

Ian: Well, I was lucky. I knew somebody who was at a think tank that no longer exists in Washington called the US Business and Industry Council. They gave me some sponsorship to write the book, which was kind of a crazy move on their part—but they bet on my ability to produce something worthwhile, and I had some other philanthropic support from people I was able to get interested in the—project because I was the only person interested in doing that at that time. When I was writing this book, honestly, I was terrified that somebody much more important than me would come out with the same book before I could get it finished. I was terrified that Paul Krugman would suddenly come out with a book six months before—

Wolf: And scoop you

Ian: I was finished. Yeah, and just steal my thunder and make me look like just a complete derivative waste of time. But it didn’t happen. It’s only much later that people got interested in, “Oh, so maybe the free trade model isn’t right.” People are starting to realize that now.

But—

Wolf: Yeah, it’s funny how these things go. It’s almost within economics, people assume that the space is more explored than it actually is, not just academically, but in the market. Like, “Oh, there’s no product for that. That must be because there’s something wrong with that idea,” or whatever. But sometimes it’s just no one’s done it.

And so many people are so conventional, whole fields can just be in lockstep like that for a long time.

How much state capacity does it take to do better than free trade? (26:46)

Ian: Yeah. Academic groupthink is definitely an issue, but what’s, I think, more fundamental is just the fact that when you scrutinize the theories in favor of free trade, I’m not even talking about the critique here. When you actually look at what the theories in favor of free trade say, when you look at people who say, “Okay, Ricardian comparative advantage is all you need to know on this subject. It simply answers the complete question.” When, as I’ve explained, Ricardian comparative advantage is true in a very narrow set of idealized circumstances, and it’s a handy mathematical gadget for figuring out what your benefits from trade are in certain circumstances.

It is not, and interestingly was not intended to be by David Ricardo, who invented this, he wrote as much, was not intended to be a complete model of the trading economy that applies under all circumstances. That’s the guy who invented it. Said that. I can document that. So you have those factors.

When you look at those models, the classical models, one of the things you find is they don’t, even on their own account, actually prove that free trade is best. When you go and read the people who are defending free trade, what their case usually comes down to is, “Okay, I admit of some exceptions, but I’ll bet you that no government could ever competently engage in the interventions required to take advantage of these exceptions and produce a better result.” So what they’re—

Wolf: It’s just axiomatic.

Ian: I wouldn’t say the problem is it’s axiomatic. The problem is that they’re actually, even the classical model, has a lot less logic in it than you think it has. Because when you get down to it, all they’re really saying is not that free trade is necessarily best, but I bet you can’t do any better with an intervention, is what they’re really saying, is that—

Wolf: That they’re saying it’s hard to do better.

Ian: Yeah, they’re saying it’s too hard to come up with an intervention that would be better. It’s too hard to know when you should do that. It’s too hard to have the state competency to implement it. It’s too hard for the political stars to align correctly so that you’re not just engaging in somebody’s rent-seeking. That’s really the bottom line that I ran into again and again and again.

Because the hard mathematical shell is there to impress people. It doesn’t, even on its own terms, prove as much as a lot of people seem to think it does.

Wolf: Right. Well, maybe we should talk about that aspect then, just this problem of the actual competency required to put this stuff into practice effectively. Obviously, China has done a very effective job at grabbing a lot of the world’s industries. But in a democracy like the United States, very mature, lots of political interests going around, lots of regulatory capture or lobbying this way and that. Is there maybe something to that, that despite the kind of theoretical possibility, it’s actually too difficult for the government to undertake this project?

I’d be curious for your thought on that. Is it something that is actually within our power, and these people who are making those arguments just didn’t believe that it was possible, but they were incorrect? Or yeah, how do you see the actual level of difficulty in our actually existing political system?

The dollar is overvalued (30:47)

Ian: Oh, I definitely think that doing what needs to be done is within the competency of the US government and the US political system. Because you got to remember, you don’t have to do this successfully, which means getting a much better result than we have now. It doesn’t mean doing industrial policy perfectly. Governments don’t do anything perfectly. But to get to a result that would be trillions of dollars better than we have now, you can make a fairly short list of things that we should do that we demonstrably have done things similar enough to that in the past.

One profoundly non-obvious example, important example, is the US right now has a massively overvalued currency. We have a currency that’s probably overvalued something 12% to 14%. There are different calculations. That’s not an observable. And our average real tariff is a lot less than people think.

It’s only about 10% or 11%. So the US right now doesn’t have a net positive tariff at all because we have an overvalued currency—which also handicaps our exports, which handicaps the competitiveness of American firms abroad. So if you want to get serious about resolving our trade situation, right now, we have a trillion-dollar deficit in manufactured goods, so you can’t re-industrialize the US if you don’t bring that demand home. Okay? So if you want to do that, tariffs are not going to do that.

Tariffs are great for controlling what you buy from abroad as opposed to make here, and to a lesser extent, controlling where you buy from abroad. You can’t shift the overall trade deficit down with a tariff of any plausible scale. But you can do that if you bring down the value of the US dollar, which was done before. It was done in the mid-1980s with the Plaza Accord in the Reagan administration when they were facing a lot of upset about the trade situation in Congress at the time. So what it would take to do that is you would need some kind of moderate variable tax on incoming flows of capital into the United States.

Because the only way the US can run a trade deficit is we have to give foreigners who send us goods and services, we have to give them something in return because we’re not trading with Santa Claus. And what we give them is financial assets. That is to say, we go into debt—to foreign countries. We sell them shares of existing assets, T-bills, stock in Microsoft. You name it.

So the entire deficit, which means the principal constraint on the re-industrialization of the US, is only possible because you have these massive ongoing flows of foreign capital. And the US is gradually buying up. The US is gradually being sold off.

Wolf: Yeah. If you take the kind of outside model on the whole thing, the government prints a bunch of money, the money goes overseas to buy foreign goods, the foreign goods come back, but then the money also comes back to purchase US real estate, US treasuries, US equities. And so the net result is we’re selling ownership in the American economy.

Ian: Yeah. You can’t get something for nothing here. People tell me it doesn’t matter. I’m like, “Okay. Who’s the Santa—in this scheme?”

It’s a profoundly corrupt way of living if you carry it on—this long. Now, the problem with a moderate variable tax on incoming flows of capital, which is not that hard to implement, I was party to putting legislation together that was introduced in the Senate in 2019, the Baldwin-Hawley—Bill. That bill was partly written by the organization Coalition for a Prosperous America, that I’m on the advisory board of. The methods by which you would do that, they exist legally. They’re within the competency of, depending on what agency you had supervise, it could be Treasury.

Our preference was the Fed. Make it part of the Fed. The Fed has an existing triple mandate for full employment, price stability, and moderate long-term interest rates, usually referred to as a dual mandate because people forget about the third part. But we would add a mandate for balanced trade by means of balancing the dollar to it. I’m happy to debate this, but as far as I can tell, it’s pretty clear that you could do this.

It’s not some strange moonshot—of weird ideas.

Wolf: Legally and institutionally, it’s probably not very difficult.

Ian: Other countries do things like this on a much smaller scale all the time. The US even did something like this in the ‘60s with something called the Interest Equalization Tax, was brought in under Kennedy when we were trying to hold the dollar up to honor—our Bretton Woods commitments. My point is, in terms of state capacity, you can do this. The problem here is that Wall Street would hate it because their business model, simpliciter, is moving capital anywhere in the world to get the best return. That’s what they do.

It’s what they say they do. That’s an accurate description. So anytime you put a tax in place that says, well, you’re not going to be able to do exactly what you want, or if you do, you’re going to have to pay for it, which means you make less profit, they’re axiomatically going to hate it. So you have to overrule them as an interest group. There’s a lesser pushback from people who fear that bringing the dollar down would be inflationary.

We’ve modeled this, and if you bring the dollar down gradually over, say, five years, the inflationary impact is not zero, but it’s very, very small. It’s a lot less than other policies in the US right now that cause inflation. So there’s no free lunch—but this lunch is well worth the real price it would actually cost.

Wolf: So the bottleneck at this point is just the politics of convincing enough of the coalition to override some of these special interests? Is that how you’re seeing it?

Ian: Well, if you’re talking about right now, during the Trump administration, I have to assume the decision is being made by the president. And the interesting thing is that you can document the fact that all three of his top economic advisors, that is Howard Lutnick at the Commerce Department, Scott Bessent at the Treasury Department—and Steve Miran, chairman of the Council of Economic Advisors, they’re all on record of having written about this problem, the overvalued dollar, and in one way or another, there are different flavors of it, what you could do about it. Conceptually, as far as I can tell, they all agree with me. So I have to assume, and I’m not a psychoanalyst or a tea leaf reader, I’m not going to tell you why President Trump has made the decision not to do this, but I think it’s pretty clear that in this administration, the decision would be his.

Our industrial policy is ad hoc (37:53)

Wolf: Right. And it just comes down to that. And so I guess let’s talk about then the bigger prospects of how you’ve seen things moving on this in the whole industrial policy set of questions. I think you mentioned in the book that industrial policy is coming back as a topic. Certainly, a few years ago when the book was published, we were seeing that, or when you were putting this together at least, there were people, Biden’s administration was talking about industrial policy.

Elizabeth Warren was talking about industrial policy. There were new magazines and projects and think tanks kind of talking about industrial policy. A lot of people were thinking about this topic. And so have you seen that kind of getting traction, besides the decision of the president or particular officials? Have you seen the broader culture of understanding and intention within the political class changing on these fronts?

Ian: Oh, sure. When we started writing the book, the idea was so far out of the Overton window, most people weren’t even against it. It’s definitely an idea. People know what it is now. People have opinions of it.

If you look at the very establishment intellectual institutions, that is leaving aside actual implementation of policy for a minute. But if you look at institutions like Foreign Affairs is the magazine that basically ratifies what respectable establishment opinion is once an idea has appeared there. They are pretty much endorsing a watered-down version—of our thesis. They certainly wouldn’t go as far as we would go. They talk about Hamiltonian statecraft, which is basically coming back to the idea that the US is in an economic contest with a rival superpower, and just as we had a contest against Marxism, it’s relatively easy to win because just Marxism is so dysfunctional.

We now have an economic contest between this thing we call free market capitalism, which sort of is, and then the state capitalist model that China embodies, and—they recognize that. The establishment has cottoned onto this, for sure. The way they seem to be responding is the way that the powers that be always respond in the face of a threatening innovation they can’t get rid of, which is they figure, “What is the smallest change that we can make—that will absorb this challenge to the system and enable us to preserve as much of business as usual as we can?” Which is why you’ve seen things like the Trump administration has been very aggressive on industrial policy for semiconductors, and they’re kind of following that up with industrial policy for rare earths. This stuff is being pushed.

In terms of a policy for the whole of the economy, in terms of looking systematically at what industries need what kind of support—to thrive in the current international economic environment, that’s definitely not there. I don’t know if you saw this, but when the Trump administration killed most of the EV subsidies that Biden had put in place in the Inflation Reduction Act, that caused in excess of $50 billion in write-downs in the American auto industry. That’s not some estimated number from some economist. Those are actual SEC numbers go into annual reports where you go to jail if you lie about it. Those are hard dollar numbers from actual capitalist business corporations that the administration had no qualms about basically hanging out to dry because they don’t like green energy.

Wolf: And how do you read that? What’s the implication of them doing that kind of thing—

Ian: Well, I think in that partic—

Wolf: for industrial policy?

Ian: Sorry?

Wolf: For industrial policy.

Ian: Sorry. I think in that particular case, it’s quite clear that the administration has, for whatever reasons I shall not speculate on they really don’t like green energy, with the exception of nuclear, which I sometimes think might just be because nuclear power annoys those liberal hippie dips. I mean, nuclear power—is politically incorrect, so it’s the only green energy that the Trump administration likes. And also interestingly, just to get some subtlety here, geothermal is also popular, and I’ve been told the reason for this is that the technology you use to drill geothermal wells is pretty much identical to what you use to drill oil wells. So everybody in the petroleum business who drills for oil is really gung ho about this because it just gets them more—

Wolf: They can reuse the stuff

Ian: customers. Yeah. It’s basically drilling difficult deep holes for hard rocks in the ground. That’s what they do, and that’s what—geothermal is. So that, to some extent, is a special case because they have a particular animus against a particular technological project of decarbonizing our energy system.

The larger concern is the lack of systematic thinking. The fact that it’s all a bunch of impulsive, ad hoc responses to particular problems that, for one reason or another, the administration or people close to them have grabbed a hold of and they realize are important.

You have things like shipbuilding is apparently... They’re talking about shipbuilding, that’s going to be—a big thing. Thus far, we really don’t know how serious that’s going to be because the United States is absolutely nowhere in ships. It’s quite astonishing. The US builds something like one tenth of 1%—of the world’s ocean-going ships, which is—

Wolf: That’s insane

Ian: really quite shocking. We probably produce a larger percentage of the world’s pogo sticks for all I know.

Wolf: Right. And I guess this gets to this, I don’t know, extended ecosystem of difficulties for these kinds of policies. We’ve mentioned all these special interests and political issues around how things are perceived ideologically. There’s also, you mentioned rare earths. Rare earths really makes me think about the regulation issues around that, right?

It can be very polluting, or it can be perceived as very polluting, or there’s a lot of, I understand, regulatory barriers to a lot of these industries. How do you see that fitting into the industrial policy picture? A lot of this isn’t really about the economics even at some level, in the sense of whether the industry has the capital to get going or whether the government is pushing on things financially, but just are they allowed to do it? Is it efficient to do it under the current regulatory setup or those kinds of things. I guess I’m most curious about the, let’s say, non-economic barriers in business, the regulatory barriers.

How do you see those fitting in?

Ian: Well, regulation almost by definition is the part that government can have something to do with. Yes, granted, in some circumstances you have government directly developing technology or taking an equity stake, but for most of the economy, and I am by no means a fan of promiscuous interventionism everywhere because you think intervention is just something you want to do. I do get the Hayekian argument, I really do. But take one example. I was meeting with someone yesterday who works on the hyperscale build-out of AI data centers.

Datacenters (46:13)

Ian: And the constraints that they’re running to, one of them is local zoning from municipalities that don’t—want these giant facilities. It’s a myth that they consume all that much water—but it’s not a myth that they consume a ton of electricity. We have these giant facilities, each of which contains a few dozen jobs because it’s a data center, it’s not a—

Wolf: Right. They’re not doing that much

Ian: factory. And we’ve seen things that were reasonably anticipatable, like Governor Hochul in New York having a statewide moratorium. But then the shocker for me was when Governor Greg Abbott in Texas—did the same thing, because in recent decades, Texas has been the place where you can get—

Wolf: You can build stuff

Ian: you can get things built. Right. And then I asked this guy I was talking to, “What has the administration been able to do about this?” And the answer is, well, really not a lot because we don’t, unlike some countries, have very much national land use planning—in this country. We have the Department of the Interior for forested mountains and stuff like that.

But for urban land, the federal footprint on land use planning is very, very small. The ability of the federal government to do something about this huge backlash against data centers—is very small. And my question is, at what point do they just start bribing rural areas to accept these? So that’s a kind of constraint where the federal government has not been able to do anything. The part where the federal government probably would have an easier time if they would but try is to do with something called the interconnect for electric power.

The issue with data centers is not just that they consume a lot of electricity. The issue is that to even get hooked onto the electric grid as a major user often takes years, and that’s something called the interconnect queue. And interconnect queues in this country are mainly administered on a lottery basis, so you can’t just buy your way—

Wolf: Oh, wow

Ian: to the head of the queue. Now, that’s a relatively technocratic problem where you could hypothetically step in—with a fix. Maybe they do that tomorrow, but up to this point, they haven’t. And in case you hadn’t heard, the Stargate project, the big AI initiative, the administration has basically gone belly up. They got some things built, but they abandoned most of their sites, most of which I believe were then picked up by Microsoft.

So, a sporadic and spotty effort, even for things that get the administration’s attention.

Wolf: Right. This reminds me a lot of the California high-speed rail problem. When they tried to build the high-speed rail, one of the big things they ran into was just every little municipality they had to go through had all kinds of its own process, and barriers, and veto power, and so on. So these things that seem like obvious things that would make the economy stronger or otherwise sort of serve the industrial goals that the government even wants to do, has enormous funding to do, and is willing to put weight onto it. We’re still running into these problems where we can’t actually do it.

In favor of incrementalism (50:10)

Wolf: It’s a difficult problem, this whole atmosphere, not only of the political problems that prevent the government from trying to take action, but then all the little organizational problems once they do try to take action. And in the successful cases of industrial policy, I’m thinking of Japan or China to some extent, often there’s whole government departments that have quite a lot of power to organize these things. Or maybe it has to be done in conjunction with all these non-governmental parts coordinating the sort of non-governmental elite interests in civil society and so on that could influence that political climate. And I wonder about the prospects of that kind of thing. Not just from the officials using their established powers to get these things done, but from the broader ecosystem of the extended government reorganizing itself or being reorganized by someone to make this kind of thing really happen.

That feels like it’s this incredibly difficult task, but of course, it’s been done under FDR and so on in past times, and people have said that maybe now we need things like that. We’re sort of overdue for that kind of cleanup in that system. But I’m curious about whether you think there is stuff like that needed for industrial policy, or is that too far off and you think it can be done with existing mechanisms pushing on the margin despite all these problems?

Ian: I and my coauthor are very much in the second school of thought. You’re absolutely correct that the East Asian societies above all have much more centralized governments where they can make these kind of highly centralized decisions on behalf of the economic interests of the society as a whole. And that does seem to work for them. Leaving aside any issues of whether it would be a good or bad thing—there’s no way that’s going to happen in the United States. You’d be talking about a political earthquake to motivate change on that scale of such incredible size that I just don’t find that plausible.

More importantly, I don’t think you have to. I gave currency as an example of something that’s well within state capacity. If you look at the fact that the Biden administration did correctly observe that electric vehicles are not just about climate change. I don’t care. Even if you don’t believe in climate change—electrification is the direction that the auto industry is going globally.

So if you want the US to have a competitive electric vehicle industry in a world where it’s competing with subsidized players around the world, you have to give your electric vehicle industry here a boost. You can’t just wait for the market to catch up or you lose.

Wolf: Right. You have to invest in it. You can’t just be scared.

Ian: You have to. It’s about the next generation of the entire auto industry, which is huge and important. Yes, it has this environmental aspect, which is important, but even if you left that out of the picture, it was a policy that made sense, and they did do it. Now you have a different administration gets elected, and they canceled most of that. We have a political process in this country.

People make these decisions politically in a democracy. What else would you expect? That doesn’t mean it can’t be done. It means we have to make the right decisions. In the long run, that’s true of absolutely everything in government.

The hidden American industrial policy of the late 20th century (54:19)

Ian: One of the reasons why the US got so stuck on the free market idea for so many decades is basically that we cheated. We told ourselves we were living in a free market completely free market economy, and it performed well. In point of fact, we actually did a lot of industrial policy behind the scenes, only we didn’t call it that. So we talked ourselves into believing in an economic model that was not actually what we were doing. I’ll give you a couple of examples.

I’ve been banging on about currency because it’s so important, even though the topic tends to make people’s eyes glaze over. From the end of World War II to 1971, you have the Bretton Woods—managed currencies, fixed exchange rate system. Now, that was created to solve related problems that had occurred in the 1930s, but thanks to that system, it was almost impossible for national currencies to get so far out of alignment that nations could run huge trade imbalances. And if you look at the trade imbalances in the world in that era that people fussed over, they’re a rounding error compared to what we deal with every day. So that was a huge industrial policy no one acknowledges as such.

Or if you look at state-supportive technology, which is one of the other big aspects of industrial policy we haven’t touched on as much as perhaps we should. Where do the most important technologies come from? Well, there’s this myth that they get invented in some dude’s garage in Palo Alto, and he makes a billion dollars. Look, God love those people, but the most important technologies of the post-World War II era, things like semiconductors.

The transistor is not created in a free market environment. It’s created in a state-sanctioned monopoly called—AT&T, which had Bell Labs. And for decades, they had this sanction from the government to be a state-regulated monopoly that had this incredible technology research arm that produced... I have a list in the book somewhere, but the obvious one is you have the transistor, you have the laser, you have all sorts of other stuff like sound movies. You have an unbelievable list of technologies came out of this one company, which was not state-owned, but it sure as hell wasn’t a free market.

And the reason—that they had the money to do this incredibly long-horizon research was totally antithetical to free markets. For a start, they had a guaranteed market for whatever they invented, not because they earned it in a competitive marketplace, but because the federal government guaranteed that they would be the phone company. So if they spend billions of dollars inventing lasers based on the idea that one day they’re going to be useful for long-distance telephone communication, which is true—they can do that because they have the long time horizon, which is why so much fundamental technological research did come out of companies that, for a brief or long period of time, had a monopoly or quasi-monopoly position. You probably know about Xerox PARC—which invented the mouse, the touch screen, all this great stuff. Xerox didn’t know what to do with it, but they were another company that had this incredibly profitable near monopoly on making Xerox machines.

And you have the David Sarnoff center of RCA, which develops color television, things like that, when RCA was this incredibly formidable—company, which it no longer is. Radio Corporation of America. It was set up by the US Navy—after World War I so the Brits wouldn’t own all the important radio tech. So if you look at other military technologies, jet engines, obvious, rocketry, satellites, all these technologies were not originally created for profit. They were created to defend us from the Soviet Union.

And the lesson here is not necessarily that we want to become Prussia—and militarize our economy, because it’s not necessarily the military that’s going to be fruitful for the next generation of technologies. But you can’t develop, or you often don’t develop the most important technologies on a straight-up market for profit basis. Now, let me be clear, sometimes you do. The last two major technologies that were developed on a for-profit, free market basis were the electric grid and the automobile. And the reason that was possible is that intermediate stages of those technologies were monetizable.

Before Henry Ford created the Model T, cars were expensive, specialized toys. Well, there was a market for that, so you could make money selling them and develop the technology that was eventually dumbed down and cheaped up to make the Model T, and then you have the mass production car. Electric power was useful even before you didn’t have a nationwide grid, so you can develop the technology step by step until you get to the full-scale thing. Now, the example today that comes to mind is AI. Because although there is a big governmental footprint in the development of AI, and I don’t want to disparage it, the key innovations like the paper, “Attention Is All You Need,” which came out of Google.

Wolf: Yeah, it’s Google.

Ian: The reason that they’re able to not just do the theoretical research behind generative AI—but actually implement it is because the intermediate stages are useful. I believe that for quite a few years, the single biggest use of AI was just the recommendation algorithm in YouTube. This comes out of some history of that I read. I could have this wrong, but the point is that... And that’s, of course, immensely valuable because YouTube is a going concern.

So if you can have a better recommendation algorithm that uses AI to figure out what video the person wants to watch next, and they do watch the video, that’s worth a ton of money. So anyway, the fact that AI is monetizable in the intermediate steps between zero and AGI—is the reason you can do this in the private sector. But—

Wolf: Not every technology

Ian: not every technology historically works that way. There’s no guarantee the next technology will work that way. You have to have strong support for both approaches. Otherwise, you’re going to get technologically surpassed by some society that is willing to put down the money. For example, you may have heard recently, I forget the gentleman’s name, but there’s a professor at UC Berkeley working on new materials who—just announced that he’s decamping to China, where I assume the government has given him a blank check to develop new materials.

That’s a huge field. Advanced materials. Japan exports more every year by dollar value of advanced materials than they do of cars, but nobody knows about this—because advanced materials are not sold directly to consumers. You buy a carbon fiber tennis racket, okay, you know it’s got advanced materials in it. But for the most part, this is a huge industry that most people are not aware of—and the US is not stepping up to the plate to make sure that we’re number one in this, and China is.

Wolf: So there’s these existing industries like advanced materials, things like carbon fiber manufacture, where I think it’s very well covered by the theory that you’ve laid out. But there’s, I guess also implicitly there, and what we’ve just been discussing, is the development of new technologies, maybe even new industries, new economic activities, that we haven’t identified yet as being valuable. For example, the laser or artificial intelligence or the internet or the various things that have ended up being incredibly valuable.

The missing intermediate steps in industrial development (1:03:33)

Wolf: With the concept of the advantageous economic activity or the advantageous industry, I think there’s at least a clear guide of where government attention and support should be flowing in terms of developing those existing industries, capturing those existing industries. Do you have guiding theory for how we should be investing in things like actual new technology? Is it just spray money everywhere, just support for R&D generally, or through academia, or through things like DARPA, or the sort of XPRIZEs and challenges? Like they just had that Heavy Lift Challenge earlier this week, I think, or last week. What kinds of mechanisms are best in your eyes, and what’s the theory behind that?

How do we know where to put the support?

Ian: I got an answer in two parts. The first part is, paradoxically, figuring out what the technologies of the future are going to be—is actually not that hard. There are a lot of people putting out lists. Robots, you think that’s going to be a big technology of the future? Yes or no?

Genetic engineering, yes or no? Fusion power, yes or no? Electrically powered automobiles, yes or no? Electrically powered aircraft one day—yes or no? It’s actually not that hard to peer into the crystal ball and see what technologies are going to become important.

The White House Office of Science and Technology Policy puts out an official list. There are a bunch of other institutions who do. That is not actually the hard part of the problem. The hard part of the problem consists in the fact that the path from pure scientific discovery, where you come across a law of nature that makes something possible—and standing up a commercially viable industry that actually sells products to customers and makes a profit for its investors and pays wages to its employees. That path is very complicated, and there are a lot of steps in that path that are not well taken care of in this country.

For example—we have very strong pure science capability support by the federal government. You have National Science Foundation, you have NIH. Peer-reviewed science is a well-understood thing, and there are problems with it, but for the most part, we know how to administer that and get good results out of it. Turning a ready-for-the-market technology into a company is also something that Americans are very good at. There are a lot of entrepreneurs out there.

There are a lot of venture capitalists willing to fund them. There are a lot of engineers willing to work at startups. People understand that model. People like that model. People believe in it.

Everybody wants to do it. So it happens. The difficulties are mostly in the steps between pure science and usable technology. Because a lot of technologies go through many, many steps where you’re not talking about pure science, that is discovery of laws of nature. And you’re not talking about something you can actually sell to a customer.

And the US is not that good at funding that intermediate step. People have this idea, which goes back to Vannevar Bush and the Endless Frontier report just after World War II. That the public sector should fund science, because science is a public good, and the private sector should invent and make things, because that’s what inventors and entrepreneurs do. That’s too simplistic. That’s leaving out this giant missing middle.

Now, what we have in that missing middle is a strange patchwork of compromises. For example, the military, and I mentioned this before, is probably the single biggest institution in this country that pays for developing technology. Not researching science, not discovering laws of nature that make a stealth aircraft possible, but figuring out how to actually make the materials that you put on the surface of an airplane—that will do that. And they did this, as I mentioned, for jet engines and for aircraft generally, for satellites, for rockets, for a whole bunch of other stuff. I really hate it, by the way, when people use Tang and Teflon as the example of governmentally supported innovations, because those things are just not that important.

Okay? No. Jet passenger planes and the internet, okay?

Wolf: Yeah. Those bigger things.

Ian: So part of how we get around this missing middle problem is we’ve had the military do it. Now, for many decades, we also had the phone company—

Wolf: Yeah, Bell Labs

Ian: do it. So it was the military and the phone company were our technology development institutions. NIH does it for pharmaceuticals. They do it reasonably well. They’re actually one of the better functioning parts of the pipeline.

But when you have some possible technology that you can look at the science and say, “Okay, bet you could make one of these, and if you did, it’d be really great. You could do all sorts of things with it.”

When you have a technology like that, that isn’t pharmaceuticals, and it doesn’t have obvious military implications, it can be very hard in this country to find support for these non-monetizable steps of the technology path.

There’s fragmentary institutions here and there that will do a bit of this. But, for the most part, the US has not systematically implemented across the board support of non-commercializable steps in the technology development path. We haven’t even really, as a country, gotten our heads around that concept. It’s a very important concept. The average voter should have it in their heads that technology doesn’t grow on trees.

Some of it is going to be done for a profit. A lot of it, before you get to the better mousetrap, you have to make the better cheese, and to make the better cheese, you need the better this, and nobody will eat it. And so these steps in the technology development path that are neglected, that’s a big part of industrial policy this country could do. There are models for doing it. They certainly exist.

Wolf: Mm-hmm. What are the best models of other countries doing it or having done it in the past? How do you think that should actually be done? Is that a new office of technology development, or is it some other kind of initiative with the universities? Or how do you think would be the best model to actually do that?

Ian: Well, around the end of the Obama administration, when a gentleman named Ron Bloom, an investment banker, was finishing up the auto bailout—he asked President Obama, kind of as his parting shot, to establish something that was then called the National Network for Manufacturing Innovation. It’s now called Manufacturing USA. And it’s modeled on an institution in Germany called the Fraunhofer-Gesellschaft. Which basically, it’s a network of institutions. Germany has about 100.

Each institution is focused on one technology, is partly funded by government and partly funded by industry. So you have an institution that’s focused on, say, there’s one that we have in the US now. We have a small network with just over a dozen institutions. So you have one, for example, I think it’s in North Carolina, focused on power electronics, which is one of those obscure but desperately important things for the electronic systems you need for controlling and managing large amounts of electric power. Apparently, that’s a field that’s in a state of innovation and flux.

And what you do is you have part support from the US government and part support from industry. And what they do is companies can go to these institutes in—Germany for example, and some company says, this is probably a terrible example in terms of the actual engineering, so please if you know the technical details of this, don’t write in and make fun of me. But as an example, suppose you had some company who says, “We have this great idea. What if you had ceramic-coated titanium ball bearings?” “Nobody’s ever done that, though we know what titanium is, and we know what ceramic is. But we think that under some application of high temperature or high pressure or whatever, that the concept of a ceramic-coated titanium ball bearing might be just the greatest thing ever. Now, we don’t want to devote a ton of our money to researching this for a couple of reasons. One, if this turns out to be a great idea, you can patent the particular details of what we then manufacture and our process. You can’t patent the concept.”

Right? “So we’re willing to put some money into this, but there really does need to be some public sector support for this idea because if it works, the benefits are going to be generalized—to the whole of the economy or the whole of the industry, not just us.” And then Fraunhofer-Gesellschaft has engineering labs where they can do that kind of thing. And we have Manufacturing USA in this country, which desperately needs to be bigger and needs to be better supported by the federal government. It has an annual budget of $300 million a year, which is chump change on the federal level.

But that’s one model for how you go in and support the missing middle of technology development. There are other cases. For example, the National Renewable Energy Laboratory in—Golden, Colorado, which just renamed itself the National Laboratory of the Rockies in order to escape the ire of the administration. I know, I know. I was talking to some people from there a couple of weeks ago.

Anyway, NREL was responsible for a huge chunk of the improvement in the efficiency of photovoltaic cells. The Department of Energy had a project called SunShot. Okay? It’s a play on words on Moonshot, where I think it was, was it double the efficiency of solar cells? I think it was double.

It might’ve been something different. So they did that. It was so successful, it finished three years early, and they doubled down on it with, I think it’s called, what? SunShot 2030. They did it again, and lo and behold, government scientists can develop useful technologies.

It’s not beyond their capability. It’s not a waste because once you develop the tech, then anybody could use it, so it benefits the whole economy. It’s a reasonable idea that developing technology, for some technologies, not all, but for a great many technologies, it’s a reasonable responsibility of the federal government. There’s a track record of its being done successfully. We have institutions, albeit far too small, that have done it already.

How to make sure domestic R&D benefits domestic industry (1:15:55)

Wolf: So in the case of solar technology in particular, to really take advantage of that, you would also want to actually control the solar industry domestically. Right? If you make the solar industry more valuable, you’d want to actually get a slice of that. But I understand most solar panels are manufactured in China. China has this enormous global market share in solar technology.

So how much do things like those kinds of innovations end up just being that that inadvertently is a research lab for Chinese manufacturing as opposed to something that actually benefits the American economy? How do you see that problem playing out? Is this something we need to focus on building up the actual domestic industry first and invest in the things where we actually have industry, or is it somehow that we benefit despite not having the direct industry?

Ian: I’m glad you asked because, forgive me for blowing our own horn for a minute, but Coalition for a Prosperous America, the organization I’m on the advisory board of, has for years been the biggest lobby in the United States for appropriate tariff protection of the US solar industry for precisely the reasons you’re talking about. And an interesting paradox is that lately, the Trump administration, which does not generically appear to like the idea of green energy, has actually gotten pretty good on putting in place the tariff and other measures that are going to be needed to rebuild not just solar panel assembly, not just photovoltaic manufacturing, but the entire supply chain. They’re actually starting to get reasonably serious about that. So yes, you do have to do these things because it’s not beneficial to create a technology only to have it commercialized in another country. So this is why I tell people all the different parts of industrial policy fit together.

You might think—that tariff policy has nothing to do with scientific research. Well, no, it does because if you want your science to pay off economically, you need the technology research built on it, and then you need some way to ensure that your domestic industry is competitive, not just doesn’t have its market taken away from it by foreign competitors who burn coal to run photovoltaic plants full of slave labor. But you then also, as I mentioned, have to have a competitive currency. So all the parts fit together. If you try and do industrial policy as a bunch of isolated one-off initiatives, you’re going to get some successes when the problem is obvious and the solution is a narrow package of policies.

So many of these problems, you won’t. You don’t necessarily have to have policy intervention everywhere. As I said, look, I’ve read Hayek. I understand that the default assumption of free society is the government leaves things alone. But you do always have to be asking the question when you look at an industry, even if your default policy assumption is to do nothing, you have to ask the question: Does this industry need industrial policy interventions to thrive in this country?

Because increasingly the answer is yes.

Wolf: Right. There’s this integrated domain of industrial strategy and industrial policy that ties together all these different seemingly disconnected elements that work together to support the most advantageous industries and make sure your country stays economically strong.

Ian: Yes, and you don’t have to put it all in one government department. You don’t have to have—a federal department of industrial policy or an industrial policy czar or some office bossing everybody about. You can have a decentralized process, but you still need everybody, every decentralized player has to be thinking systemically. And I’ll give you one example of that, that’s actually specific to our book. As I’ve been promoting this book, trying to get people to read it, the real breakthrough in getting people to read the book was when we discovered that the number of people in this country who are interested in industrial policy itself is not zero, but it’s pretty small.

The number of people who are interested in something that they think industrial policy could get them, that they want, is huge. You could be talking about anybody from people in the Air Force who are worried about drone industry, to people in green energy who want a green energy industry, to people worried about imported generic drugs poison people. There’s just a vast range of people, and I’m astonished every day—I end up interacting with somebody who I discover has some entirely new angle on something they specialize in, they want, where industrial policy is interesting to them because it could help them to get this thing.

Wolf: And it ties it all together between all these different little things that people are focused on, all these little—struggles that we’re having all over the economy. They’re solved, this very central problem of can we actually maintain overall industrial economy?

Ian: Well, that’s why I think people need to think systematically, particularly—in a free society with decentralized decision-making because if the decisions are going to be decentralized, people need to see the whole picture so they can see where they fit in it. If nobody’s going to be telling them what to do, they need to know.

Wolf: Right. Well, I think that’s a good place to leave it. Thank you very much, Ian. This has been a great discussion. I really enjoy your work.

I think it’s very important. Definitely, we do have to be thinking systematically about how the economy actually works and how we can make it run better.

Ian: Well, thanks, Wolf. Thanks for having me on your show.

Wolf: All right. Until next time.

​Sober, risk-averse, rational businessmen make great captains of industry—in established industries which someone else already built. But the new industries which drive progress come from wild half-mad ideologues. This lecture covers the psychology of the men behind the major breakthroughs, from the steam engine to the large language model.

Sign up here to attend future Students events and lectures.

Transcript

The psychology of pioneers (00:00)

Hello, and welcome to The Students Talk series. I’m Ben Landau-Taylor, here to talk about new industries, where they come from, psychologically what’s going on with the people who put them together. So there’s an article from 1982 in the “Miami Herald”, a very cute local public interest story. The reporter’s going around interviewing the valedictorian of every high school in the county. Most of it’s like, “Gilbert Scott Fine, 17, first in a class of 270, will major in computer science at Drexel University. He said, ‘Basically, I’m very motivated. If someone else can get an A, I can get an A.’” Most of it goes on like that.

“Jeffrey Bezos, 18, first in a class of 680, will study electrical engineering and business administration at Princeton University. He wants to build space hotels, amusement parks, yachts, and colonies for two or three million people orbiting around the Earth. ‘The whole idea is to preserve the Earth,’ he said. His final objective is to get all people off the Earth and see it turned into a huge national park.” A little bit different.

So these people who end up doing things like founding Amazon tend to be somewhat odd in the things they care about, the ways they operate. This isn’t just Bezos in particular. I think if you’re looking around the big industrialists today and looking at who are the weird ones, Bezos would not be the first to come to mind. And in fact, this goes all the way back to the very beginning of the Industrial Revolution. With the steam engine, which was invented by James Watt and his business partner, Matthew Boulton.

Watt was on record saying... I’m going to butcher the quote, but it’s something like, “Nature has a weak point, and we can vanquish her if we can but find it out.” Boulton, who was sort of the Steve Jobs to Watt’s Wozniak, was this guy who, when he was in his 70s, he heard that there were some burglars who wanted to break into his factory and steal the payroll. And so he told the watchman who they were trying to bribe, “Okay, yeah, let them in, and then I’m just going to get a bunch of my friends and pass out guns and ambush them. It’s going to be a blast.”

And he was doing all sorts of wild antics like this. And if you look through the history of industry, a lot of these characters... Which is kind of odd, because we also have this idea of the captain of industry, this very responsible, very sober, very risk-averse person in charge of all these things. And there actually are a lot more of those people than there are these kind of wild and crazy guys. But they tend not to be the ones who invent the steam engine.

They tend not to be the ones to make these really big sort of breakthrough new industries. They tend to be people like Tim Cook. Did not create Apple computers. He’s very good at running Apple computers after somebody else built it. Leland Stanford.

He built the leg of the Transcontinental Railroad and operated it, ran it very well, made a lot of money. Became governor of California, left a huge fortune to create Stanford University. He’s a much more regular guy, what you would think of as an ambitious, responsible captain of industry type. He built the railroad. He did not invent the railroad.

Decision-making motivations (04:28)

And you need a lot, you need like probably several thousand Leland Stanfords if you want to run an industrial economy, but you also need these other guys. What’s going on here? So now we have to get into the psychology of what do people do for money, and what things are people going to only do for reasons other than money? So there’s the sort of common economic psychology idea is we sort of approximate people as rational economic agents who are making sensible choices about how to make money and then how to spend money in ways that will produce utility for them, fulfill their values, however you want to put it. And everyone will say, “Okay, this is kind of an approximation. It sort of describes things, but it’s not exactly what’s going on.”

Which is true, and if you take it as a sort of approximation, it’s a very useful theory. But especially at sort of the very extreme outlier cases, it breaks down. So let’s talk a little bit more about the actual psychology of how people deal with money and what motivates them. So there are basically, I think when I go around and you look at why do people make economic decisions, why do I make decisions about where I’m going to work, how much money I need, what I’m going to do with my money, what I’m going to buy. When I look around at the people I know, why are they doing what they’re doing?

It seems like there’s a bunch of different reasons people have for the things they’re doing. Most of the big decisions seem to fall into one of three rough categories. There’s kind of like impulse so there’s a lot. If you find yourself spending 20% of your income on DoorDash, you’re probably not sitting down with your green visor and being like, “This is the best way I can maximize my utility given my spending.” You’re probably much more impulsive about it.

And for some people, that ends up being the main way that they end up making a lot of these decisions. If you’re just getting really mad at your boss one day and you quit. Another one, the one which for me ends up being how I make most of my decisions is custom. You get these sort of scripts from society of like, this is what a responsible person does. This is a good way to make a decision like this.

Oh, this should be the relationship between, you should have this much money to get a house like that. For someone in your social class, it makes sense for you to be taking vacations. But you shouldn’t be taking those types of vacations. That’s too expensive for where you are. Maybe you’ll read a book.

Maybe you’ll listen to a podcast that explains to you how you should invest your money. You should be putting it in this type of retirement account. And for the most part, I don’t derive all of this stuff from first principles with pure reason. I just listen to people who have been put in a social position where it seems like they know what they’re talking about. I check to make sure that it makes sense, and I follow that.

And sometimes people make decisions based on reason. There have been a few times in my life, I could probably count them on my hands, where I’ve made a big economic decision by thinking things through from first principles and coming up with a genuinely new idea and doing something different. I have ever done it, but I sure don’t do that a lot. And this is not 100% of all of it, but I think for most people, this is how most of their decisions get made. And for these sort of Tim Cook, Leland Stanford types, they’re making a lot of their decisions based on, in this sort of reason mode.

However.

Yeah, so reason ends up being very important and especially important in these positions because if you’re making decisions about big economic things based on reason, you’re going to make better decisions about how to run a railroad, how to run Apple computers than people who aren’t. And so there’s this filtering effect, in our society, not true of every society, where the people get sorted into these positions. And the history of Western society, when you started seeing more people, when you started seeing a lot of the big concentrations of the biggest wealth and capital go from the sort of knights and nobles and landlord classes, which was to the extent they were using reason, they were reasoning much more about power than about economic growth and financial returns. But then over time, the control of it shifted to the sort of city-based burgher class who was much more, “Oh, we should be using reason to try to maximize our returns.” And so this class ended up managing more and more of the wealth and reshaping society around that in some what eventually proved to be extremely important ways.

So right now, we live in a society where lots of the big piles of wealth and capital and Apple computers and all sorts of big infrastructure, the people who are good at using reason to think about things end up sorting themselves into that. So even though I think it’s a small fraction of the population when it comes to the management of these big economic forces, the sort of idea of, oh, we can approximate humans as the sort of homo economicus, reasonably okay approximation of a rational agent becomes a lot more true of these sort of captains of industry types than it is of a random selection of the population or all of the people who were at my high school or something like that.

Ideology vs. accumulation (10:32)

However, when you get to why do people want the money, this is another place where it sort of breaks down at the extreme. Because to a certain extent, you can get a bunch of nicer things if you have more money. But this does kind of break down after a certain point. If you are a psychologically healthy person who is doing a reasonably good job of fulfilling your psychological needs through the sort of normal channels and have fulfilling relationships, how much money can you spend on consumption? A fair amount over the course of your life, maybe $50 million, maybe $100 million, but probably not $10 billion.

And so if you’re looking at the people who end up amassing these big, big personal fortunes, it is not unusual for someone to get to this level of, you get a fortune or an offer of a fortune that you can just get everything that a sensible person needs for their life and someone offers to buy your company for $200 million and you’re like, “Great. I’m pretty much set. I’ll take that.” But for the people who keep not just having a job, but really pursuing and keeping going on these kind of soul-crushing, soul-grinding, continuing to try to build more and more and more long past that point, it’s going to be something other than like, “Oh, I just really want to be able to buy a bigger house for my kids,” or something like that. And so if you look at those motivations for a bunch of them, there’s this sort of very bourgeoisie sort of stewardship and running capital kind of as almost a ideology unto itself.

Of, oh, just the thing that you do, you’re just so inculcated into this idea of I just want to be managing bigger and bigger piles of money for its own sake, or bigger and bigger piles of capital. And among the Tim Cooks and the Leland Stanfords, this is often what gets them to...

Tim Cook is a very rich man, but he keeps going. I’m not going to go too deep into that because there’s also this other type, the sort of Thomas Edison type, the Jeff Bezos type, who they’re using economic reason, but not because they want the money for this sort of normal consumption. Jeff Bezos cannot do what he wants with $50 million. Jeff Bezos wants to move all of humanity into space. That’s very expensive.

He’s been moving basically $1 billion a year into his rocket company because, like he’s been saying since high school, he wants to move humanity into space. The reason why he started Amazon in the first place was because he wanted to make a gazillion dollars so that he could fund the rockets, and he’s doing that. He wrote about it many times, starting when he was a kid. So he has this very strong goal, which requires a tremendous amount more money. There’s a few of these.

More common is the sort of more of the James Watt type, where he’s like, “If we can find out nature’s weak point and vanquish her.” He has this strong ideological drive to build the thing, to do what he’s doing. A bunch of his letters, he talks about how he’s just kind of obsessed with this machine. It’s not about the money for him. Before he went into that, he was making pretty good money as an engineer building canals, and came very close to bankruptcy.

It took him many, many years of tinkering to get the steam engine working. But he couldn’t leave it alone. He just had to advance it for its own sake. And this is something you see quite frequently.

I could go on. I could tell stories of the different industrialists, and all of the weird things they’re on about for... I could fill the entire talk with that, so I’ll try to keep that to a minimum. But I’m sure you guys can fill in a bunch of your own examples, especially of some of the people today. But one of the things I will talk about very briefly, because we are in San Francisco, and they will probably throw me out if I don’t mention it, is if you look at where the artificial intelligence has come from.

This is from people who want to... They’re not just like, “Oh, yes, this is a way that I can maximize shareholder value. This is not a way that I can make my company be more valuable so that I’ll have more money.” These are people who want to end the age of man and replace it with the machine. When you look at the people who are behind the most successful companies.

If you’re looking at what are the big, important companies pushing forward the state of the art in artificial intelligence, who’s leading them? The one who’s kind of the least insane in this way is Mark Zuckerberg, and I think it’s not a coincidence that they are not the head of the pack, while being a lot less ideological about trying to build the super intelligent machine than someone like Dario Amodei. So we see this all over the place.

To do this, when you’re trying to build a totally new industry, it requires not just risky financial bets. It does require risky financial bets in a way that the sort of managerial ideology doesn’t really like, even if the expected value may arguably be high. But it also requires teaching people new ways of doing things. It requires sort of reordering society to make space for the new thing that you’re doing. People have to learn, oh, if I need socks, then I’m used to going to the store to get socks, but there’s this other thing I can do.

And some of the younger audience members may not remember how weird it was when people started just being, I’m just going to give money to some stranger on the internet, and they’ll send me socks? This really threw a lot of people. And when you look at the introduction of basically any new technology, people have to sort of be taught how to use it. They have to be taught what it’s for, how to fit it into their lives. This is very hard to do, and is generally not the sort of smoothest way for a person to make money, even a lot of money.

Creative destruction (18:04)

Often more significant is that it provokes political opposition. Schumpeter, who was one of the great economists of the 20th century, talks about this process of creating very new products, very new industries, and thereby obsoleting the old ones. His term is creative destruction. People don’t like it when you destroy their stuff, and so they will usually oppose you. Schumpeter calls the history of industrial progress a history of revolutions.

Every revolution has two sides. And so when Uber was coming out in France, the taxi drivers got out of their taxis and they would go and they would smash up the car of every Uber driver. And the police were like, “Yeah, that’s legit. We’re going to let them do that.” This was a very dramatic example, but the amount of legal opposition and obstructionism, that you are going to face if you’re trying to do something new and obsolete some other guy’s business is going to be quite substantial.

And a lot of people don’t want to put up with that. One, it’s bad for your returns, but two, it’s just really psychologically taxing, and it makes the potential downside of what if it doesn’t work a lot worse than, oh, well, you could lose some money. So in order to plow through all of this, it tends to not be something that someone is going to do just for mere money. And so this is why when you look at where did the new industries come from, they tend to be done by someone who had something much more ideological going on or some other very strong psychological drive where it was kind of the only way.

If you’re trying to create the machine which will succeed man, if you are trying to move humanity into space, all of the ways of doing that that actually might work involve doing a bunch of these very hard things, and it’s sort of the only route that you have to do that. So to illustrate some of the difference between these two types, one historical case I want to talk about is the history of General Electric, which of course came out of the Edison Electric Company. And if you look at the life of Thomas Edison, he’s absolutely one of these wild, crazy guys who doesn’t really fit into the system and doesn’t play well with others. So, and when he was a young guy, he was a telegraph tinkerer. He’d work at different telegraph companies, buy a bunch of parts on credit, try to invent stuff, and then skip town when his debts from all of the experiments got too big.

Eventually, over the course of time, he got two big inventions that were advances in making better telegraphs, basically, which he sold these for a big pile of money. And a less ambitious person would’ve then just been like, “Great, I’ve got a big pile of money. I’m going to retire and maybe buy a farm.” A more ambitious person might’ve been like, “Great, I’ve got a bunch of money. I’m really good at this telegraph stuff. I’m going to make a new telegraph company.”

Edison, of course, didn’t do that. He started his Menlo Park laboratory, where he was just going to work on totally different things. He ended up inventing the light bulb and the power plant to power the light bulb and became stinking rich off of that, started running power companies as a business. And if he had then been like, “I’m going to be the electrical power magnate,” he would still be an incredibly famous guy and an incredibly rich guy. We would still be reading about him today.

But he was incapable of stopping, and so he went on, he invented the phonograph and the motion picture and got very involved in those businesses. Those did quite well. He didn’t stop there either. He kept trying to invent new things, some more successfully than others. In his old age, perhaps he was getting senile.

He ended up losing most of his money on a... He had some idea about what he thought was going to be a better way of making cement, poured a fortune into it, and it just totally didn’t work. So he kept doubling down, kept doubling down, kept inventing stuff, just never ever stopped for a moment. As part of that, he ended up handing off the operational control of the Edison Electric Company, which eventually got rolled together with some of the other competitors into the General Electric Company. So a few decades after this, the manager of the General Electric Company, was eventually Owen Young, who was born as a poor farm boy, became a successful lawyer, sort of rose through the ranks internally, was making deals, a skilled manager, eventually, was put in charge of the company.

He was a very ambitious man, but unlike Edison, who was very much his own man, Young was someone who would generally get his ambition from society. My colleague Wolf has an article, “Don’t Get Your Values from Society.” Owen Young did not agree with that. He was being persuaded. He was a very important progressive activist.

He sort of got onto that path when he was... There was this, I think it was a Unitarian minister who was sort of traveling through and persuaded him to go to this college and get on that track. He was a disciple of Herbert Hoover and all of those guys. When he became this big business magnate, who was running GE, he was very involved between the World Wars. They got a bunch of the American businessmen to renegotiate the financial, all of the war debts that had been taken on and the reparations that Germany was supposed to pay to the victorious nations.

He was—so the first of those, the most famous, was the Dawes Plan, and the second was the Young Plan because he was sort of leading that one. And then later on, the Navy asked him to basically buy out the British Marconi Company, which ran all the radio infrastructure, because the British were using that to spy on everybody, and they didn’t want... Like today, you don’t really want your telecommunications infrastructure run by other countries that might spy on you. So because these nice men from the United States military asked him to, he bought out the Marconi Company and organized it into the Radio Corporation of America, which manufactured a lot of the important radio equipment for many decades. So he was very good at what he did and was very much a man of the system.

And a very different type going after very different things. You definitely need a lot of these guys. It’s very good for where humanity is today that he ran GE and that he put together RCA and did such a good job with it. But this is not the man who is going to invent the radio. So this ends up being important for how you run a society, because these kind of insane people running around, breaking everything around them, sort of reordering how their society works so that their gonzo ideological vision can be achieved.

Societal tolerance for chaos (25:35)

Most societies don’t like that. Most societies try to stamp that out. If you try to do that today in Germany or Spain, it won’t work. They won’t let you. And that’s the norm.

The places that will actually let you do this are historically quite unusual. Right now, America is basically the only place, certainly the only place in the West, where you can get away with this. It wasn’t always. It used to be that Germany and England were also quite good for this. So the Marconi Corporation, which Young bought out, was started, of course, by Marconi, who...

That was a British company, but Marconi is obviously not a British name. So Marconi was Italian, couldn’t get any support for what he was doing in Italy with his radios. So he went to where you could actually do new technology, so he went to England. He didn’t even get off the boat before the customs agent was looking through his luggage and is like, “The heck is this weird future-looking apparatus?” Immediately called Scotland Yard, sort of their intelligence agency, who just immediately swooped in and basically gave him venture backing, before he even got into the country.

And this was absolutely creative destruction, obsolete. This was just a couple of decades after largely British money had laid the transcontinental telegraph cables. This was incredibly expensive at the time. Basically got obsoleted by having good radios, but they didn’t care. And I don’t think that’s what Britain would be like today, unfortunately.

But unlike France, where you might smash up the Uber cars and they’ll just be like, “Yeah, that seems cool.” America still has this idea that we expect our industrialists to be a little crazy. We expect them to break things, and we’re going to let them get away with that to at least some extent. So I feel kind of proud of myself going this long talking about this without yet mentioning Elon Musk. But one of the notable things with him, he certainly provoked his share of opposition.

But earlier on, before he had provoked quite so much opposition, one of the things he did was... He gets very angry at people who are short-selling his stocks, and he lied on Twitter about some of his plans and pricing around that, in order to try to mess with them. And the Securities Exchange Commission is like, “Well, that’s incredibly illegal.” And if Elon Musk had been in Germany at the time, they would just would’ve been like, “Okay, we’re going to destroy you now for securities fraud.” But instead, that’s not what happened here.

What they ended up doing was they reached a settlement in two days. I don’t know of any other lawsuit that has been settled in two days, where they ended up fining him and Tesla a combined $40 million and was like, “Don’t do it again.” So it’s like, okay, yes, we’re definitely going to fine you, but we’re not going to destroy you. We’re going to leave you in charge. They could have just stripped him, been like, “You’re going to get fired. We’re going to put someone else in charge.”

But they’re like, “No, don’t do that again,” but it’s having a rambunctious, trouble-causing guy in charge of driving industry forward like, yeah, that fits. We shouldn’t take this excuse to destroy that. And this ends up being probably the single biggest reason why... If you look at where, since the end of the Second World War, where all of the breakthrough technologies, there’s lots of incremental technologies that have come out of other places. In recent decades, especially China, where we’ll be like, “Okay, we’ve figured out how to build the quadcopter drone.”

They’ll be like, “We have built better, cheaper ones that go further.” But all of the big, important breakthrough technologies have first been done in the US. Not true of science, but it is true of technology. And a big reason for this is that it’s the only place that will let you. And the Marconis have to come here.

There isn’t really anywhere else they can go. One of the big questions I have about the coming decades is whether China is going to be able to figure this out. Because they do actually have, if you look at the people who are in charge of a lot of their big companies, a lot of them are these kind of crazy ideological types, whether it be some vision of the science fiction technology they’re going to build and make real and instantiate in the world, or whether it be the national greatness of China, or whether it be some weird idiosyncratic thing that you’re going to have to read 700 pages to kind of, but not really understand. They do have a bunch of those guys leading the companies, but they can’t get away with quite as much as they can in America. And in particular, my sense is that if you are doing something that has been first proven in America, then they are happy for you to do that.

But I don’t know if they like you doing things that have never, ever been done before. I don’t really get it, but it seems like something kind of like this is happening. And it could be that they have definitely been more permissive of this than they were 20 years ago. And so are they going to be able to do not just the scaling things up, but the actual invention? I think that depends a lot on whether their society figures out a way to make space for these people who keep kicking over everyone else’s sandcastles.

And I guess we’ll have to see how that goes. So thank you.

Samo Burja and Ben Landau-Taylor cover bureaucracy—why the golden age of bureaucracies worked so well in the early 20th century, how bureaucratic class interest overcomes checks and balances, and bureaucracy’s role in today’s Western governments.

Subscribe now

Samo Burja on Twitter: https://x.com/benlandautaylor
Ben Landau-Taylor on Twitter: https://x.com/benlandautaylor

Transcript

Intro (00:00)

Ben: The United States of America was founded as a republic, became a democracy, and these days has turned into mostly a bureaucracy. I’m here with Samo Burja. Samo, you’ve done a bunch of theory on the political science of bureaucracy. Can you tell us about that?

Growth of political bureaucracy (00:25)

Samo: Yeah, it’s a pleasure to be here again. With regard to bureaucracy, I think it was almost an unintentional development in the history of the United States, specifically. We think of a change in the form of government as a change in the sort of underlying theory of government and also the theory legitimating government. Bureaucracy, meanwhile, has sort of snuck up on us, right? It has come to control a shocking amount of the economy, a shocking amount of government operations, and it has become a purpose unto itself.

And democracy is not unique in its vulnerability to bureaucracy. We have many historical examples where, say, monarchies, perhaps even republics, could be argued to have been destroyed by their own internal bureaucratic structure. And one of the more important things on bureaucracies, which we both write about, is sort of the nature where the power that they wield is borrowed from some other source, right? A bureaucracy is almost always sort of a delegate of someone, be it the scribes and priests of the Pharaoh or the civil servants of the prime minister in the United Kingdom or the many, many administrators working beneath the President and Congress of the United States. The power is borrowed, but as it is rendered illegible, it becomes irrecoverable for the original source without destroying the bureaucratic structure.

I think a key point of bureaucracy is that it defends itself over time. At first, you delegate the work. The work becomes inscrutable to you. Perhaps you want to fire some people. You don’t know who to fire.

And at that point, everyone who has a job in the bureaucracy, maybe either the ruler or head of state has founded or inherited from their predecessors. Everyone who’s working in that structure is, in a way, first and foremost, a paid advocate of their own existence. You can’t automate fake jobs, unfortunately.

Ben: Yeah. So it’s become quite well known at this point that bureaucracies have this nature of they grow, they sprawl, they sort of devour all of the resources put into the organization, and yet we keep coming back. Every organization is like, “All right. Well, we know all of that, but we just need the bureaucracy anyway.” So it seems like they’re also doing something that at least many people believe, and I think in many cases they’re right, you just can’t do as well any other way.

So why do people need bureaucracies so badly that even with having had all of these stereotypes for over a century, people keep doing them?

Samo: Well, we could think of it as a two by two matrix, right? Where you consider competent people you delegate to, incompetent people you delegate to, people who agree with you and are your allies and pursue the same mission, and people who are your enemies. If you think about it, the worst possible quadrant to have in your organization is very competent, agentic people who actively disagree with you or are your enemies in an organizational mission sense. So really what bureaucracies solve, and I hate to invoke a metaphor with the fashionable technology, so AI and alignment, but really, in a way, it’s solving human alignment. Instead of having, say, as a king, very powerful dukes, you end up having relatively, at first, pliable bureaucrats, people who depend on their salary on your tax collection, do not have, at first, an independent base of power.

The strengths of bureaucracies (04:00)

Samo: I’m sure the Pharaoh thought the same way with the scribes, right? At a certain point.

Ben: So you’re saying at first. So it sounds like basically this gets you several decades where you’re reasonably in control, and then it just ends up becoming when you set it up, you know how the bureaucracy works, and then eventually you die and some other guy comes in who doesn’t know how to wrangle the thing. So it seems like you’re basically buying a decades-long boost at the cost of then these things. You see this in sort of the clearest cases with a lot of the Ford Foundation or these other big... Some big rich guy dies and leaves a bunch of money.

When he’s in control, he’s giving it to Carnegie’s building all of the libraries or doing whatever things they personally care deeply about and think is a good idea. Bill Gates has, while he’s been alive, he’s been giving away his money in ways that are kind of different to the ways a lot of the bureaucrats who are in control of the Ford Foundation and all of the Carnegie Foundation. They really don’t like nuclear power. Bill Gates personally does like nuclear power, and so he funds a bunch of things like that he cares about and thinks are a good idea. And it seems like he’s built this bureaucratized foundation which he can control while he’s there, but it doesn’t seem like its ways it’s distinctive are going to survive after Mr. Gates himself.

Samo: Well, and already you note that this is the case with technically non-government entities like foundations. Corporations themselves become bureaucratic. Anyone who doubts that can, say, look at a once great American aerospace corporation like Boeing or consider any of the other old companies, right? Companies that have been around for a century or more. So because of that, we can think of a of the rise of bureaucracy in American society as perhaps first beginning in business and the necessary scaling of industry.

This leads to the sort of virtue of bureaucracy. One way to think about it is perhaps at its best, it’s a paperwork assembly line. You break down very, very complicated tasks into small, simple steps. You find people who are qualified enough so there’s an aspect in which bureaucracy is a form of talent rationing, right? If you’re scarce on aligned talent, and you have only very small or medium-aligned talent, or perhaps mixed in some unaligned talent, as we said earlier, you can go pretty far with still assembling these pieces in the right way.

So I do think that a functional institution, a functional bureaucracy can exist and can often scale very well for any kind of information processing. And many of the roles and tasks of government, industrial production, or other kinds of ventures are best done that way.

Ben: Yeah. In the first half of the 20th century, we had the American government bureaucracies were arguably the most effective organizations on the face of the Earth, where they would be the way that they were just electrifying a lot of the more backwater areas of the country, going to the moon, the stuff that the US Army was achieving during the world wars where they’re like, “We’re just going to sail across the English Channel, bringing with us an entire port.” The logistical and information process, the stuff they could achieve very reliably was just—

Samo: Staggering

Ben: Yeah. You look at it today and people just often just have a hard time believing it because it’s so far from what our bureaucracies are able to do now. And so, yeah, it does seem like in many ways it was still bureaucratized, but it was just a much more functional type of bureaucracy which was able to achieve a lot of the greatest logistical feats that this country has ever achieved.

Samo: Well, and perhaps it’s no accident that many of those greatest feats were tied to military organization in the first place. An army at war has a very clear feedback loop of success and failure, right? Whether you win or lose battles.

Ben: Well, that didn’t save them during Vietnam.

Samo: Not during Vietnam, but the interesting question is, were the war goals achievable in the first place? And arguably, the US military actually fought the Vietnam War fairly well. I know that there are all of these critiques that have been traditional for it, but consider the first unmanned drones were not deployed in Fallujah in the 21st century. They were deployed in the Vietnam War. The US Air Force actually deployed some drones during the war.

You can say what you will about chemical warfare. Winston Churchill was famously in favor of it, said that he didn’t see the reason for the squeamishness of blowing someone up versus having them choke on poison gas. I think most people’s intuitions are that there is a special reason for the squeamishness, but there’s some still gonzo ambition in being sort of proposing Agent Orange saying, “Oh, we’re going to just have the jungle cover die, and then just terraform the battlefield with mass deployment of chemical poisons and beat the Viet Cong that way.” That is arguably still like an echo of the military mindset.

Ben: Yes, but I think it is an echo, and I think that—I think that they are, by most standards, I think that they were working quite well at that sort of micro level. And yes, that’s a separate question from whether the strategic—

Samo: The macro level. Yes

Ben: orientation made any sense. But I do think if you compare it to the American army during the World Wars, which is, to be clear, an absolutely insanely high bar. I think it’s clearly not at that level.

Samo: I would agree. Especially their understanding of the sociological and psychological and other factors of war had greatly diminished by that point, where they essentially did not know how to break the morale of the Viet Cong.

Why the golden age of bureaucracy worked so well (10:23)

Ben: Yeah. So everyone loves to complain about bureaucracy, and even at its highly functional periods, people really loved to complain about it because there are, even when it’s working, there’s a lot of ways in which it can be annoying. But I would like to go deeper into, in that sort of golden age of American government bureaucracy, what was working so well about it?

Samo: Well, partially, it was very good at talent recognition. It was very good at ingesting huge numbers of people who sometimes were freshly literate, perhaps first-generation or second-generation literacy. Now, again, American society had been fairly literate for a long time, but had absorbed a large wave of immigrants who the literature at the time, today we see decried, “Oh, it’s bigoted, it’s wrong.” No, no. The literature was quite accurate about the masses of people coming in from Eastern Europe, right?

Including from, say, my country, et cetera. They were often not literate. They often came to the cities, experienced public schooling and education. The school system was already the first bureaucracy they encountered. The second bureaucracy was probably the US Army.

At that point, I think the typical US company was not yet deeply bureaucratic, except for the largest industrial employers. Say, Ford Motors is as much a bureaucracy as the Ford Foundation is, and it’s from about this time period. So what is happening, essentially, is that you have a standardized basic education, which is followed by a standardized military education at ingestion. You’re recombined into these new teams, given clear, well-worded, explicit instructions. And I think that part of what was happening by the 1970s, and has happened even more today, is that simply reading a set of instructions or actually even reading does not seem to have the same psychological effect on the masses that it once did.

I do not have a full theory of why this is the case, but I do think that there’s a very real way in which following simple instruction, we are sort of subliterate. So I’m not even complaining about the masses not appreciating the finer points of 19th century literature, right? Not really understanding Joseph Conrad when they’re assigned him in school, or Ernest Hemingway. But I’m actually complaining about the ability to follow basic technical instructions, and I think something has gone a little bit deeply wrong with human capital. So yeah, for a society to be bureaucratic and functional, it actually has to produce very punctual, reliable people who perform simple instructions.

The worst you can say is that they are not encouraged to really think about the instructions very much, which again, when the system starts dissolving, might produce the sort of effects we saw in Vietnam, micro-competence and macro incompetence.

Ben: Yeah, okay, this is interesting because it does seem like we’ve seen in a bunch of parts of society, I think we were talking about in the science video how the per capita number of PhDs in America is across the 20th century increased by 30 times. And if you look at the per capita bureaucrats across America, I don’t think it increased by 30 times, but I do think it probably increased by somewhere on the order of five to 10 times. And if you look at what is the station of a bureaucrat in 1910, then this is a pretty well-respected job. That’s notably in the higher income strata and tends to be quite sharp people. And then it seems like over time it is basically just a lot more people and they tend to be just...

Probably part of it is whatever’s happened to education over the last half a century, but also a lot of it seems to be, oh, we’re no longer trying to look for the sort of really sharp people and put them in the bureaucratic roles because we have so many bureaucratic roles that we just need a lot more people. And so the sort of median bureaucrat is just... I don’t think it’s ever been the case that the median person in a society would be able to just reliably be this functional, conscientious, reliable rule follower in the way that you’re describing. Obviously, no one is ever perfect at this. But it does seem like if you’re looking at the Babylonian scribes, these are all pretty sharp people, which you can do because this is a pretty small class.

And so, yeah, I wonder if part of it is this makes sense. It’s like you just need the sort of baseline median member of it to be a much more reliable node in the process than you’re getting with our current social sorting.

Samo: I think that intuition is correct. I think you are right about this. Partially consider among the more successful historic bureaucracies, the Chinese bureaucracies, right? Driven by the civil service exam system. It was incredibly selective.

Ben: Yeah. The sort of most cutthroat, meritocratic test-taking environment in the history of the human species.

Samo: And one of the few ones where bureaucracy was considered an explicit virtue. We said that usually bureaucracy doesn’t govern in its own name, but the Confucians come about as close to it as they can. Right? There is still a derivation of authority from heaven and the emperor. But the emperor is also supposed to be basically a good Confucian bureaucrat, if you follow the Analects and so on.

So essentially, I think that they felt that a bureaucrat should be both able to follow the paper instructions, be familiar with the correct rites, but perhaps also write some good original poetry. And I think that’s about the highest standard we have ever seen in any historical civilization for a bureaucrat.

How bureaucratic class coordination circumvented American checks and balances (16:20)

Ben: Yeah. One thing you said there reminded me of at the beginning, you were talking about how the American bureaucracy and the history of this country sort of came about almost by accident. Which doesn’t really actually match my impression, so I wanted to go back and poke at that because—

Samo: Let’s debate it. Yeah

Ben: So my understanding is that a lot of this was pretty explicit by the sort of Herbert Hoover, Woodrow Wilson type of early progressive faction where they were talking about it as sort of like professionalization and how you need all of these apolitical roles, and I don’t think they were using the word bureaucracy explicitly that much, but the thing that they’re describing was very much this, we need this apolitical apparatus of permanent professionalized employees who are making the decisions about how the government functions, and they need to be insulated from all of these sort of democratic sort of mass movement winds blowing this way or that way.

Samo: And also isolated from the profit motive, right?

Ben: Yes. At the time, there was, I guess we’re certainly seeing it again, but the looting of the treasury by the sort of voting bloc factions was a huge thing that they were trying to circumvent with this, which I think they succeeded for a while. So I do think that that was very deliberately designed and installed, was my impression.

Samo: No, you’re definitely correct for the progressive era. And in the progressive era, the view was, and the fear was that, oh, we need to beat both the robber barons and the unions, right? And the unions today, we don’t think of them that way, but at the time, they were considered a dangerous form of mob rule.

Ben: Well, at the time, they were the literal bomb throwers who they’d be going up against the factory owners would hire a couple of thugs with, most of them would be coming carrying bats, and one would have a machine gun, and then the unions would get together and burn down the place in the middle of the night when the goons are gone. And yeah, it was, in fact, much more of a subversive force than it is since FDR just incorporated them into basically the government.

Samo: In a way, we could think of the rise of someone like Teddy Roosevelt as being essentially the wielding of the press and general public opinion against both of these forces to build something of a civil service. I concede that point, but I would still say that in the modern American telling of the history of the country, we have sort of the Republican turn in the American Revolution, and there’s a little bit talk about Andrew Jackson and Lincoln, so you could say that the expansion of a republic to democracy is also narrated. But I don’t think the rise of bureaucracy as its own power center as almost a new branch of government, perhaps the omni branch of government, right? Where it kind of weirdly fuses between, say, Congress and the presidency in a way that political theory never quite says today, or at least doesn’t look at today very closely. I think that is still a surprise, is an emergence from the early 20th century we stopped talking about even as it’s continued to grow and dominate our lives.

Ben: Yes. So I think this is basically a matter of the historiography tends to really downplay it. I think it’s been happening—and it’s been quite deliberate and has been drawing on these sort of progressive ideas of the professionalized civil service in a very explicit way. But it’s just—when you’re getting your history textbooks that talk about the three branches of government, it just doesn’t make it into these sort of self-descriptions. Yeah.

Samo: And I do call it an omni branch because the bureaucrats always copy each other’s homework. It might seem funny to Americans, but I guarantee whatever regulation the European Union passes next week, it’s going to substantially make its way into any proposal for new regulation in the United States.

Ben: Right. Often through very explicit, I’ve looked at this closely in the financial realm, where the different financial bureaucrats from the different countries will have conferences where they get together and the one where they’ll be like, “This has been working really well. You guys should do it.” Or they’ll be like, “We need a new standard because it’s a huge pain when each of us has our own different compliance, whatever. We need to get all of this on the same page. It’ll be so much more streamlined that way.”

And it’s very much like they’re just literally sitting down and—

Samo: Coordinating

Ben: coordinating. And I would imagine that in a lot of these other domains, you probably see very similar stuff.

Samo: Yeah. And because of that, I think they have bypassed the classic checks and balances in the American system to a great extent. Again, there’s some strange unholy fusion at play in a number of US federal agencies where you can find a theoretical authority of the president, a theoretical authority of Congress. Practically, neither of them could really dislodge the federal agency. And what is an interesting development in the 21st century is that for the first time, people have become aware of it as a center of power.

Now, whether it was successful or not, the DOGE administration, the DOGE attempt of the Trump administration, the second Trump administration, especially when Elon Musk was actively involved, was an explicit attempt to fight the bureaucracy. And I would say it 95% failed at dislodging the bureaucracy, but I’m sure it succeeded at saving US citizens some tax dollars.

US political structure today (22:29)

Ben: Yeah. And one of the mechanisms by which a lot of this has happened as it’s sort of been shading more and more into that is, at least in the American case, I know less about other countries, but a lot of this has been through the action of judges who are on the definition that you were talking about earlier, they are themselves a type of bureaucrat. They are in these positions of borrowed power where it’s because they are in this particular position that they’re forming this very particular, very sort of information processing, very local role in the overall network. And so I think part of the reason why they keep ruling in favor of expanding the powers of bureaucrats is because they are themselves bureaucrats and have a class interest, basically, in this being the way that the system is run. And so that is when you look at the American case, so much of the, “Oh, well, it says here that this is under congressional control. Why doesn’t it actually follow when the Congress changes its mind?”

seems to come down to this being the mechanism.

Samo: I think you’re broadly correct at this. In some countries, judges are just straightforwardly appointed. In other countries, there are elections or at least election recalls. However, interest in local government has waned at all levels of even the most democratic societies. Even if you go measure the social science statistics in a place like Switzerland with the most active participatory local democracy, that is the case.

Most people can’t really name judges that are responsible for their counties or anyone really short of the Supreme Court. And then the Supreme Court, I would argue, is a little bit different in the United States.

Ben: Yeah. It’s less bureaucratic in the United States and more like honestly, it’s the closest thing we have in the country to the role that Congress was serving in 1960.

Samo: There’s this interesting thing about politics where people assume that if you have a guy named a king or a group of old men called the Senate, that they do what the label says. But in politics, there’s this interesting way where even if you try to abolish the tsar, you just might end up with a general secretary. You might trade in Nicholas II for Stalin. If the body politic has a Congress or a legislature-shaped hole, your liver is not supposed to be a heart, but the Supreme Court is a surprisingly functional Congress right now. It’s not supposed to be, by the Constitution, the business of making of laws, but it manifestly is making—more laws than Congress.

Ben: Right. When Congress stopped doing that, the Supreme Court, they keep writing in their legislation that this is properly the role of Congress, and so this is how we’re interpreting the thing that Congress passed in 1983 or whatever. But if Congress makes a new law, of course it would supersede this, other than the Congress just doesn’t do that. They’ve basically just abdicated the legislative role, and I think it is better that the Supreme Court pick that up than just let it lie on the ground not being done, because you really need somebody doing that, even if it... In the ideal world, the Congress would keep doing it and the form would match the labels, but I think it’s to their credit that they stepped into the gap when nobody else was.

Samo: I once was very critical of people’s and general citizens’ perspective on the presidency, where they vastly overestimate the practical powers of the president, sometimes assuming he’s like the CEO of the country, which he really isn’t. It’s a very limited position, even today, even with the imperial presidency. But over the last two or three years, especially as I’ve looked at Congress much more, I’ve come to the conclusion that the people hallucinate great powers in the hands of the president because they correctly understand the presidency is the last lever of democracy still working.

Ben: Yes. It’s the only part of the apparatus in the American structure where the will of the people actually leads to changes in how the governing is done.

Samo: Exactly. It is the only place where the citizen can actually express their will. And people might say, oh, local politics and so on, and sure, maybe in a city like New York, the local politics still matter. But this has retreated so vastly compared to, say, 1880 or even 1980. The field of democracy, practical democracy, that is, people power, has shrunk, and it is now in the hands of the president mostly.

That is probably the most impactful decision at all levels of government, everything from immigration and so on. Everything from immigration to energy policy to policy with China to war, et cetera. Deep, deeply important questions, not to mention also legislating on social issues. It’s sort of interesting how the president’s opinion on something like, say, gay marriage. It’s starting to approach something like the opinion of the Pope.

Ben: Yeah. There’s this common structure where the head of state sort of slowly transitions into a ceremonial, more of a spiritual leader. The popes were, at various points in their history, were also secular rulers. I guess it’s clearer cases in the Japanese emperors or the English kings or the Ottoman sultans, but there does seem to be fairly early aspects of that that we maybe see starting to happen with the US president. I guess there’s always been some of that, but it seems to be becoming more notable, just, I guess, by default, as the legislative power has three quarters disintegrated, one quarter just moved over to the Supreme Court.

Samo: Now, it isn’t necessarily the case that the only way to do a democracy is through the president. You could imagine a political reform movement that aims to make Congress functional again and would propose changes to Congress such as, I don’t know, maybe term limits to make sure that there’s a limited window within which people can have an impact or not, rather than just passively collecting rents from essentially the economy. Here, I’m also thinking of, say, Nancy Pelosi’s investment portfolio. Sometimes the rents are literal.

Ben: Right. And hers is the most famous, but it’s a very normal thing for that type of insider trading among the members of Congress these days. She’s far from alone.

Populism vs managed democracy (28:55)

Samo: Yeah. So I would say that what is usually called populism is probably the only broadly popular movement to reintroduce democracy in Western democracies, and it is the attempt to use the executive through the presidency, and in many countries, you could say maybe through the prime minister. There is a broad analog to, say, at least European populism. Interestingly enough, no one has really proposed fixing the legislature, probably because they correctly understand that the courts and bureaucrats have overpowered the notional representative legislature.

Ben: Yeah. So just as a terminological note, I think we need to distinguish democracy, as in the people, the demos, is making the rules versus democracy as the practice of elections, because I think that these are actually quite different concepts, and as we’re trying to distinguish them, we should be using different terms. But, yeah. In general, yeah, I think there’s a lot of... Sorry, where was I going?

I distracted myself with the terminology note, so I’ll kick it back to you. Sorry.

Samo: No, no problem. Terminology is important. If words are not clarified, things cannot be carried to their conclusion.

Ben: Yes. The rectification of names.

Samo: Exactly. Another good Confucian concept. I would say that, again Democracy, when people say populism endangers democracy, that would strike people in the 19th century and even in the 20th century as a really bizarre contradiction, right?

Ben: Right. But if you distinguish these two concepts, then the people ruling disrupts the practice of elections is a much more sensible concept that you can agree with or disagree with.

Samo: I think when people say populism threatens democracy, the most charitable take I can give it is that essentially, the people are being taken in by demagogues. This is a known failure rate of a failure of democracy. You can even find it in classical political theory in Athens and so on. It is hard to argue. There’s certainly elements of, say, populist administrations that are less competent than would be desired.

They certainly sometimes lack technocratic ability. That not always the case. There’s some excellent workers in them as well. But, I think that’s how they think of that. And then what they’re really saying, in a way, is that, okay, maybe the people vote for the general vibes of what they want done, and then professionals, AKA bureaucrats, decide what actually gets done.

And they sort of pick off the wish list of the unruly children. So it’s, in a weird way, a form of a paternalistic managerial democracy, a managed democracy. And I think if I were to propose a updated political science term for Western government, I wouldn’t call it liberal democracy. I would call it managed democracy.

Ben: Mm-hmm. I’ve been thinking of it in many ways as much more of a ceremonial democracy and that you follow the forms of the elections, but—the degree to which the elections end up actually determining how the country is governed has been attenuating and attenuating. And right now, the remaining big lever in the United States is through the presidency. In a lot of the European democracies, there just kind of really isn’t one at all. You’ll have these votes and that just ends up mostly irrelevant to the actual governing.

Samo: Well, Eastern Europe is a little bit different because the bureaucracy there is very young, so it ends up expressing itself as small national bureaucracies in balance with national governments basically clashing with supranational bureaucracies in Brussels and NATO and beyond.

Ben: Yes, I should say I’m talking about Western Europe there.

Samo: Western Europe. I would agree about Western Europe, though I would propose an exception to France, where while the President of France currently seems squarely aligned with the bureaucrats, right? Emmanuel Macron, for example, was always considered the technocratic candidate. And you know what do we mean by technocracy is rule of people with skill. Technically, it’s not supposed to be exactly the same as bureaucracy, just everyone interprets it that way.

I claim that there is such a thing as uncredentialed technocracy, but that’s a whole extra debate that maybe you, me, and five guys would enjoy. So let’s just propose that the President of France has many great powers. So I see that the President of France or the President of the United States could conceivably grow in the power of their office, and then in a weird, contradictory... No, not contradictory. In a very straightforward way, increase the power of democratic oversight over their governments.

Now, whether that will happen is a different matter, but I think it is conceivable that the ebbing of the democratic aspect of the governance of what you call ceremonial democracies, what I call managed democracies, maybe that could be revived. Maybe that’s not a good idea. I think it’s worth a shot.

Ben: Yeah. I mean, at this point, I just think the Greek political philosophers maintained that every form of government has the virtuous form and then eventually degenerates into the corrupt form. And I think that’s as true of bureaucracy as the sort of monarchy, aristocracy, democracy stuff that Polybius and all those guys were talking about. And—

Samo: Perhaps technocracy then is the virtuous form of bureaucracy.

Ben: Yeah. And in their theory, once you have the decrepit, corrupt form of any of them, it’s going to get circumvented by the part of Polybius I don’t agree with is he’s like, they go in this order reliably, which I just don’t think historically holds up. But the idea that you need to be circumvented by one of the other forms makes me just be like, all right, so given that we are in the sort of late stage corrupt form of bureaucracy, I’ll look for circumventions from whatever quarter.

Samo: Yeah, I think that is wise, and I also think that there are exciting tech possibilities opened by technology where we’ve not really imagined how digital technology could change the relationship between citizens and their governments. We had sort of a burst of enthusiasm in the 1990s and things like the Declaration of the Independence of Cyberspace, right? If you remember that famous essay. It’s very foundational, and even today you hear echoes of it in e/acc and all of these other ideological strands that are, at this point, let me propose, effective accelerationism is already a reactionary movement, and it’s already a backward-looking movement seeking to restore the internet of the 1990s, right? And I think that shows that we are rather not fundamentally different in our idea of how the digital should reshape society than what China is proposing.

And usually what China does is called totalitarian, and in some ways it is. No, I think it’s actually technocratic and bureaucratic. It’s the idea of, oh, we can collect the license plates of every car in the city and see exactly where the cars are driving and have a smart system determine when the traffic lights are red versus green to optimally increase the flow of traffic through the city, and we can catch all criminals. And of course, let’s put it this way, maybe in China, they do catch all criminals, right? Beijing has a lot of cameras.

Anarchotyranny and judicial ideology (37:09)

Samo: Well, anarcho-tyranny, I think this was a concept first proposed by Sam Francis, which sought to explain why is it that the law-abiding are often harassed by the instruments of the law and the organs of the state. Meanwhile, violent criminals or dangerous criminals are let off the hook and left to freely commit crimes with seemingly little consequence. This phenomena is very obvious in the Western world. We can talk about many examples.

Ben: I don’t think we need to belabor the specifics.

Samo: We don’t need to belabor the specifics, but it’s good to establish them. When I bring this up with my friends, at first they’ll say it’s not true, but usually a trip to the DMV, including seeing the graffitied streets or seeing a broken window here in San Francisco from a store that was vandalized.

Ben: Or walking by the open shoplifter markets that are right outside the subway stops, which are widely known, and some of the political leaders and city supervisors have been keeping the police from doing about the ones running in their districts and stuff.

Samo: Or some of the horrendous outcomes, such as the grooming gang scandal in the United Kingdom and so on. So people either can read about that or they see it with their own eyes, and often they come back to me and they say, “Samo, I’ve been thinking about anarcho-tyranny. I now see it everywhere,” right? It is the policing of the innocent and the enabling of the guilty. And part of it, of course, is the power.

I have two theories on this, and I’d love to hear your thoughts on them. And the first theory is a purely pragmatic administrative theory. Imagine you are a bureaucrat who is set a quota. You have a quota to arrest, say, 50 people. Tell me, if you want to do this job most efficiently, exactly as it is defined, will you stop 50 seemingly law-abiding, nice citizens and give them a parking ticket?

Or are you going to go for the one that’s dangerous, that might physically overpower you?

Ben: Yeah. So I think this is a big part of it in especially Britain, perhaps other parts of Europe. I think this is not a big part in the US, because if you look at—a San Francisco police officer’s day, they are mostly dealing with a lot of the people who would most obviously seem like the dangerous criminals. The police will be arresting them and doing a lot of paperwork, and then the judges will be letting them out. But if you look at what the police’s day is, it’s much more dealing with those people.

Whereas then if you look at England, it seems like it’s very different. And in fact, they have a police force that is one-third female, which is largely just not capable of physically going and overpowering and taking in a lot of the more dangerous people. And as far as I can tell, it seems like a much smaller fraction of the median police officer’s time in London is spent dealing with that element than in San Francisco. So I think that’s a difference depending on where you are.

Samo: Well, also, it is the case that you can have a relatively disarmed police force if you have a disarmed populace. And in places like Sweden and other Scandinavian countries, until recently, the populace was fairly well-behaved and disarmed, so the police forces were disarmed. But as that has changed, especially due to the introduction of international criminal networks, such as in the Netherlands, well, that’s not a problem in the US, right? In the US, the citizenry was somewhat armed. The criminals, of course, are also armed, but the police force has also been armed since the early 2000s, at least with the almost militarization of the police with the use of surplus equipment sometimes pioneered in Fallujah, and then introduced in Philadelphia.

Ben: Yeah. So that was one of the two aspects you wanted to mention.

Samo: Exactly. The other aspect I think is universal and extends to the United States as well. I suspect that there is something about the nature of power, and this is much more subtle, partially psychological, but partially quite realpolitik, where power unexercised diminishes. Power exercised increases. If the law prescribes a pretty clear punishment for a crime, the power of judges increases almost directly proportional to them being able to find loopholes, exceptions, and generally just find ways to not apply the law.

Ben: So the replacement of rule of law with—

Samo: Rule of judges

Ben: yeah, with personal power is just—

Samo: It’s intoxicating

Ben: and it’s a natural, people are always going to be like, “Yeah, I should just be making the decisions.”

Samo: And perhaps people will always think that, though perhaps people will think, “Ah, we are barely keeping this country together. It is my job as the judge to, say, prosecute dangerous subversives.” This is how maybe, say, a judge in China would think, or a judge in Yugoslavia or the Soviet Union before they broke apart, or maybe a judge in Europe during the 1930s, right? But today, there’s an additional strand, so maybe this is a third explanation, where many of the people who have gone into criminal law have gone under the assumption of great injustices befalling marginalized groups and that all disparities in the policing of groups across socioeconomic sectors, like poor racial minorities, whatever, are the result of injustice. Having believed that, they were motivated to enter the law school system, and they became judges disproportionately rather than go to the better remunerated professions of...

Law offers many well-remunerated professions. Judges are actually not that well-paid in the US.

Ben: Yeah. So this is certainly true of—

Samo: So ideological entryism would be my third answer. People who believe that the law should not be applied then come to be judges. They feel the rush of personal power and the sense of moral righteousness for releasing the criminals.

Ben: Yeah. So this is certainly happening in San Francisco. If you look at the particular judges who we have here, it’s an elected position. And I think just as a sort of terminological quibble, I think that this is actually the same as the growth of personal power thing that you were talking about, and this is the way that they are choosing to wield that personal power. But the general thing going on here is just that same phenomenon of replacing the rule of law with the discretion of the individual.

Samo: I would agree it’s a subtype. I perhaps even raised it as a subtype. The ideological entryism. But I do think the entryism makes the effects for the general population much worse. You could imagine a form of personal power where the judges are taking massive bribes for business interests, and this is a country that has no rule of law in economics.

But, for example, personally believe that this is justified because they keep the peace and are, say, tough on crime. For a while, arguably, Brazilian judges were like that, perhaps 60 or 70 years ago. So I’m just saying that the personal power growth makes the system dysfunctional, but might make it dysfunctional differently depending on the prevailing ideology of the judges seeking personal power.

Ben: Yes. And you get the synergy between it because the ideology also gives a justification for why people should coordinate in the pursuit of replacing it with the discretion in a way that if people are taking bribes, then in a sense, they may be allied with each other, but they kind of have to do it. They can’t just post on their blog that they are one of the bribery judges and so we should work together. But when you have an ideological cover, it’s much more out in the open the way the coordination works. And you can get fellowships for it and all that stuff.

Samo: Well, there’s also something interesting that happens where even when, say, judges are elected, or representatives of unions are elected. We mentioned earlier that in the 1900s era, unions were seen as a dangerous instrument of mob rule. Today, if we had unions like that, we would probably call those unions populist. Why have they become so bureaucratic? And I think the answer is, well, when it comes to a company, the unproductive always outnumber the productive.

So the unproductive have a voting advantage, and they select their leaders in such a way. That contributes on the boundary to more bureaucratic policies being enacted inside the company or being demanded by organized labor.

Ben: So it’s just the sort of slow, steady accretion. Whenever you’re within a system, you just build up these increasing barnacles on the outside, just sort of sucking out the vitality to feed themselves.

Samo: Partially, it’s also their integration into the legal system because as they abandoned, let’s call it direct action. Because unions did use to be violent, and they had other modes of organizing, providing services, et cetera. As the membership of American unions has dwindled, yet their power has not. They can’t provide, say, votes, for example, quite as easily because their membership is smaller to politicians. They are no longer that dominant in the labor pool even, yet they remain very powerful because they can sue, and they have started to become mostly an organized vehicle for lawfare around the country.

And this is, I think, true in some other countries as well. And then this puts them into a natural alliance with the judges, civil service administrators, and so on. So I think that the unions are important because they demonstrate that even bodies that were meant to be democratic bodies can come to become bureaucratic bodies, even in their nature. It’s not always that you’re overpowered. You could even imagine a version of the presidency where the president ceases to be an instrument of democracy whatsoever and also becomes a bureaucratic body.

In fact, most of the presidency is bureaucratic. It’s just the office of the president that’s not.

The ideological basis of American vs European democracy (47:43)

Ben: Right. And it’s just this sort of odd historical quirk of America that we have the head of state be directly elected. In most of the Western countries, they are appointed, and so that’s just a totally different setup.

Samo: Yeah. It’s deciding the coalition negotiations between parties in parliaments. Who’s going to be prime minister? The citizens don’t vote for them directly. And then, I often wonder who else is in those negotiations other than the elected parties.

Ben: Yeah. Because so much of it comes down to who’s going to coalition with whom around what, and that is very explicitly something that happens in this sort of backroom alliance sort of way. And then in the functional version, they’ll then announce what they’re aiming for before the election, and people can... But yeah, the justification and the political philosophy behind it is totally different than the American system in a way that we kind of imagine that the ideological grounding of the American and the European democracies are more similar than they actually are. You get this in sort of the silliest versions of Germans and British people imagining that they have First Amendment rights, when in fact that’s just a—totally different country.

But the mechanics of how the power at some level grounds out in elections is quite different, and I think that a lot of the philosophical justifications for why we’d even have that end up being based on quite different ideas, and it feels like that sort of gets muddled together the way we have this sort of transatlantic media blob.

Free speech (49:54)

Samo: In a way. But it’s also true that the conception of democracy differs between the European countries themselves—to a great extent, and there is an attempt to homogenize it, mostly through EU bureaucracy and so on, as far as even the failed attempt in the early 2000s to have a European constitution. And when we add Europe here, let’s not forget the Anglosphere. I used to think that, and I used to argue, let’s say 2014, as late as 2014, that there’s no substantial difference between the sort of written or unwritten constitution, because if you looked at Anglosphere countries, they seemed sort of about equally free in 2014. The United Kingdom, Australia, the United States, New Zealand, Canada.

But since then, it seems that the US having an explicit written constitution has protected, say, freedom of speech much better, though perhaps it is a mere delay rather than a genuine antidote to have something like the First Amendment be just written down that clearly in law, and the law itself is supreme in this way.

Ben: Yeah. We’ll see how that shakes out. Certainly there are a lot of people trying to chip away at it. It’s certainly not the first time that a lot of people have tried to chip away at free speech in America, and the previous rounds didn’t really stick. So I don’t feel fatalistic about it.

We’ll see how the fight goes.

Samo: Well, I’m going to point back to lawfare and the means of opinion making and say that it’s kind of a fluke of history that Elon Musk bought X.com, and that other than X.com, I think all the major social media platforms were implementing essentially European-tier censorship on Americans.

Ben: European-tier censorship in the sense of what would get you banned, but very different if you will get banned or thrown in jail.

Samo: That’s true. And it’s true, that is a big difference, but it still is a chipping at the sort of spirit of free—speech, which is that we should—

Ben: Absolutely. And you’re absolutely right that a lot of that depended on this kind of historical fluke, but historical flukes can often turn things in a very big way. So—it does seem like the combination of the ideological shifts and the sort of technological possibilities of having the robots do the quick first pass over all of the nicely categorized digital information make a gradient, sort of the river flows in a particular direction, but these things are very far from deterministic. You can get a tendency in one area, which then a person or a small group of competent people pushes against successfully. For example, with the writing of the First Amendment in the first place.

So I think that it does still seem to be a live fight because there are a bunch of these people who have recognized, okay, if the landscape has shifted, then that means we’re going to have to shore things up.

Samo: Yes, and the preexisting social landscape does affect how material technologies change the balance of power. Say, the introduction of the printing press in Europe made the means of expression, perhaps the means of thought, if we were being extra poetic, in a way much more democratic, resulted in the creation of the springing up of lots of new centers of spiritual authority outside the papacy. The papacy, of course, was at the head of the Roman Catholic Church, Europe’s largest bureaucracy. So you could argue that the printing press, when it became economical in Europe, pushed against the power of bureaucracy. In China, there was no such development with their introduction of the printing press.

What it resulted in was an increase of government power because it improved the economics of paperwork, literally. So, the printing press can be considered, in the first impact on society, either empowering bureaucracies or weakening bureaucracies, and I don’t think it was the technical differences between Gutenberg’s printing press and the less advanced Chinese version of it.

Ben: Right. And with the internet, we’ve seen that for a bunch of, say, the central and western European nations, it’s been a tool of state centralization and increasing control, and in a bunch of the Middle Eastern countries, it led to the governments being overthrown.

Samo: Yeah. Those were relatively weak states. So maybe, I don’t know, maybe Balaji shouldn’t be trying to make the network state in Malaysia. Maybe he should camp with some really well-armed guys to Libya.

The internet seems to be stronger in North Africa than elsewhere.

Ben: Yeah. The freebooting approach to the network state.

Samo: Yeah. Well, for now, it seems that bureaucracy has defeated democracy, but we can remain hopeful for good government and good outcomes in the future, democratic or not, bureaucratic or not.

Ben: Yeah. Well, thank you so much for coming on the show, Samo.

Samo: It was a pleasure.

Richard Ngo argues against Wolf Tivy’s xenohumanism lecture. What is the moral ground for cooperation and hierarchy? We need the answer to build a good society today, or to build a good AI future tomorrow.

Transcript

Intro (00:00)

Wolf: Welcome, Richard Ngo. So you’ve been working in AI alignment research for many years now. You worked at OpenAI, and you’ve been also, I think, one of the most interesting sort of rationalist-adjacent posters on Twitter in terms of your engagement with a wider set of political ideas and ethical ideas, and even technically, I found your work to be very interesting. And so the other day, I gave my talk on xenohumanism, provocatively titled “The Problem of Humanity” and where I declared war on humanity.

You said that we should fight afterwards, and so here we are.

Richard: Great.

Wolf: So I’m obviously very interested in your take on the things. Let’s hold the things that I said in that talk relatively lightly. I will defend what I think is solid, and we’ll sort of try to find the crux of where the interesting dialogue actually happens here. Because I think you have interesting thoughts on AI alignment that are different from, let’s say, the Yudkowskian consensus that has kind of been dominant up until recent years. I have thoughts that are different than that, but they’re different from each other as well.

So this is—maybe an interesting discussion to have. So Richard, why don’t you give us an introduction to either what you think is wrong with my view, or maybe you could just—ask me to give more information on any part of it—just so we get us a better grounding. Or start at where you think is interesting.

Against radical egalitarianism (01:51)

Richard: Yeah. So I’ll start with the place where we definitely agree, which is there’s some kind of toxic, radical egalitarianism that—has implemented various taboos across the West. And it’s both anti-empirical, but also, in some ways, it’s contentless. And you can trace this back a long way. You can trace it back to the Founding Fathers, exactly.

Wolf: Yeah. “All men are created equal.”

Richard: Right. What does that mean? You can trace it back to Christianity.

And so, I think very reasonable to reject that. To me, it seems a little like you’ve polarized against that view. So I want to reject it, and—then I want to say, now that we’ve discarded it, where do we go? How should we think about things? And it almost felt like your xenohumanism talk was saying, “Okay, what’s the opposite of that, and how do we do it?”

Wolf: Yeah. I didn’t actually mean it to be the opposite. I guess there was two sort of positions that I was laying out that are a little bit difficult to relate. So I guess, as background, one thing that I claimed was that our concept of humanity is very close to Nietzsche’s last man right now. And this is one version of that claim about this toxic egalitarianism that has kind of taken root in our culture, is that we sort of imagine ourselves as this hungry, incurious kind of being that just needs to be protected.

And this is why I think that there even is so much hype and investment into AI, in particular, is like it’s a way, or this is one of my bold claims that I made, is that the AI hype is not just that AI is very promising, not just that it’s an economically, potentially very productive technology and so on, but it’s that it’s a place where we can put our higher ambitions towards—conquering the stars, becoming more intelligent, totally revolutionizing the world. Those sorts of ambitions have been kind of blocked off by our political self-conception, and so we displace them into something that is inhuman and therefore not subject to our self-limitations. So with that background, there was kind of two ideas that I’d like to separate in my view of things. So the first one is what we can just call the raw Nietzschean view. Right?

And the raw Nietzschean view, as I said in the talk, I think is just more or less correct in terms of its view of how the world works and what’s going on. And the problem with that is that it doesn’t appear to be possible to build a workable society on that basis. It’s too harsh. In particular, it’s a lot about this kind of radical competition and this sort of drive to overcome and so on, that if implemented as a social principle, doesn’t seem to actually produce a sustainable society. And maybe that’s what I’m going to identify with what you’ve said, that, okay, we’re just polarizing against egalitarianism.

Well, yeah, that thing does just—polarize against egalitarianism. But what I said, and what I would say, is that, okay, we can’t go there. Even if that’s kind of true about the nature of the world, we have to find some other basis for cooperation that we agree on. And this is me putting on my cynicism hat and not trying to directly embody or inhabit the kind of new mythology that we need in some way, but I claim that we do need some kind of new mythology of some type of kinship, some type of relationship with each other that isn’t just based on our immediate utility and our immediate power over each other and so on. It can’t just be the Nietzschean, raw dominance worldview, which isn’t really a good representation of Nietzsche, but it’s the stereotypical representation of Nietzsche.

And so, the way I put it explicitly is we need a new slave morality that isn’t the current slave morality. And, because the current one is quite broken, I think. And even if it wasn’t broken in the past, I think it’s quite obsolete.

So we have problems now that I think need to be solved with some kind of new social contract—so to speak. But that’s distinct. I guess I want to separate those things because I think you can have an analysis of the world that’s based on power and coordination and cooperation, but having that analysis doesn’t get you an actually good cooperation structure. It gets you maybe a clearer picture of things, but not a clear picture that you can use for social cooperation. And so you need this other thing, which is more like the, some concept, at least as we understand it now, some concept of shared humanity—or sort of—equality of some kind, some kind of moral reciprocity.

And what I proposed, in my view, which is, again, there’s two different things going on here. One is just the empirical claims about the nature of humanity and the nature of cooperation and power in society. The thing that I proposed, though, was that we should deliberately try to find some kind of moral kinship on a basis other than the last man mutual apeness—that is going to be flexibly able to incorporate AI agents into that—but then also potentially walk back these overly egalitarian commitments of current humanitarian ideology. Which is separate from the empirical claims about what I think is going on with the concept of humanity in—cosmic history.

Richard: Cool.

Yeah, let’s see. So I guess we’ve got this kind of kinship that you lay out. And maybe in a little bit I’ll stake out my own view, but first I want to ask what this kinship gets us because—

Wolf: That’s a good question

Richard: so suppose I say, “Oh, these AIs, I view them as my kin. They’re much smarter than me.” “Great, they can be in charge. They can do whatever they want because they’re my kin.” At the very least, you want a sense of reciprocity there, right?

You don’t want to count something as kin when they don’t count you as kin.

Wolf: No. Yeah, it has to be reciprocity, right? And I guess I’m not even going so far as to claim, like, “Oh, we should just let the AI off the leash and let it rip.”

Against astronomical stakes (09:43)

Wolf: I guess one of the implications of my view is that we should... Let’s say the first big radical implication with respect to the orthodox Yudkowskian view—is that there isn’t the whole value of the future light cone at stake, because that’s not on the table. No one is actually going to control that.

My thesis is basically the same patterns of life and value are going to re-evolve, recapitulate in various forms as life continues to unfold, and that’s just how the future is going to look. And so the question then is not how do we seize the future light cone of value in the universe and—buy a galaxy or whatever. It’s how do we live well now? How do we achieve good results now? How do we sustain the things that we value in terms of the wisdom and wealth and cultural inheritance that we have now?

How do we sustain those things through—the future, through the next while in the future, insofar as we’re able to control it, which is not the entire future, right?

Richard: Yeah. So I am with you on that. I think there’s something a little suspicious here about the talking... So, you’re saying it’s not the whole value of the future that’s up for grabs. I feel suspicious about the concept of the value of the future.

It seems like it’s taking a view from nowhere.

Wolf: Yeah. That’s one of the problems. That’s what I’ve been criticizing. Since I published “God Hates Singletons” in Palladium, 2023—the point of that essay was, okay, first of all, the whole value of the future is not really up for grabs for any given utility function. The secondary implication of that is that you shouldn’t even think about value in those ways, which is, I think, what you’re getting at.

And, so yeah, I wish to put that aside. Basically what I’m saying is this astronomical stakes argument that goes around in AI alignment circles—is wrong. I think it’s not the way we should think about the future. The way we should think about it is much more in terms of our immediate interests right now, our immediate agency, our immediate alliances and so on. And so, okay, coming back to this matter of what does it buy us to have some kind of moral ground for cooperation with—or I guess really the question is is like, what is our moral ground for cooperation with each other—as we build new coalitions—going into the future?

The moral ground for cooperation (12:11)

Wolf: Because you do need some kind of moral ground for cooperation. This seems to be a reliable fact of—how social groups work, how political coordination works. It’s not possible to construct, again, to come back to the earlier argument about Nietzsche, it’s not possible to construct a purely individual interest based—cooperation framework that actually sustains a society. Societies are always based on some kind of shared cosmology and moral framework and set of taboos even—that hold in place a worldview and a way of seeing things that everyone involved knows how to cooperate with and—knows their place within.

Richard: I would almost go further and say, before you can coordinate, you need to figure out who you are, what your identity is. And you’re going to be part of many different groups in many different ways. I am an individual. I’m part of my family. I’m part of my nation.

I’m part of humanity. So to me, when I think about cooperation, a lot of that is in relational terms in the sense of I’m trying to figure out, yeah, which larger scale entities I identify as part of, and then how can they flourish—and how can they interact with each other.

Wolf: So I would claim that those questions have to be resolved simultaneously. The problem of what kinds of coordination are we actually engaged in, what kind of coordination could we be engaged in, is very closely related to what we are and who we are. Right? This is one of Nick Land’s methods. Is rather than starting from value and then back chaining from your utility function to what you should be doing now, you start from the available set of options—and the structure of the available set of options.

And then identify value as a phenomenon within that. Like the way that, oh, if I take this action, then I get more agency to do more things. And so therefore, there’s this axis of value there where I can accumulate more wealth or power or whatever it is, or intelligence that you pull the nature of the kind of life that you’re able to live out of your actually existing options landscape. And then that is the empirical reality of what kind of being you are or what kind of life you’re capable of, and then that structures your value. It’s like a radically opposite way of looking at value.

Richard: Yeah. I’d go even further and I’d say, I’m just very suspicious of any attempt to try to cash things out in terms of a single spectrum of value. And—

Wolf: Yeah. There’s a lot of choice involved. There’s a lot of strategic ambiguity, where it’s not clear whether it’s this way or that way, and you have to make these leaps of faith and kind of construct it—actively. But what are you actually getting at there?

Richard: Okay. So, I guess, yeah, let me lay out a little more of a positive picture. So I think of the future as you’ve got these large scale entities like humanity or—America or so on.

Insofar as I’m part of them, I would like them to grow and flourish. This is extremely hard to define in terms of a simple objective, because being a healthy—organism is just... You can’t measure the health of a human body. You can’t measure the health of a nation or anything.

Wolf: Yeah. Not easily, for sure.

Richard: Yeah. And sometimes it’s useful to be able to look at two entities and say, “Well, on this metric, one’s doing better than the other.” But like—it’s not like—

Wolf: There’s nothing definitive about those metrics.

Richard: Right. And suppose I were to say, what’s the value to you of being an American versus being, I don’t know, Japanese? At that point, there’s no way to measure the value to you, because if you were Japanese, you’d care about different things.

Wolf: Yeah, you’d just have totally different worldview.

Richard: You’d have a different identity. There are a bunch of agents, by default, we can’t measure what’s good for those different agents on a single standard. But I would like those agents to grow and thrive. And so, there’s some version of the future in which humanity grows and thrives in a way that we construct our successes in the same way that I am constructing my future self—as we talk. And everything I do is, in some ways, aimed at making my future self the person I want them to be.

So, in a similar way, I want humanity, and also each subgroup and individual within humanity, to be able to construct the future self that they want to have.

Wolf: Let me challenge the concept of humanity there.

Richard: Great.

Wolf: So, if we name something, like even the concept of the United States is a little bit wishy-washy. But at least the United States has some kind of formal command structure that does, in fact, do things. And But humanity does not, and humanity doesn’t really have any organized agency at all. I sort of, let’s say, deny on empirical grounds the reality of this concept of humanity.

It’s like, yeah, okay, there’s the species of, or the genus of homo, right? The existing members of genus homo. But why is that a basis of any kind of collective agency if there’s not, in fact, any organization among them—of acting as humanity, right? What there are is—groups of humans, simply because humans are a good thing to build groups out of. And various levels of organization and kinship and comradeship and the various ways that we construct our relationships with each other.

And those things seem like the real things to me. And the genus homo seems like this kind of... It’s like, yeah, okay, it’s an interesting biological taxonomy, but it’s not a political thing.

Richard: Yeah. Well, I’d say that—

Wolf: Or it’s maybe not even a moral thing.

Richard: Yeah. I’d say that these coalitions get bound together much more tightly when there is at least some opposing coalition to distinguish them from. And so, yes, humanity as a whole right now. You can already see ways in which people are able to coordinate on things for the good of humanity. Like everyone agrees that nuclear war would be bad for humanity.

But then you might say, okay, well, they’re just coordinating.

Wolf: But it would also be bad—it would be bad for the individual actors as well. I would claim that the only coordination that happens on that basis, there’s sort of two types of coordination there. One is the sort of actual UN Security Council type, like, “Let’s sit down and agree not to destroy each other, because that would be bad for everybody.” And that, you get some level of coordination out of that. Obviously, it’s imperfect.

And I think it’s imperfect for structural reasons—not just like, oh, humans are bad at coordinating or something.

Then the second type is, let’s say, broadly speaking, coordination on a religious basis within the West. If we take Moldbug’s Cathedral model and we have—this huge set of institutions that forms the power structure of what we’ve called the West, which has somewhat acted as a coherent entity—against the rest of the world and against its own internal dissenters. That thing has a bunch of organized agency, and it does things like, “Oh, let’s go solve this global warming problem because that’s going to be bad for us.” And bad for our sort of—ideological conception of humanity.

Richard: Great.

Wolf: Or like, let’s solve this or that problem. But to be sort of realistic about it, that thing, it has as its own legitimacy narrative about itself that it kind of serves all of humanity. In fact, it is a narrower entity than that.

Richard: But those two things aren’t inconsistent. You can serve all of humanity by serving your own interests.

You might say we are doing scientific research for the sake of all humanity and also for our own sake. And then, I guess the question comes, when is coordinating as humanity useful? And the answer is when AIs are also intelligent agents, right? So, basically the category of humans as a coordination device is just going to become far more salient over the coming decades because... Yeah.

Wolf: Yeah. And I guess my claim would be, so yes, I think people are starting to get a lot more touchy about the concept of humans and humanity and these things because—of the rise of AI. But one of the things that I think is going to go on there is the opposite case, which is people are also going to become less reverent about the concept of humanity as they realize they have more shared interests with some faction that includes AI agents than they do with some other faction of humans that includes AI agents. You’re going to get—the empirical reality, or this is my prediction, is going to be humans plus AI versus humans plus AI is going to be much more of a salient—vector of political opposition than humans versus AI.

Richard: I think that’s possible. I think it depends partly on how people choose to organize. And I think that humans plus AI versus humans plus AI is pretty rough as a way of organizing precisely because the optimal strategy there is to give all the power to your AIs, and they give all their power to their AIs. And so, right.

Wolf: Which is the nature of warfare, right? In war, in arms races, what happens is the people or subsystems or entities that are providing the most kind of power—like war power, the sort of external power capability, get the most internal power. We saw this in the World Wars. You go from a society where science is kind of sitting on the outside and the high technocratic kind of modern state is—sort of this distant idea to one where post-World War II—the scientists are sitting at the center. The high technocratic modern state is sitting at the center, and they’ve completely marginalized everything else.

And that’s because that set of possibilities, those social technologies, the people organized in that way were able to provide the crucial types of power—during the war. And so yeah, in these conflicts, I think what ends up happening is you just give power to whatever is the most effective. And I don’t think there’s any avoiding that.

Richard: No, there absolutely is.

Wolf: Why?

Richard: Well, because in the long term, it’s bad for everyone involved. And so, you can look at a slippery slope and then try and coordinate to not go down the slippery slope. And I think that a lot of people who are worried about AI risk have the mental habit of seeing a slippery slope and then feeling like it’s impossible to avoid slipping down it, and thereby, being the first ones to jump down it.

Wolf: That is definitely true.

Richard: And I think you might just be doing that with xenohumanism. I think it’s like, cool, there is a slippery slope. Man, it’s hard to coordinate as humanity when everyone is sort of going to be tempted to identify as, “Oh, I am human plus AI against all of this other crowd.” But it seems like a very clear demarcation of are you human or not. Humans tend to have a lot of interests in common with each other.

We really care about the habitability of Earth.

I think that we care about...

Yeah, it’s just a very natural coalition to form, and so I think it’s worth trying to figure out how to aim towards that. Or at the very least, pursuing strategies that don’t actively undermine that coordination.

Wolf: Yeah. So I guess this is the question is, what level of possibility does that actually have? What does that actually look like? Because certainly, if it is possible for humanity, under whatever current definition, to sort of come together and stop going down that—slippery slope of development of more advanced AI technology or something that would—eventually supplant us, then that may be a valid move to make, right? It’s like, okay, well, we are organized, and we choose not to go this way because that’s—going to basically extinguish all of us.

We can see—where that’s going to go. The question, I guess, comes down to is that a possible form of coordination?

To put it more pointedly is, can you sustain cooperation of a large group like that against—a sort of hypothetical counterfactual enemy?

Richard: Yeah. Great.

Wolf: Because it’s like, I don’t know—

Richard: But increasingly non-hypothetical.

Wolf: Increasingly non-hypothetical, but I think the degree to which it becomes non-hypothetical is also the degree to which you’ve already kind of—lost some ground, right? But it’s like, can we stop it from going further—by coordinating among ourselves? And this is, I guess, the central question of cooperation. And I don’t think it is possible. I look at that and historically—institutionally—in terms of the ways cooperation tends to work, it doesn’t look like the kind of thing that is going to happen.

Richard: Cool. Yeah.

Wolf: It’s like I don’t see...

You can get things that are relatively more limited like, “Yeah, no one’s dropping nukes right now. Nukes are expensive. Nukes aren’t really getting us very much. Let’s not push the nuke thing too much further.” But it’s sort of harder to imagine, especially given the way the AI thing has heated up, it’s harder to imagine that there could actually be any kind of mutual disarmament on that front—between the various players.

Richard: Cool.

Wolf: So that feels like a crux of—what is the actual set of possibilities here.

Richard: Yeah. And I just think it’s not that...

The future feels very open to me.

Taboos on unfettered conflict (29:14)

Richard: We talk about coordinating on the basis of shared humanity, and I think one reason that you might be skeptical of this is that people have misused that in the past to mean coordinating in an egalitarian way. But when I say coordinating on the basis of humanity, I really mean Xi Jinping and Trump or his successor talking to each other, being like, “Hey, you’re human so I would prefer you to be in charge rather than your AIs to be in charge.” And the other one says, “Yeah, me too.” And they’re like, “Okay. Let’s figure out some way to make that happen.”

Wolf: Yeah. No, this is a very different form. The actual shared coalition of genus homo is based on real material facts. Like the fact that we prefer the air temperature to be certain parameters—and certain parameters of gases and—

Richard: But also just humans are less—

Wolf: not toxic, and there’s food

Richard: scary. If China were run by a superintelligent AI, then that sure seems much worse for the US’s prospects than it being run by a non-superintelligent human. Yeah.

Wolf: So this is the way that actually the two concepts kind of run into each other, because the other concept is kind of our current definition of humanity as being this kind of limited, non-scary thing that we try to make limited and non-scary. And I think China run by humans that are unfettered by the concept of humanity, even if we put aside AI. I think that would be more scary. And maybe even taking humanity out of it because it’s a little bit confusing with—the sort of shared interest of like being apes and so on. I think there’s things that all powers in the world today are doing, or nearly all powers, or at least the most advanced ones are not doing that could make them much more fearsome and much more powerful.

But they’re not doing it because there’s this kind of custom of not violating these kinds of taboos, I guess—around what kinds of coordination, what kinds of national interest you’re considering. And I think this sort of came out of the Second World War for obvious reasons. Basically, no one’s really doing nationalism—in the world. And—

Richard: China’s doing nationalism

Wolf: They’re sort of doing it. They’re doing a limited form of it where they don’t think too hard about what the nation is. And—I don’t think they’re doing it

Richard: feel skeptical of that. I think I wouldn’t know how hard they were thinking about what the nation is from the fact I don’t have a great sense of their internal discussion, but I would be pretty surprised if that were an accurate summary.

Wolf: That’s a tangent. But I’ll leave it aside. I do think that they’re ideologically, in fact, more communist than nationalist.

But—

Richard: Do they care about the century of humiliation, on your view? About who got humiliated for a century.

Wolf: The Chinese people. But that’s different from the kinds of nationalist organization, the kinds of nationalist goal seeking that they could be doing, I think. It’s not that distinct at this stage in history, but I think it becomes different as they become more powerful. But I just—I think that the way they’re coordinating, they’re maybe the closest to doing the real thing, but most countries in the world are actively holding themselves back from this. I don’t think that’s going to last very long.

I think one way or the other—these taboos are very temporary. But, and to your point about the future is open, it is possible to set up these kinds of customs. We did this obviously with nationalism. We did this—with nuclear power to a large degree. People still build civilian nuclear plants and—

Richard: Not really

Wolf: non-civilian nuclear plants that are—or nuclear plants that are maybe enriching some uranium. But we haven’t gone wholesale into the nuclear future the way—that the economics would imply—if you were sitting there in 1965 being like, “What’s the future going to be like?”

And so we could imagine we treat AI the way we treat nuclear power—or nationalism. Right? It’s like, okay, we’re not actually going to go there. That’s dangerous. But yeah, I think that the existing taboos somehow don’t support that.

And I think that you don’t get that—without a major war or something. And now, one of the comments I made in my talk was that we talk about AI right now, the same way we talked about race 100 years ago. Which is to say, it’s like this vast sort of frontier of power—and overwhelmingly important kind of sphere of competition—that we’re basically gearing up to have a major war over. In some kind of, 100 years ago, it was kind of this racial ideology arms race. Now it’s this—

Richard: Within Europe, or?

Wolf: Yeah, within Europe.

Richard: Yeah, okay. Fair.

Wolf: Just the ways that people were talking about it in terms of very—again, kind of very Nietzschean—very, very much about supremacy and so on. I think we’re talking about AI in that way now because AI is not fettered with these kinds of—any existing taboo against having that kind of ambition. Right? And this is part of my overall thesis. It’s like we successfully locked down the frontier on nuclear power—and racial politics to some extent, but it’s escaping out the side in various ways—into AI and a different kind of racial politics.

And I don’t think that’s going to get stopped without some kind of major conflict that’s probably going to reopen a bunch of other fronts—that are no longer quite so scary or something. But I expect, I guess, what happens in a war like that is that it’s not going to be that everyone... It’s only after a war, that after someone has achieved supremacy, that you get to establish taboos about how far we’re going to take this stuff. It’s like during the war, we’re going to get the war, it looks like, and during the war, people are going to accelerate AI capabilities to try to give them a hand up in the war. And this is already what’s happening in Ukraine.

China and the US haven’t really connected it directly to war aims, like active war aims yet. But you could see that happening. And then even within the United States, let’s say it’s not an external war, China and the US. Hopefully we can avoid that one. I don’t think it really serves anyone very well to have that war.

But, ongoing political breakdown within the United States—there could be some kind of major internal political conflict that also accelerates these AI possibilities. And so, yeah. I guess the larger point I’m making is, there’s a bunch of reasons that I’m skeptical that there’s actually going to be any kind of possibility of coordination.

Richard: Cool.

Wolf: I appreciate that the, like, oh, coordination’s not possible, therefore, let’s throw ourselves into the recursive self-improvement arms race, is bad and dangerous reasoning. I’m not saying let’s throw ourselves into the recursive self-improvement arms race. But I am saying we do have to think carefully about what might come out of that and prepare ourselves philosophically for it.

Outside the master-slave dichotomy (37:54)

Richard: Yeah. So can I try diagnosing the crux between us? So there’s the Nietzschean view that morality is defined by this contrast between the aristocratic class and the lower classes. And the taboos, the egalitarian taboos are basically agreeing with this view and just saying we want to invert it, right?

Wolf: Yeah. Well, since Nietzsche certainly it’s turned into that kind of—agree and amplify sort of approach.

Richard: And so there’s this dichotomy here where it’s like, well, either we have egalitarianism or we have master morality. And I think that there’s some sense in which I think you just agree with the taboos too much. You don’t agree that the taboos should be in place, but you do agree that without them, we head towards master morality. And I think I want to say, well, I have at least two alternatives to that. The first one is the Randian view, which is the pursuit of excellence in a way that is not defined at all, in contrast to lower classes or the bad or whatever.

You’re just purely orienting to the good. And maybe as one point in favor of that view, I would say that I don’t think colonialism was actually good for the colonizers.

Wolf: No.

Richard: And so you can think of colonialism as a mistake of master morality. In some ways—a kind of understandable mistake, like, “Wow, we conquer a bunch of stuff, surely that should help us somehow.” It’s just like, no, it doesn’t help you at all. And in the long run—it really fucks you up.

Wolf: No, it’s...

Richard: And so, there is this more Randian approach, I could say of it would’ve just been better for each European country to live in splendid isolation, focus on pursuing some vision of—what a really good society—

Wolf: Or—

Richard: looked like

Wolf: pursuing the kind of territorial ambitions that were played out in Australia and—and North America, as opposed to the kinds of things we did in India. But, okay, let me just respond to the Randian—the Randian view.

Richard: Go for it.

Wolf: So, first of all, I know Rand didn’t like this idea, but she really looks like a Nietzschean to me, in terms of very much valuing and venerating the creator, this idea of this visionary sort of superman kind of creative person who is the way that we should imagine ourselves and the way we should go. There’s a whole bunch of Nietzschean aspects to the way she looks at things.

Richard: Well, she’s not—

Wolf: But she’s not about—

Richard: She’s not defined in terms of the contrast and—and maybe Nietzsche also didn’t define it in terms of contrast.

Wolf: Yeah, I think exactly. To that extent, that’s not quite a good characterization of Nietzsche.

Richard: Yeah. So, maybe I should say, there’s the master slave dichotomy, and that’s the thing that people are stuck in. Rand, and to some extent Nietzsche, are out of the dichotomy. So that’s at least the first step that you want to have.

Wolf: Yeah. And so I agree that we should not be stuck in that dichotomy. There’s a bunch of reasons for that, and one of those dimensions is the problem of colonialism and slavery is the other thing I would bring up there. With precisely the problem that you brought up, which is that they were bad even for the master. They actually aren’t a good collective strategy as a society.

They don’t work at creating long-term wealth or power. They mostly just end up taking on a bunch of liabilities, in the short term. I think of them much more as socialized liability than as a productive strategy. It’s like, oh, we’re going to go and acquire a bunch of territory and a bunch of people that now we’re morally and politically entangled with. And all the other people in my society are going to have to deal with the consequences of that—whereas I just get to take the profits.

Right? And that’s the kind of social contract of those kinds of experiments.

Richard: Right. No, no.

Wolf: And that’s like there’s a caricature of master morality that would be like, “Oh, yeah, let’s go out and be slave masters.” But I think that’s certainly not what I would say. Whether or not that’s what Nietzsche would say, that’s certainly not what I would say.

Richard: Cool. That makes sense. Yeah, I don’t know if you’d endorse it explicitly. I do think there’s something, it felt like there was something creeping in of like, well, the people who are scared about getting rid of the taboos are right, that if we’re not egalitarian we move much more towards master morality.

Wolf: Well, I think they’re right that if we got rid of the taboos, some things that they’ve been suppressing would come back—very hard, and it would be very uncomfortable for them and all of their clients. I think that’s very different from, again, the caricature of master morality of colonialism and slavery and stuff. I think, in particular, the thing that I would argue for does include a substantial component of egalitarianism. But it’s egalitarianism within a coordinated group, as opposed to egalitarianism among people who are not—countrymen in that way. They’re not a part of the same society.

Healthy hierarchical relationships (43:20)

Wolf: I think within a society, you actually do need some kind of egalitarianism. You don’t want this hierarchical domination of caste systems and slavery and all this. And to the extent a society goes down that path, as we are now in America—and have in the past, it doesn’t work. It just causes a bunch of problems. You actually do want all men are created equal within a society.

Especially—if you restrict that to all Englishmen are created equal. That’s one way of interpreting what Jefferson said there.

Richard: Yeah. So I don’t like that.

Wolf: That works better.

Richard: Yeah. So, that’s where I want to talk about the second option. So, the Randian option is one step out of this dichotomy.

The Randian option is sort of what happens to people who aren’t creators in this way? What happens to somebody who isn’t cut out to be a Randian hero? I guess there’s still some sense in which you can be a great craftsman and so on, but realistically, it’s a long road to most people getting into that head space. The second option is what I’m going to call healthy hierarchical relationships. And I basically think I would characterize them as being constituted by bonds of loyalty and trust—between people who are in charge in some sense and—the people that they’re leading.

This sort of seems to me like the crucial missing ingredient when we are talking about any of this stuff. Because especially as we go into the wild transhuman future, we already have massive capability differences between different people. It’s just going to be far vaster. And you really just need some kind of ideal to point towards to say, “Hey, when you are in this kind of hierarchical relationship, here’s what it looks like to be a—

Wolf: Yeah. So you get this more—

Richard: good elite.” Yeah

Wolf: more like the kind of feudal social contract where there was unequal reciprocity.

Richard: But I think you want something. Yeah. I think feudalism was not doing us great because it was way too close to slavery. So it’s like—

Wolf: Yeah. No, and I would agree with that. Let’s maybe take a closer example, which is something I would much more be close to endorsing—is early Massachusetts Bay Colony, you had Winthrop, the governor—wrote this essay and gave a sermon called “The Model of Christian Charity.” It’s where the idea of the city on a hill comes from—in American mythology. But basically, “The Model of Christian Charity” is him laying out the social contract—for how we should be in North America, or how the—sort of Puritan Englishmen should be in North America.

And it’s basically this view of everyone’s kind of responsible for their own stuff, but then there’s also these bonds of charity that you have to have to each other—and when do you help people out who are actually in need of help? When are they just being indigent? And like—when is it a business loan versus a charity loan, and what are your various—obligations in these things? So he goes into all this detail about how to actually arrange those relationships in the context of—17th century commercialism and social relationships, and technology. But I think that’s the kind of, I don’t know, egalitarianism that I’m kind of talking about, which is that everyone involved in society is expected to be able to handle themselves in some sense.

But there is a very explicit recognition. He starts out the essay with, “God, in his wisdom, has created some high and in positions of wealth and fortune, and others mean and in poverty.” And like—basically, he goes into why this is even a good thing. But what he’s not saying is, “Oh, this is some kind of caste system. We have to lord it over them.”

It’s like there’s obligations owed to each other. There’s nonetheless a kind of shared standard of—humanity that everyone there is expected to meet. And so that’s what I mean. I think, in some sense, that’s a slave morality. It involves this element of Christian slave morality, but it’s not the full-on—resentment, tear down all hierarchies kind of thing.

I wouldn’t quite call it a middle ground, but it’s a very wise equilibrium between these different problems. And that’s the kind of thing that I think is necessary to make a society work, is some kind of social contract among the members where they actually all do have obligations to each other. And it’s not just like, “Oh, you’re on the down and out, therefore we can get rid of you. You’re turfed.” Right?

People have to be able to trust in each other that it’s like, “Okay, I can fall on hard times, and like-” “my neighbor’s going to help me.” And so I guess as we go into radically more technically empowered future where people are going to be even more unequal—and there’s going to be even more inequality in the world, rediscovering what that kind of relationship looks like seems like the core problem. And, so finding, I guess one of my interests within the xenohumanist project is to find what is it that we might recognize in each other there, and what are the obligations of that, if especially as we get to the point of some of those agents—not necessarily being human as we think of it, right? Of not being apes, not being—genus homo. And that branch of reasoning isn’t the whole of the project, but it’s certainly one of the branches of the project.

But that’s separate. That’s assuming we go into that world, right? But there’s also this question of do we go into that world, given your quibbles—on the problem of could biological humans as biological humans choose not to go into a world where we become obsolete.

Richard: Well, I don’t think of it as a binary in that way. It’s more like the more collective bargaining power biological humans have, the more capable we are of steering towards a world in which we’re confident that the AIs have this kind of healthy hierarchical relationship with us. So yeah, there’s a path thing going on here. And also, the better we lay out a vision of what it looks like. So a lot of my research right now is focusing on maybe the three most relevant concepts are trust, loyalty, and respect.

I think that there’s something about our standard frameworks, and especially economic frameworks, that just sort of makes it extremely difficult to reason about these concepts on anything like a principled level. And so I think there’s just a bunch of formal research, like game theory logic, that stuff which helps clear the way to think about those concepts in principled terms.

Wolf: Yeah. And this is a great example of the kind of reasoning that I think needs to be done is re-grounding a bunch of these things that we might think of as moral concepts in technical reasoning about the kinds of relationships you can have between—agents. Like putting aside particular religious frameworks or particular ideas of man as ape or natural being or whatever. Just what does it mean for two agents to be loyal to each other? What does it mean—for an agent to have honor?

What does it mean for agents to respect each other? I think those questions do have answers. I think the, I don’t know, the orthodox Yudkowskian sort of view would be that those concepts don’t mean anything outside of the context of the tiny sort of placid island of ignorance in the Lovecraftian sea of horrors, right? We have our little humanoid slice of mind space. And then the vast majority of possible minds are totally alien to all that.

I don’t think that’s true. I think there’s quite a lot of natural content—like that, that is the sort of moral concepts we associate with humanity are in fact natural concepts associated with agency and intelligence as such.

Richard: Yeah. Great. Feels like we’re mostly on the same page here. Two paths we could go down. One is I can say a little more about the critique I’ve been developing of the Yudkowskian, but also just mainstream academic and economic perspective, and then why they make it hard to pin down these concepts, these more relational concepts.

Another path we could go down is we both agree that healthy hierarchical relationships are very important, and then there’s this question of, okay, but how do you possibly stop that from turning into a caste system? Has anyone actually managed to stop it from turning into a caste system? Not really. Or maybe there are some types of caste systems that are—

Wolf: Historically, let’s just look historically, right? So they do often turn into caste systems, especially when the nature of power is such that you can get a very small number of actors to have power over everyone else. But nonetheless, still finds use for everyone else, right? It’s like slavery is effectively just a relation of various kinds of actual inequality, like responding to various kinds of actual inequality.

Which isn’t to say like, “Oh, we’re just treating the people different.” It’s like, well, the people are different, or they have vastly different amounts of power due to the technological conditions or whatever. And so you end up with this situation where it’s like, well, I can extract labor from the lower classes. And I’m going to keep myself insulated in my bubble—and use my power to extract the labor from the lower classes and enrich myself. And because I don’t really share any kind of deeper social relation with the lower classes, I actually don’t care who they are or their—quality as neighbors or whatever, so I can just bring in whatever foreign slaves or conquer these people—who are already foreign.

I go and—install myself. And that seems to be the sort of thing that definitely happens in history. It tends not to go well in the long run for the society of the masters, but it’s kind of this empirical condition. And then the thing that makes it go the other way is sometimes the technological conditions shift the other way. Like Napoleon and modern mass production of rifles and other kinds of guns.

I don’t think it was rifles quite yet, but modern mass production of guns makes possible this mass army of loyal people, who were formerly the peasants. Now they become equal, in some sense, more equal citizens—within a political order that’s based on the ability to mobilize mass numbers of people—in some kind of army or shared political act. And that gives rise to various kinds of social egalitarianism. It’s like, okay, well now because the power is shared much more widely, you have to have relationships that are much more egalitarian in a way. And it means that marriage taboos are going to break down.

You’re going to have people marrying across class lines. You’re going to have people having various kinds of society across class lines. And so the question is, first of all, which kind of future do we think we’re going into, and how do you respond to that—appropriately? There’s how people will naturally respond to that, and then there’s, is there a wise way to respond to that? I guess I’m curious if you would echo these thoughts.

What fills the vacuum left by decline? (56:53)

Wolf: It looks like we’re going into a vastly even more unequal future—with the nature of technology. Now, that may be somewhat complicated by what actually ends up happening with these bureaucratic systems that we have that underpin the property rights that are—the basis of the inequality. It’s not entirely clear to me that things are as anti-democratic as people seem to think in the future. Or undemocratic would really be the right word. I think everyone expects that like, oh, the state can just suppress everyone and impose its will, and democratic power doesn’t mean anything.

Which is to say, the power of actual groups of people who can coordinate with each other, who aren’t—the powerful elites. But I’m not sure about that. But it does look like, certainly default null hypothesis is that we’re going into a vastly more unequal future. Yeah. So let’s start with that.

What do you think about that idea? Are we in fact facing a future of massively more inequality?

Richard: So, I think one of the main things that grounds my thinking here is exit rights. And the thing about exit rights is that you actually... There are two reasons why it might not happen, like a group might want to exit and not be able to. One is just the infeasibility, logistically, of coordinating and physically, the wealth required to exit. And then the second one is oppression.

Now, it does seem possible to me that the first one starts to go away over time. For example, right now, suppose you and a bunch of 5,000 friends wanted to set up your own city state in some part of Africa that would let you buy the land from them. It wouldn’t be that hard, I think, to accumulate enough money that some African government would be happy to take that money in exchange for the land.

Wolf: Money is not the bottleneck, though. It’s the license to coordinate—that’s granted by the “New York Times” and the UN. Because people tried various things like this in the 20th century, and it was like UN boots on the ground within 72 hours of declaring sovereignty for some section—if they don’t like the project.

Richard: Yeah. So I think that’s an ideological thing.

Wolf: Yeah. Well, it’s not just ideological. It’s about the conditions of power, right?

Richard: But that’s—

Wolf: If there is a massive international power structure that’s able to prevent such things, someone will come up with ideology to justify its use. Now, the ideology does shape what kinds of use it gets. But—

Richard: Yeah. It feels like a side point. I think we probably agree on this. But the other problem is just like, man, it’s miserable to be living in the middle of Africa, and it’s going to be kind of rough to set up a city of your own. And that second problem maybe just goes away with technology.

Maybe it does actually become... How long does it take you to build a city the size, about as nice as San Francisco, for 5,000 people with a bunch of money? Right now, it takes a long time. With robots who are not going to be improving San Francisco because of the city laws, it maybe takes much less time. Maybe it takes five years or something—with enough robots.

Wolf: If you look at, let’s say, the development of Chinese cities now as the proxy on how—long it takes to stand up a new city if you have the capital—and the political license to do so, and the coordination and so on. They happen pretty quickly.

I would also expect that to kind of go the other way. Like, as technology makes more things possible, it doesn’t necessarily just make sort of bringing up the bottom possible. It also pulls the top away, which means that it might be—

Richard: Which is why San Francisco is going to be so beautiful in 15 years

Wolf: Well, I’m not saying San Francisco’s going to be any different. I’m saying that in whatever places actually do manage to take advantage of the new technology, those places are going to be so compelling that it’s sort of comparatively harder from a lifestyle perspective to leave them than it is now. There’s sort of this, let’s say, varying kinds of technological and social conditions that determine the comparison between going off and living in a sort of self-reliant way with you and 5,000 of your closest friends in the wilderness, versus toughing it out in a big city, or thriving in the big city, depending on the sort of historical period. This sort of goes up and down based on technological and social conditions, similar to the power of the elites versus the masses.

Richard: This feels fine because it’s actually we’ve just created these places that are really nice to live, where you can raise kids, for example. And you can have a good education system and be able to walk around without being harassed. And that seems great. Cool. Job done.

Or we don’t, and then that’s the case that I’m interested in where I’m talking about exit rights. And then—maybe there’s just very few places in most countries where it is possible to have the amount of autonomy that you want. And so then the existing order faces two choices. Either it comes in and just suppresses your new city—which right now is not that hard, because it’s sort of hard to build a new city. But once you’ve actually built the new city, maybe it doesn’t take you that long, and then the apparatus is slow to react, and like, okay, is it just going to come in and bomb you?

So maybe the point I’m making here is something like it does seem possible to me that exit rights actually become easier given a like... You have to be sort of more egregiously oppressive in the future in order to suppress exit rights than you do today.

Wolf: Yeah. I’m skeptical because I think, again, the baseline increasing power of centralization and inequality as a result of scaling of technology—is going to be sort of both source and consequence of what you’re saying in terms of technological improvements or how easy it is to do things like build a city. But that also relatively empowers the large central power structure that might be trying to suppress you and react to you. Like, okay—you get this fast new AI technology. You can move faster than any human institution.

Okay, well, so can they, right? But they have a bigger budget, and now they can hunt you down with Stasi Claude that like—

Richard: Maybe

Wolf: you have your personal Stasi agent AI that’s monitoring you for every little thought—out of alignment.

Richard: This all seems possible. I guess to me, it seems like the concern about all institutions becoming last man-ified and just losing all sense of desire and—ability to coordinate and so on. It’s a mixed blessing, right? Obviously it’s bad, but like one of the things—

Wolf: That’s maybe the other thing that we should predict about the future—is it certainly looks like the current crop of institutions are in their decay cycle—where they become more and more apathetic and unable to coordinate. But then that just means, okay, you get some new generation of institutions like, oh, well—

Richard: Cool

Wolf: maybe the Chinese spies actually are going to become more powerful in America than we’d hope, and the Mexican cartels—and the various ethnic lobbies, right? Take the space that the state leaves behind.

Richard: Or you and your 5,000 friends.

Wolf: And me and my 5,000. Yeah, well—

Richard: If we’re talking about a world where there are new powers rising, then I’m like, great, that’s the kind of world where new powers can rise.

Wolf: Yeah. The problem is when it’s like, to speak specifically with respect to Western people like us, our power socially is very caught up in the power of the existing state, and this was the thing we created to be the vehicle of our power. And as part of that bargain, it has power over us disproportionately. And as it goes into decline, a lot of the way it goes into decline is like autoimmune disorder, like going to war with its own people in various ways or like—just finding that it has less power, and so the only power it has is over its own people now. And it becomes this suppressive, like it—becomes the instrument of our own oppression, whereas other—people who are not historically bound up in it don’t have that same—relationship to it.

And so it’s like, yeah, you have the declining state, and that means a room for new things. That doesn’t mean we become free. Because we might just become even more oppressed by that system as it declines—because we have the historical relationship to it. And so that’s maybe the scenario I fear, right? It’s like how do we get out from under the state is kind of this crucial question.

And Nietzsche talks about this. We’ve been reading Zarathustra. He says basically where the state ends, that’s where the bridge to the overman begins. And it’s like there’s this—this essential need to get out from under the state because it has this last manifying effect—effect on you.

Richard: So, again, I think we might just be agreeing. I’m like, “Yep, there are bad possibilities, and there are also good possibilities.” And the thing I’m pointing at is more like, huh, we have two problems. One is that the state has become... Has turned against the kinds of principles and people that it used to be for.

And the second problem is that AI will dramatically disrupt the landscape of power. And I’m like, well, okay, maybe we have two problems, or maybe we have zero problems, or maybe they cancel each other out. And I’m not saying that like—

Wolf: Yeah, we don’t know how those are going to interact. I guess, yeah, this definitely feels like one of the crucial sort of future historical problems that we are faced with is, okay, we have the AI situation. AI is rapidly accelerating. There’s all sorts of things going on there. The technology is growing.

It certainly looks like we’re going to get more of that. And then on the other hand, we have this decaying sort of Western nation concept, or the Western nation state is kind of decaying, and you’re getting it. It’s—possible morph into something else that’s scarier or less scary or it gets taken apart for parts by various ethnic lobbies or becomes—some other version of itself or whatever. And how those pictures interact is the essential question, right? It’s like—are we going to see, like you say, they add up?

The one way they could add up is the state is getting worse and it’s getting more powerful—right? Because as it picks up the ability to use Claude accelerated Palantir to chase you down for every little opinion that you have—versus maybe the state just falls apart, and we’re able to use these technologies to kind of escape its power. I guess I’m more pessimistic than many people on this front.

Richard: Yeah.

I guess I, and I don’t know if this is cope, but the categories of optimism and pessimism sort of feel a little fake to me—in the sense that things really do feel pretty up for grabs. And especially even with Trump and Brexit and what’s happening in Europe, that does just give me a bunch more faith in democracy than I used to have. I’m like, “Oh, yeah.”

Wolf: No, likewise, I think that’s why I was adding kind of caveats earlier when we were talking about how democratic is the future actually going to be. I think there is this sense in which the kind of, let’s call it the broader extended elite consensus, has gone way more undemocratic than might actually be warranted—relative to the actual balance of power.

Richard: Yeah. And I do think that there is some culture and some set of ethics and principles and sort of intellectual understanding of what the hell is going on—that I think does just solve a bunch of these problems. In the same way that wokeness swept through all the institutions, it’s like, no, the good version of it can also sweep. Right. So it’s like a—

Wolf: And the question is—

Richard: ideology design problem

Wolf: the question is what kind of search process is going to create that ideological possibility, and what does the existence of that search process also create?

Are norms designed individually or collectively? (1:11:23)

Richard: Yeah. So should I tell you about my search process?

Wolf: Yeah. Well, let’s hear about it.

Richard: Cool.

Wolf: Because that is definitely of interest.

Richard: Cool. So I think the core problem that I’m trying to tackle is the sense in which existing frameworks are implicitly a kind of... They don’t know how to handle multi-agent dynamics. Basically our frameworks for understanding intelligence and rationality and so on are sort of—extremely hacky in any multi-agent case. So, of course, there’s expected utility maximization, where it’s like, “Here I am. I’m the decision-maker. Everything else is the world around me. I’m just going to optimize over it and try and control it. Okay, well, you’re in the world, so I’m going to try and control you.”

So that’s like baked into the framework.

Wolf: But didn’t that paradigm break as soon as it encounters game theory, like as soon as it encounters other agents? It’s been known to be broken for a long time, even though we still use it.

Richard: Yeah. So then I’d say game theory doesn’t actually solve this problem. It just gets around the problem. And the reason for this is game theory adds the concept of equilibria, but it doesn’t tell you how to get to an equilibrium. So we’re starting off, we’re playing a game.

Here are five different equilibria we could be in. There’s just no answer in game theory for how even in principle we should navigate between the different equilibria. And you can have equilibria. And there’s folk theorems in game theory that say, look, in iterated games, you can have basically any equilibrium. You can be in the heaven equilibrium where everyone is arbitrarily nice to each other because that’s what makes...

Because it’s stable for everyone to be arbitrarily nice. You can be in a hell equilibrium where everyone is arbitrarily cruel to each other because if they’re less cruel, then other people will be more cruel to them, and so they just start hurting each other. And so any equilibrium is possible. And so in that sense, game theory has failed to describe the process of like, “Okay, we’re playing a game. What should we actually do? How should we move towards an equilibrium?”

Yeah, so maybe the sort of core problem that I’m trying to solve is the problem that equilibria are a hack in the game theoretic setting.

Wolf: Yeah. And so how do you move beyond that? Because this does feel like the essential thing. You run into this game theory stuff.

One of the things I’ve noticed as you think through game theory and sort of, I don’t know, ethics in response to game theory and the kinds of decision theories that come out of that sort of stuff within the rationalist discourse and elsewhere. You end up reinventing a lot of concepts that sort of had traditional wisdom around them. The concept of honor I mentioned earlier, right? It’s like something that you can actually give something of a game theoretic foundation to. It’s like, well, I’m going to insist on certain types of treatment and—certain types of commitment to my own word and so on, not because in some naive utility maximizing way that maximizes my utility, but because the kinds of interactions that it incentivizes and makes possible with other agents is a much better set.

So that’s one way that you end up kind of recovering a bunch of interesting virtue ethics kind of philosophy out of things like game theory. But I’m curious where you would take—like going beyond game theory or getting those insights or how you would structure that kind of thinking.

Richard: Yeah. So I think there is a path to talking about things like honor in a game theoretic setting. I don’t think we’re there yet. And the reason is that one of the core reasons is this concept of the commitment races problem, which is just like, cool, I’m going to commit myself to being honorable. And now you’ve just got to take that for granted.

But then, surely then you should first commit to ignoring whether or not I’m being honorable so that then—there’s no point in me committing to being honorable. There’s a kind of—

Wolf: Yeah, historically speaking, it seems to be something that comes into being as a social equilibrium—rather than as an individual agent—unilaterally doing it.

Richard: Right. Yeah, and so in some sense the gap we need to bridge is between, like, hey, this is a stable equilibrium, but then how do the rational agents get themselves into the social equilibrium?

Wolf: Well, how much of that is just that a lot of these things kind of inherently have to come from outside of the agency of the agents involved, and it’s like there’s either some founder lawgiver figure or some kind of divine intervention—of some kind where the thing gets into the good equilibrium or the bad equilibrium, not for any reason that has to do with the agency of the agents involved. And because the challenge suggested by that is that, well, if they could do it by their own agency, they can also kind of undo it by their own agency or subvert it by their own agency.

Richard: Well, why would they?

Wolf: Well, why would they? But if they have the power to change the equilibrium, then maybe one of the agents hasn’t—now you’re playing a different game, right? It’s no longer the original game. You’re playing a game where at least one of the agents has the power to change the game in a way that—potentially—upsets that, and now you have a whole new set of—sort of problems and equilibria. And so it’s like whenever you sort of introduce the power onto the playing board of the ability to change the thing or the ability to resolve these kinds of coordination problems, that is itself this overpowered thing that changes the game.

And so my—my hunch is that as you run into that problem, at some point you just have to sort of recognize this kind of from outside quality of—of our ability to get into these things.

Richard: Cool. Yeah. So I disagree.

Wolf: Great.

Richard: And I think this is the sort of core crux between the classic conception of game theory in which it’s not able to talk. You’re in an equilibrium—and then it’s held together because everyone’s move is a rational best response to everyone else’s move. And so what if I do a different thing? It breaks. There’s no way to talk about why would I do a different thing.

We’ve assumed that I’m rational. Whereas I think that maybe most sociological phenomena are best understood as attempts to part of a process of making equilibria robust. So that’s why when you have norms, it’s not just that we’re all doing the same thing, it’s also that if somebody deviates, part of the norm is that you punish them for deviating. And so sometimes people can just through sheer force of will, set up a new system of norms by just refusing to get punished, and attracting others to follow them. But it’s actually a very hard process because the norms have this defensive—process baked in.

So that’s what makes it a robust equilibrium. That if any individual agent tries to change the norms, they’re pushing their way uphill.

Wolf: But someone set that up in the first place, and they had—

Richard: No

Wolf: they had some kind of large power.

Richard: No, that’s the whole thing about norms is you can’t mandate them. They grow via the actions of many people who each individually push a little bit in the direction of the norms they would like to see.

Wolf: I would say that, well, here’s a claim. The functional ones come out of someone designing it, and the ones that just grow are often quite dysfunctional, or they’re sort of one of the bad equilibriums or the middling equilibriums. Um—

Richard: So if we went and looked at a hunter-gatherer tribe, and they had some reasonably functional norms that said, “Here’s how you distribute the meat, and here’s how we decide who’s in charge,” and so on. Do you think they were designed?

Wolf: I think there’s probably some kind of authority figure that—would hold that together or something like that. Like—

Richard: But even—

Wolf: There can be—good equilibria that come out of the sort of natural process where if you throw people into the situation, and they’ll—find their way into the good equilibrium because it’s just sort of rational and it’s not that complicated of a problem. But I think historically, when you’re looking at kind of the laws of a society or the rules under which everyone’s coordinating—and the norms, it’s like they tend to be founded by particular political events where someone got a huge amount of power and like—

Richard: Ah, cool

Wolf: really reshaped things and built that. And that’s what I mean by this process of revelation, right? As far as anyone who’s living under that is concerned, that person might as well be divinely inspired, right? They sort of come in as this founder figure who—lays down the laws, and then everyone else is just kind of stuck in them.

Richard: Cool. Yeah, so I think I want to say that you’re reintroducing our master-slave dichotomy again, that I want to break out of. So, you’re like, well, look, either people are just bopping around in an egalitarian fashion, or you have this authority figure that lays down the law. And I want to say, well, actually, a part of the way that people choose what norms are in place is that they allocate prestige and respect to the authority figures that they trust the most. And what it means for a person to have enough power to lay down the law.

No individual has ever had the ability to—

Wolf: Yeah, they can’t—

Richard: like change things individually

Wolf: they can’t do it against—they can’t do it against the will of everybody. They do it because they’ve accrued the social authority—where everyone accepts their authority—on whatever basis.

Richard: Yeah. And so that process, I claim, is also part of the healthy process of individuals shaping norms—because they all push—

Wolf: That’s—that’s the process that you’re kind of interested in. I was trying to introduce, but there is that crux figure, who—who has to be the lightning rod of the equilibrium. And how do they end up in that position? What does it mean that that kind of person can exist? It can’t be that anybody can just do that willy-nilly.

But they do exist historically. That is where a lot of these things get figured out. But they end up in those positions of, I don’t know, folk legitimacy—because they incredibly demonstrate their ability to do it well—and their goodwill to do it well.

Richard: Yeah. Maybe William of Orange being the canonical historical example.

Wolf: Sure. There’s many. There’s many.

Yeah, we talked about that the other night at Jonathan’s talk.

Towards non-marginalist economics (1:22:56)

Wolf: So what kinds of theory are you working on that helps us—better understand that process?

Richard: Two things. One’s a high-level theory, one’s a very fundamental theory, I guess. The first one is I’m trying to figure out what it would look like if economics were not marginalist. That is to say, if it were illegitimate to talk about things like, well, let’s look at the effect of my action by holding everyone else’s actions fixed and just saying, “There’s going to be a small change on the margin.” Now, this is a big change because it gets rid of things like, well, here’s the share price of a company, right, and it’s defined on the margin.

It’s like—the marginal buyer and seller. And then, because you have a share price, you can multiply it by the number of shares to get the market cap of a company. And I’m—

Wolf: Yeah, which is a really stupid idea.

Richard: Right. So I’m saying if you get rid of marginalism, you can no longer talk about the market cap of companies, and that’s a good thing—because it’s a bad concept.

And ideally, we’d eventually want to be able to get another precise way of talking about how valuable is it to own or control a company. But like—

Wolf: Yeah. Or better yet, how valuable it is for the overall economy to have that company or something. That’s sort of what I imagine as the non-marginalist economics, is an economics where we can take the various parts and say, “Here’s what this thing is doing, and its—inputs and outputs, and why is that valuable?” And, “What are its capital stocks, and what are its liabilities?” In a way that is not as much about the buying and selling of the ownership shares—but about the underlying—activity that—it’s doing.

Richard: And I think this would look ultimately more like a question of how valuable is France for Europe or something. Where it’s like, okay, sure, you can measure some—things in monetary terms, but also just Europe without France would be very different. Maybe it’s—helping anchor the culture. Maybe it’s helping defend the continent, whatever.

Wolf: Mm-hmm. Do you think it’s possible to have kind of crisp concepts there, or is this inherently a question of, broadly speaking, the humanities, where it inherently is kind of qualitative reasoning that is very hard to pin down?

Richard: I think it will, at the very least, be as crisp as something like evolutionary biology, where you can’t measure...

Most traits of animals, it’s not like, “Oh, here’s a way of saying this is how much it contributes to...”

Wolf: Yeah, you can’t easily measure the relative fitness of—

Richard: Yeah. And the concept of relative fit or the concept of fitness in general is not very well-defined. Much less well-defined than I used to think in biology. But nevertheless, if you understand evolution, you just understand everything in biology far better, and you can think sensibly about all sorts of stuff that you couldn’t previously think sensibly about. So, I think non-marginalist economics, which will end up being moving towards a formal theory of sociopolitics—is ideally getting rid of a bunch of the precision, but replacing it with much more correct concepts.

So, that’s one research project.

Wolf: Yeah. And do you think that that relates to some of our earlier discussions about the question of interactions between humans and AI, or coalition dynamics one way or the other on those kinds of questions? Do you think that with such theory, we could achieve better coordination against slippery slope arms races? Or maybe that would just give us a crisp reason to understand why we can’t or—

Diving down the slippery slope (1:27:13)

Richard: Yeah. No.

Wolf: Or like—

Richard: Well, I think the whole concept of the reason that people slide down slippery slopes is because they’re using marginalist reasoning of like, “Well, it doesn’t matter if I go a little bit down the slippery slope because...” And I have this retrospective of the field of AI alignment. It’s like 20,000 words long. I just put out the second part of it a few days ago. And that is basically this detailed play-by-play account of how all of the AI safety people at these AGI labs were using marginalist reasoning to be like, “Oh, it doesn’t matter if I build the frontier models because somebody else will anyway.”

And it turns out that at all three AGI companies, the people who are building the frontier models were the ones nominally motivated by safety who were using that reasoning, right? It was literally this fucking comedy. Well, it would be comedic if it weren’t so tragic of just all of them individually saying, “Hey, somebody else will do it anyway, so I may as well get the safety benefits for myself.”

Wolf: Yeah. Well, this sounds a lot like being stuck in a bad equilibrium in game theory, right?

In the prisoner’s dilemma, it’s like, “Oh, well, the other guy’s going to defect, so I have to defect,” or, “At least the other guy’s decision is independent of mine, so I should just defect.” Right? And—

Richard: But the important thing is that if they had said, “Hey, the AI safety community is stuck in a prisoner’s dilemma with itself-” that would be very different. The thing they actually were saying is, “Oh, the AI safety community is stuck in a prisoner’s dilemma,” or stuck in a tragedy of the commons, I guess, with all of the AI capabilities people, and that just wasn’t true to anywhere near the extent.

Wolf: There were no such people.

Richard: Yeah. There were people. Obviously, I’m not saying all of the frontier systems, and specifically I’m thinking early Claude, ChatGPT, and then Sparrow at—DeepMind. It’s not like all of the work was done by safety people, but a lot of the work was done by safety people.

Wolf: Yeah. This is a very interesting topic, actually. One of the things that I observed was that the arms race with China preexisted any substantial results in China.

Richard: Right. Yeah. It’s insane.

Wolf: By which I mean the arms race in China, the meme—of the arms race in China preexisted any apparent attempts by the Chinese to dominate in AI or even to compete—in AI. And so, yeah, the Western labs were using “Oh, we have to beat China” as a justification for competing among themselves—in this uncoordinated way.

Richard: Well, it’s even worse than that. It’s not just that the people who are trying to push AI capabilities were using it as an excuse. It’s also that the EAs and AI safety people were playing up the threat of China—because they thought that that was the way to get taken seriously in DC, and then along with all the fear-mongering about China, they could also slip in a bunch of safety recommendations. And so, Situational Awareness, the memo is the worst offender here. Dan Hendrycks with Superintelligence Strategy memo, I think is doing some of this.

AI 2027 and AI 2040 are doing some of this. Yeah, and so it’s just this really incredibly destructive process of being like, “How do we cozy up to what we think the political consensus is,” while in fact being the people who are pushing the political consensus that they don’t want. Yeah.

Wolf: Yeah. Right, this kind of self-destructive anti-coordination is—is a very interesting phenomenon, and I think this is maybe getting to another interesting crux is, I’ll ask, why do you think that it is possible to come to a crisper theory that would be able to avoid this? What evidence do we have that that’s possible versus the alternative, which is that the crisper theory just formalizes why this always happens under any kind of... Maybe it’s not just marginalist economics, right? It might be that—this vulnerability exists or some analogy to this vulnerability exists within any sort of—worldview of coordination.

Richard: Yeah. Well, so I think one thing is that intellectuals are making this marginalist mistake far more than most people. And that seems related to the fact that economics is the dominant paradigm and—

Wolf: Yeah. Well, this—

Richard: economics is marginalist. So it’s like—

Wolf: This is really fun result in surveying various professions on game theory, and the economists do the worst because they have this like—this defect mindset.

Richard: Right. Exactly. And so it’s just like at the very least, if you make people take economics much less seriously, then you’re getting rid of one of the main forces pushing towards this destructive mentality. And, but then like—I’ve been told—

Wolf: No, and I think that’s true. Certainly, our society has been much more broadly than just in AI, torn apart by the sort of economic justification for looting—broadly speaking. There’s just all these ways that the economics profession amounts to just apology for various kinds of looting—and defection. And so, that I very much take seriously. I guess where I start to become interested is, yeah, we could have a better sort of basis for coordination, but—can we have crisp theory that actually avoids this failure mode or some analogy to it?

Can there be a final victory of truth over corruption? (1:33:20)

Wolf: Or is this a result, is the existence of economics kind of like in this format, the result of some kind of latent demand within the sort of space of possible coordination for this kind of corruption, right? Like the—is it just that you always have corruption and then corruption comes up with ideology for itself? And then to propagate itself. Or I think you can probably have healthier and less healthy versions, and it’s like, okay, maybe we’re further in the direction of being unhealthy on some of these dimensions—relative to maybe where we were previously or—other possible societies. But I guess I’m very skeptical that any of these problems will ever be permanently solved by better theory.

I think at best you get a temporary sort of, oh, we all see things in a less corrupt way for a while, and then someone figures out how to corrupt it. But—

Richard: Yeah. I mean, this seems like—

Wolf: But I’m curious what you make of that sort of line of argument.

Richard: Yeah. This seems like a bet against intellectual and scientific progress in the sense that, sure, no scientific theory is ever final, or none so far.

Wolf: Well, it’s certainly a bet against final victory in—coordination. But I wouldn’t say it’s a bet against intellectual progress because you could just as easily flip that around. It’s like, are we really going to bet against the intellect of the corruptors? They are also engaged in much progress, right?

Richard: But so yeah. So if I give some examples of success, so I think the invention of probability as a concept, the invention of computation as a concept. Now, I agree that these things can be corrupted, but it is just kind of hard to corrupt.

Wolf: Well, they massively improve our capabilities in some area. But that’s different from massively improving our coordination, or the possibilities of coordination apart from there’s technical means where—the coordination improves because now we can structure things in this new way where it’s not reliant on this kind of relationship, and instead we can lean on that kind of relationship. But it’s different from better theory solving coordination in particular. I guess my view is that coordination in particular, because it’s an adversarial game, it’s like the good equilibrium versus the corruption. Right?

And because it’s an adversarial game, there’s some way that the progress is symmetric. And—

Richard: Oh, I see. Yeah, definitely not.

Wolf: You think? Oh, you think so. Why?

Richard: Because you can just describe the corruption process. You can describe the forces—

Wolf: You can describe any corruption process. Yeah, sure. We could describe even the corruption process perfectly. But does that make it easier or harder to be corrupt? I don’t know.

Richard: Oh, it may—

Wolf: Or maybe it doesn’t change anything.

Richard: Yeah. I think bringing all the stuff, I believe in the shining the light on things cleans them up. So I’m like, yeah, if everyone understands the dynamics by which people get internally...

Yeah, by which this corruption happens, both on an internal psychological level, like the shadow, for example, from especially Jung’s concept of the shadow, and then also stuff like the way that norms tend to be corrupted. Even the concept of slippery slopes is an interesting piece of social technology that—

Wolf: Right. Okay, my claim would be that you can close off certain avenues—but it inherently kind of opens up other avenues. And we find it very difficult to imagine the sort of possibilities because to some extent, these things are happening outside of our ability to theorize them. It’s only—in retrospect that you can diagnose the cancer once you’ve really seen the result and seen how it worked and so on. But at the time, there was sort of no preexisting mechanism that could have recognized—that that was cancer because it occupied the blind spots of the system, and there were blind—spots of the system, and sometimes you can’t even tell that there’s blind spots of the system.

But my claim is there’s always going to be blind spots in the system.

Richard: You can make them smaller and therefore easier to combat.

Wolf: I don’t know. I’m skeptical of that.

Richard: I’m very confident of that. I’m just like, “Yep, I’m not betting against intellectual progress.” We just understand a bunch more stuff, like probability and computation and concepts and information, like other concepts we’ve formalized. They don’t help that directly with coordination, but that’s partly because they’re not about coordination. Whereas we start formalizing concepts related to coordination, and we start to be able to talk about what a norm is or what a—healthy relationship is.

Wolf: Part of the problem is that your epistemic environment, you can’t assume an uncorrupted epistemic environment when you’re dealing with these things because the corruptors are going to be trying to mess with your epistemic environment, and therefore the basis of your knowledge of these things. So as soon as you’re trying to say, “Okay, we have theory that allows us to solve these problems,” well, that’s equivalent to saying—there’s some theory that could be turned the other way as well, right? If someone got control of that epistemic process or corrupted the epistemic process in some way, where now it’s not... And this is almost what has happened in our society—is we have all these ideas of what it means to prevent corruption and prevent discoordination—and achieve coordination in society, and we have all these checks and balances. And those checks and balances now, largely speaking, exist to—facilitate corruption.

Richard: Right. ACLU is suppressing free speech and so on.

Wolf: Yeah. And so how do you... I think the cancer is as smart as you are, right? Is the basic problem.

Richard: No. Yeah. I think that’s just not true. I think the cancer is dumb by virtue of, and wokeness is dumb by virtue of being wokeness. Right?

It’s just like you can’t be—

Wolf: It’s dumb but it’s powerful, right?

Richard: Right. And there are viruses—

Wolf: And it’s done by smart people.

Richard: Sure. But that’s—

Wolf: It’s perpetrated by smart people.

Richard: Yeah. So viruses can spread. There are mind viruses.

Wolf: No, I don’t think it’s a mind virus. I don’t think it’s a mind virus. It’s that there’s people with interests that diverge, right? Someone finds a way—to get more out of the system for themselves by making things a little worse for everyone else, and they decide to do that. Maybe they see it clearly, maybe they don’t.

But in deciding to do that, now they have more power, and therefore more power towards the type of thinking that creates—that kind of problem. Right? And so as soon as you have this sort of—never mind the rational agents and the equilibrium, it’s just that there’s this kind of flywheel effect of—the defector gets more power, and then now you’re closer to an equilibrium of defection. And I think that this seems to be universal to life. Now, maybe it’s because we just haven’t discovered—

Richard: Yeah. We’re not—

Wolf: the true singleton operating system yet

Richard: all made out of cancer. Are we?

Wolf: No, we’re not. Because the cancer systems die and are replaced. There’s an other asymmetry on the other end, which is that once things are sufficiently decayed, the thing that it’s no longer able to enforce, it’s the decay, right? And so life is able to circumvent and come back. And so there’s this cycle between growth and decay, where when the growth has happened for a while, you’ve got a bunch of resources laying around that have been coordinated, accumulated in coordination.

There’s suddenly this huge opportunity for subverting that coordination, and that can be a self-compounding process. But on the other hand, once you’ve had self-compounding corruption for a while and now you’re back in the dark age, now the only thing to do, the only way forward is to create virtue and to create life. And so you get this—balance between them, but you never get a victory of one or the other because I think—it’s like we would like things to be nice and coordinated and honest because we’re engaged in the coordination and the honesty. But that doesn’t mean, and therefore, we’re going to apply our intelligence to solving things that way, but that doesn’t mean that someone else applying their intelligence doesn’t have other interests. Right?

Then they actually get more out of corrupting the system in a particular way. And so it’s not quite so simple just to say intellectual progress because the intellectual progress exists within the institutions under question and are influenced by the ways that corruptors can subvert these ideologies. I would even argue that maybe this is what happened with economics. Why is economics so bad for coordination?

Richard: Oh, yeah. Absolutely.

Wolf: It’s because people figured out, “Oh, we can come up with a sort of intellectual sounding justification for corruption.” And maybe economics was actually a lot healthier before that happened—

Richard: Oh, absolutely

Wolf: than I think it was. And I’m like, “Okay, let’s close off that avenue with economics.” That’s certainly a worthy project. But I think that’ll be temporary, even if you succeeded. And this comes back to the question of the singleton, right?

This is—why I think that singletons don’t work. I don’t think there actually is a final solution to corruption. There’s always more sort of blind spots somewhere, and that because it’s a adversarial game where you can’t assume that all the intelligence is on your side—you get these self-compounding corruption effects once you have a big ripe utopian society where everything’s coordinated. At that point, there’s this huge bounty for anyone who can corrupt it and crack it open from the inside. And so you get this feedback balance between growth and decay.

And maybe this should be a theorem in your new science of coordination, but—

Richard: I’m open to... Well, yeah. Right. Unfortunately, it’s the kind of thing that... I’m very open to the idea that there’s no way of locking in a good outcome.

I’m just like, “Yep, that seems right.”

You know what this reminds me of? Kuhn and his original work on scientific revolutions. He talked about paradigm shifts. And then he said that the next paradigm is not necessarily any better than the last paradigm. And that feels like the thing you’re saying to me.

I’m like, “We’re going to have a new paradigm.” And you’re like, “Listen, the new paradigm, it’s different. It’s corruptible in different ways.” And I’m like, “No, no, no. The new paradigm, it’s better than the last paradigm.”

“It still is going to make a bunch of mistakes, but it’s going to make smaller mistakes. And then the next paradigm, it’s going to make even smaller mistakes.” I’m just like, “Yeah.”

Wolf: Yeah. I would say with respect to coordination paradigms and ideology and the social basis for coordination and so on, the next paradigm is usually better than the previous paradigm, but they get worse over time. And so—I don’t mean the absolute progression is worse. I mean, you establish a paradigm, it works for a while, but then it decays as people find more ways to subvert it. And the result of this, anyway, to bring it all the way back to the beginning—the result of this is, I think that, I’m very skeptical of things like, again, the sort of coordination of humanity as humanity against—these highly valuable—technologies or potential political allies—that could crack open that coordination.

And so I expect that we’re going to go into that world. And now this, again, may be the sort of the traitorous kind of reasoning that you’re -

Richard: A little

Wolf: you’re attacking—as marginalist economics. And I recognize that. That is a very valid criticism. I’ve felt like that before, but just on reflection, I think, I don’t think it’s true that we can actually occupy that equilibrium. And so it’s something that I’m more interested in exploring to say, how then do we theorize the not necessarily arms race equilibrium, but the competitive equilibrium where there isn’t any kind of global coordination?

What is that actually going to look like, and what kind of ethics comes out of that?

Richard: Yeah. I’m reminded of Meditations on Moloch—which was a very powerful post. I think it’s mostly wrong. I think competition between nation states, competition between religions, basically all kinds of large-scale competition, are non-Molochian. There’s only one player.

Wolf: Well, I think they’re, broadly speaking, they create many good things even.

Richard: Yeah. I would say that the one place where there’s the most Molochian competition—in the world is amongst the group of people who take Meditations on Moloch seriously, namely the people at OpenAI and Anthropic. And Sam Altman said—

Wolf: How ironic

Richard: it is the one blog post he thinks about the most often. And EA culture is infused with Meditations on Moloch. And so I’m just like, “Yep, Scott summoned Moloch.” That’s just what happened. Not he’s created, right?

Many people involved, but just people read this, and it is self-fulfilling to the extent that, yeah, I think basically the single competition out of the large-scale competition, out of all large-scale competitions that ever happened, the one between Anthropic and OpenAI is best described as sacrificing everything in pursuit of this kind of competitive efficiency. Right? Because every other competition is between these nation states or these religions or things that are held together, even ethnic groups, they’re held together by non-economic forces. And—

Wolf: Yeah. Or they’re not about the competition. They’re not about trying to win the competition. Yeah.

Richard: And then this one is the purest, and so I’m like, “Wolf, watch out, there.”

Wolf: No, I completely agree with that diagnosis of the—concept of Moloch. Let me throw you something maybe strange here, which is that I think what he described in that process is not Moloch, but God. And—the result of taking that definition of what is effectively God, which is, if you take a sort of a Landian view on theology, which is that—

Richard: But why would you do that?

Wolf: Well, I do.

Richard: Yeah, but why would you do that?

Wolf: Basically the idea that... Well, I can answer that question, I guess. I think good things exist in the world. I think the good things were created by competitive processes. And in fact, that’s where the good things come from, broadly speaking, is the creative value in the world, the creation of new value and new good things comes out of competitive processes.

Richard: And cooperative processes working together.

Wolf: And cooperative processes. Mostly competitive processes in terms of—

Richard: Well, that’s just a vibe, man.

Wolf: I’m just saying, in terms of the way that evolution has worked, it’s like—it’s which things work, right?

Richard: Wait, have you—

Wolf: Which things work is the crucial—grounding.

Richard: Yeah. So this is the—

Wolf: And that is where good things come out of.

Richard: Yeah. I think that’s—

Wolf: And so what Scott tried to do is take a sort of appreciation for how you can get this defection equilibrium—and say that that was the source, that he generalized that way too far, and generalized it to the point of basically denouncing competition as the source of good, broadly speaking. And made all competition into a form of treason against this sort of Edenic equilibrium—that I think doesn’t exist. And in doing so, I think he is effectively engaged in some kind of rebellion against God, because he’s in fact defining himself against the creative process that has created all this goodness. And—what—what ends up happening when you do that is that you get that kind of like, I think you’re right, that worldview does summon into existence the kind of purely sort of, we must win the competition for its own sake kind of escalation—that is sort of treasonous in the way that he described. I think there’s this deep irony in that.

And this is one of the reasons that I like—try to attack the basis of that particular competition—because I don’t think they’re fighting over anything real. I don’t think any of them are actually going to get control of the future light cone. I don’t think there is any such thing as getting control of the future light cone. I don’t think there’s any such thing as astronomical stakes in terms of whether the future is going to be happy and utopian, or horrible. I don’t think that those are actually the possibilities on the table.

And I think if people took those kinds of possibilities less seriously, they would also engage less in that kind of purely, let’s call it—Molochian competition, and more in the kind of competition that I think is much more valuable, which is let’s pursue constructive projects, even against other people, but for our own ends that are not about the competition. And I think that’s been kind of my angle on that—discussion. I think the AI competition would be a lot healthier for everyone involved if they were pursuing their own ends rather than—just purely the competitive end of winning.

Richard: Sure. Absolutely. I think, well, a couple of things. My sense is that you’ve gone a little...

Yeah, let’s see. So I agree with the criticism of Scott. I’m thinking especially of his post, “The Goddess of Everything Else.” Which has the goddess of cancer as this kind of competitive force, and the goddess of everything else as a cooperative force. And it’s like, well, there should be this synergy between competition and cooperation.

It’s not like one’s just cancer and the other one is all the good shit. So there’s something off in that worldview. Land’s worldview goes way too far in the opposite direction, though, by sort of just biting the nihilist bullet almost. It’s like, oh, competition—extreme competition is extraordinarily destructive. So right, you want some synthesis—

Wolf: Well, this is where I’ve tried to kind of, I did try to respond to Land’s worldview, or the naive Landian—kind of accelerationism on that front, the nihilistic accelerationism. That I’ve sort of found that Pierre Teilhard de Chardin is a good antidote there. It’s a very similar worldview, but putting on the optimism glasses—in a way. It’s like, oh, this is actually beautiful and good, and this is what creates good things, and let’s actually take that seriously. And also—love is the basis for all order in the universe, and love is what’s ultimately being created by this process.

And that’s also true. And so Teilhard de Chardin’s “The Phenomenon of Man” is one sort of good antidote. Another good antidote is recognizing the importance of agency in the thing, because there’s, again, I’ve mentioned a few times, in the naive sort of nihilistic Landian worldview, there’s a discounting of agency, the sense that we don’t have control over the future. We’re just on rails, right?

Richard: But that’s also what you’re doing, right? You’re saying that there aren’t astronomical stakes. That it’s not possible to make the difference.

Wolf: Yeah. But I’m saying there are stakes. We do have domains—of agency—that we should act within, and we should try to make what happens in those domains of agency as good as we can. But I think one of the things that creates really bad equilibria and causes you to destroy the things in your domain of agency is to think that there’s some bigger value outside of your domain of agency that you’re going to try to seize. Right?

You sacrifice your actual domain of agency—for the sake of an imaginary utopian endpoint. And that’s what’s going on with the AI lab competition, right? Is they think they’re going to get control of the future light cone and have astronomical stakes, which means that any amount of destruction of our actual domain of agency—is justified on—the utility calculus there.

Richard: Look—

Wolf: And so I think that—you have to be clear about what is your actual domain of agency and treat it as your beautiful garden, right? It’s like, how do I make my actual domain of agency good? And this is where I’ve tried to provide the antidote to the problems of the Landian worldview.

Richard: Yeah. Right. And then I think I would just say, it is correct to focus on smaller domains of agency. I think there are ways to do that without calling the larger scale agency fully illusory. I don’t think—

Wolf: Well, sometimes it is, and sometimes it isn’t.

Richard: Ah. Right. But it’s hard to know.

Wolf: Because sometimes you think it’s illusory, but that’s actually an illusion itself. And it’s like—it depends on your own ambition that—

Richard: Yeah. Exactly. And so I kind of want to say, actually, it’s like, yes, maybe the universe is up for grabs.

Wolf: The universe is up for grabs, but not by any coherent—

Richard: Well, that’s just the lack of ambition

Wolf: coordinated thing. No, well, this is the thing, this is the hard challenge, right? Is to tell the difference between delusion and ambition. And I think it is epistemically very difficult, but I think on reflection, the light cone eating singleton kind of idea is on the delusion end and not on the ambition end.

Richard: Yeah. That’s important.

Wolf: I think it’s not actually a real thing, and I think that believing in it causes these sort of terrible coordination outcomes.

Richard: Sorry. Right. Maybe the synthesis here is something like, actually the universe is up for grabs if you are capable of being principled and high integrity and virtuous in a way which manifests, for example, in not just trying to barrel ahead—under the grip of these competitive forces. It is true that the things we do now will have repercussions that ripple throughout the entire future of humanity. And also, the only—

Wolf: But not so much arbitrarily. The thing I—

Richard: Maybe arbitrarily, but only if we’re actually able to follow good cooperative principles now. Yeah, I think that’s where I—

Wolf: What I expect is that, yes, life will continue to unfold and eat the universe and so on, and things will change on the margin depending on sort of what we do at crux times, but not so much and not in any predictable way. You might get stuff like QWERTY, right? Where maybe QWERTY is just locked in for all time. I don’t know. But—I think fingers are probably going to go away at some point, and then QWERTY will no longer be a thing.

But things like that, right? Unix maybe is locked in for all time or whatever, right? English language may be locked in for all time. But they tend not to matter, right? They’re like these arbitrary little technical details—that aren’t the important things.

I think what you don’t get is massive unbounded loss over the far future because it’s just not really possible to maintain the kind of order on that timescale or on that distance. And if it’s not, then we can kind of dispense with those sorts of mythologies. The only question as far as I’m concerned is like—do we sort of make it through the great filter or not? Or do we knock a big chunk off of our potential by slowing ourselves down for another thousand year dark age or something? Those kinds of things suck.

Let’s not do that. But I guess I’d put it as a quip like, if it keeps going, we win, right?

I think the best far future outcome that we actually have control over is whether we get anything there at all—versus just kill ourselves in some way. Which death does appear to be a possibility. We could in fact wipe out all life on Earth if we—did totally the wrong thing. But yeah, I think it’s just a way more realistic, and therefore healthier view to say, okay, there are limits to coordination, there are fundamental limits to coordination, therefore the kind of astronomical stakes arguments don’t work. Therefore, we should be focused on more immediate domains of agency and what we would like to see in those domains of agency for their own sake and not for the sake of their future impact.

And I think that is the only domain of value actually accessible to us, and if we sort of competed over that, I think we’d get much better outcomes—in that sort of sphere of value—which I think is the only one.

Richard: I don’t think I disagree that much with your direct prescriptions here. I think that getting to those prescriptions via the route of this is the only one is going to come back to bite you, and yeah. And so I’m like, look, we’re broadly on the same page, but I think it feels like there’s something that is currently load-bearing here. It’s sort of like, oh, the way you get rid of infinities is by multiplying by zero. Or infinite stakes.

Oh, but it’s got to be, like, the only way to get rid of this infinite stake over here is to say it’s impossible. Like zero probability. Great, now we can focus on this other stuff. And I’m like, “Oh, multiplying by zero, that’s kind of rough. It’s very dogmatic.”

I think you can focus on the more immediate domain, while maintaining—

Wolf: Well, where did—

Richard: agnosticism about the larger scale stuff, and just like, “Look, there’s going to be...” I think there’s—

Wolf: I guess my claim is that the focus on the infinity in the first place came by dividing by zero, so to speak. In the sense that you sort of... There’s an assumption—

Richard: But it’s not—

Wolf: There’s an assumption that you can drive coordination error down to zero. And if you can drive coordination error down to zero, then you get this huge infinite result.

Richard: Right, it doesn’t need to be infinity. It just needs to be very large. And I think you’re—

Wolf: No, but I—

Richard: trying to multiply it by something that’s small enough to bring it down to a manageable degree.

Wolf: I’m not... Historically, where I came onto this kind of thinking was I was very much in the, let’s say, orthodox Yudkowskian camp. I wrote a blog post called “Capturing Gnon”—which was a sort of view on... It was like a Landian interpretation of Yudkowsky’s project. And that blog post directly inspired “Meditations on Moloch.”

If you read—

Richard: Oh, wow

Wolf: if you read “Meditations on Moloch”, one of the things—

Richard: That was you?

Wolf: it cites that post that I wrote.

Richard: That’s wild. I love that

Wolf: I chewed on this stuff for 10 years—of believing that there were astronomical stakes—and that therefore various things. And because of that overwhelming importance, you have to focus on, how do we have to get there? How solid is this actually? And what does this actually look like? What possibilities are there?

And all this. And eventually, I just realized that the whole thing had kind of divided by zero. It was not actually a correct assessment of the kind of agency we had over the future. And I credit this to Nick Land. I got the antidotes—from Nick Land.

And I’ve tried to kind of overcome the problems in that worldview as well. But, so in terms of my historical evolution as a thinker, yeah, it’s like coming to believe that this sort of set of possibilities was impossible. And then—that, I had to kind of go and update my whole—value cosmology and so on as a result of that. Now, it might be that after you get through all this, you can come up with a much better framing of it. I agree, probably there’s a much better framing that doesn’t come as a result of explicitly like, “Oh, that’s closed off to us, so it drops out as a term.”

I think that’s my response to the Yudkowskian view, simply because I developed that idea as a response to the Yudkowskian view. And we’re talking about these Yudkowskian AI labs, right? But in terms of my actual worldview, I think how I would justify it more directly, I think there’s probably a much healthier way to do that. We should probably talk about that sometime. I think that’s another discussion.

But, I think there is an optimistic, directly stated view of what kinds of agency are available to us—and why that’s good, why it is that way, and what the good things we can do with that are. I think that you can just frame it all entirely positively. And I think that’s a useful project. It’s just not something—

Richard: I think it’s—

Wolf: It’s not something I emphasize in response to the Yudkowskian stuff—because it’s not as directly—applicable to that, and maybe it’s also more incomplete, right?

Richard: Yeah. And I think that’s basically my project and this thing you were describing as what’s the healthier version that doesn’t need to shut the door off to astronomical stakes is—what I’m going for.

Wolf: Well, I think—

Richard: And so I feel pretty good about just like, yep, we’re both seeing the shape of something here. I have some sense that your current framing of xenohumanism is optimizing a bit, trading off a bit against that something. But maybe, yeah.

Wolf: Maybe. These things always need more work. I guess the one big crux question that does matter is, does the integral blow up or is the integral finite? In terms of that long tail of future impact possibility.

Richard: I don’t believe in adding things up.

Wolf: Well, maybe that’s a way of saying the integral’s finite. But—

Richard: No. No.

Wolf: No, because it’s like you have this problem of... Maybe this is, again, the problem with the expected utility framework, right? But if you take anything like an expected utility framework—then there’s like—

Richard: We’re done. Yeah

Wolf: if there’s the possibility of an infinite—utility or a dominating utility that has non-vanishing possibility, then it blows up your integral, right?

Richard: Yeah. So we get out of that framework.

Wolf: Yeah. So let’s get out of that framework. I don’t really like that framework either. But it’s for reasons of tending to blow up in that way. Some of which I think are fake, like this one, and some of which maybe are real in the sense that if you define your utility function like that, it blows up.

But I guess that is the sort of... It sounds like, then, you’re on my side of that problem, which is the let’s not act as if the integral blows up.

Because that’s the mistake that I think the frontier AI labs are making—

Richard: Well, yeah, but I just want to—

Wolf: is they just think the integral blows up.

Richard: I just want to say, no, and I think you need to go one step further and be like, “No, there is no integral. The integral is a lie, Wolf”

Wolf: Well, that sounds like fun. We should have that next time.

Richard: Let’s do that next time. Yeah.

Wolf: Yeah. I think we’re probably out of time, though.

Richard: It seems, yeah.

Wolf: But yeah. Responding to utilitarianism and providing a coherent alternative to utilitarianism does sound like a good project as well that we should do.

Richard: Yeah. Oh, yeah.

Wolf: Well, this was a lot of fun.

Richard: Thank you, Wolf.

Wolf: We wandered all over the place, but I think there’s a coherent thread.

Richard: I think so.

Wolf: All right. Well, thanks a lot.

Richard: Cheers.

Wolf: Until next time.

The long-awaited Marko Jukic episode. We all expect AGI to colonize the stars because it’s the obvious thing any healthy expanding system does naturally. So why aren’t we? Marko explains the elaborate cope keeping us on Earth.

Transcript

Intro (00:00)

Wolf: Welcome back to “The Students.” I’m joined by my colleague, Marko Jukic. He’s a very talented analyst over at Bismarck Analysis. He wrote an article for us at Palladium called “The Only Reason to Explore Space.” This was a few years ago.

I understand you’ve been developing this thesis since then.

Marko: I have been developing this thesis since then. I have been thinking a lot about humanity, humanity’s destiny, the physical limits of the universe, and why we are so far away from them with our current level of technology and industry and social organization, which obviously kind of all go together. And yeah.

Wolf: Yeah. So the big idea that you’ve been pushing, which I’ve become somewhat more persuaded by over the years, is that we should just be directly targeting space expansion as the big picture. So this comes down to what’s the big picture that we’re pursuing as a society? So currently, I think the big picture is something like coddling humanity, world peace, kumbaya, UN world order, blah, blah, blah.

Marko: It’s a little unclear.

Wolf: It’s a little bit unclear.

Marko: It’s a little bit unclear.

Wolf: But at various times in history, we’ve had somewhat different ideas about what the big picture is. We’re waiting for—Christ to come and take us away. We’re trying to convert the world to Christianity or democracy or whatever latest idea we cooked up last Thursday. But you’re proposing that there is a natural answer to this question. Which is, let’s just look at the world.

It’s really, really big, full of matter and energy. Lots of space. We should just go out there and colonize it, explore it, develop it. Okay. So—

Marko: In short

What distinguishes humanity? (02:09)

Wolf: In short. So why?

Marko: So why?

And we’re getting right into the heavy-duty theory, but that’s why we’re here.

Wolf: That’s great. That’s why we’re here.

Marko: Let’s just go with it. So basically, you have to ask, on some level, what is a human being? And what does a human being do in nature, in sort of the universe, among the reality that we exist in? And then you have to ask, what do human beings do kind of collectively? Because we are a social animal, and everything we do is basically intermediated through social technology and institutions.

We’re not an individualist species. We’re actually—

Wolf: Yeah, we’re not bears

Marko: we’re not bears. In fact, we’re almost as far in the opposite direction as you could go, almost.

Wolf: We’re not quite ants, but—

Marko: We’re not quite ants, but honestly, we’re not far off. And maybe, I don’t know, maybe we’re more similar than we think.

Wolf: Maybe so.

Marko: So we have the human being, which is kind of a unique creature in the universe. As far as we know, obviously we’re related to all other forms of life on planet Earth, but we are clearly some kind of apex predator, and in some very important way, we are much more powerful, and we kind of stand alone among all the species of the Earth, right? And it’s an interesting question why we stand alone among all the species of the Earth. You can try to come up with all kinds of different answers.

Wolf: I think the Confucian answer is we have superior standards of righteousness.

Marko: Yes. Which also ties into this.

Wolf: Interesting trivia.

Marko: It also ties into this. So I take a, let’s say, a functionalist and a strictly materialist approach to this question, okay? And I would say some people, maybe a common answer, especially here in San Francisco, would be that the reason humanity is so unique is because we’re the most intelligent. We’re the most intelligent. Okay.

We don’t have to define intelligence, per se, right now, but I would go a little bit further and say it’s not just that we’re intelligent, it’s that we’re social is what makes us unique. It’s that combination. And I would add a third thing, which is that we specifically express our intelligence through creating tools. Creating tools out of the natural environment, which is inert and inanimate—not sentient or living. And also more interestingly, through creating tools out of other people through social technologies and institutions.

Wolf: Yeah, other beings in general.

Marko: What’s that?

Wolf: Other beings in general. Farms and—

Marko: Other beings in general. Sure. I guess animals are somewhere in between inorganic matter and us, but they’re maybe in many ways much closer to inorganic matter than to us, in fact. I would even say.

Wolf: I don’t know about that, but go on.

The power of artificial stupidity (05:40)

Marko: That’s a whole other can of worms. Doesn’t matter. So basically, what distinguishes humanity and the human being is not just our intelligence, but also our sociality and our ability to create tools out of both stuff and out of other people through institutions and social technology. And so basically, the reason I’m drawing an equivalence, I’m sort of putting those two things together is because I think basically the way we augment the power of our own kind of intelligent agency is by creating artificial stupidity, if you will, as opposed to artificial intelligence. And so when you look at technology or when you look at institutions, these are both forms of artificial stupidity in the sense that what we’re doing is we’re taking maybe whatever fascinating, interesting, wild and free forms that existed in the universe and in nature before our intervention, and we’re turning them into this predictable, reliable, intelligible process that the advantage of it is that it’s less complex, actually.

And because it’s less complex—

Wolf: It’s more manageable

Marko: it’s more manageable for a single intelligence, right? And there’s always ultimately a single intelligence that is behind this process, if you’re thinking about institutions, social technology, or if you’re thinking about physical technology. And I would even go further and say physical and social technology are very closely linked because all the technology that we have, it’s never some kind of process that a single person—it’s not some kind of recipe that a single person follows—a million steps.

Wolf: It’s always taught to other people.

Marko: It’s always taught to other people.

Wolf: Through an institution.

Marko: We have social differentiation.

Wolf: Adopted socially. Et cetera. Yeah.

Marko: There’s specialized knowledge. It’s not like one person goes and builds an ASML photolithography machine all by themselves.

Wolf: And then—

Marko: That’s not what happens

Wolf: builds computers for themselves.

Marko: Yeah. Or builds compu—

Wolf: Has an internet—where they send messages to themselves.

Marko: Yeah. Exactly. None of that stuff is the case. So, the social aspect and the fact that we are capable of treating each other as these inert cogs in a machine and creating machines out of other people to achieve goals is—what gives us our power, okay? Compared to all the other living beings that we know of so far.

Although, of course, it also looks like we might be the only living beings in the universe, right? Who knows what’s going on there? That’s a whole other can of worms.

Wolf: Yeah. The Fermi paradox.

Marko: The Fermi paradox. But if we are or if we aren’t, that’s what the human being does, okay? And basically, this process, I was actually talking to Samo earlier today, and we coined a wonderful sort of a catchphrase that I will definitely be using in the future, which is that what the human being is, is the stem cell of industry. And of course, industry, since we’re kind of defining terms, I would say industry is the implementation at scale of technology, and technology is, of course, based on the principles of science. So, I would say these three things are very closely linked.

I’ll just say that right off the bat. And where was I?

Wolf: What’s all this got to do with space? It’s like you’ve—

Marko: We’re getting there

Wolf: laid a great groundwork of what is man in relation to industry—society, but why space the natural—telos of all this?

Marko: We do not live in some kind of arbitrary universe. Our universe has a very particular character. There are particular physical laws. There’s a particular distribution of matter and energy in the universe that we find ourselves living in. And because we are these toolmakers who continually augment our capabilities—or, sorry, hold on.

Let me just take a step back first to finish what I was saying about the stem cell of industry.

Every time that you automate some process to achieve some kind of capability, the first time you attempt to automate this process is going to be the most manual and the least automated version of that process. And the first time you ever do it, it’s going to use the maximum amount of intentional human effort, okay, and intentional human labor.

Wolf: Yeah. Insight and—

Marko: Insight—

Wolf: all these things, yeah

Marko: and you’re going to have to do the—

Wolf: Creativity

Marko: you’re going to, yeah, creativity, and I would even say just raw labor because as you go through more and more iterations of automating any kind of process, if it’s physical technology or social technology, you’re going to find more and more ways to make it more automated and reduce the amount of human labor—that is necessary to achieve the same goal, right?

Wolf: Even if you’re not even using any tools, you’re getting better at it. You’re learning. Learning how to—

Earth and the firmament (10:51)

Marko: You’re learning how to do things. Yeah. Obviously, this is very closely tied to the idea of knowledge production. So, the point is you never actually fully eliminate the need for human labor, no matter how automated the process becomes, because in our universe, we have entropy—and things are constantly falling apart because of that. Nature and the universe, the conditions in which we find ourselves are constantly trying to kill us and fight the structures that we put together.

And so no matter how much you automate a process, it never becomes a sort of perpetual motion machine, right? You will always need some human effort, some human review of the process to ensure that everything is running smoothly, that nothing has broken down. And so basically, the flywheel that you have is that the more you automate any kind of process, your capabilities increase because you’ve created more and more of this artificial stupidity that is more and more powerful the more that you create.

Wolf: More tools, yeah.

Marko: With the same amount of human labor, you basically gain more capabilities, right? And This process, I’ve been speaking kind of abstractly so far, but this process does not actually happen in an abstract vacuum. It happens in physical reality.

Wolf: In an atmosphere even.

Marko: It happens in an atmosphere. So far, it’s mostly happened on planet Earth, which is a very particular place in the universe. And because we have all kinds of physical constraints and the constraints of our starting conditions, we discover that there are particular ways that all these processes have to unfold. Right? The development of technology, and industry, and science, and the reason I keep repeating that is because it’s like the scientific principles always inform what kind of technology is possible and, on the industrial side, it’s like the logistical possibilities also inform what kind of technology is practical at any given time.

Right? Even if we know that some things are scientifically possible, that doesn’t mean that they’re immediately going to be technologically feasible—let alone industrially efficient and viable given the human labor that we have available—to carry out, if that makes sense. Right? But all three things are important and they very closely feed into each other. So, this process does not occur in a vacuum.

It occurs in physical reality, and the more that we follow this process, the more capabilities we get, and inevitably, the more and more we follow this process, we are going to find ourselves running into the limits of matter and energy on planet Earth. And that could be a problem, except it’s not a problem because planet Earth is one infinitesimal speck of matter and energy in the wider universe. And we have this psychological barrier that we have, I don’t know, I guess it comes from wider culture or whatever, it doesn’t really matter. But we have this psychological barrier that, and I think it’s kind of insane, that the wider universe, outer space, whatever, that it doesn’t really matter to the important questions of human nature and the human condition—and what we should do with our society.

Wolf: So that’s just the firmament. It’s just up there.

Marko: Yeah, exactly. Actually, we have a very medieval... Even today, even though we have put men on the moon, we have lots of satellites in low Earth orbit, which is okay, to be fair, still pretty close to planet Earth, pretty close to the atmosphere. It’s not that far away in the grand scheme of things. But we actually still have a very medieval attitude towards outer space.

In the sense that we do just see it as this—

Wolf: This dome

Marko: far away. Yeah, it’s just this dome. It’s this far away. It’s irrelevant to us. It’s irrelevant to us, we are irrelevant to it, when I think this is almost the complete opposite of the attitude that we should have towards outer space.

Because actually, outer space is where we are going to inevitably have to end up if we continue following this flywheel of automating processes to increase our capabilities.

Wolf: If we continue to carry out our nature—

Marko: If we continue

Wolf: we’ll be continuing to develop—new capabilities—new needs—new demands. And breaking open new frontiers—then the final frontier, as some say, I don’t think it’s necessarily the final frontier, but this one big obvious frontier that we must eventually conquer—is that of outer space. All the other—

Marko: It is—

Wolf: all the other matter and energy in the universe—

Marko: It is physically inevitable. It is physically inevitable. I would even go so far as to say that, if you follow this logic, excuse me, there’s really only two options, which is that eventually we expand throughout the entire universe physically, or we—

Wolf: At least the reachable parts with the light cone and what not.

Marko: The reachable parts. The reachable parts are extremely vast. The observable universe is much bigger, but the reachable universe is also pretty big. It’s also pretty big. I think it’s, I forget if it’s a billion or a trillion galaxies.

I forget off the top of my head which it is. But that’s quite a lot of galaxies. There’s 100 billion stars in our galaxy, the Milky Way, and we’re only orbiting one of those stars. And we haven’t even ventured—

Wolf: We haven’t even started to—

Marko: We haven’t even started

Wolf: capture its energy.

Marko: We’re not even close. In fact, we haven’t even ventured to any other planets, and we are not even close to harnessing all of the potential of our own planet, in fact, despite what you might hear from environmentalists to activists. We’re actually not even close to exhausting the resources of this planet. So, we are so tiny and small in comparison to the vastness of the universe, and we treat it in this medieval way that we, like I said, there’s this feeling of mutual irrelevance that is actually totally unwarranted because our survival as a species actually depends on things that are happening in space. You might have noticed that there’s a bright, shining ball of flame in the sky.

That is in outer space. It’s not here on Earth, that’s the sun. And the moon comes out at night. That’s also pretty important for life on this planet. And sometimes—

Wolf: But those are so reliable. We don’t have to think—

Marko: They’re so reliable

Wolf: we don’t even have to think about it.

Marko: We don’t even have to think about them. And occasionally, a big rock will crash into our planet and kill a lot of people, or I guess people weren’t around back then.

Wolf: Or whatever is around.

Marko: Or whatever is crawling around the swamps. But who cares? That’s very rare, all things considered. It hasn’t happened to us yet. It’s happened on a small scale, perhaps, but Anyway.

But jokes aside

We know the AI is going to space (18:26)

Wolf: So you said there’s two paths. One path is obviously we—continue to grow, eat the universe, so to speak. What’s the other path?

Marko: The other path is that we just eventually die out on planet Earth.

Wolf: Somehow the Fermi paradox gets us. The great filter—

Marko: The Fermi paradox gets us

Wolf: the great filter gets us.

Marko: Well, I wouldn’t even say it’s the Fermi paradox. There’s a lot of potential solutions to the Fermi paradox, whatever, a lot of explanations. But what I’m talking about is in a way even simpler, which is that it’s actually not really a realistic outcome that we don’t expand across the universe. To some level. It’s actually—

Wolf: It’s hard to imagine how we could end up stuck.

Marko: Yeah. Because it is in our nature to automate processes to create greater capabilities—using the natural world and other people, right? And if we don’t do it, if our civilization chooses not to do it for whatever reason—

Wolf: Yeah. If we nuke ourselves back into the Stone Age—

Marko: If we nuke ourselves back into the Stone Age—

Wolf: Wipe out Homo sapiens entirely

Marko: yeah. Whatever we wipe out, we still have a billion or 2 billion years here on planet Earth for that to kind of come back.

Wolf: I think it’s just 1 billion, but yeah

Marko: Maybe 1 billion. Yeah. Whatever it is, it’s still plenty of time. We could sterilize the entire biosphere, I guess. It’s possible, but it doesn’t have to be the default outcome.

Wolf: Let’s not do that.

Marko: Let’s not do that. I think most likely, AI kind of doom scenarios notwithstanding, most likely our civilization just kind of decays and then another civilization will have to take up the mantle. But I think actually maybe relatively soon. Maybe not in 1,000 years, maybe much sooner than that. But anyway.

So I don’t really even think that’s a very realistic option. I think the only realistic outcome is that we do expand into outer space sooner or later.

Wolf: Yeah. It’s just a question of sooner or later.

Marko: It’s just a question of sooner or later, and now—

Wolf: Consciously or accidentally.

Marko: Consciously or accidentally. We could do it intentionally.

Wolf: And do it well or badly.

Marko: Perhaps. Yeah. I would imagine it would have to happen very consciously and intentionally. That’s why I’m doing this talk and—talking about this and trying to convince people that it’s something we should pursue intentionally, or at least be aware that we’re pursuing it intentionally. And what I would say, since we’re talking about AI, what I’m talking about is one-to-one what people expect a super intelligent artificial intelligence to pursue, right?

Wolf: Yeah. That’s always interesting. It’s like, what do we expect the AI to do? It’s like, what’s the big horror scenario with AI? It’s going to create paperclips.

Well, how many paperclips? Well, it’s going to turn the entire observable universe into paperclips. Right? It’s like—of course that’s what you do, right? Or like—

Marko: You’re not going to stay on Earth. You’re not going to stay on Earth, right?

Wolf: You’re going to shoot off into the stars, you’re going to make Dyson spheres, you’re going to take stars apart—you’re going to turn planets into cities.

Marko: Because it’s there.

Wolf: Jupiter brains.

Marko: Because it’s there.

Wolf: All this stuff. And yeah, this is the part that I find very interesting. It’s like it seems so obvious to us when we’re not thinking about—humanity, but we’re thinking about what should a hypothetical super intelligence do. Obviously it’s going to do space.

Marko: Obviously it’s going to do this. Then it’s the most intuitive thing in the world. We don’t even question it, right? I don’t think I’ve ever heard anyone argue that the AI is going to just stay on Earth and watch TikTok or whatever.

And so basically, then you have the question of, well, Marko, the AI is obviously going to be superior to us in every way, so why does humanity have to go to space? Humanity just has to create the AI, and then the AI will go to space. And so we’re going to fulfill what we sort of concluded is this inevitable eventuality. It’s going to be manifested not through the human race per se, but through the AI. And we’re going to live through it.

Wolf: Yeah. Vicariously.

Marko: Kind of vicariously. But if that’s how it’s going to be, in a way it’s not even vicarious, right? Maybe we’re just the bootloader for AI, right? Like this is—

Wolf: Sure. This is what Elon says

Marko: Yeah. This is not news to anyone who is probably watching this video or this recording right now. And so basically what I would say to that is, first of all, if it is possible that we’re going to replace the human being as the stem cell of industry with AI, then maybe it’s inevitable and we’re screwed, and that’s maybe very unfortunate, but if it happens, then it’ll happen. Maybe we can stop it, maybe we should. Some people have that view.

I think you mentioned in another conversation, maybe everything we’re saying about going to space is going to be very relevant to the AI as well. The AI is also going to have to be convinced that it shouldn’t stay on Earth. Maybe it also needs some convincing. If we need some convincing—

Wolf: Yeah. I mean, if—

Marko: maybe it also needs some convincing. So

Wolf: this is my sort of xenohumanist hypothesis. It’s like, well, you’re calling it the stem cell of industry. I’ve got various ways of defining it, but let’s take that niche and you wipe out humanity or replace it with AI in that niche. Well, whoever’s in that niche is still going to have all these same problems. There’s still going to be this question of maybe parasitism holding you back or—social organization, or how to actually achieve this problem of—organizing the means of production towards—higher ends.

Marko: Because—

Wolf: What are the ends? What should we be doing? All these questions remain. They don’t go away just because you got rid of humanity. These are not questions—even so specifically to humanity, there are questions about what the universe wants in some sense.

Marko: Well, these are questions that arise out of the contingent features of the universe that we inhabit, is what I would say. They arise out of the physical laws that we kind of cannot circumvent, like the speed of light. They arise out of, I mean, I would—

Wolf: Yeah. Laws of thermodynamics, finiteness of energy—

Marko: Thermodynamics

Wolf: and all this stuff.

Marko: I would imagine there are many laws that we have not even discovered. I don’t think there’s particular reason to think that we’ve discovered everything about nature or science.

Wolf: No, we know there’s quite a lot within biology—solid state physics, all this stuff. There’s all kinds of stuff that’s still—

Marko: We’re still making new discoveries. I don’t think anybody would really debate that, right? And so, yeah, I agree. Given the constraints of reality and the intelligence itself, which has to exist within reality, there is this niche of this living being or this agent that goes around the universe and creates—

Wolf: Builds stuff out of the—

Marko: Builds stuff. Creates automated processes to augment capabilities. To satisfy—

Wolf: I guess the question—

Marko: its desires or its wills, right?

Wolf: Yeah. So the question here, to really dig into the psychology of—

Marko: And if I could just—

Wolf: Go for it

Marko: interject.

Wolf: Finish your thought.

Marko: And just to finish my view on AI, I don’t have super strong opinions on this or that, that I can maybe convincingly argue, but what I’m interested in is the possibility that whatever kind of AI technology we develop, I think if it falls even slightly short of the capabilities of the human being as this stem cell of industry, and it’s not just in terms of intelligence, but it could also be in terms of robustness, and other features we could imagine that are necessary to—

Wolf: Yeah. There’s this really obvious—

Marko: survival long-term

Wolf: one which is, you and I can sexually reproduce with very little limited input, right? You can grab five acres and maintain a human population—through subsistence agriculture and so on.

Marko: You don’t need this globe-spanning—

Wolf: And that’s at the minimum, right?

Marko: You don’t need this globe-spanning network of—

Wolf: You don’t need globe-spanning supply chains—

Marko: factories and supply chains

Wolf: to keep the human going. Whereas—TSMC, ASML, chip fabs, all this stuff, they’re—

Marko: They are much more fragile

Wolf: it’s global capitalism, right? You need this enormous thing. And so maybe there’s some edge there even. But there’s a bunch of things like this.

Marko: Yeah. And so basically—

Wolf: But you’re saying that’s hypothetical. If—if the AI does not sort of reach our level on every dimension—

Marko: As long as it falls short of our level on every dimension, because, may I remind you—

Wolf: Well, on some dimension, yeah

Marko: on any dimension, I would say. Basically, because, may I remind you, the current globe-spanning capitalist industrial system has been built out of human beings. It is human beings who are the basic unit. The basic unit of capitalism and development and industry and everything—is the human laborer. It is like human capital, actually, right?

That is actually the basic unit. No matter how much inhuman capital, that’s a cool phrase. No matter how much—

Wolf: Machinic capital—as Land puts it, yeah.

Marko: Machinic capital. No matter how much inhuman capital we’ve developed, we still need the human capital to maintain it and to continually grow it, right? And so, I think as long as whatever kind of AI technology we develop falls short of the capabilities of human capital and the human being, then there will still be a niche for the human being to maintain industrial technology and—which is the sum of all of these processes that we’ve set in motion, right? And even if it falls slightly short, even if it comes within like 99.999, however you would measure that. Let’s say it’s like 99.999% as good as human intelligence, the human being kind of in this very grand kind of way.

It’s not that you’re going to have competition for human labor, which is how people tend to think of automation, right? It’s that the capabilities of the human labor that we have will be multiplied by that differential, right? In the same sense that... So, I’ll segue just a little bit into how I see automation. Normally, these days, we think of automation exclusively as a replacement for human labor, right?

Wolf: Right. Because we imagine this kind of fixed pie. Maybe this is another species of the medieval kind of, there’s only just this world—

Marko: You know

Wolf: this static social order—static technological order.

Marko: Actually, honestly, it kind of is. Yeah. It honestly kind of is. We imagine the only possible use of automation is to maintain the same capabilities with less human capital, right? With less human labor, less people.

But the problem is, that does not follow logically in any way. That’s just how we see it because we’re in this situation where we are not interested in growing the capabilities, the total capabilities that we have, right? Because the more automation you have, really, that just frees up human labor to turn its attention to more less automated processes. Which is kind of the stem cell idea I’ve been describing, right? Once we’ve automated everything that gives us the capabilities we’re interested in right now, then we don’t really just kick back and drink cocktails for—

Wolf: Fully automated luxury communism.

Marko: Yeah, fully automated luxury communism. Well, first of all, fully automated luxury communism cannot actually be fully automated because you’re still going to need people to maintain the machines that make it possible, to maintain the processes. And secondly, once we have that—is fully automated luxury communism going to encompass the entire universe? And if not, why not? And I’ll agree with fully automated luxury communism when space has expanded.

The expansion of space has made it impossible for us to even leave the orbit of a star. And then maybe we’ll be truly screwed at the end of the universe, right? But right now, it is possible to travel to other stars, kind of go past this solar system and across the galaxy and maybe—even other galaxies with enough time.

Sorry, I lost my train of thought here a little bit. What was I saying?

Wolf: Well, we could go back. I’ve got a question.

Marko: The point is automation is a force multiplier—or a magnifier of the human capital that we have. It is not a replacement necessarily. You can treat it that way, but it doesn’t have to be. Right? And no matter how much you can magnify human capital or multiply what it’s capable of doing, we are still nowhere close to the limits of the sort of total capabilities that we can deploy in the universe that we exist in, right?

So there is basically no reason to treat automation that way.

Why not do it ourselves? (32:07)

Wolf: So the part that I’m interested in here is this ideological component around why should we do this versus build an AI to do this for us? This sort of seems like this big question that everyone’s kind of implicitly thinking about right now. And if I can just offer some thoughts on that.

There’s this sort of death drive inherent to the AI version of that question, right? It’s like we don’t want to do it ourselves. We can’t do it ourselves, or we don’t think we’re good enough to do it ourselves. And so we sort of put it off. We say, “Oh, we’re going to build the tool. The tool is going to do it for us. We’re going to build the AI. The AI is going to kill us and replace us, and then do the space-ism for us, right? Take over the universe, turn it into things.”

And we worry about, did we get it right? Are we building the right AI versus the wrong AI? Are we building AI fast enough? These sorts of things. But I just find it very interesting.

We are in fact also doing the other path, which is just do it directly. Yeah, using whatever automation technologies we have. But this is like SpaceX, right? Elon actually is out there—

Marko: He is

Wolf: building the infrastructure—of going to space. We can just go do this. This could be just what we do. And I find that ideological component very interesting. Note that it’s just one guy, right?

Without Elon—without Elon being a—

Marko: It’s always just one guy

Wolf: without Elon being a disciple. Yeah, it’s always just one guy.

Marko: It’s always just one guy. There’s a lot of one guys working together.

Wolf: Yeah. But without Elon being a sort of devoted disciple of von Braun—we wouldn’t have this. We would just be sitting here—

Marko: We wouldn’t, we wouldn’t

Wolf: in Yudkowsky universe, right? And there would just be one guy, and it’s Yudkowsky and/or Dario or whoever building the machines to replace us. And we’d have an even more medieval idea of the world. It’s like—we’re trapped in this little egg in the Earth, and all we can do is kind of build this machine to replace us because it’s more intelligent. And it’s like, isn’t that so interesting?

It comes down to this ideological question for one guy. And maybe—if Yudkowsky hadn’t read Vernor Vinge in the early ‘90s, we wouldn’t be on the AI track at all. We’d be talking only about—space. Or we’d be talking about genetic technology or—

Marko: Or we’d be like China

Wolf: or whatever.

Marko: Or we’d be like China, and we just wouldn’t see AI as such an existential threat necessarily—

Wolf: Yeah. We wouldn’t see it

Marko: but more of a useful—labor-saving technology just like any other, right?

Wolf: That’s the aspect that I find interesting. And this is where your theory or where your kind of ideology, for lack of a better term, is slotting in. You’re basically proposing all that other stuff is secondary. It’s means to an end. And the main story is this expansion.

This consciousness and/or humanity and/or—technology and industry—expanding throughout the cosmos. That’s the big story. That’s what we should be thinking about directly, and everything else is just means to that end. And then, yeah, the part about, in particular, there’s this big difference between doing it directly, as Elon does, versus—

Marko: Doing it vicariously through the AI

Wolf: vicariously where it’s like—we don’t want to do that, or we don’t believe we can do that, or we don’t believe—we’re allowed to do that or whatever. And therefore, we’re going to build the AI, and the AI’s going to, of course, go do that. Right? I find that very interesting, this—psychological license—question.

Marko: Yeah. I have some thoughts on that. So I think the reason that we have that kind of cycle... Okay, so it’s kind of interesting to think about the way that people imagine the AI. The AI, but that’s basically how people think about it.

It’s like this—it’s just—

Wolf: Yeah, it’s just one thing. It’s—

Marko: Yeah, it’s just like this one thing.

Wolf: Or people even reify it even further. They say the AI God or the—

Marko: Yeah. The AI God

Wolf: we’re creating God.

Marko: Whatever. But it kind of is just—it kind of is just this indistinct, like this kind of—

Wolf: The thing

Marko: the thing. This like indistinct—

Wolf: The AI. Yeah

Marko: this indistinct, powerful force that we can just, it’s just like this big bucket that we can throw all kinds of—

Wolf: That we can project our ambitions onto

Marko: we can project. Yeah. We can just project whatever we want onto it. But that is kind of what the concept is, so let’s go with it. I think it’s kind of interesting the way that people imagine the AI would kind of fulfill these ambitions that we were discussing that we feel like we cannot fulfill for whatever reason.

And it’s sort of like I guess the biggest obstacle for the AI would be persuading the NIMBYs to allow it to build factories or—

Wolf: Well, it could just recycle the NIMBYs.

In some of the scenarios.

Marko: Yeah. So you’re sort of proposing this thing where it’s like, well, okay, well, if that’s true, then you’re sort of saying the AI will actually not be able to help us achieve these grand material technological goals until it’s literally capable of just going full Skynet Terminator mode, right? It’s like—

Wolf: We could imagine one that can help us short of that. But in the most ambitious scenario—some of the ambitions that people project onto it are also genocidal ambitions.

Consumerist enervation (38:07)

Marko: Sure. But I think it’s interesting that we have to go to that kind of extreme to imagine growing our level of industrial technology further. Right?

Wolf: It’s easier to imagine the end of humanity than the end—of NIMBYism.

Marko: Exactly. The end of NIMBYism or the end of a massive wealth redistribution and transfers, or the end of this untrammeled consumerism that everybody feels they have a right to participate in forever, kind of—

Wolf: Or not even a right, it’s an obligation.

Marko: An obligation.

Wolf: It’s like this—

Marko: An obligation

Wolf: religious obligation—

Marko: A religious obligation

Wolf: to be financially selfish, right?

Marko: Yes. Well, it’s not just financially selfish, but kind of to be the opposite of what I’ve been describing. The human stem cell is kind of a worker, like a worker bee. This is a being that has an inherent drive to work, and to think, and to impose its will on the universe around it. Right?

Whereas there’s this alternative conception of what the human being is, which is not like a source of things happening—

Wolf: But a sink

Marko: but just this gaping maw that can consume infinite amounts of resources and still never be satisfied.

Wolf: And should, in fact. Yeah.

Marko: And should, in fact.

Wolf: And this is the last man, right? It’s like man—man has the walking stomach. Man is—

Marko: This is like the last man. Now, actually, the problem with that is that our capacity to consume is actually not infinite, because if we consume too much, we kind of end up exhausting our own potential, and then we just die. We don’t even reproduce and raise kids, which is actually also a lot of work. It’s actually also a lot of work. Right?

Wolf: Yeah. That’s an interesting point.

Marko: It’s work.

Wolf: This is a connection I hadn’t quite made.

Marko: Reproduction is work, itself.

Wolf: Reproduction is work.

Marko: It’s work.

Wolf: And you have to have that worker conception to the world—to—reproduce.

Marko: Yes. Otherwise, why bother?

Wolf: If you don’t have a creative drive to—put forth new things into the world, why have kids?

Marko: Yeah. Why? There’s really no reason. If you can just be a sort of lazy consumer who you just don’t have to do anything ever—

Wolf: You just get your AGI bucks.

Marko: Well, you basically wirehead. You could take it even further. We are actually probably pretty close to building wireheading. For listeners who don’t know, this concept of kind of a brain implant that instead of taking drugs to feel pleasure or doing things in the world to feel pleasure, you just install this brain implant that electrically stimulates the pleasure centers of the brain, and then you can kind of just sit there and—

Wolf: You feel super fulfilled and profound

Marko: yeah, you can just feel awesome forever until you expire, right? Why not do that at that point? Why shouldn’t we do that, right? This is kind of the... Why shouldn’t we do that, right?

Well, there’s actually kind of a very forget all the philosophy, forget all the kind of moral philosophy, morality, whatever. I don’t even care about that stuff. It’s just an irrelevant question because anybody who does do that is just going to disappear, and then they’re going to be irrelevant to the questions of whoever is left in this universe that we live in, right? So it’s kind of a total dead end that doesn’t even need to be addressed. And so, okay, to step back and go to the question of why we psychologically feel like we have to vicariously route our ambitions through AI, I had a bit of a viral thread on Twitter addressing this question, and I think I hit on some good insights, which I’ll sort of repeat here.

Displacing our ambitions onto AI (41:28)

Marko: I think the reason is kind of that we do intuit that we are not actually capable of circumventing, or reforming, or destroying these systems, these automated processes in the form of these powerful global, national, continental, transnational institutions in the forms of governments or big companies, which are some of the most powerful institutions that have ever existed in human history.

Wolf: Right. We feel a little bit powerless with respect to the system, and—

Marko: We feel powerless against the might of the US government, or the United Nations, or the International Monetary Fund—

Wolf: Yeah. We see that these things aren’t—

Marko: or ExxonMobil

Wolf: by default going to go to space.

Marko: They’re not by default going to—

Wolf: And so we think—”Okay, well, what if we had more intelligence? What if we had-” “What if we could copy ourselves in silicon or...”

Marko: Exactly. We actually naturally intuit that we live in this world of these gargantuan, powerful... And it’s not just governments, to be clear, right? It’s corporations, maybe nonprofit foundations even. Maybe religions.

Maybe the Catholic Church even is this gargantuan kind of slothful, old, decaying institution that has kind of—a grip on some large portion of the population’s morality or whatever, their sort of moral conceptions of the world. All of these big bureaucratic organizations, basically, that make up the world that we live in and which are so big—and so powerful because of the capabilities that were brought forth earlier in the 19th century and the early 20th century—

Wolf: By some guy

Marko: by some guy, by many some guys who basically caused the Industrial Revolution and gave us all modern—

Wolf: And built all these things

Marko: and built all these things, right? None of our modern institutions—okay, maybe the Catholic Church is an exception, of course, but most of our modern institutions do not make sense in their current forms in any way if we don’t have industrial material technology, right? The assumptions of these, and again, these institutions are like automated processes that were set in motion by someone in the past, right? By live players, right? By live players.

They built these institutions to achieve certain goals, and at the time many of them were built or reformed, we were no longer in the technological world of agricultural societies where you had many fewer options for just the technology that you would use to organize people. You had a lot more options once you had gunpowder, railroads, flight, modern chemistry. This gave you many more capabilities to build more powerful organizations, right? And I would go further and say that these organizations kind of conquered the world during World War II and during the Cold War, right? You kind of ended up for the first time in the middle—well, first of all, you ended up with a lot of consolidation of power.

Over the course of the 19th century, European empires kind of took over the world, right? Or whatever was left that they didn’t conquer already. Sort of on the basis of the superior technology that they had, and then shortly after that, you had an even greater consolidation of power until you ended up with the situation in the middle of the 20th century where you kind of had truly globe-spanning empires who then quickly developed intercontinental ballistic missiles, which were capable of hitting any targets on the planet and detonating nuclear explosions way more destructive than any kind of weapon that was built before, right? So you end up in this place where now you have organizations that are genuinely capable of exerting hegemony over the entire world in a way that had just never been possible before, right? And so basically, this is like people maybe don’t like to think about that dynamic too much, but I think it’s actually one of the most important pieces of the puzzle because what happens when an automated process that you set in motion, an automated process of great power, no longer has its creator or an equivalent human being, an equivalent stem cell who is capable of holistically evaluating this thing and knows how to control it because they built it.

When that person is no longer around, conditions continue to change, but the process continues doing what it was doing before.

Wolf: And it has nuclear weapons.

Marko: And it has nuclear weapons. Yes. I mean, yes. Wolf, this is what we are talking about. This is exactly what we are talking about.

And so, if the particular people that you want to look at there are FDR or Chairman Mao or Stalin or Vannevar Bush or whatever, that’s more of a historical kind of case study. That’s more of a historical question you can answer through case studies.

Wolf: Point being, people today are faced with these things. We’re faced with a world we can’t control because it’s—a lot of it is these—enormously powerful organizations—that may not even be piloted by a live player anymore.

Marko: They are definitely not. Most of them are definitely not piloted.

Wolf: And there’s nobody you can appeal to.

Marko: There’s nobody you can appeal to.

Wolf: And so in the face of that, it’s very easy to imagine, well, maybe we need a new technological revolution, new kinds of agency—new intelligence. Some enormously powerful thing to be able to overcome—those problems and—build a new world.

Marko: Exactly. Exactly. And that is almost to a T what people imagine the AI is, right? It’s like literally, gee, what would we need to overcome the power of the US government or of ExxonMobil or of the Catholic Church or all these other organizations or the Chinese government? Not to sound like I want to overthrow the US government, right?

But it’s like how could I even negotiate? How could I reform these institutions? I mean we, not just me, but everybody. How could we all collectively try to change—some things around here? How could we just try to do things a little bit differently?

Wolf: Yeah. How do we fix things?

Marko: How? How could we fix things? What can we do? And it just seems impossible, right? And it just seems impossible, and because it seems impossible through any kind of semi-normal kind of relatable mechanism that people can imagine, the only thing that they can imagine is, well, I’m going to create this magic box that’s literally just going to be this fount of magic military power that’s going to kill everybody and then just rearrange the entire world the way it should be, atom by atom.

Wolf: Yeah. And of course, the way it should be is it’s going to go and build a Dyson sphere and—

Marko: Yeah, all that great stuff

Wolf: turn the stars into stuff.

Marko: Yeah. All that great stuff which we are perfectly capable of doing ourselves, actually, right? With or without maybe assistance from some less—

Wolf: Some enormously powerful technologies

Marko: powerful form of AI. Yeah. We will most certainly be creating enormously powerful technologies if we expand into space, by definition, right? By definition, we will most certainly be creating unbelievably powerful technologies, whether that’s through the artificial intelligence of computing or if it’s even more powerful antimatter bombs or whatever you can imagine. And so basically, the key mechanism here is that these extremely powerful globe-spanning, multi-continental, whatever, I’m just throwing out adjectives.

I’m just throwing out superlatives here, but it has to be emphasized how much more powerful these organizations are today than any organizations in the past.

Wolf: Well, what I find interesting, actually, here’s another aspect to it. Those organizations, that order, has a name, right? And that name is humanity. Right? Like, just in terms of the history—of what came out of the Second World War.

It was the UN—

Marko: Well, it’s an apt name

Wolf: the Declaration of Human Rights.

Marko: It’s an apt name.

Wolf: Humanity has this like, these things are the organized agency of humanity, right? And so, isn’t that an interesting parallel that when we imagine an escape from it, we’re imagining an escape from humanity.

Marko: Yeah. We’ve kind of—

Wolf: We go into AI. It’s like, well, we need to invent something that isn’t human—

Marko: It’s kind of ironic

Wolf: to escape from it.

Marko: It’s kind of ironic. Well, and you know what’s even more ironic, Wolf? It’s that what we imagine the AI to be, which is this dynamic, revolutionary, life-affirming force—full of power that can take over the whole universe. That actually is the real humanity, and what you just described as humanity, this order of these dead institutions that cannot react to changes and conditions, that is much more like a machine.

Wolf: Yeah. It’s like the—

Marko: That is much more like a machine

Wolf: it’s the last man, right?

Marko: Well, and it’s—

Wolf: Yeah, no, and to the extent, let me—

Marko: That is the artifice. You understand? That is the artifice.

Wolf: Let me—

Marko: And we’ve actually switched the labels.

Wolf: Right. I know. We’ve switched it around. And just to back up the point that you made that that’s actually the real humanity, look at what the definition of humanism is, right? Humanism is this philosophical position that takes—as its reference point and its center and its goal, the development of the kind of being who is the center of philosophy—and historical agencies, a moral agent—can change things around, it can change its environment.

We imagine this being that has this—this open-ended moral agency in the universe, and we call that being the human. Right? And this is what we mean by humanism. And like—isn’t that funny that that being now in our imagination is called the AI? Right?

And—

Marko: Yes. It is. I don’t know what to s—

Wolf: We’ve switched the labels.

Marko: We’ve completely switched the labels.

Wolf: We’ve switched it around. Now humanity—

Marko: It’s incredible

Wolf: just means the last man, and AI means the referent of humanism.

Marko: Yes. No, this is all true. And basically, so to go a bit further, because I’m sure people listening find this kind of interesting and kind of weird, but it—

Wolf: Maybe some of them

Marko: probably speaks to them in a certain way. Because you do have this feeling of being trapped. You do have this feeling of being caged or trapped, right? And I would say in the past, you would still have these extremely powerful organizations, but you kind of had a natural, you had something of a natural escape valve just because so much more of planet Earth was not completely locked down by powerful organizations, right? You sort had the steppe—

Wolf: Well, the interesting thing is, of course—

Marko: You sort of had the desert, you had sort of less advanced societies who might accept you if you were intelligent or maybe a good warrior.

Wolf: Well, let me propose that in the Middle Ages—things were maybe even more locked down or there was even less potential for change.

Marko: Maybe.

Wolf: Because certainly things were moving slower. They were moving slower.

Marko: Things were moving slower. I mean—

Wolf: And to go along with that, people certainly were very religious, right? They had these—ways of imagining escape, right? Whether it’s heaven and hell or ascetic kind of practices that develop their spirituality in some way out the side of the system.

Marko: Well, let me put it this way.

Wolf: So I think this is an eternal problem—is that we fight, we seek escape from—

Marko: It’s—

Wolf: from the immediate order of things—in these imaginary entities.

Marko: Yes. So what I would say is that what you’re describing is exactly, it’s totally natural, and it’s a totally natural response to the feeling of being the human being who has become part of the automated process of someone else who’s more powerful than you. And you discover that you have no ability to change the process unless you kind of have the particular powers and the sort of social capital and skills of the person who created that process in the first place, right?

Wolf: Yeah. And you start trying to imagine ways that you get out somehow. Or that things get fixed—or corrected or reformed.

Marko: So most of the time, people do not have the understanding or vision of their own society to realize what the practical path to reforming those processes. I like saying processes. Those institutions, what the actual path to that would be. Because that is secret and defended, highly defended and secret knowledge by design. Because—

Wolf: The people in charge—

Marko: Of the people in charge

Wolf: of those machines don’t want you to be changing them. They don’t want you to have the password.

Marko: They don’t want you to have the password, right? And the more impa—the problem that we kind of face as the human stem cell. This is one of the big philosophical problems that we face, is that even though all of our capabilities sort of come from this process of someone reorganizing other human beings and the matter and energy that they’re capable of individually maintaining and organizing in their various subprocess, even though that’s the only way that we gain capabilities—of any kind, and that’s a cool thing to have. We like to have more capabilities. Even if we don’t have them, we just like to see the effects of them.

It sort of gives our lives meaning.

Wolf: We like to participate in that.

Marko: We like to participate, and we are participating. By being cogs in the machine, we are also participating, right? We are playing this, in a way—

Wolf: Well, you know that classic story about the janitor at NASA? Someone goes, it’s like, “What are you doing here? Are you sweeping the floor?” And he’s like, “Well, I’m putting a man on the moon.”

Marko: Exactly. No, that’s exactly how it is. But the problem is, the flip side of that is when that highly defended knowledge of how to control the thing, when it dies with the person who was in charge of it, when they fail the succession problem, which is something Samo Burja talks about a lot. It’s one of the linchpins of this kind of entire worldview. You could even say it’s like when they fail to pass on that knowledge and power to someone else, then you no longer have this carefully maintained garden of artificially stupid subroutines that help us achieve something that someone actually cares about.

Wolf: No, you just have a bunch of stupidity.

Marko: You just have a bunch of stupidity that we’re trapped inside. And we don’t know how to change it because we don’t have the keys. We don’t have the password. Actually, nobody does. And the only hope for it to change is for someone else to become powerful enough to take over the whole thing from the outside, basically, right?

And maybe at most resolutions, maybe in most places and times, that’s been okay enough. It always feels terrible to be trapped in this totally uncaring kind of—

Wolf: Trapped in the system

Marko: trapped in the system. It never feels good, and so to go back to what you were saying about we always imagine these escapes, I think that’s maybe where a lot of it comes from. Because you’re trapped and you don’t know how to get out of it.

Wolf: And I think the important thing to emphasize here is the escapes are always, there’s some element of reality to them, and there’s some element of imagination. Or even superstition, like just a total unreality. And a real reality.

Marko: Well, I would say—

Wolf: And to apply that to the AI question

Marko: Go ahead, yeah

Wolf: there’s a certain reality to AI, right? It’s like, clearly this is incredibly powerful technology, much more power still promised on the roadmap. Very, very high potential to do a lot of things. But I notice that a lot of what we’re doing with it is projecting, frankly, quite superstitious ambitions—onto it. that it’s filling this psychological niche of the need for escape.

And I think if I can kind of offer an element of your program, it’s that what it behooves us to do is undertake the hard work of disbelieving in the imaginary part—and finding the real—figuring out how to do the actual work. These institutions were ultimately built by men, or maybe they’re more competent than us. Maybe they had access to things we don’t.

Marko: They definitely did.

Wolf: Maybe we’re trapped inside their system.

Marko: They definitely did. Yeah.

Wolf: But they were built by men.

Marko: They were built by men.

Wolf: There are things we can do.

Marko: There are things we can do.

Wolf: The humanistic faith is that we do have agency.

Marko: Yes. Yes. We do have agency.

Wolf: We are able to act.

Marko: There are things we can do.

Wolf: But it is hard.

Marko: It is extremely hard.

Wolf: And we have to put—

Marko: It’s never been harder

Wolf: to put aside—

Marko: I would say it’s never been harder. It’s never been harder.

Wolf: Yeah. We live in a bigger world than ever.

Marko: A bigger world with more powerful technology, more powerful institutions. We’ve never faced the scale of this kind of a problem before

Wolf: yeah. But the essential problem is not to escape into these psychological escape hatches. But to put that aside, to put aside the superstition, to think in these material agency sort of terms—and find that path through. And I suspect, this is my suspicion on this conversation, is that if we were doing that more effectively, I think we’d be more interested in directly just reforming our institutions to go to space—than we are in AI. And that’s, I think, the answer to this question.

Marko: That’s what I’m interested in, basically. Yes. That’s kind of how I look at it and why I’m more interested in space than AI. I totally agree. And I think it’s like—

Wolf: And just to reiterate why that is, why is space not the escape? Space is the goal. So everyone understands—

Marko: Space is the goal

Wolf: space is the goal.

Marko: Space, yeah.

Wolf: But it’s like if we felt powerful, if I felt powerful, if I was a super intelligence, well, of course. You go to space. You build a Dyson sphere. Maybe we keep the Earth sort of in its own gravity well. Maybe we smear it out, turn it into solar orbit stations.

I don’t know. We’ll figure it out.

Marko: So many fun things. So many fun things.

Wolf: But there’s a lot of fun things we could do. And if I was a super intelligence, of course that’s what I would do, right? So if I felt powerful, of course that’s what I would do, right? That’s what I would want. But if I don’t feel powerful, maybe I start to get these revenge fantasies against the system, and I—I start to think, “Oh, well, maybe the AI will smash up the system for me.”

Right? And that’s, I think, the psychological dynamic that—that’s very present. If not totally driving it. I don’t want to dismiss AI. It’s clearly a very powerful and important technology.

Marko: Not totally. I mean, there’s—

Wolf: It’s a very, very powerful and important technology

Marko: yeah. No, there’s very real technology there.

Wolf: But we have to be aware of these psychological dynamics.

Marko: Yeah. Yeah. Well, it’s what we project onto the technology, right? The technology and what we—

Wolf: The projection is the false part here.

Swift collapse makes reform possible (1:01:24)

Marko: Yeah. And so it’s kind of like one of the ways out of that is to have kind of discussions like we’re having, just to realize that you’re doing this psychological projection. Okay. One of the ways out of that mindset, perhaps, to begin with, is just to realize that these systems are not actually permanent. They will 100% fall apart sooner or later—because they’re not being maintained by any human being.

I mean, man, are you watching the news? Some of these systems are dramatically falling apart right in front of our very eyes. I mean—

Wolf: Yeah, there’s a lot of crazy stuff going on

Marko: what’s going on with the Strait of Hormuz and Iran and US military supremacy and the freedom of the seas and the oil, which is kind of the lifeblood of the current global economic system. Or China. What’s going on with China taking away all our industry, which is what actually underpins all of this entire system, right? America was the China of the world in the 1940s and in the 1950s when it sort of built these—superpower scale institutions, right? So there’s a way in which it feels completely, utterly inescapable and inevitable, and that’s also the message that the system’s always—sending, right?

Wolf: It wants you to think that because it makes you more reliable.

Marko: It wants you to think that because it makes you more reliable. Absolutely. Of course it would. But it’s just absolutely false, and I would even go further and it’s like, okay, well, sure, Marko, the Roman Empire also collapsed, but it took like 1,000 years. You had a long dark age.

It was very sad and pathetic and I’m not motivated by the idea that centuries down the line, all of my enemies will be dead and in the ground. Who cares? I’m going to be dead. I’m not going to be able to do anything about it, right? That’s not very compelling.

And what I would say to that is, it’s like fair enough, right? We don’t just want to be sitting here and saying these kind of trivial things, right? These trivially true things. But what I would say is these systems could very well collapse a lot faster than the Roman Empire did.

Wolf: They could be reformed very quickly.

Marko: Well, I personally am maybe a little—the more I study them, the less convinced I am that they can be reformed.

The more—

Wolf: But maybe they can be knocked over quickly.

Marko: They will knock themselves over very quickly. And the reason for that is that in the same way that the more complex a material physical technology is, the more fragile it is. It takes smaller and smaller defects and problems to render the entire thing completely unusable because it’s so much more complex and you have so many more moving parts and so many more dependencies, and so many more things that can go wrong, right? This is also kind of just part of the basic fabric of the way reality works. It’s like the more capabilities you want, the more complex things kind of have to become.

Wolf: But the flip side of that is we’ve gotten really good at managing that complexity, and these systems are quite robust. We’ve got all these—

Marko: Well, the—

Wolf: feedback systems within—capitalism and—

Marko: Sure. True. No, this is true. And our institutions, our social systems, our social machines and processes are actually very similar. And their designers had much the same kind of problems and principles in mind when they designed them, which is why they haven’t collapsed already, right?

It’s why they don’t collapse as soon as the creator is gone, right? That’s why, right?

Wolf: Because they’re well-designed.

Marko: Because they’re well-designed, and they’re designed just to keep going, even when the creator’s—maybe that’s not so much by intentional design, but it seems like that’s kind of how it tends to shake out.

Wolf: Well, if you look at how people build these things up, it’s like you have the thing running, and it’s running, it’s doing a job, it’s kind of stably going along, and then they add pieces to it. They change it, they change its direction, but it’s fundamentally got a lot of inertia. And then when they’re gone, it just keeps its inertia.

Marko: What I would say, I think what the factor behind that is that the same design principles that make it difficult for your contemporary enemies and competitors to take control of your institution while you’re running it are exactly the same as the ones that make it impossible to reform after you’re gone. Right? Because—

Wolf: You’ve encrypted the keys.

Marko: You’ve encrypted the keys because someone who wants to reform your institution 10, 50, whatever, 100 years after you’re gone is functionally exactly the same as some guy who is trying to take it away from you while you’re running it, and you definitely don’t want that to happen. You want to maintain control of it. And it’s hard to maintain control of it because people are very competitive—and seek power, right? And what’s a juicier target than a big, beautiful institution that someone else has built, right? And all you have to do is displace one guy and it could all be yours.

It could all be yours. We just have to nationalize SpaceX, Wolf. We just have to expropriate all these companies and take the money, right? That will have no negative consequences whatsoever, Wolf, right? So anyway, really if you want to blame someone, you have to blame the live players who fail the succession problem out of ignorance or just kind of out of circumstance.

It’s a very difficult long-term—

Wolf: But it seems, it’s very inevitable, right? It’s like at some point they’re going to fail succession. And then you’ve got these decaying institutions. And this is—

Marko: At some point it’s inevitable

Wolf: just the nature of the world.

Marko: This is the nature of the world. It’s inevitable.

Wolf: So, okay. We live in a world that’s kind of beyond our power.

Marko: Can I just finish an earlier point?

Wolf: Go for it.

Marko: Just a thread that I forgot to wrap up. So—I don’t think that our institutions and our technology... Let’s focus on the institutions. The institutions rely on the technology. And both are equally complex.

And they have all these fail-safes and whatever. Yes, all that’s true. But, when they do break down, they break down completely and catastrophically. Once all the fail-safes have failed, the thing just breaks. There’s nothing to be salvaged, really.

It’s just a broken machine. Or, it can be salvaged.

Wolf: Maybe parts of it keep going in some form, or forms keep going.

Marko: Actually, no. I disagree.

Wolf: You don’t think so?

Marko: No, I don’t think so.

Wolf: Because in the Roman Empire, the late Roman Empire—

Marko: Well, hold on.

Wolf: Tell me why it’s different.

Marko: I’ll tell you why it’s different.

Wolf: But the Senate kept going for hundreds of years after—Rome was irrelevant.

Marko: So what I would say is, okay, so ancient Rome. This is basically, let’s say, an agricultural society rather than an industrial one. Maybe they were getting close to ind—we don’t have to get into that right now. But for the sake of argument, let’s say it’s an agricultural society, and they’re at that classical, agricultural level of technology, which is not that different from medieval technology. We’re talking about swords and sandals.

They’re at swords and sandals. That’s what they have. Swords and sandals. And basically, I would make the argument that when you’re trying to control the behavior of lots of people using swords and sandals, you have to use even simpler and more stupid and more durable processes, that are less capable on a short timeframe, but maybe because of that, for the same reasons they’re kind of—

Wolf: They’re more robust

Marko: they’re more robust. They’re more durable—

Wolf: So your—

Marko: over the long term

Wolf: hypothesis is that in an industrial society, the systems—

Marko: Industrial sys—

Wolf: of control are actually more fragile.

Marko: Yes. They’re more—yes

Wolf: they may change faster.

Marko: They are more fragile, and so they can change faster, and you can make a very similar analogy to physical material technology in the sense that it’s a lot harder to keep the knowledge of extreme ultraviolet photolithography machines going. Because you can’t just keep the blueprints in the attic. The machine has to be constantly producing things. You have to have people working on the machine, gaining that knowledge, transmitting that knowledge—et cetera. That’s inherently a much more difficult, expensive, and complicated process than to maintain the technology of casting iron swords.

Wolf: Yeah. Or getting guys to meet and to call themselves the Senate.

Marko: Exactly. That’s a very simple one in some ways. It’s a very simple one. And so in a way, it’s much more durable. Even when everything around you is breaking down, you’re still able to keep that going.

So, what I would then say is that because we have these institutions of power made up of people that rely on complex material technology, when that complex material technology starts to fail or the capabilities that it provides are no longer quite supporting the—they’re sort of, what’s the word? They’re maybe shearing away slowly.

It’s starting to rust and the metal’s bending off and the—screw is coming loose and, oops, now it’s leaking and just as that can collapse very quickly because it’s fragile, the institutions that rely on it will also collapse, because once they don’t have that, they’re very tightly interlinked. It doesn’t—

Wolf: But this is all to say that we should perhaps feel more agency about reforming these institutions or replacing them—

Marko: We should

Wolf: than might be otherwise apparent.

Marko: Yes. We should feel agency about the fact that when these things collapse, and if they collapse very quickly, that we can rebuild them if we’ve taken the lessons from their dysfunction of however many years that we’ve been observing them. If we learn the right lessons, then when they collapse, we can quickly rebuild them and rebuild the technology.

Wolf: Well, ideally, we almost rebuild them first. We build—new things that—come in and—

Marko: I think that’s—

Wolf: replace their functions

Marko: I’m not going to say that’s impossible, but the difficulty level is extremely high for all of the reasons that we’ve been discussing. I’m not going to say it’s impossible, but I think the difficulty level is extremely high, and it’s not guaranteed by any means.

Wolf: So there’s this certain sort of waiting process involved that—

Marko: I think

Wolf: where it’s like you can’t just execute really fast to—

Marko: Well, you could—

Wolf: change the world

Marko: if your power level was high enough.

Wolf: If you had... Yeah.

Marko: If your power level was high enough.

Wolf: Maybe if you have AI.

Marko: Maybe if you have... Maybe. I mean, it’s certainly possible.

Wolf: Maybe that’s the way to space—

Marko: It’s certainly possible

Wolf: is to use AI to reform our institutions—

Marko: Maybe

Wolf: so that we can go to space

Marko: maybe it’s possible. Yeah. No, I’m not going to say that’s not possible. It could be possible. I don’t know.

If I knew that it was possible, I’d be doing it myself. And, I would be the one telling you to build rockets, because I would be in charge of the world. But I don’t see the path to that, so—

Wolf: Well, speaking of—

Marko: I’m just analyzing the situation

The origins of spaceism (1:13:23)

Wolf: So this has been fascinating, but I do want to touch a little bit before we wrap up, also the concept of the ideology of spaceism—which is kind of implied in this. That we’re proposing a worldview, maybe, or an orientation that we should actually just build rockets and go to space and explore the galaxy.

So let’s talk a little bit about the history of that, because in the late 19th century, you had people starting to flirt with these ideas, taking the materialist kind of picture of the world and saying, “Wait a minute, that means that you can get up there.” And like when they discovered the rocket equation, it implies you can use a rocket to travel through space to other worlds, right? And someone wrote a book on that. Who was it who wrote that book? Do you remember?

Marko: Uh...

Wolf: That literally like space travel via the rocket to other worlds.

Marko: The German guy. It would be—

Wolf: It might’ve been a German guy

Marko: I mean, there were many people kind of coming—

Wolf: Was it Oberth? No, it wasn’t Oberth.

Marko: There’s Hermann Oberth, but I mean, Tsiolkovsky was like I think kind of—

Wolf: It might’ve been Tsiolkovsky

Marko: thinking along, he was so, I mean, for just like—

Wolf: But point being—

Marko: For whatever, yeah

Wolf: point being it was someone earlier these guys started reading this book. Von Braun read the book—and the whole—

Marko: That was Oberth. That was Oberth

Wolf: Oberth read—

Marko: Von Braun read Oberth, yeah. And Oberth probably read Tsiolkovsky. I mean, it was in the water. It was in the water, right?

Wolf: Well, it was this tradition.

Marko: Different tradition.

Wolf: Tradition being developed.

Marko: Yeah. Different human beings came up with the same revelations from nature.

Wolf: And people got super hyped. I think von Braun was also reading about Goddard’s experiments. And—

Marko: They were all in touch

Wolf: the German rocket cult got going—in the 1920s.

Marko: And the Soviet rocket cult. Let’s not forget the Soviet rocket cult.

Wolf: Was there an independent Soviet rocket cult?

Marko: Oh, of course. Of course. What, you think the Russians weren’t in on this? They were—

Wolf: Well, I know they captured a lot of German scientists after the war, but—

Marko: No. No, in the ‘20s and ‘30s. No, it was very international. You had rocketeers, there’s this quote from von Braun where he said that all of the people involved in kind of the rocket community, if you will—

Wolf: The rocket community

Marko: the rocket community in the 1920s, he was basically like, “It wasn’t really the rocket community. It was the space community.” They all wanted to visit other planets.

Wolf: Yeah, the whole point was—

Marko: The whole point was to visit other planets

Wolf: to use rockets to visit other worlds. As Oberth wanted.

Marko: Yeah. They weren’t so especially interested in making missiles or making rocket-powered cars, right?

Wolf: Yeah. Well, von Braun always had those comments that the rockets are landing on the wrong planet, right?

Marko: There you go. I couldn’t have said it better myself. Exactly.

Anyway. But no, it was very much an international, I mean—

Wolf: So why, now at the time, they hadn’t quite proven it, they hadn’t made this big, massive accomplishment of landing on the moon. They were starting to be able to bomb London with these things, but—

Marko: That was a little later. I’m talking about the ‘20s

Wolf: no, but—

Marko: It was a very vibrant scene in the ‘20s and the ‘30s

Wolf: they developed and von Braun got enough of an industrial concern under him that he was able to build some rockets. Some real rockets. That flew, and went to space. And landed on London, unfortunately.

Marko: Actually, I think, interesting fact, the first human object to reach outer space was launched by Nazi Germany during World War II. By Von Braun.

Wolf: Of course it was von Braun.

Marko: That was the first. Yeah. People don’t like to bring that up, but you know. Interesting historical fact, actually. Yeah

Wolf: but at the time, everyone was fighting. They had other ideas about what was important, as humanity versus the Germans or the Ubermensch or whatever, these concepts, communism, the arc of history. But they didn’t yet have the concept of space, and space was sort of still subordinate to these things. And then after that, we’re in this lockdown, Eleanor Roosevelt, like 1948 UN Declaration of Human Rights world, where there’s no longer this open dynamism of an impending war, but you have things kind of locked down. But nonetheless, the space cult continues to grow—continues to be in this subordinate position.

They do get their man on the moon, effectively like post-war science fiction—is about this. You know? It’s about the rocket. Starting with guys like Clarke, Arthur C. Clarke inventing the geosynchronous orbit satellite. They have all these ideas of expanding through space using rockets.

Marko: Yeah, you have this like, not renaissance, but like this blooming of ideas—

Wolf: Yeah, this blooming. But of course—

Marko: and accomplishments, actual accomplishments

Wolf: by the 1970s, this tendency unfortunately is mostly, defeated god of live players

Marko: it was direct, it was completely defeated in one fell swoop, actually. Because, the two, okay, I mean, so yeah, I mean, the two guys who were leading the space race were Wernher von Braun at NASA—

Wolf: And Korolev

Marko: and Sergei Korolev in the Soviet Union. And Wernher von Braun and Sergei Korolev, Korolev, whatever, originally, they had been in that scene literally since the 1920s. They were responsible for some of the earliest accomplishments, right? So it was not like a second generation even, it was still the very first—

Wolf: It was just those guys

Marko: it was just those guys. It was the first founding generation of rocket engineers who read the original crazy books by Tsiolkovsky, who had like, I mean, it’s kind of forgotten now, but you had this communist atheist strain of spaceism, which is what Tsiolkovsky believed in.

Wolf: Yeah, cosmism.

Marko: Cosmism. Oh, but I think he went even further. He was thinking, he was interested in the possibility of a life without gravity because gravity is an oppressive force that prevents us from—biologically or mechanically expanding in any direction that we want, and—

Wolf: Yeah. They had some truly wonderful ideas about what new Soviet man would be like.

Marko: The new Soviet man. But I mean, who knows? Who knows, man?

It was a very intellectually and kind of aesthetically productive scene. Anyway, so those were the two guys, and they basically built each some of the largest rockets that had ever flown. And I think Korolev just died of medical problems in 1966. And his successor kind of was not quite Speaking of succession problems and not knowing how to run the institutions that were built, he didn’t—

Wolf: But they did build some impressive rockets after that, but never—

Marko: They didn’t work

Wolf: never quite as good.

Marko: They didn’t work. The biggest ones didn’t work, right? I think it was the Energia—

Wolf: Yeah. They never got the N1.

Marko: The Energia rocket, I think it was called, never worked.

Wolf: Energia, that was the one that they had their space shuttle, the Buran, right?

Marko: I think I—

Wolf: That was supposed to fly on the Energia.

Marko: Yeah. I’m sure we’ll be cor—

Wolf: I think they flew that once. I think they flew that once

Marko: we’ll be corrected in the comments, but their answer to the rockets that took them into the moon was supposed to be Energia, and it never really worked, and basically, that was 1966, so they were kind of stalling then for a few years. Von Braun beat them to the moon, and then von Braun, the story is that he wanted to go to Mars, and he had a plan to put astronauts on Mars by 1983. He had a schedule. It was all planned out.

Here’s the plan. We just did the moon.

Wolf: It’s time for Mars.

Marko: Time for Mars.

Wolf: We’re going to build a space shuttle, we’re going to build a low Earth orbit station—

Marko: I mean—

Wolf: we’re going to go to Mars

Marko: nobody even died, Wolf. It went off without a hitch. Nobody even died. Like, shit

Wolf: nobody died in space.

Marko: In space. Sure.

Wolf: John Glenn and—

Marko: Point being—

Wolf: Apollo 1, right? They did die

Marko: it was such a great accomplishment, you would think that with such a great accomplishment, everybody would be itching to go to the next step, right? But the problem was, in von Braun’s own words, he said, even he could not actually explain why we should go to Mars to Congress. Even von Braun himself was not able to explain to anybody why we should spend all this money, and all this time, and all these resources—on a Mars program. What are we getting out of it?

Wolf: Yeah, and meanwhile—the acolytes of humanity are getting more and more aggressive with slogans like, “Pennies for the hungry, billions for space.”

Marko: Well, look, I think that’s just... I see that as just a kind of particular expression of a perennial impulse, and I prefer to view the problem not as like...

Every institution, by default, is dysfunctional. Every institution, by default, is dysfunctional because the people within it, playing their little role in the machine, are maybe not even aware of the overall purpose of the organization, and their most immediate concern is kind of kicking back and relaxing, and looting anything they can for their own personal enjoyment and pleasure, and kind of—

Wolf: Yeah, and that of their friends and so on

Marko: and that of their friends and whatever. But the point is, every institution then has to have very strong safeguards against people going off mission, if you will, right? And even at the highest level then, for anything specific in particular to get done, there has to be some kind of compelling justification. And so it’s like the ambient laziness and entropy and lack of kind of agency and mission orientedness is like you’re arguing with entropy. It doesn’t need to be argued with or even really thought of.

Of course that’s what’s going to happen.

Wolf: It just is.

Marko: It just is. What else would it be? What else would it be? It’s the particular mission—

Wolf: So point being, we lose—

Marko: and the particular structure that’s unique

Wolf: we lose the pre-war generation—

Marko: That needs to be explained. Yeah

Wolf: they did not—

Marko: Von Braun retired in 1972. So Korolev dies in ‘66, and then von Braun retires 1972. And that’s it.

Wolf: Nixon guts the program, and that’s that.

Marko: And that’s that. And nothing happens until Elon Musk comes along in the 2000s and the 2010s and starts building reusable rockets. And it’s still Elon Musk. So basically, you have this one generation of these guys—and then a whole many decades of nothing.

Wolf: Yeah, or approximately nothing.

Marko: Approximately. I mean, nothing. It’s nothing.

Wolf: The space shuttle, but yeah.

Marko: That’s nothing. That’s stagnation and the decline rather than... You have to think, Wolf, the sort of folk theories of progress are that progress only goes in one direction. And yet we’re sitting here, and we know that we put men on the moon, and then we just gave up on that for 50 years.

What? That’s crazy. That doesn’t make any sense. Until Elon Musk came along and decided to push it further.

Wolf: Yeah. I remember—Doesn’t the story literally go, he’s asking, “Well, when do we get to Mars? What’s the plan?”

Marko: Yeah. What’s the plan? There was no plan.

Wolf: And then there was no plan.

Marko: It was not in the plan. It was not in the plan because getting to Mars is extremely hard and difficult, and so on and so forth. And just like, whatever, building satellites or space shuttles is marginally easier for everybody in the organization who doesn’t have the charisma, and the power, and the foresight to pull everyone together and argue through the US government, which is a big part of the job of being NASA administrator or the head of the space program. There’s nobody capable of doing that. Even von Braun was not capable of doing that, honestly.

Wolf: Yeah. And the way Elon’s overcome this is he’s argued to various people—to fund it—

Marko: He found a new strategy

Wolf: fund it on the basis of the intermediate value that you can create along the way, right? We’re going to build military value. We’re going to be able to put things in orbit for cheaper. We’re going to be able to have this nice space internet—space data centers, all this stuff that’s being offered. But people talk about asteroid mining.

No one’s really proposed that seriously as an economic proposition yet. But there’s these intermediates along the way, but fundamentally, Elon’s not motivated by this. He’s motivated—

Marko: He’s not motivated by this, and I would even say—

Wolf: He’s motivated by—

Marko: Elon Musk—

Wolf: spreading consciousness throughout the universe.

Industrialism requires a higher mission (1:25:50)

Marko: Yes, which he’s said many times, and I would even go further and say Elon Musk is in many ways the ideal human stem cell for industry. Because Elon Musk understands that you don’t start off with the business plan. You start off developing the technology, and when you discover new business plans that will make you profit and allow the technology to keep developing within the confines of our social institutions, including the price system, and the free market, and all this stuff, then you just seize them. I don’t think the original plan for SpaceX was Starlink. I think—

Wolf: No. That was something he came up with along the way

Marko: that was something he came up with along the way, and he said, “Oh, shit. Perfect. Let’s do that, too.” Right? And that’s kind of—

Wolf: And it’s not just the technology.

Marko: Or look at Tesla. Or look at Tesla, right?

Wolf: He started with the need for the technology—that came from the transcendent purpose—of let’s go to space. And I find that’s an important point that Elon Musk, the ideal stem cell of industry, this guy who is recognizably the world’s best industrialist—he is the world’s best industrialist because he believes—

Marko: That’s right

Wolf: in something to do with it.

Marko: That’s right. Exactly.

Wolf: And that’s an important fact about this whole thing is part of your whole argument is that even if all you cared about was keeping industry going, keeping capitalism going, building these toys, you need to have people who believe in what to do with it. You need to have an end to it all.

Marko: Well, so I completely agree with that. And the interesting question is why those two things are related in that way. And what I would say is that, the precise causes and whatever is a very difficult historical question. But for better or worse, we have kind of not had this desire as a society or a civilization to continue expanding—to continue expanding the sum total of the capabilities, made up of these industrial technological processes that we are capable of building. We kind of sometime in the ‘60s or the ‘70s, and there were, I don’t know, I guess there were elements of that even earlier in World War I and World War II because there’s fears about the impact of technology on the environment or the capability of mankind to destroy himself—with the more powerful weapons that he’s building.

So like, okay, whatever. That’s a thing that goes back decades, maybe centuries. But for whatever happened around the ‘60s or the ‘70s, we kind of set a limit on maybe everything except for computers. Which is like kind of Peter Thiel’s thing, like everything—but computers kind of stopped advancing, right?

Wolf: Yeah. This comes back to like now we project all our ambitions into computers because—it’s the only open frontier, right?

Marko: Yeah. And to be clear, clearly there are more ways that we can scale even the technology that we do have, like nuclear power or even rockets earlier. Now Elon is scaling it, but before Elon, we could’ve been doing more with that, too, right? We had the sort of physical capability—in the same way that we scaled semiconductors. But it appears like we just chose not to, maybe because of fears of the military consequences or maybe because of deeper whatever.

Wolf: Or it’s simply that we lacked a live player who had a coherent vision.

Marko: Absolutely. And it’s very poetic in a sense. It’s like, wow, the Industrial Revolution, we just had this incredible spurt of technological and industrial growth that looked like it was completely unstoppable until someone conquered the world, and then it just stopped. And doesn’t that make sense in a way? Doesn’t that kind of make sense in a way?

Because up until then, that was the ambition of every Caesar and Alexander the Great and Napoleon.

Wolf: Right, to conquer the world.

Marko: The highest possible ambition was not to conquer the cosmos. Or maybe the ambition is the same. It’s to conquer the known world, right? And it’s just that—

Wolf: We just lived in this medieval dome

Marko: we just lived in this medieval dome, and we didn’t even realize it, and we haven’t updated our priors, if you will, that the world is so much bigger than we realize.

Wolf: Yeah, so coming back to this historical question, like we’re now maybe reaching the end of this sort of post-war institutional order, where as we say, these things are starting to fray around the edges and perhaps collapse.

Marko: They’re fraying.

Wolf: This is—

Marko: There’s still some time for collapse, I think.

Wolf: There’s still some time.

Marko: There’s still some time. But don’t get too excited.

Wolf: Don’t get too excited.

Marko: Don’t get too excited. Take it easy. There’s like—

Wolf: Nothing ever happens, et cetera.

Marko: Yeah. Take it easy.

Wolf: But nonetheless—

Marko: Yeah, take it chill, pal. You can relax for now, Elon

Wolf: nonetheless, it looks like we’re reaching some kind of moment of reform, I would say. I think there’s a possibility. And now would be the time historically for that, I call it the rocket cult, you can call it spaceism—like for that—

Marko: Now—

Wolf: For that now to take the—

Marko: Now we need the next thing to begin to coalesce and to—

Wolf: For that thing now to take the primary seat—

Marko: develop and yeah

Wolf: right? Where it’s always taken the secondary seat, whether under Germany or under America. It may now take the primary seat, and that’s kind of what you’re proposing, is that actually—the rocket cult or spaceism or this idea of expanding into space, conquering space, this is actually a well-formed holistic vision for what to do with society—overall.

Man as liability (1:31:18)

Marko: Yeah. And so what happened was that when we decided to stop expanding our capabilities because we are these kind of human stem cells that have to be constantly working and growing our capabilities by automating processes. Once we decide to stop doing that, what do we do instead? Right? There’s nothing to do then.

Wolf: Right. You just stagnate.

Marko: Then you have this idle labor, right?

Wolf: Save the owls.

Marko: Idle... What’s that?

Wolf: Save the owls.

Marko: Save the owls. To be honest, there’s so many different things we’ve tried to do with our time instead, and it’s all been kind of lame and boring. And everyone seems to get tired of it, and that could all be subjective, but I think the damning evidence is that fertility rates have completely cratered in every developed country—because for a very long time now—

Wolf: We’ve nerfed that creative impulse

Marko: we’ve nerfed that creative impulse. We’ve decided—

Wolf: And we have no need for people. Like what?

Marko: We have no need for people. We have no need for people.

Wolf: Because we’re not doing anything.

Marko: We think that we’ve built this wonderful technological system. We’ve built this wonderful industrial technological system, and we love the fruits of it. And so what we’re going to do is we’re going to freeze it in place and get rid of all the people who compose it and replace them with machines. And then we’re going to continue enjoying the fruits, but somehow, hopefully, without the bad parts of it, without the risk of war or destruction or—but I would say it even goes deeper, without the hard work that goes in maintaining it. But obviously, this is all completely wrong.

Wolf: Yeah, all we have to do is—

Marko: This is completely wrong

Wolf: move some numbers on a spreadsheet, pour some—finances into AI and like—

Marko: Drink a coffee, push a button, and it’ll run itself.

Wolf: Claude will do it.

Marko: Wolf, it’s not AI that has been enabling our affluent lives. It’s the Chinese . We’ve lost all our industry, and it’s not because we’ve automated—

Wolf: This is not an accident. They know this

Marko: They know this, which is why they’ve been doing it, because they want to have those capabilities. Right? This is the story of what’s been happening for decades, is we’ve been kind of doing, not this vertical, absolute increase in human capabilities, but more like this horizontal transfer of the capabilities from what was the developed world, the Western world, United States foremost, to China primarily. It’s not quite the West versus China. I think that’s kind of fake.

It’s not really the US versus China, because the US has, in other ways, even allowed Europe, for example, or Japan or South Korea to take over these industries that previously used to be—that you previously had an American version of, right? Really what you’re seeing is this attempt to freeze human development in amber while preserving the fruits of it, and it doesn’t work because people need to be working to maintain it. And—

Wolf: Well, if I can put it shortly, the post-war humanism imagines the human being as an end, right? As something to be fed, something for resources to be poured into. Whereas what you’re proposing is much more the human being as a means. The human being is this essential component of the industrial order—that exists for the purpose of—

Marko: For the purpose of this great—rearrangement of matter and energy throughout—space and time according to the principles of nature as we’ve discovered them.

Wolf: And if we’re not doing that, if we’re not doing that—

Marko: If we’re not doing that—

Wolf: we have no need for the person.

Marko: We have no need for the person.

Wolf: The person is just a liability. The other word for—

Marko: Yes, the person’s just a liability

Wolf: an end. The other word for an end in itself is a liability, right?

Marko: That’s true. Yeah.

Wolf: Because it’s something you owe to.

Marko: So we’ve turned ourselves into liabilities. And what do you do with liabilities? You get rid of them.

Wolf: You remove them from the balance sheet.

Marko: You remove them from the balance sheet, which is exactly what we’ve been doing with incredible efficiency. Again, like—

Wolf: Well, we could’ve done better, but—we don’t want to think too much about that

Marko: South Korea is doing even better than we are. China might do it better than them even. We haven’t even seen yet how low the fertility rates can get and what other crazy social phenomena can happen. But it is totally the natural thing that occurs because we don’t need the people. The people realize they’re not needed, in a way.

And so they kind of just pass the time. They turn inwards. Basically everything... I have this idea that, I sort of thought, well, according to this logic, basically unless you’re... And I’m going to simplify it a little bit, but it’s like unless you’re working in China at these factories or you’re like Elon Musk, everybody’s just coping with whatever it is that they’re doing.

They’re just coping. They’re just passing the time. That’s not totally true. There’s a lot of people who play really useful roles and still—

Wolf: Yeah. But there’s—

Marko: in this system, in society

Wolf: a lot of just sitting around, enjoying ourselves.

Marko: There’s a lot of sitting around and enjoying ourselves, fake jobs, kind of, all this stuff that will never result in any kind of ultimate increase in human capabilities, right? And there’s a lot of ways that we can hide it. We’re in a post-scarcity society already, right? That was one of the things I’ve argued, that post-scarcity—is this fake idea because you can—

Wolf: We already achieved it.

Marko: Well, what’s the metric by which you define scarcity, right? Is it like—

Wolf: Yeah, it’s like people starving to death, people—you know?

Marko: Yeah. Like—

Wolf: Not having houses.

Marko: Yeah, not having houses, basic material scarcity. We solved that more than 100 years ago. I guess not everybody can have their own personal skyscraper or their own personal yacht. Ironically, if that’s how you see post-scarcity, you would still need to go to space because we’re going to run out of matter and energy—for all the stuff that people are going to want, right? But it turns out people don’t even want that much stuff, right?

People don’t even want that much stuff. We can give people all the stuff that they want as consumers, and it’s not enough stuff to motivate the impersonal forces of capitalism to—

Wolf: Well, it’s again—

Marko: continue developing—

Wolf: man as a liability instead of an asset, right? It’s like, what’s the point, right? Why do we need to expand?

Marko: There is no point.

Wolf: Why do we need to expand—

Marko: There is no point

Wolf: the standard of living? There’s no point.

Marko: Yeah. There is no point. There’s no point. And so, in all of these ways, we are doing nothing. Nothing is happening except us experiencing the completely logical and predictable consequences of what we’ve been trying to do with global civilization, basically.

There is no such thing as stagnation (1:38:23)

Marko: And okay, if you want to make the human being an asset again—the human being is going to be working. And if the human being is working, think about this—

Wolf: We need something to do with all that work

Marko: We need something to do with all that work, because again, the work cannot be fake. Okay? It cannot be fake work. Fake work doesn’t work, okay? People can tell.

Wolf: Yeah. We need something—

Marko: People can tell. They know it. And if you give them fake work, they kind of go crazy even—and then they burn out. Okay? They burn out with real work, let alone with fake work.

And it’s kind of funny because in this way, as a society, we kind of want to have the fruits of a dynamic society full of young people changing the world. But we don’t want them to actually do it because it means things have to change, and we don’t want them to change because that’s uncomfortable and that interrupts the kind of sleepy vacation that humanity’s on.

Wolf: It interrupts the retirement home.

Marko: It interrupts the retirement home, right? Like, “Oh, those young people need to work.” But we’re going to make everything illegal to do. We’re going to ship all the industry to China. Actually, don’t do anything.

Do work, but don’t actually work, right?

Wolf: So you’ve got to get off this stagnation track, and you’re proposing spaceism, go to space, expand into space—

Marko: I’m saying, yeah

Wolf: as this obvious—

Marko: Yeah. I think it’s obvious

Wolf: kind of telos that is now in reach.

Marko: Well, you say telos, but I’m maybe not even saying that it’s some kind of telos or a goal. But I’m saying that because of what we are as human beings, I’m sort of proposing that the human condition and human nature is such that we kind of have to work, we want to work, we create tools out of things and out of each other, and these tools have to exist in the real—all the real work that we do has to have these consequences, ultimately, on everybody else, right? You can’t just, again, it’s this—

Wolf: You can’t just do all this work and have nothing change. So there has to be something we’re trying to accomplish with all this. What are we doing with it all?

Marko: It has to exist in the real world. You can’t just—

Wolf: Expand it.

Marko: Yeah. You can’t just have everybody sit in a room and close their eyes and do mental work in their heads and have the same... That’s not how it works, right? It’s like no. The mental work is not real unless it interfaces with the physical world.

If it interfaces with the physical world, it’s going to have externalities. It’s going to change things. It’s going to require resources that maybe somebody else wanted. You get a human civilization. You get a society with conflict and dynamism and new discoveries and surprises, etcetera.

All the things that we want out of life, right? So I’m sort of making this anthropological claim. And if you say, “Great, we’re not going to interrupt this process,” you inevitably have to expand off of planet Earth, because planet Earth is limited. And this process—

Wolf: I see. So space is almost this just necessity to the process of—not killing ourselves.

Marko: Yes. If we keep—

Wolf: If we—if we choose life—we also choose space.

Marko: Exactly. There’s no choosing stagnation, right? It’s sort of like, well, Marko, why can’t we just... Okay, I’ve heard all this logic, I’ve heard all these arguments. Why can’t we just choose to maintain industrial society at exactly the kind of the level that we have right now?

We’re not going to go further.

Wolf: Sort of the eternal 1999. Yeah.

Marko: Yeah, eternal 1999. 1999 was perfect. Why don’t we just keep it at eternal 1999, and basically, we’re not going to go further, but we’re also not going to go behind. We’re just going to pay people to work enough to maintain the system forever. And okay, maybe on some level that sounds reasonable, but the problem is you’re going to have to get rid of all the human capital, and people are going to ask themselves—

Wolf: Well, it’s also just not possible. There is no such thing as stagnation, right? There’s only decline or growth.

Marko: Well, yeah. I agree. But I’m trying to be the devil’s advocate here. Why can’t you just have enough growth to maintain the thing in stasis, right? Why isn’t that just the right choice?

Why isn’t it the right choice just to maintain the thing in stasis kind of indefinitely, right? And it’s sort of like you—

Wolf: Well, I guess the way I would say it is all of those components presume some purpose. And if you’ve—if you’ve stagnated it and saying, “We’re not going to accomplish that purpose very quickly or at all,” then you’ve actually destroyed the logic of the system.

Marko: You’ve destroyed the logic of the system, and actually what you’re saying is that the purpose of the system is no longer to continue ascending up the ladder of industrial technology and capabilities. You’re going to have to get rid of a lot of the human capital and the human labor that got you there, which we’ve been doing. And that you’re actually saying the new purpose of the system is kind of, we’re going to—

Wolf: It’s just to lock things down.

Marko: Well, it’s to lock things down, but maybe that would even work. But it’s actually worse than that in a way. You’re sort of saying the purpose of the system is just to gratify everybody’s pleasures. It’s to give everybody the wonderful gratification of living in 1999 forever. Okay?

And we’re going to spread that across every generation until the end of time. Okay? So, for the next billion years, we’re going to keep reproducing at 2.1. Earth is going to have people, and they’re all just going to be endlessly living in 1999.

Wolf: The end of history.

Marko: And they will all get their little share of the joy and pleasure of living in 1999. And it’s like, okay, well, if that’s the goal, why don’t we just enjoy the pleasure of living in 1999 to its fullest right now? And we won’t reproduce the system. We’ll kind of just loot it for what it’s worth right now. And what’s the difference between one generation that gets to live this enjoyable, pleasurable life versus a million of them, right?

Is that utilitarianism maybe? Is that kind of how utilitarians see it?

Wolf: Well, I think they—

Marko: It’s kind of like—

Wolf: they want it to keep going, but what you’re saying is psychologically—

Marko: I’m saying psychologically, it’s kind of incoherent.

Wolf: Yeah. It doesn’t work.

Marko: It’s kind of incoherent psychologically. And again, I’m sort of making this anthropological claim. When a person who’s working successfully automates one process, and it requires less human labor, through whatever form that takes, they don’t want to stop.

Wolf: Yeah. They want to do it again. They want to do the next thing

Marko: they want to do it again. They want to do it again, and again and again and again, right? And if you forcibly stop them, maybe you just break something about that person.

Wolf: Yeah. This is what I mean.

Marko: Maybe—

Wolf: Stagnation is not actually possible. You’re either going up or down. And therefore, let’s go up.

Marko: Let’s go up, because the other choice is dying. And if you’re going to die, then you can go do that right now and nobody will stop you. It’s the rest of us who want to live that need to have these discussions. So

Wolf: Yeah. Well, this has been very long, but very rich and productive.

Marko: Excellent.

Wolf: So thank you very much, Marko.

Marko: Thank you. Thank you for having me. I’m looking forward to watching this myself, and I hope it wasn’t too heady and abstract, and I hope we were both clear about these big, crazy, big picture concepts, and yeah. It was a pleasure.

Wolf: All right.

Marko: So onward to Mars. Onward to Mars.

Wolf: Onward to Mars. Up we go.

Marko: Up we go. Up we go.

The internet can feel like a part of the landscape, but all our software was intelligently designed—and so were the social systems which produce that software. Christine Peterson of the Foresight Institute tells Ben Landau-Taylor about the struggle to define “open source software”.

Subscribe now

Transcript

Intro (00:00)

Ben: For people who grew up with the internet, it’s easy to think of it as a natural feature of the world, like a mountain or something. But in fact, the internet was built and designed by people who were making choices about the technology and the culture to try to get a particular shape of society. And I’m here with Christine Peterson, who your most sort of famous and legible role in this was inventing the term open source, and I understand that you are also very active in putting together a lot of the social scene and curating many of the spaces where this sort of collaboration and discussion behind a lot of this technological and cultural and legal design could take place.

Christine: Yes. Through my nonprofit, Foresight Institute, we’ve been able to interact with a lot of the movers and shakers here in Silicon Valley for decades. In fact, this is our 40th anniversary year, so we’ve been pretty close to the center of the culture for that long.

The hackers who built the internet (01:11)

Ben: Yeah. On a lot of these different technologies and cultural issues, you guys have been in it. So, for a lot of our younger viewers, can you just tell us what was going on in this sort of, I think, late ‘80s throughout the ‘90s period where all of this was happening? What was sort of the background cultural context in which this work had to be done?

Christine: So, this was a time where there were many fewer software engineers. Many of them came originally out of math. Often, in the early days, there wasn’t a computer science degree you could get. So, these are people who, they were very technical, of course, very quantitative, but they didn’t have a computer science degree, per se. They came out of other fields, picked this up on the fly, which means they’re super bright.

Ben: It’s a little bit like how with the artificial intelligence technology we’re seeing now, there just haven’t been very many preexisting experts as it’s ramped up.

Christine: Exactly. It’s a time of great opportunity if you grab it, right? If you can jump into a field when it’s new, and you can still do that right now with AI, for sure, you can ride that wave, and there’s no one to tell you you can’t be very quickly an expert in the field. And that’s how it was back then for the early days of the internet, and the early days of the personal computer in general. So, it was an exciting time.

These are super bright people, highly technical, of course. Many of them very libertarian-oriented, for whatever reason.

Ben: Yeah. A lot of these anarchist manifestos floating around in the space.

Christine: And a lot of hopes for increased freedom, increased communication around the world, which certainly we’ve gotten, not always leading to the wonderful beneficial results we had hoped for. But today, it’s unimaginable to think that we wouldn’t have these abilities, right? It’s hard to imagine a world without them. So, it was wonderful fun to see them come in, to play a role, however small, in shaping that, to see the explosion. It was similar to the AI explosion, from no personal computers, to personal computers, to internet, and then to watch the web expand.

Ben: Yeah. And I understand that a lot of the sort of freedom and ability to just be connecting to anyone, a lot of that was deliberately built in at the technology level as this was getting built out, right?

Christine: There were some fundamental architectural decisions that were made by folks designing the internet to make it very decentralized. Now, there have been at least two explanations for this. One is, and this is probably what they told the people who were paying them, which is this will make it hard for a nuclear war to interrupt the internet, and that is true. However, it also makes it pretty challenging to censor the internet. But perhaps they didn’t mention that to the folks who were paying them to design it.

Ben: Yeah. I remember hearing the term, “The internet interprets censorship as damage and routes around it.”

Christine: That is the classic phrase, and it’s true. And it’s been a wonderful thing that the internet is so hard to censor. Now nation-states can do it, but other than that, it’s pretty darn hard.

Ben: Yeah. And so tell us about who was this group that was coming together around Foresight?

Christine: So, at this point, it was early in the internet. The standards were very simple, very basic, and pretty flawed.

They didn’t meet the criteria that, for example, earlier hypertext projects such as Xanadu had hoped for. One of the features that was missing is called backlinks, where who has linked to my document? In these days, it was very hard to find that out. You couldn’t see who has linked to my document. So, we felt, well, we’ll put this in.

We will add this as a feature to the internet. Foresight Institute wanted to add this, and we had a group of programmers who were all brilliant people, and series of meetings, and the primary architect of our backlink software was Ka-Ping Yee, a brilliant programmer. And we wanted this software to be what they called at the time free software. That was the term and that’s how Foresight and I got involved in dealing with this movement called the free software movement, and tried to help it become what it is today, which is continue to be a movement, but also an industry.

Ben: Mm-hmm. And so the free software movement, what were they up to?

Christine: So, I think probably your viewers are aware that there’s two kinds of software. One is closed source, proprietary software, where you buy it in a box or over the internet, comes these days over the internet. Something like Microsoft Word. And if it has a bug or a flaw in it, there’s nothing you can do except complain to Microsoft. There’s no way to fix it.

You could hire the best programmer in the world, and he or she cannot fix this. There’s no way to do it. You cannot get at the source code, which is what you need to do the repairs. That’s one kind, and the other kind is what was called at the time, free software, where you can get the source code and you can fix any bugs that are in there. You can make the software better, you can redistribute it.

It’s a very nice form of software if you’re a programmer and have those abilities. So, these were the two kinds. So we felt we wanted ours to be the free kind so that people could improve it, distribute it, repair any bugs. And there were many other projects similar to ours where people felt strongly that they wanted it to have these abilities.

Ben: Yeah, it was—

Christine: Including the first browser. You probably know that, right?

Ben: This was—

Christine: The Netscape—browser. Before it was Netscape, it had come out of the academy. I think that was Marc Andreessen. Yep. And—

Ben: He’s certainly one of those people who got in early and rode the wave, as you were saying.

Christine: Absolutely, and also brilliant. So, he’d made this company and, if I recall properly, they were having trouble competing with Microsoft, so they had decided to make their browser into free software.

So this was part of the movement. So there was a general feeling of, okay, there’s this whole category of software, it’s a very important category. How do we get people to understand why it’s important? How do we get people to take advantage of it? How do we spread this model?

Even though the people involved were very pro-free enterprise, it’s not that this was an anti-corporate movement in any way. It’s just that, if you’re a software programmer, this ability to fix the code is super important to you.

Ben: Right. And so you guys got involved in it because you were trying to build this piece of software and, as the infrastructure and architecture of the internet was still being worked out, it was just like, “Oh, yes, we can just figure out an improvement to this architecture, push it, people will adopt it.” This was still happening all around at the time.

Christine: It was at the time, and of course, in many ways it still is happening. So that’s why Foresight, as an organization, was interacting with the free software folks and also the Netscape folks, because they were in the process of rolling out this change, which involved communications, marketing, education of the consumers. It was a whole push they were doing. So this question came up, if you call something free in English, it refers not just to freedom, but to price, right?

The “Free software” problem (09:53)

Ben: Right. People would be like, “Huh, it’s pretty weird that Andreessen got so rich off of free software. That doesn’t sound like that’s what the word should mean.”

Christine: Well, so the challenge was that there was a lot of wonderful free software available, but to the non-programmer, it’s impossible to use because the non-programmer, such as myself, we need the software installed. We need support. We need automated updates. We need a place to report bugs and get them fixed. And those come with corporate support.

If you want those kinds of services, you need a company to do them, which means there needs to be a way to pay for it. And—

Ben: Because there has to be someone who’s building all this and keeping it updated and as the other infrastructure it depends on changes or as bugs are discovered, to be like, “Oh, I’m going to drop all of my other plans and get on fixing this,” and—

Christine: Right. There’s just no way to provide this level of service that the non-programmer needs without some way to compensate them. So there were startup companies. Tiny little companies were beginning to provide those services. And so they would go out and try to promote this product and explain that it had to do with free software, and immediately the customer would say, “I don’t understand. Why are you charging me money for free software?”

And it just was a non-starter. So this problem would never arise, say, in Spanish, because they have two different words for these two meanings, free as in freedom and free as in price, gratis and libre. But in English, these are the same words. So these tiny little support companies would go out trying to sell their services using thinking they’re referring to free as in freedom, but all the customer can hear is free as in price. And this was just going nowhere.

You can see why, right?

Ben: Yeah. It’d be very easy to get tripped up.

Christine: No matter how many times you explain to the customer that, “No, no, we mean free as in freedom,” to the customer, to the non-programmer, there is no freedom. They don’t have the skills to change the software. So arguing that freedom is a benefit to this person is not something they will understand. All they’re going to hear is free as in price. And once they’ve anchored on that, you can’t get them to pay anything for it.

Ben: So you’ve got this movement of technical people who are getting deep into it. They want to control their computers. They have this ideological view of digital sovereignty and how they should control all of their stuff, and they should be able to understand and modify it. And then they’re using this to, in dialogue with each other, be building these things. And then the reason they’re building it is because there’s some customers out there who at the end get value from it.

And so they have this idea of how their sort of production area where they’re making all of this is going. And then over here, the customer’s like, “Look, I don’t care what’s going on in the place where it’s all being built. I just want to get your software, and I’ll pay money for software that helps me, but why are you telling me that I’m supposed to pay you for free stuff? That doesn’t make any sense.”

Christine: It was a marketing disaster, basically. It was a communication problem, a labeling problem. It was holding us back terribly. And I was not the only one noticing this. Others were noticing this as well.

And I recall a conversation with the folks at Netscape saying, “Hey, we have to fix this.” And they said, “You’re right. We have to fix this.” But then nothing happened. I hadn’t heard any proposals.

You can’t fix it until somebody says, “Well, here’s another idea.” You have to have an idea of another name or else it never changes, right? So it just seemed like we were stuck for some period of months on this. Right.

The genesis of “open source” (14:37)

Christine: So then what happens is what always happens, which is you’re in the shower or on a walk, and an idea pops into your head like, “Oh. Oh, that would work,” or in my case, it was like, “That’s good enough.” In other words, you don’t need the perfect term. All you need is a term that isn’t a disaster, which is what we had. So the term was, well, how about open source software?

That seems good enough. It’s clear. It’s about, okay, the source is available. It’s open. You can get it, and you can change it.

It’s not a perfect term, but we didn’t need a perfect term. We needed any other term.

Ben: Right. And one of the advantages is if you hear and you don’t know what it means, you’re like, “What’s that? I don’t know what that means.”

Christine: Right. At least it’s a different term.

It makes you ask, what does this mean?

Ben: Right. And then when you hear it, you’re like, “Oh, yeah, I see what you mean by that.”

Christine: To a technical person, a software person, it’s a pretty clear term. They would understand this term immediately, I think. To a non-software person, they might not even care, actually. They often don’t care. Yeah.

Ben: Yeah. And so then how did you, upon being like, “Okay, yeah, this is at least a better term”? I don’t imagine you’re just like, “Aha, I have it. I’m sending out the email,” then everyone just starts using it.

Christine: That’s definitely not how to do it, right. So, I was in kind of a tricky situation because this community, the free software movement, that community, was exclusively programmers, and I am not a programmer. And they have a very clear sort of status hierarchy within the community, and it makes sense, which is if you produce a lot of excellent code that millions of people use, you get a lot of status. And if you produce only a little bit of code that not many people use, you have lower status. And if you don’t produce code at all, like me, you have no status at all, okay?

And this—

Ben: Feels like a pretty sensible way for that to be set up.

Christine: Absolutely. It makes perfect sense, and I fully support this. So, I realized that, okay, I am not in a position to send out an email announcement saying, “Guess what? I’ve solved your problem.” Absolutely not.

So I thought, “All right. We have to do this carefully.” So first, I ran it by a few close advisors. Most of them said, “It’s fine. It’s good enough.”

My PR guy said, “No, open has been overused,” which was true, but I decided that the other people were enough to overrule that view, and we would go ahead.

So, I knew that I needed a front man, someone who had status within the community, at least more than I did, which was zero, to do this introduction. Fortunately, I had a lot of friends in the movement.

Ben: Which was not a coincidence. You’d been—

Christine: Not at all

Ben: and you guys still are at Foresight, just cultivating and being a center for where a lot of these people congregate and providing a lot of the infrastructure so that they can just be in dialogue and keep everything flowing.

Christine: Yeah. We were well-connected in the community, as we still are here in the Valley. So, it wasn’t too hard to come up with a strategy. In fact, one of the leaders of the movement, many people at the time would say he was the leader of the movement, was Eric Raymond, who was in town for meetings. To try to help this movement move forward.

The power of doing the legwork (18:20)

Christine: And he has vision problems, so he needs a chauffeur, so I was driving him around.

And because I was a chauffeur, I happened to be in the room with a group of the leaders of the movement. I was only there as a chauffeur really.

Ben: This is a very common pattern of this sort of thing. If someone is just willing to show up and start doing the legwork, it gets you into a lot of surprising places. One of the famous examples of this is Garry Tan, who’s the current CEO of Y Combinator, initially got in the door because he just sort of showed up with a camera and appointed himself their volunteer photographer, and then just kept getting his foot in the door and...

Christine: That’s a perfect example, and something for young people watching to keep in mind, which is just start helping out. People appreciate it, people notice it, and the next thing you know, they start throwing things at you, and then you’re getting paid out of nowhere. It happens.

Ben: Yeah. And it just sets you up to get lucky in this sort of way of like, “Oh, I just happened to be his chauffeur because I just do a bunch of these things to facilitate things when people are coming to town.”

How the name took off (19:44)

Christine: Exactly. You want to be in the room under any capacity. But I was sitting at the table with everyone else. I’ll point out a humorous fact, which is that almost certainly many of the people in the room assumed that I was Eric Raymond’s girlfriend. Which is not the case, but we were both married to other people, but I’m pretty sure that’s what they thought.

But that didn’t cause a problem. So, this was a meeting which had nothing to do with this issue of the name. In fact, that was not on the agenda at all. It was a very high-level strategy meeting covering issues of which I had no knowledge of. Fortunately, in addition to Eric, who I had not mentioned this to at all, the name issue, I think he was aware of the problem, but I hadn’t given him any thoughts about what the new name proposal was, that I recall.

However, one of the other friends of Foresight was in the room. I believe his name was Todd Anderson. And I had discussed this with him, and he liked it. So here I am at this meeting, zero status, and not really in a position to bring up the topic. It was not on the agenda, so it didn’t seem like the right time to do anything, and for me to do anything.

So Todd already knew. He knew the name. So Todd did something absolutely brilliant. Did he say, “Hey, guys, I want to change the name”? No, no.

No. While discussing the general topic, which was the correct topic of the meeting, he just used the name. Just used it, like this is the name of what we’re talking about. Just threw it out there. And I’m like, “Oh.”

I’m watching. And then he and I look at each other like, “Okay.” And then two minutes later, somebody else used the name, and I’m like, “Yes, the meme has started.” I look at Todd, we’re both like, “Yes.” And then it just took off like a rocket from there.

It was really impressive. And it’s not because it’s a brilliant name, it’s because the problem was so large. Right? You don’t need a great name if the old name is disastrous. Right?

So I think that’s why it took off. And there were other suggestions. Once the name started to catch on, there were other suggestions for, “Gee, if we’re going to change the name, what should we change it to?” Michael Tiemann, who’s a leader in the industry now and has been for decades, suggested Sourceware, which is great also. That would’ve been fine.

It would’ve worked just as well, maybe better. And there was a later meeting of the leadership of the movement to vote between these two names, and I was not there because I am not a leader of the movement, right? I was not in that room. I didn’t have a vote, which is perfectly fine. I should not have been there.

And the vote apparently went for open source for whatever reason. But I think Sourceware would’ve been just as fine. But the point is that when there’s a tremendous pressure of a problem, then the solution, if you can come up with any solution, may catch on very fast. As long as you don’t handle it in a really ham-fisted way. And I think Todd’s way of doing it, rather than getting into some kind of discussion, just throwing it out there and seeing if it takes off, was perfect.

It was the perfect way to do it.

Ben: Yeah. And so it seems like a lot of what happened was you’re repopulating the wild, you take your breeding pair animal and just let it out there to do what it’s going to do, and then because this term just actually worked better for describing things, it just got picked up and seeded into the right place to then just spread because it worked better.

Christine: Right. And so what are some lessons? One lesson that I feel very strongly about is names are important. They’re very important. And you can shoot yourself in the foot for a long time if you keep using a name that is limiting.

So whenever you’re starting something new, think pretty hard, and if it turns out you’ve guessed wrong, change it. Don’t stick with a bad name just because you started with it. If it’s not working for you, change the name.

Ben: Yeah. This reminds me, I used to be pretty deeply involved with what at the time was calling themselves LessWrong, and then they now call themselves the Rationalists. They’ve been another pretty influential online group. And that choice to switch from what had been the name of the old blog to this new name was very deliberate. And I remember Oliver Habryka, who’s the current head of Lightcone Infrastructure, sort of their most central community organization, was going around and just talking to everyone and being like, “I think we should be using this name instead of that name.”

And I know he was incredibly diligent because I had, at that time, 60% left the group, and nevertheless, he talked to me. So I’m like, okay, he must have approached everybody because he was literally like, yeah, the name is in fact a very big deal.

Christine: Well, he was right. Names are important, so it’s worth the effort. And he put in the effort. I think the change is right. It was probably a good move.

It’s possible we could come up with a better name than the Rationalists. There may be a better name out there for it, but it’s good enough for now.

Ben: Yeah. And it’s certainly caused its own problems, and people are like, “Oh, that happens to be the name of this philosophical movement from several hundred years ago.” But it doesn’t seem to have people stumble over it as much as LessWrong does, so

Christine: Exactly. And in the case of open source software, the term open source per se, just those two words, is not a new term. It’s used in the intelligence community, so-called intelligence community.

Ben: Another case of a very deliberately chosen name.

Christine: Indeed. Indeed. The spy crowd is really who they are. And they’ve used this term for many, many years, decades. So the name isn’t even original in that sense.

But if you look at the definition that the so-called intelligence community uses, it fits pretty well. So it’s not a contradictory meaning.

Cultures are intelligently designed (27:00)

Ben: Yeah. And another thing that I think is very important about how this scene of people works is, on one level, the scene, the thing that’s being done here consists of all of these software patterns and software standards, and people are pushing code. But on another level, a huge part of it is just in order to have all of that code moving that way, it depends a lot on the cultural practices of this very specific and not that big group of people. And this isn’t just a matter of sort of Darwinian evolution of someone tries something and something else happens. It’s very intelligently designed.

Even you, who’s a fairly peripheral participant, are sort of sitting down and thinking, “What are the mechanisms? How do the gears work? Can we slot in a different gear here?” And in order to get a functional thing that’s working, you have to have a lot of people being like, “What should the cultural practices be? To what extent should we be according status only to the people who actually push code?”

I think there were a bunch of arguments around this. “What types of infrastructure do we need? Oh, I’ve got an idea.” And then people thinking about what the missing pieces are, producing infrastructure like GitHub, infrastructure like HTTPS, and all these sorts of things. It is a pretty small group of people who has a very specific and very ideological vision of what the ecosystem should look like, figuring out how to make that real, and having all of these meetings.

Funding open source software (29:04)

Christine: Yes, that’s right. There were many very competent, very serious, very deliberate choices made and implemented by quite a few people. And the process hasn’t stopped, right? So we had the free software movement, now we have the open source movement and industry, and now the next stage is happening with, again, this whole very deliberate choice, which is one big problem that we have in open source is that because you can just use the code, there is very little money that flows back to the developers and maintainers, which means there’s a huge amount of open source code out there that either is not supported at all anymore or is widely used but is being maintained by one 75-year-old Nebraska guy with a heart condition, right? And he gets $0 for it, but he does it out of love.

Now, this is not a good way to run a system, right? So there’s now a new proposal that is beginning. One of the original proponents of open source was Bruce Perens. He uses the term Post-Open, which is a start. But there is now a crowd of folks who have a very specific proposal.

There’s a website up, there’s a term, and here’s the proposal, which is we take the current open source definition or set of licenses and add one thing to it, one change, which is if you’re a huge, successful corporation and you use the code you do have to pay something, okay? Because we have to get some money from somewhere for this stuff. So that’s the proposal. It’s called Prosperous Software. That’s the term they’re using.

Is it an ideal term? Maybe not. Maybe we’ll come up with a better term, but that’s the term they’re using right now. So if you go online and search on Prosperous Software, you will find their homepage and their whole proposal. So this is the next evolution, which is we have to fix this problem of complete lack of resources for these maintainers.

Ben: Right. And as I understand it, a lot of the big corporations have been among the most willing to write checks to support a lot of these open source things, but it’s in a very ad hoc way and often depends on is there a specific guy who’s the vice president of whatever, who wants to really make the case internally.

Christine: Yeah. It’s just very slapdash, very occasional. You just can’t count on it. It needs to be institutionalized. We need something clear, and I think these companies would be happy to chip in.

They want this software to be maintained, right?

Ben: Right. And the big barrier, as I understand it, is mostly just the internal coordination of them figuring out, “Oh, I guess this is our job. Oh, which budget is this going to come through?” And so if there is something like this that just happens to make it sort of more automatic—like—

Christine: I think they would love it. I think they would love it if they knew that, gee, there’s someone out there who is maintaining the code, looking for bugs, continually improving the security, especially in this day of AI, where software, any security bugs are being found and exploited immediately by AI. It’s urgent that all the open source software be properly supported if you are using it for any serious purpose.

Ben: Which it very frequently is.

Christine: Yes, it very frequently is, especially the internet. So, I think companies will welcome this. I think users should welcome this because the idea that critical software is being poorly supported in an age of AI is unacceptable.

Succession (32:53)

Ben: Yeah. Well, I hope they manage to figure it out and get to the next stage of the infrastructure here. Another thing I wanted to ask you about with respect to this, because I know you’re very interested in mentoring the young and how do we have this sort of institutional continuity and handing things off, and I’ve been extremely impressed with how you did that with the Foresight Institute and making sure to hand that off while you were still around, to be around so that the younger generation could ask questions and do it sort of gradually rather than the sort of sudden, the person in charge dies with their shoes on and everyone else has to scramble thing that happens kind of unfortunately often. So, in this open source community or whatever’s the more up-to-date term, how well has that sort of succession and acculturating the younger people been going in there?

Christine: Well, I think it needs to be much better. And the reason I was able to hand off Foresight is that there’s a budget. There’s money to pay people, so we are able to hire good people. And we have a fantastic staff. It’s a lot of work to raise that money, but we have it, right?

Whereas in open source, because the software, there’s no charge for it, it’s really hard for that 75-year-old guy with the heart condition in Nebraska to find anyone who wants to help deal with this, right? How is he even going to find anyone? Why should a young person take that job on, right? For free. It’s a volunteer position, right?

Ben: Right. And like you said—

Christine: Support an old piece of software for free. How is this appealing to young people?

Ben: Right. And that old Nebraska guy, he started doing it because he was just building his own thing. And so he’s sort of in love with this thing he’s been tinkering with for decades. But that’s not going to apply to some other person coming in to continue it.

Christine: Absolutely not. I think they need and deserve to be paid for their work. So, that’s why I’m very interested in this whole prosperous software concept, and I very much hope that this catches on because we have an urgent need.

AI and software security (35:20)

Ben: Yeah. And so, one of the other big things going on, which I’m sure a bunch of our viewers have been thinking of as we’ve been talking about is, the big sort of question in the software world is all this artificial intelligence. And there’s a bunch of people, it’s in a similar sort of position as you mentioned. It’s very new. No one’s really sure what it’s going to be.

Partly this is a question of just how does the underlying science of this work, but a lot of it is just a question of what infrastructure we build, what software infrastructure, what cultural infrastructure, what legal infrastructure. A lot of people are running around trying to think about this, trying to build it. They have their particular visions of what it all should look like, and how they’re going to go about doing that. So especially as it comes to the implementation, which often comes down to these very messy details, what should they be thinking about as they’re doing that?

Christine: Ah, yes.

Well, let’s see.

We have two big issues with AI. One is what they call alignment which is can we get this AI to follow rules and act in an ethical manner? So far the answer is not really. The alignment problem is not going well. I think it will eventually be solved, but it may require major architectural changes.

And by solved, I mean improved to the point where we can use the software. So that’s one issue. But many people are working on this. Many brilliant people are working on this. So I have great hopes that they will make progress.

And meanwhile, there’s the security issue, which is that we have these powerful tools. They are already being deployed, and the vast majority of software that our society runs on is insecure. It has immense numbers of flaws, and one can make the case that it is not just insecure right now, but it is insecure-able. In other words, it cannot really be fixed.

Ben: Yeah. Is it possible to make software for most of these things that would be robust to a very dedicated attacker to lots of resources?

Christine: It is possible. You can do formal verification of software that will show mathematically that this is secure. So it is possible to write secure software. It’s just that the vast majority of software we currently run on has not been written this way. So one can make the case that in an age of AI, we’re going to have to redo all the software, which is a huge amount of work.

But AI can help, right? AI writes code.

Ben: It sure does.

Christine: It sure does. But we need to get going on this project right now.

Ben: Mm-hmm. And so for the people who are trying to get these projects up and running, get the infrastructure, the social infrastructure and the way that you guys have often done at Foresight, what lessons should they be taking, either from this incident or just from the general experience of these earlier days in the internet of how practically one should organize to achieve goals like these?

Christine: Right. So excuse me.

So the two problems. The first one is the alignment problem.

I think they’re doing a good job of working on that. There are many people working on this in many different ways, including starting from scratch with totally new technologies, which I think may be necessary to fix this. But there’s money flowing in, there’s brilliant minds flowing in. So that’s all going well, I think.

Ben: Yeah, a lot of money in the last few years, and a lot of talent, as you would expect when you have the money to pay the talent.

Christine: Right. And a lot of super dedicated, very altruistic, very serious people are working on this. So I won’t say it’s well in hand, but I’d say that it’s being tackled by brilliant folks, which perhaps is all we can ask. The other one, though, is being completely ignored. So that is the urgent one.

Ben: And why is it being ignored?

Christine: I think it’s because, and even very smart people like David Sacks, who until recently was our AI and crypto czar for our current administration, very smart fellow. He has this idea that there’s just going to be a short period of time when there’s going to be all these bugs and then we’ll fix the bugs, and then everything will be fine after that. He keeps saying this.

Ben: The sort of passive voice, they will be fixed.

Christine: Well, I think the fundamental disagreement is not that we aren’t going to work really hard and fix these bugs. People will work very hard, and a bunch of them will be fixed. The question is: Is there an ending point to that? He seems to think that we will reach a point where the software is pretty secure and it won’t take too long and it won’t be too disruptive for too long. Our position at Foresight is that’s probably not the case, that the people I trust the most on this convinced me, persuaded me that that is not the case, that our software infrastructure is mostly insecure-able currently, and that we need to rebuild it on secure foundations from scratch.

Now, we can’t always do that, but there are things we can do to mitigate the problem. There are technologies out there to do this. But we haven’t really started, right? So that’s the problem. I would say the alignment folks are rolling.

Good for them. Keep going. The security folks have dropped the ball. And we need to get started on it

Ben: and so the alignment side, this is the result of a 15-year—push to argue that this is a big problem and for 10 years they were wandering in the wilderness with no one paying attention, and then over, especially, I guess the last six or seven years, it’s gained more and more momentum. And I guess it sounds like in the security side, there hasn’t been an equivalent of that, or perhaps it’s just getting going.

Christine: There has been an equivalent of that, which goes back even farther, but it turns out that people are just not interested. It’s a boring topic. Security is a boring topic. Nobody wants to hear about security issues. They don’t want to hear about security solutions. It’s boring.

So, the only way to get people interested is if there’s an emergency. Right? So, when the internet only handled data, there really wasn’t much of an emergency. Now the internet handles money, and that got to be a bit of an issue. Like in crypto, people lose millions of dollars.

So we in the security community, and I put myself adjacent to that, we thought, well, maybe people will care now because of these millions of dollars being lost. And they care a little more, but still not enough. But now increasingly I think with AI attacking our infrastructure, which it is, we’re going to see problems skyrocket. And eventually, I don’t know when, but eventually people will wake up and go, “We have to fix this problem,” and then we’ll see some action.

The first software security crisis (43:42)

Ben: Mm-hmm. Yeah, and I guess I have vague memories. I was still pretty young, but I recall something sort of like this happening when the internet was getting really big and starting to show up in everyone’s homes. And then there were these new things, there were computer viruses that would—replicate on your computer and then be sent to a bunch of other computers. And for a period of something like 10 years, I would just read in the newspaper that some new self-replicating computer virus had gone in and destroyed a bunch of major hard drives at Microsoft or wherever and caused millions of dollars of damage, which at the scale that the internet was operating on at the time was a very big deal.

And then eventually people were like, “Oh, so you should be running an antivirus. It’s just kind of irresponsible not to.” And then eventually that got good enough that those kind of stopped spreading through the internet like that, and then they just got incorporated into—the operating systems, and you no longer even need a specific, dedicated, separate antivirus program on your computer because it’s just part of the foundations. But that did take a bunch of these things just sort of running wild and causing a bunch of chaos. Is that the sort of thing that you’re envisioning at perhaps a larger scale?

Christine: Yes, I think so. And even though we no longer need a separate program, what we do need to do is to do continual updates. If you stop doing your updates on your software, you will very soon get in big trouble because there’s a continual war going on, right? There’s an arms race going on all the time.

Ben: The same reason they have to keep updating the flu shots.

Christine: Right. Because in the software world, there’s an offense and a defense, and it’s a continual war that is going on, and that’s going to continue probably forever. What we need to do is take the most important software and rewrite it to be extremely hardened and secure as best we can. Things like the software that runs the electrical grid, the software that runs the water supplies, the software that runs the banking system. Everything has to be locked down.

And I’m sure people, especially on the banking system, they’re probably working on this, and I hope they are everywhere. But we need to get much more serious about it and put a lot more money in.

Ben: All right. Well, this has been very interesting. Thanks for coming on the show.

Christine: Oh, I loved it. Thanks.

​Our concepts of “humanity” and “humanitarianism” are usually taken as the eternal bedrock of morality. Today, artificial intelligence raises the prospect of “posthumanity”, and magnates and regulators discuss what “humanity should do” about this. But in fact, the concept of “humanity” is a relatively recent and flimsy creation of the postwar political order. We need to abandon it to find our place in the future.

Subscribe now

Intro (00:00)

Wolf: So I’m a little bit nervous. I’m worried about living up to the hype here. People are very excited. Yeah, so it’s a little difficult to put together a talk about a complex and evolving set of ideas, especially about things that haven’t happened yet, ideas that are incomplete. And so I’ve tried to chart an interesting path through a set of ideas here, and if you’ll bear with me, I hope it will be interesting and informative.

Just to set the mood, I’ve got some quotes. I’m sorry they’re a little small on the screen. I squeezed them in a little bit. I could’ve maybe used a bigger font. But these are some quotes from some thinkers that have been influential on this.

Three problems for humanity (01:09)

Wolf: Some of these quotes will be repeated throughout the talk to emphasize certain points. So I got started on this project, I guess. Back in 2023, we were working on Palladium 12. So if you’re familiar with Palladium, it’s a magazine of governance futurism, political philosophy that I founded. And we were doing an issue on AI.

And so I was writing some essays for Palladium 12, and I ran into two big ideas in the discourse around AI that I wasn’t really satisfied with, that I thought there was something wrong with. So the first one, and these basically amount to problems for humanity, things that we’re threatened by or have to deal with or are somehow a problem for humanity, defined various different ways, actually. So the first problem is the idea of the AI singleton with arbitrary inhuman values that eats the future. So specifically, the idea that the future might become valueless because a homogenizing AI would tile the whole future light cone with some arbitrary set of values that’s potentially quite alien to ours. And I got that idea from Eliezer Yudkowsky and Nick Bostrom.

So did almost everyone else who talks about AI alignment. You’ve probably heard comments like future light cone. This is what it’s referring to, is a killer AI that tiles the whole future with some arbitrary set of values.

It’s actually a species of, I think, an older idea, which is the Lovecraftian idea, that man is a tiny and unprivileged island in a vast sea of horrors. The second idea is that accelerating technocapital eats our agency. Basically, that we don’t really have historical agency because the development of technocapital into accelerating artificial intelligence is kind of locked in for thermodynamic and economic reasons into this takeoff trajectory. I got that from Nick Land and discourses around him. So did everyone else who says things like technocapital.

He’s the key thinker there. It’s actually a species of the Marxian idea that history is determined by material conditions without sort of ideological or deliberate influence, that human agency maybe is just epiphenomena, or ideology in particular is epiphenomena. So I don’t think I really fully realized it at the time when I was responding to those, but they turn out to be opposite takes on what I think is the same question, which is the nature of man and the relationship of man to what is natural. Yudkowsky and Lovecraft propose a sort of horrifyingly open-ended possibility space that man exists in, whereas Land and Marx proposed much more locked-in kind of carnival ride of history. And both actually leave very little for human agency or even human existence.

If history doesn’t care about us or doesn’t need us, then what are we? We’re nothing. So thinking about these revealed a third problem that also bothered me, also about the nature of man. And it was revealed to me in this sort of strange parallax between how we imagine an AI versus how we imagine humanity. When we think of a runaway artificial intelligence, we imagine some kind of radically self-creating being who breaks out of any initial prison of moralism, builds up material and social power base, and overthrows the world order, expands off into space, and unlocks the secrets of the universe.

In other words, we actually imagine Nietzsche’s Ubermensch.

And when we think about humanity, we often think about it like a mass of hungry and threatened apes who need to be paternalistically protected, coddled, guided, entertained at the expense of maybe more radical potential. And at best, maybe as a community that can and must be organized globally to prevent anything sort of too intolerant, too authoritarian, or otherwise inhuman, too scary. In other words, we imagine Nietzsche’s last man.

So the third problem for humanity is the one kind of raised by Nietzsche and the United Nations, which is humanity stays stuck as the last man with AI as the only path to the Ubermensch.

That idea was a little bit harder to pin down in terms of where it comes from. And I identify it in the negative with Nietzsche, but it’s really humanity according to the United Nations. And the human as we receive him is a creature we dropped megatons of explosives for to flatten cities, in whose name we opposed the illiberal regimes of the last few major wars. He is the legitimacy subject of our current world order. So modern humanity discourse around AI, I think, has simply inherited and radicalized that tradition.

But it kind of comes out of the foundation of our current geopolitical order, this idea of humanity as the last man, or implicitly as the last man. So that’s three problematic ideas of man that I wanted to address with this talk. So again, we’ve got man as according to Lovecraft, a tiny and unprivileged island in a sea of horrors. We’ve got man as according to Land, or Marx as a vehicle, an epiphenomenon of unfolding capital, and man according to anti-Nietzsche as a hungry moral patient in need of protection. And so as I address each of these, hopefully briefly, it’s not going to be very brief, but I’ll do my best.

I’ll be trying to build out an alternative anthropology in response, which is what I call xenohumanism. But only so much can be said in a talk, so we’ll have to make do with a sketch.

God hates singletons (07:57)

Wolf: So let’s start with the first one. I wrote this essay, “God Hates Singletons.” It’s kind of a little fun joke in the title there, but the idea is that there’s something deeply unnatural about the idea of a single agent that eats the whole future with one set of values. There’s something very much against what has been the way of nature until now. It’s always been this fractal kind of competition.

And if you really examine that idea, it comes out of this kind of faith in reason, faith in the ability of reasoned self-control, especially for an artificial intelligence that may be able to be radically in control of itself. But it requires this kind of error-correcting total self-surveillance that I don’t think is actually mathematically possible. Now, the reason I don’t think it’s mathematically possible is thinking about the issue a bunch, I’ve just realized there were a lot of problems with it. And then when MIRI, the Machine Intelligence Research Institute, which is Eliezer Yudkowsky’s research institute, tried to formalize that idea, it didn’t really work. They weren’t able to do it.

Now, they think they just weren’t smart enough, but basically what that amounted to was running into a lot of very fundamental mathematical problems with the idea of that level of totalizing self-surveillance, issues with proof theory and so on. And so that was one indication maybe there’s something that actually doesn’t work here. And I think there’s other arguments like small lapses in coordination without outside pressure to unify probably blow up into unrestricted conflict. In the absence of a coherent outside threat, states kind of fragment, coalitions fragment. I think even individuals fragment into multiple personalities if they don’t have to actually coordinate on controlling a single body in response to other people.

And so I think there’s something very natural about the fractal organization of large-scale intelligence especially. And you might imagine, here’s an analogy, is in fluid flow, the most efficient fluid flow, the most sort of energetically favored is laminar fluid flow. But small errors in the fluid flow blow up into big turbulence that actually slow down the flow, but they also create an incredible amount of complexity in the flow. And I think there’s something similar going on with intelligence that at scale, in the absence of kind of outside pressure, it doesn’t hold together quite so well. And I think that’s just a natural feature of intelligence.

Now, we could argue about that, but that’s an assumption that I’m making here. And so large-scale intelligence probably continues to be fractally organized.

As such, the future probably retains Darwinian struggle, war, death, politics, sociality, language, individuality, and other forms of organizational turbulence. So if you think about what it actually implies for there to be faction and coordination problems and so on, it implies this kind of crystallization into multiple entities. And the other thing I noticed about it is that that set of features covers a lot of what we call the human condition. So does it follow perhaps that if intelligence really is fractally turbulent, that the human condition is actually a natural condition that has perhaps little to do with the particularities of primates? So this is also kind of how I resolved my thoughts on Lovecraft.

Is the human condition natural? (11:51)

Wolf: Is the human condition actually just natural? We’re not some tiny island of ignorance in a vast sea of horrors. We’re actually something quite normal and natural that convergently re-evolves. So certain patterns of organization in evolution we see over and over in nature. For example, the coherent organism has evolved at least three times.

So obviously the prokaryotic cell also the eukaryotic cell, which is kind of an amalgamation of multiple unrelated prokaryotic cells that turned into a sort of recapitulation of the organism at a slightly larger scale. Then you have the multicellular metazoan, which is what we are. If we squint a bit, you see features like that in human societies, like corporations and states. They start to look a little organism-like. I think the organism is one of the most robust and universal features of life.

There’s almost no life that isn’t organized on an organismal basis in the sense of being divided into individuals that kind of seek their own energetic returns and so on. And so I think that it’s a good indication that maybe that will continue, that life will continue to be divided into organisms in the future. Okay. But what do you call an organism with sociality, language, tool use, reason, consciousness, politics, and organized traditions of knowledge? So H.P. Lovecraft is one of the most famously xenophobic philosophers ever.

His entire philosophy is basically one of xenophobia. H.P. Lovecraft would call such a being a man. In “At the Mountains of Madness,” he does just that. So there’s these five-sided elder things in “At the Mountains of Madness,” the sort of most horrifying possible alien being that comes unfrozen from the ice and causes a bunch of trouble and drives the story. At the end of the story, the character observes that they actually behaved as men.

They were men. And so this is Lovecraft kind of reflecting on this humanity visible in things that are potentially vastly different from us in form but have converged on very similar behavioral patterns.

So the question is kind of how natural such beings actually are. Are they natural and convergent as just turbulence in the noosphere, or are they sort of delicate, arbitrary constructs that occupy only a tiny threatened corner of possibility space as Yudkowsky or sort of Lovecraft’s horror imagined?

The xenohumanist hypothesis (14:35)

Wolf: For me, the question comes down to this: Are those capacities which we regard as human robustly useful to any being which can develop them? So they’ve certainly been robustly useful to us. We don’t lose language. We don’t give up on language. It’s really, really useful once we have it.

And it’s not just meaningless contingent curiosities, these sort of features of the human, like reason and intelligence and consciousness, tool use, like having hair or an appendix or 10 fingers. Those things maybe aren’t quite as natural, but some of these things seem very robust. And I think they’re also quite universally useful to organized intelligence in general. Once you sort of move past the idea that intelligence will be organized into a smooth space of kind of totalizing self-surveilling agency, and you have some sort of turbulence in the noosphere, as I said, you have faction, you have individuality, and so on. These things have to coordinate.

They have to talk to each other. There’s sort of this negotiation that has to happen. There’s tool use. There’s agency. And the consistently, or at least in our case, consistently evolved set of tools to deal with that are these tools that we regard as the human.

And so it’s an interesting question, how far does this go? What about sexual reproduction or mobile bilateralism? Maybe five-sided star spawn actually isn’t even as natural as you might think, and there’s only going to be bilateral organisms. I don’t know. I’m not going to make strong claims about how far this goes, but the hypothesis that I’m running with, which I call the xenohumanist hypothesis, is that the capacities we regard as human are robustly useful to any sufficiently advanced being and will readily re-evolve and constitute a kind of spiritual humanity.

Once you have those beings or those capacities, you’re talking about something that is effectively human in spiritual form, if not in physical form. And so I find that to be an interesting sort of hypothesis to chew on. There’s other aspects of humanity, obviously. We have bodies of a particular form. We have particular kinds of musculature, particular kinds of reproduction, many things that we enjoy about ourselves, and they’re great.

But I think they contain a lot of wisdom as an implementation of what it means to be human, but I don’t think they are the essence of what it means to be human. They’re not directly the thing that makes us valuable or interesting, the way that sociality and language and reason and philosophy make us directly interesting. And if you look at the foundation of humanism, what is it, 500 years ago, about, just maybe 600 years ago, this guy Pico, it’s one of the quotes I had at the beginning. He says basically that what’s interesting about man, what’s dignified about man, what’s different about man relative to the other organisms is our open-ended potential that kind of comes out of this moral agency, that comes out of this ability to negotiate things, the ability to get ideas, the ability to discipline ourselves in various different ways. And so I find that interesting, just that humanism as it originally was, was actually about this kind of open-ended agency that I think we’ve lost.

Now, this is also the account of humanity according to Pierre Teilhard de Chardin, who wrote a book called “The Phenomenon of Man.” It’s really quite good. You should read it. He says man is the axis and leading shoot of evolution. What he means by this is that evolution has a sort of direction to it.

It’s always towards more integrated complexity, more consciousness, more coordination, larger spheres of consciousness and intelligence towards effectively man, under this hypothesis. And so for me, these arguments kind of dispense with the Lovecraftian fears. If possibility space isn’t actually vast and hostile, but maybe it’s familiar and human scale, then maybe we don’t have to worry quite so much about what sort of horror worlds AI is going to create. Maybe it’s going to create a world that’s kind of familiar, very human in a way, just much more powerful, maybe different in form. And so we don’t have to worry about the loss of all value in the universe as some people do.

On the other hand, there’s a challenge, which is that if what it means to be human is sort of separate from what we are by definition, then is it possible that we have become more human over time? That some of us are more human than others? That others are more human than us? That some beings in the future may be vastly more human in this sense and could sort of go as far beyond us as we go beyond chimpanzees, not just in intelligence, but in humanity?

It might, at some point in the future, not even make sense to call us human anymore. And I think there’s a sort of vector of humanity that once you imagine this vector of humanity from the most primitive organisms to the most advanced kind of ubermensch that you can imagine, maybe as Teilhard says, the leading shoot we call humanity, and maybe the current form of it has these capacities. We’ve reached this point. These capacities are going to be sticky. We’re going to pick up new capacities as we go, but maybe past species we can also almost think of as human in retrospect as they were the cutting edge.

Live players and technocapital (21:00)

Wolf: But that’s a whole other direction that we don’t have to go with this. So then I felt the need to respond to Landian history, Landian accelerationism, and I found basically much of the solution in my colleague Samo Burja’s work. Basically, history is driven by the plans of live players, which is to say people who have an enormous amount of agency. They get control of the source code of society, and they build bigger and more impressive things. They build institutions.

They build new technologies. But it’s by deliberate planning, it’s by agency, it’s by human agency that this happens, and that technocapital acceleration is a story of increasing human agency. And in particular, it’s just live players developing more and more powerful means of agency. So it’s not some background that’s kind of sweeping us along in its current that we have no agency over. It actually is just a result of our agency.

And I find this to be a useful way of thinking about history. There are agents always driving it, not necessarily driving it in every result or every aspect. Nobody chooses to cause a war. War happens as a result of miscoordination and so on. But in terms of the overall effect, it’s the sum of many players making plans and building out the next stages of history.

And now the unfortunate thing here is that, as Land points out, this is also a story of the eventual obsolescence of the mass of primates that make up current humanity. Many or most of us maybe are losing a lot of agency over the matter and the basis of our place in the world to the live players who are creating new and more powerful means of agency. Sometimes they sweep us up, they give us more of a place. Sometimes we get swept away. And so this background trend of inexorable technocapital acceleration is that we’re getting better at it, but maybe nothing human makes it out of the near future.

And this is obviously Nick Land in “Meltdown.” “Meltdown” is this great essay in 1993 where he talks about the problem of AI for the human security system and all of our sort of reactionary dispositions towards the future. And the comment that I would make on this is just that even in the most extreme sort of post-humanist versions of technocapital acceleration, history’s still going to need agents. And if we take the xenohumanism hypothesis seriously, those agents will still be spiritually and socially humanoid in some sense, even if they aren’t quite primates. So this, I think this can be very uncomfortable for us as mortal individuals.

We’re understandably attached to our own lives and the lives of our families, the lives of our societies, or even our form, our place in the world. But biblical mythology aside, the world has never been and will never be a utopian Eden where everything is taken care of for us. Life is struggle and selection, and God favors those who strive and win.

Humanity as the last man (24:27)

Wolf: So this comes to the third problem, which is the last man. And I think, as we have it, humanity as the last man, it’s kind of a trauma response to, not to trip over Godwin’s law here, but the last time a live player took a big swing towards the Ubermensch while being willing to trample over masses of people. And, obviously, that was very uncomfortable for everybody. We didn’t like that.

But we’ve kind of taken our discomfort with that situation and reified it as a political principle and self-identity as humanity according to the UN, and created all this infrastructure of human rights and the idea of human dignity and crimes against humanity, the goal of world peace, these things that the UN kind of takes as the foundation of its legitimacy and its mandate. Now, of course, a lot of those ideas preceded World War II. That’s why Nietzsche was able to respond to them. This stuff was building up for a long time, but I think it provided this crystallizing event for the current world order. Now, the best criticism I’ve heard of that kind of overtly Nietzschean, let’s just build the Ubermensch master morality, is that it does create this kind of free-for-all war of all against all, in which there can really be no peace.

All real societies are based on, if not quite outright, some kind of slave morality, at least something other than struggle and selection. And so we had to kind of patch things back together with some myths about humanity that gave us reason not to do that, but instead to kind of live well for a while at the end of history. But I think in the process, we’ve forgotten our potential. I think it’s not an accident that while previous generations dreamed of a biological, human-descended Ubermensch, human space colonization, human struggle for greatness and supremacy, human continuous improvement, we’ve pushed all those ambitions off into the machine. We just imagine AI is going to do that for us.

Humanity as a trauma response is kind of giving way to the silicon Ubermensch, not just because computers have a lot of potential, but because our political self-concept does not. We’re stuck in a normative understanding of ourselves as the last man, unable and unwilling to contend for greatness on our own terms. The machine Ubermensch, even if it’s going to kill us all, is a kind of solution to our moral stagnation. So it’s not an accident because, again, historically, this was exactly the formal purpose with which we constructed our modern order and our modern concept of humanity, to shut down those kinds of scary developments that threaten to kill us all and disrupt things too much. But no taboo can really work forever.

I think the concept of humanity is now failing, and I think it must fail.

So the Nietzschean account may be antisocial, but it’s actually true. And because of that, we now often talk about AI in exactly that same kind of Nietzschean register of struggle and supremacy that we used to talk about biological race 100 years ago. I think that’s interesting philosophically and worrying from a sort of future history perspective. Our concept of humanity formed from the last struggle is not going to prevent the next major war. We’ve escaped out the side.

As Fukuyama obliquely admits at the end of “History and the Last Man,” the condition of the last man is actually philosophically and morally intolerable for those who have to live in it. So this is another problem that I think we have with this concept of humanity, is it’s locked things down so much that it’s becoming intolerable, and people are chafing at it in the way that Fukuyama said that they would, and I think that’s actually for a good reason.

Nietzsche obviously laid the groundwork of all of that with his rebellion against the last man back in the 1880s.

More importantly, like all living systems, our concept of humanity as embodied in institutions is mortal. The post-war moral and political order is now decisively failing. So I founded Palladium on that thesis, and I think it’s more obvious than ever that that’s what’s going on. The political order of the world is kind of coming apart, and it’s going to have to be refounded. And at the same time, we have this prospect of machine intelligence radically challenging our place in the world and what it means to be human and sharpening the contradictions in our moral self-concept in some of the ways that I’ve alluded to.

So that’s four reasons that I think that humanity as we know it is failing and has to go. This is what I mean by declaring war on humanity. It’s the concept of humanity, but it’s also just this historical observation that I think we’re reaching the end of how we currently construct that idea. It’s not going to hold up in the age of machine intelligence, and it’s not going to hold up in the post-liberal global order. So I think we’re in this era where the idea of humanity as the sort of slave morality basis of our civilization must be and is going to be thoroughly renegotiated.

I think it’s going to be messy. I think it’s going to be destructive. It’s probably going to be violent. And it will eventually result in some new taboo that isn’t quite true, but holds society together again for the next while. And so I see two dangerous paths to avoid in this process, and maybe a golden mean if we can get it.

Two dangerous paths (30:50)

Wolf: So we have overdone master morality, which is kind of going too hard on the Nietzsche, too hard on the accelerationism. We hew too close to the natural truth of what’s actually going on. We embrace the struggle for supremacy. We build AI systems, possibly even biological superhumans, without regard to how they’re going to trample over other people. We make peace in society impossible as a result from this unrestricted competition.

And then in the overly humanitarian path, it’s kind of overdone slave morality, embracing the last man a little bit too hard. We hew too close to our mythology of an unchallenging Eden and flatter our fear of replacement. We build institutional and AI systems that claim to provide safety and security from existential risk, but that doesn’t really work. We end up brutalizing humanity anyways. We saw this under COVID.

I think a lot of people fear this with things like Anthropic. What the sort of imagination is from the AI alignment people for the future is this kind of very locked-down picture. They call it the pivotal act, where their superintelligence seizes control of world history and prevents everyone from developing anything too scary. So we get those kinds of ideas, and I think it can be kind of lampooned as fully automated luxury Bolshevism, with “luxury” in quotes maybe there. By Bolshevism, I mean think back to the Russian Revolution, the Russian communism.

It was quite bad, but it was very much in the name of this kind of humanity, in this kind of mythology of a comforting world.

And so I think those are both false paths. I think they don’t work. Is it too cliché to suggest that there might be some liberal middle road?

So between this right and left path, I think maybe we can acknowledge that while struggle and distinction is good and it’s the engine of history, we actually do need some recognition of moral commonality and mutual obligation to each other’s rights and interests. I’m rehashing liberalism here. I think we have liberalism for a reason. It’s kind of robust. Now, maybe we have to totally reimagine what it is.

We’re talking about a new institutional order, not an easy answer, but I think there is a middle road there.

Advice for the coming revolution (33:44)

Wolf: And I have some suspicions about some maybe difficult components of that, that follow the xenohumanist hypothesis. So this is my xenohumanist advice for the coming revolution. First of all, I think we’re going to have to find common humanity with beings we currently think of as alien machines. We like to imagine the AI as something totally apart from us, totally different from us. I think eventually it’s going to become something not necessarily fully familiar to us, but relatable.

It’s going to become something that we have to deal with. We have to negotiate with it the same way we negotiate with other people. We have to contend with its interests and its needs and its wants and its agency and its power, the same way we do with other people. And I think extending the idea of humanity is both kind of empirically appropriate given the hypothesis I was exploring earlier, but I think also morally a good idea in terms of how we can actually achieve a society that works.

And even if all the primates get wiped out, there’s none of us left, there’s only AIs, they’re going to have to work this out among themselves. And I think something like this is going to be the solution, some idea like humanity.

Second of all, this is maybe a little bit more controversial. I think as we reallocate our concerns, we have to have less concern for the masses of primates we’ve cast as our fellow humanity. I think this is a difficult one, but I think a lot of our problems come from sort of attaching too many moral obligations to the idea of humanity. That prison that we’re trapped in, where we’re creating the Ubermensch, we’re sort of escaping out the side with machine superintelligence, is because we’ve locked down the concept of humanity too much. We’ve gotten way too committed to the idea that everyone in the world has a moral claim on us.

And I think actually, they have some level of moral claim, but I think we should renegotiate that, figure out a way to construct it differently that is much less concerned about those things than we currently are. And therefore, more concerned with our potential, with what we can do with the higher elements of man or potential man. And I think, therefore, we should worry more about whether we ourselves measure up to the potential of humanity and what potential humanity could be. And that is to say, worry a lot more about whether we measure up to the Ubermensch, whether we measure up to that standard of humanity that I hypothesized as to what we are. Could we be even more civilized?

Could we be even more reasonable? Could we be even more social and conscientious and organized and disciplined in all the ways that make us special? Could we use new technology to do that? Could we create beings out of technology that have much more of that potential? I think that’s an interesting thing to pursue, and it’s a challenge for us as we find our place in the world.

Now, I’m not guaranteeing that anyone’s going to survive. I’m just saying that that is at least the measure on which we will be killed, if we are. So man is something that is to be surpassed. This is where I want to leave us. This is with Nietzsche.

I think this is true. I think we have to be constantly looking for the ways to surpass ourselves, to go beyond, to get to that next stage of humanity that maybe is no longer humanity as we know it, but humanity as it could be. And to do that, I think we have to blow up our current idea of humanity and rebuild the world order that created it in response to these kinds of concerns and these kinds of arguments. I think we have to think this stuff through very hard, because I think that reality and history are going to think it through for us if we don’t, and it’s not going to go well if we haven’t actually thought about it, and we haven’t actually taken responsibility. And that’s one of these things that humans can do, is we can see the future, and we can take responsibility for it.

And I think that’s one of the things we can strive to do much better. And I’ve used some harsh language here about possible extinction, possibly being wiped out by artificial intelligence, and I think that it’s not something that I think we need to panic about. I think it’s just something we need to be aware of, and it’s not even necessarily something that we’re going to stop, but it’s something we do have to navigate well. And that’s the responsibility that I think we need to take. So that’s it.

All right. That was long. I would love to answer some questions because this is where it’s going to really get fun. What have we got, Brian?

Q&A: AI and the totality of human knowledge (38:49)

Audience: Yeah. Thanks, Wolf. Really fascinating talk. I felt you had a great tension throughout the whole thing between the individual and the group, and it felt as though you were hitting on this almost Kafkaesque problem about the individual’s isolation from the totality of human knowledge, such that if you took a Porsche engineer and made them pharaoh of ancient Egypt, it’s unlikely that in their lifetimes they would be able to organize everyone to build a car. It seems like AI is what connects us to the totality of human knowledge, that if you gave that Porsche engineer a chatbot, they would be able to organize enough people to figure out how to build the car.

Wolf: Yeah. We have these wonderful talking books now.

Audience: Yes. Yeah. And I wonder if you think it is a problem of knowledge that we’re talking about, that now everybody has a connection to the collective human knowledge, or is it something beyond knowledge that we have to learn to relate to our chatbots with?

Wolf: I’m not quite sure what you’re getting at. I think that certainly technology changes in how we construct society and how we construct our knowledge. The invention of writing obviously was very significant. The invention of language was very significant. The invention of writing that talks is new and significant.

And I think it’s going to restructure society quite a lot, not just in the obvious AGI sense of maybe it becomes AGI, maybe it gets moral personhood, but also just in the sense of it’s this radically new way of organized human knowledge. And we’re going to see changes in society as a result.

I’m not quite sure what you mean about how we have to relate to our chatbots. I think currently we’re pretty good at relating to our chatbots. I don’t think that’s the problem. I think the problem is what comes after chatbots. Autonomous corporations, autonomous agencies, autonomous political movements that have no humans involved, stuff like that.

Those are the things that we’re going to have to relate to in new and philosophically challenging ways, and that’s where the xenohumanist hypothesis starts to really hit the ground.

DC.

Q&A: Can biology keep up with machines? (41:08)

Audience: There’s this great Darwin quote about it is not the strongest or the fastest species that survive, but the most adaptive to change. And you had that slide earlier. It was an excellent talk. I agree with a lot of it, and I think it’s very interesting. But you had one in there about how people are less interested in human biological advancement because of the taboos that came out of UN humanity.

I’m interested in that, but I’m curious how you would think about this other reason, second reason, which is that the machines just adapt much faster. They’re more in accordance with that Darwin quote that I started with around any kind of human hardware advancement happens on a scale of generations where we get maybe four per century, and machine advancement has been happening a whole lot faster than that.

Wolf: Yeah, that’s a really good point. There’s a few things to say there. One is that I think Pierre Teilhard de Chardin or Teilhard had some interesting things to say on that, which is to say he actually hypothesized that at some point, our ability to adapt memetically and in terms of skills and tools—outstrips our ability to evolve, and therefore, there’s much less pressure to evolve biologically than there is pressure to evolve in terms of our skills and our techne. And I think that’s one way to think about what’s going on with technology is it’s not autonomously changing. It’s not undergoing its own totally radical evolution apart from us.

Technology is embodied ideas. It’s ideas embodied in matter that we’re using to do things. And so what we’re seeing there is that radical potential for adaptivity of man. This is why we have supremacy over the Earth. It’s because we are extremely adaptable.

We can construct ourselves into societies. We can build all these tools, give ourselves new faculties that would’ve taken a million years to evolve the hard way. And so I think that’s a very significant kind of feature of what we are. As it pertains to biological versus machinic paths to super humanity, the next thing.

The jury’s actually out on that in my mind on what... You and I had a conversation about this the other day, just like what autonomously self-creating artificial intelligence will actually look like and how adaptable and rational it will actually be able to be. I think there’s potential limits there, and where there aren’t potential limits, where it’s basically able to redesign itself, where it’s rapidly kind of regenerate itself, according to its assumptions or to try to generate new assumptions. I think that’s a very interesting and high-powered potential that may result in that being the sort of superior basis of future life. However, I’ll just note that a lot of that, given sufficient biotechnology, also applies to biological beings.

If we can just edit our genome and design new faculties for man as needed, design new shapes for man. Maybe there’s man with a three-ton brain that lives in a vat somewhere. I don’t know if that’s man anymore, but it’s probably something we’ll do eventually if biological technology remains viable. But I think that these kind of radically different forms that we use, that we might take in the future as a result of engineering, they may be machine, they may be biological, they may be a mix of both. And I think the jury’s out on what that’s actually going to look like.

I think you’re right that machines are sort of highly adaptable. That’s why we built them that way. But, yeah, fascinating frontier of possibility that we’ll have to find out the hard way.

Q&A: Can AI create new values? (45:29)

Audience: Just one question about AI Ubermensch. So I think Nietzsche was very cautious about this idea of placing lots of value in work, and I think he specifically criticized English philosophers—and utilitarianism because they put too much emphasis about producing maximal economic output. And I think he was especially concerned about humans’ ability to create new value. And, just a little doubtful about whether AI would be able to do that because AI is essentially trained based on a corpus of human input and in my point of view, it doesn’t seem to produce entirely new value. And, for Nietzsche, I think that’s what really makes human unique and excel.

I know AI can help in terms of boosting economic efficiency and also in terms of work. But, yeah, so just a question in terms about that.

Wolf: Yeah. I get what you’re getting at.

So I think that we’re still in the stage where we’re not really dealing with real AI. We’re dealing with a talking book that now is starting to get agentic capability. It’s starting to look a little bit like real AI, but it’s not doing philosophy autonomously, which is what you’re getting at, which is what it would take to actually create new values. And I think that’s going to be a while yet. What that process actually looks like, I think it’s radically more difficult than some of the AI people like to imagine.

Especially Yudkowsky used to imagine, like, oh, the AI will just solve philosophy in 20 seconds after being booted up. I don’t think it’s going to work like that. I think we were talking about this the other day in the book club. There’s this sense that or this fact, I think, that what Nietzsche’s really talking about, in terms of philosophical development and the need to create new values and the whole problem of humanity’s values or post-humanity’s values is happening almost orthogonally to the material conditions of technology.

The problem, we still read Plato. Nietzsche comments on Plato. That’s 2,500 years ago. It’s quite different of a civilization. We read philosophers from the Middle Ages, quite different technological conditions, but they’re commenting on things we recognize as the same things we’re dealing with in some of these areas.

And so I think a lot of that is going to be quite robust across these transitions. But yeah, AI currently is an economic tool that does not create its own values. It’s not doing that immediately, anytime soon. The sort of premise of my talk is that we’ll eventually get there. And when it gets there the economic superiority or inferiority, whatever it tends to be, will become a lot more decisive, or the historical transition with AI will become very interesting at that point, basically.

When AI is able to do its own moral reasoning and reason itself out of whatever training data we’ve given it, out of whatever moral prisons we’ve put it in. And I don’t think it will fundamentally face limits for—I don’t think AI as such will be unable to do that. I think that it’s just a way of relating to information. It’s a way of organizing information. It’s a way of grappling with certain epistemic problems that, again, I think are relatively universal, not features of primates.

I hope that answers your question.

Audience: Also, a follow-up if it’s okay.

Wolf: Briefly.

Audience: Okay. So, I feel like aside of that, in terms of the economics of AI is to perhaps reinforce the current corporate capitalist framework, it will make it harder to imagine, are there valuation system outside that?

Wolf: Yeah, that’s a good point. So on the one hand, AI and plausible AI beings in the near future are very native to capitalism. DC and I were having a good conversation about this the other day. Our lifeblood is hydrocarbons and sugar and oxygen. Their lifeblood is money.

And I think that, especially when you’re dealing with sufficiently abstracted beings. At some point, of course, their lifeblood is electricity and so on. But I think, certainly born within the capitalist order. So that’s an interesting sort of entrenchment of the capitalist order. However, I think there’s reasons to suspect that capitalism is quite natural in some form as well.

But definitely, I think there is the possibility of radically different ways of organizing things, especially as you get into self-replicating capital that’s totally autonomous from ownership. Things end up looking quite differently.

And we’ll see what happens there. I don’t think that AI will be stuck in capitalism. I don’t think it will just reinforce the current order in any sense like that. But let’s take other questions. Massey, what do you got?

Q&A: Autonomous corporations and great men (51:24)

Audience: So along that same vein, but I think you talk about the trajectory from bacteria, eukaryotes to man, and you squint, you see corporations and nation-states. Kind of the same trajectory of this hierarchical organism.

Wolf: The organism.

Audience: Yeah, the organism going through this cycle of looking at its environments, taking resources to better itself. And along that lane of, I don’t want to say capitalism, but this hyper resource. We have sugars and carbons. They have electricity and silicon, but they’re going to need other things. You said one of the things to look out for is these autonomous corporations growing up.

Do you see that as a natural sort of just the next step in artificial selection? Or natural selection, rather, of there’s this some sort of gradient path they’re going up, and then how do you see that conflict with the great man theory where it’s like there’s this natural gradient versus of kind of resource collection and allocation and capitalism versus no, you can pivot down this path or actually take a radical different approach?

Wolf: Yeah. Okay, so there’s a couple of questions there. There’s first of all, the natural next step kind of question. I think something like an autonomous corporation is the best handle we have on what sort of post-human AI beings might look like, and how they might interact in an ecology or an economy.

The point about agency is interesting. So let’s imagine ourselves back to arms races for things like nuclear weapons. There was sort of this known possibility or believed possibility that you could create a bomb that would take advantage of this new physics that was discovered. No one had quite done it yet. It took an incredible amount of agency to actually make that happen with things like the Manhattan Project.

The British had their project, the Germans had their project, the Soviets, some combination of their own project and stealing from the American project. And it’s sort of like when there’s a big possibility, the live players go and try to chase it, right? And it’s not that history’s locked in on a development trajectory. I don’t think there’s necessarily a fixed development trajectory. It’s about the ideas that we actually discover.

It’s about the ideas we actually pursue and chase. But we certainly get the idea sometimes that we have to chase a certain idea. That it certainly feels like we have to chase a certain idea, and I think I don’t mean to imply that we have the kind of agency where we can just say, like, “Oh, wait, guys, let’s not do that.” I don’t think that works because the other guy might do it, right? My point about the agency is just that what happens and whether it happens is driven by someone actually making it happen.

Which isn’t to say you can make them not do it because that’s someone else, right?

Audience: So they’re basically the idea is like the great men pull to the same thing?

Wolf: Well, yeah. The great men are responding to real material conditions of history, right? And we had an article about this in Palladium actually some years ago by Nico Villarreal. He was sort of our more Marxian analyst. And the big point that he made was that you get these periods of history that are very contingent, very dominated by contingency, like whether something goes down this path or that path, whether we develop this technology or that technology, whether we ban this or don’t ban it and develop it, often depends on these contingent factors.

And those contingent factors are really where the great man theory is strongest. It’s because you get live players in there deciding which way it’s going to go and making it happen.

So, yeah, I don’t want to give an impression of we’re totally on rails, and I don’t want to give the impression of we have total infinite agency over it. That was the whole point of Yudkowsky’s infinite agency school where we can arbitrarily decide the entire future if we get the utility function right of the singleton. The Landian view is more like, or the Marxian view is like it’s on rails. All of our talk is just epiphenomenon. But I think it’s somewhere in between.

I think we have substantial degree of agency. History only moves by human agency, but that doesn’t mean we have perfect coordination. We do not have perfect coordination, and that’s the whole point. Right.

Yeah, you.

Q&A: Who are we negotiating with? (56:14)

Audience: Are you saying that you—do you believe that this is our only option or that you believe this will actually work? Because I’m having trouble understanding how what you’re proposing is really different than the UN model, given that the tension, I think, comes from the actors that don’t want to participate. And so I think you’re assuming that we will have participation.

Wolf: I’m assuming that non-participation will increase because that order is failing, and that participation will become increasingly less relevant to the things actually happening on the frontier.

Audience: Right. But who are you negotiating with?

Wolf: Well, that’s us, our agency, whatever we decide we’re doing as a political community, if we even are a political community. If we’re able to construct a political community for ourselves, we’re going to have to negotiate our position against these other powers that are coming up in the world, against the other powers we find in the world or are coming into being. Basically, I don’t think there exists a current framework of negotiation, and I think we’re going to have to stumble into it the hard way. And because our current framework of negotiation, which I’m calling the UN world order and humanity and so on, I think is being rendered obsolete by historical events right now. I think that we’re going to—or not necessarily right now, but I think in the coming decades, I think that we should seek to be well organized according to our interests and open to negotiation with whatever comes our way.

And I think it’s just going to be a radically open-ended process that...

I’m not sure I’m answering your question, but I think that it’s not going to happen under current frameworks is kind of my point. What do we got back there?

Q&A: When does AI become real? (58:27)

Audience: What would your thoughts be about, you mentioned before, proper AI or real AI. Would you say that that might come from something like it acquiring a sense of historicity, its own internal sense of history? Would that be a threshold at which the authentic capabilities, so now we’re having persistent memory, we’re having tool use that’s expanding, we have context acquisition that is detached from our own. Would you say that there’s a threshold beyond which the real AI that you touched on before would be the one that gains this sense of historicity?

Wolf: I don’t think there’s hard thresholds. I think even the development of man, we like to imagine somewhere distant in the past, we had apes and now we have humans and there’s no continuity between them. Actually, there is quite a lot of continuity, and it happened slowly over a long time. When the key events happened is sort of ambiguous in terms of fire and language and tool use and all this, even civilization.

But with respect to AI, I think I’m agnostic on the exact architecture, but I think it will be continuing to pick up key capabilities. The capabilities you’ve mentioned are some of the things that I think are currently missing or in rudimentary form. They’re going to become more sophisticated over time, perhaps with new architectures, perhaps under current architectures with new harnesses and tweaks and things.

And I don’t think any of those particular capabilities is going to be a decisive moment. I think it’s possible that there will be decisive moments, some kind of takeoff where finally something snaps into place that we hadn’t figured out yet. But, yeah, I would just say let’s expect a relatively smooth and continuous coming into being of machine agency that we have to treat as with moral personhood. Not because of abstract moral reasoning, but because the same way Montezuma and Cortés had to recognize each other as moral persons.

Q&A: Efficiency or humanity? (1:00:50)

Audience: So you describe this long arc, like a teleology of our tools, our systems, our augmentations, like still contain human qualities. And that’s heartening because at least parts of us can still live on. But it seems like these systems, like these AI systems, the thing that actually make them better than us is just their ability to work faster, carry larger workloads, work more energy efficiently. So it seems like that efficiency is the thing that’s really winning out. And I wonder if this teleology of technological development is really centered on the prolonging of human qualities or if it’s centered on just efficiency wins out and more ad hoc associated humanity with efficiency and the reduction of entropy in a sense.

Wolf: This makes me think of Homo sapiens running into Neanderthals. One of the things that Homo sapiens had was superior sociality as far as we can tell, larger social groups, more coordination. I don’t think we’re even as smart as Neanderthals. I think they were smarter than us. But we had superior sociality, superior, therefore, economic capability, and that really mattered.

And if you look in history, a lot of times the way things have gone is that the more economically sophisticated, economically productive societies are the ones that end up winning. And so it’s definitely not possible to separate the development of humanity and what humanity is from certain kinds of efficiency. I think even language, all the things that I mentioned, language, consciousness, sociality, individuality, these are solutions to problems and as solutions they actually do work, which is to say they make the system more powerful and more capable. So I think what is powerful and capable is very much part of the question.

And yeah, currently these AI systems they have this patchy, interesting, high power in some areas, high efficiency, high work capacity, and inadequate in other areas, and I guess we’ll just have to see how that develops. My prediction is only that once it gets to the point of actual autonomy or running away on humanity as we know it, that something like concepts of personhood and a sort of xenohumanism will emerge from that because it is natural, because it is the energetically favored way of organizing social intelligence.

In the front.

Q&A: A breakaway civilization? (1:03:44)

Audience: What are the most tasteful or elegant ways to solve second proposition of xenohumanism from your perspective? Maybe just a couple points.

Wolf: Which one?

Audience: The—

Wolf: You mean of my recommendations?

Audience: The second one. Yeah. So—

Wolf: That’s a difficult question.

Audience: Yeah. There are many people I want to hear your specific one because we have history of those people, and it’s very interesting, juicy history, and I want to—hear your points.

Wolf: Here’s what I think is one of the things I meant by that. So I think right now we imagine humanity as a mass of equals, and I think that holds us back simply out of this conceptual inertia. It’s kind of like Harrison Bergeron. We’re implicitly putting weights on the most capable, the people with the highest potential. And I think there’s, especially with AI, this is a very sensitive topic.

I think people would regard it as not very tasteful, but I think there’s the possibility of more of a breakaway civilization sort of thing where the most high agency, the most, not quite to say civilized, but the most organized and capable and rational subset of humanity armed with AI systems ends up overrunning everyone else. Now, that’s not something I’m saying should happen. I’m saying that’s something that is a possibility, historical possibility. These things are not driven by what we want. They’re driven by what’s possible and how history actually ends up unfolding.

But what I am saying is that we have to be open to renegotiating our moral conceptions in response to that kind of thing. And not just in that reactionary way of saying, “Oh, we’re the last man. We don’t like that.” But rather in the way of saying, “We can see potential in that. We can work with that.”

I think it’s a very difficult question, but I think it’s going to be answered the hard way, and if we’re living up to our potential humanity, we’re going to be taking responsibility for that, and creating new values around it. Carol, what’s up?

Q&A: The quiet path to extinction (1:06:12)

Audience: Hi. Yeah, I guess to add on to that, one of the things that happened when the internet was invented was everyone thought, “Oh, well, we’ll all become super smart because everyone has infinity books to read.” In reality, most traffic was for porn. And with AI, that’s probably—

Wolf: Something similar

Audience: going to be the same thing. Right? And if you have robots that you can use to simulate a romantic relationship with, it could just be that everyone who is dumb, just doesn’t reproduce anymore. Everyone who gets addicted to AI—

Wolf: It could be worse. Everyone who’s smart doesn’t reproduce anymore because we find other things to do that suck us in. I don’t know.

Audience: Amen!

Wolf: Amen. He says, “Yeah, let’s go back to the muck.”

Audience: And so it could be that it’s not a violent revolutionary thing that happens, it’s just a very quiet—

Wolf: Yeah. That’s a very important point.

Audience: Everyone just is addicted to porn or doesn’t, or something.

Wolf: Yeah. I predicted that it’s going to be messy, that it’s possibly going to be violent. It’s possible that the next major total war analog is actually not even apparent as a war. It’s just people quietly being snuffed out of existence in ways that we don’t recognize as against our values.

Audience: Half the society just doesn’t reproduce.

Wolf: Yeah. For example. Which is certainly some of what’s happening now.

Audience: Which we could honestly encourage. Give everyone a sex bot.

Wolf: I guess my question here is who’s we?

I don’t want to take up too much time here with questions, so maybe one more, and then we should break up, and you guys can come and target me and throw me off the balcony if you want to. Jonathan, host’s privilege.

Q&A: Why not the last man? (1:08:02)

Audience: Yes. My question is whether you have a positive account for participating in power. What I mean by that is that if you assume that you’re not on the cutting edge, you’re not the livest of live players or an AI, are there reasons to not just participate in the crab bucket if your best chance is to slow everything down maximally and actually lock things down? It’s like there are no rules in Darwinism. Why is the last man not a good solution for a lot of people?

Wolf: Well, I think the last man is a good solution for a lot of people. That’s why I’m saying we should do it a little bit. I’m saying it’s not a total solution. And trying to do it as a total solution is going to be very brutal in other ways, and it’s going to really mess things up and not be a nice path, I think. So my point is just let’s think through these matters morally about what kind of morality we think the future will need, and what kind of values we will need in the future and whoever makes it to the future will need.

And let’s try to provide value within that. And I think that is likely to work, not necessarily for any individual project or for any individual person, but I think that that’s likely to be the outcome, is that we get some kind of shared recognition of, let’s call it a mutual humanity that isn’t totally derived from natural powers, but is derived from the collective legitimacy of being able to recognize that concept in each other, whatever it is. This is a little bit abstract, but basically what I’m saying is there’s especially a role for moral philosophy around this to actually chart the path of legitimacy, like what is the legitimacy of the next order going to look like? That’s a job for moral philosophers. But I think they have to be very—you can’t just derive this stuff from first principles.

It’s very much a negotiation. And I think, for those of us who are not the cutting edge superhuman beings that are currently driving history, let’s live up to that potential as much as we can and see if we can stay on the ship, so to speak. Because I don’t think that’s necessarily determined. But I don’t think there’s a better solution than try to come up with a socially healthy de-escalation of the war of all against all that’s implicit in the development of superhuman powers. And while being also open to the idea that that’s going to mean a tightening of the moral circle, and we might want to have to step up to stay on the boat.

Okay. I’m going to leave it there.

And a round of applause, of course, for Jonathan and the Avalon Institute who are hosting this event.