Ben Landau-Taylor argues that techno-singularitarian predictions are unfounded, while Wolf Tivy defends his xenohumanism program. They cover why theology is inevitable, and why the dominant theology keeps predicting techno-apocalypses.
Ben on Twitter: https://x.com/benlandautaylor
Wolf on Twitter: https://x.com/wolftivy
Transcript
Intro (00:00)
Wolf: So Ben, we’ve had a bit of an obsession with AI on the channel a little bit, and more broadly in the discourse these days in San Francisco, everyone’s talking about AI. AI this, AI that, AI theology, AI doomerism, AI optimism, all kinds of things going on like that. Your writing over the past years has been very cautionary on getting over-hyped in AI, and in particular, you’ve written about how people project onto this apocalyptic view on, and this is just the latest incarnation of the apocalypse, which we’ve seen many times, or that we don’t actually know anything about the future of AI or the future of what happens after AI, the real sort of AI that might come about. So you’ve been kind of a voice of caution, and I wanted to have that conversation then between us on is AI all it’s cracked up to be? Is it actually going to kill everyone?
Can we know anything about what’s happening after AI or with AI?
Why don’t you give us your take on what you think is actually warranted in—in AI prognostication and what is out over our skis? Important stuff.
The case against imminent machine god (01:45)
Ben: I do think there are a lot of people who are getting a lot further with it than is justified in patterns that we have seen several times in the past on a bunch of subjects, including previous incarnations of AI. Also, I think it’s a very important technology, probably the most important technology that has been developed in my lifetime. Certainly top two. Quite plausibly top one. So, it’s a very big deal, and when people are like, “Oh, this is an important new technology and understanding it is very important to understanding the world and what’s happening.”
100% there with it. And then there’s this further thing that a lot of people do where they’re like, “This is ending the age of man. This is going to seize the light cone and determine everything that happens in the galaxy until the heat death of the universe.”
Sometimes people call it AGI, that term started getting walked back. People now call it super intelligence or there’s a bunch of different terms, but the infinitely powerful mind that can do everything—and sort of just wipes humanity out of the water. If humanity persists, it’s kept around as pets of this... There’s this idea that people have of this god mind artificial intelligence—which just totally changes everything, and that people frequently just sort of assume is happening quite soon. And I think that’s—
Wolf: Yeah, people talk about short timelines, like—”Oh, my AI timeline is three years from now,” and—which they mean we’re all going to be dead three years from now.
Ben: Yeah, I was hearing the three-year timelines five and six years ago, and—
Wolf: Right. Yeah, I remember.
Ben: Yeah. And now, a lot of the same people still have three or five-year timelines and good often seem... Anyway. So this is a separate question from the, is this thing important now and economically important for political power? Yes, it is.
And then there’s the separate question of, are we going to get this god mind that ends the age of man and just takes over everything? And I keep asking people who believe this for what’s the specific reason why you think that this is going to happen soon? They’ll come up with this sort of, here’s an argument that this type of mind is philosophically possible. Here’s why these things aren’t unique to the human mind. Here’s why—all of these things could be done with the source of software in principle, and there’s no reason to think that what is done in the human mind is the pinnacle.
There’s certainly a bunch of individual areas where the software is, and has been doing better than the human stuff. Since before I was born, it was better than any human at multiplying things quickly. And the list of things at which it is better than any human has just expanded and expanded and expanded. And so clearly it can be better than humans in some ways. Will it be better than humans in all ways?
And people think that this is coming quite soon, and when I ask them why, they don’t really have any sort of mechanistic explanation. The closest I can get is that it’s currently moving quite fast—in that direction. And like—
Wolf: It is, in fact, moving quite fast.
Ben: In some sense, yeah. So then I’m like, “Okay, well, if you’re looking at whether something is going fast, you have to be like—
Wolf: But how far?
Ben: relative to the dis...” Yes, exactly. How far? It’s like we’re moving at 110 miles an hour. Is that fast? If you’re going between San Francisco and Los Angeles, it’s pretty fast. If you’re going from the Earth to Venus, it’s extremely slow. And I’m like, “Okay, so you see that there’s this certain pace of advance.
Why do you think that this is large in terms relative to the distance that we have to go?” And I’ve just never been able to get a clear answer. I have had quite long conversations with a bunch of the sort of inner circle AI doom people. I have had shorter conversations with many, many of them up to and including literal Eliezer Yudkowsky who is kind of more open than a lot of people are about this stuff, that ultimately it just kind of comes down to vibes. And like, or deferring to the vibes of various people who are purported to have more reliable vibes on a subject like this.
Wolf: Right. So there’s a lot there of different questions in the space, of very different types. Implicitly, there’s other questions behind those questions. I want to get into all that. Why don’t we just continue down on this thread, though?
The case for imminent machine god (06:38)
Wolf: Why do we think AI is coming soon? That seems like a good place to start. So I’ll just give a bit of the devil’s advocate here. Now, for my own view, I think I put a lot more stock into at least the question of the god mind type stuff than it seems you do. That was not raised in what you were saying, but I think there’s also this, you have a sense that maybe we can’t even tell.
We don’t know what the far future is going to look like. Some people make these very confident predictions of god minds of various shapes and sizes. I think we can make some projections there, but not the particular projections that some of these guys do make. But more practically, we have this sense that AI is moving fast and that we feel like we’re close. So let me try to ground those intuitions and see how you respond to that, because I think that would be interesting.
Since the deep learning revolution, we’ve kind of delivered on a very large fraction of what the original AI program was. Back in the ‘50s, when they did the original conference, they basically said, “Yeah, we’ll get some grad students, and over the course of the summer, we’ll solve vision, we’ll solve locomotion, we’ll solve speech maybe at some point.” And like—
Ben: Yeah. They’re still working on locomotion, but they’re making progress.
Wolf: Yeah. But like—
Ben: And they’ve got vision, they’ve got speech. It’s notably getting there
Wolf: But even locomotion. Have you seen the Beijing Robot Olympics? This thing that’s been happening. It’s super exciting. I mean, I love—
Ben: Very cool stuff.
Wolf: I love seeing these super fast bipedal robots, running with these ridiculous gaits and then crashing and literally bursting into flames.
Ben: Right. And it seems like the sort of thing where you just take normal engineering process, give it another single digit years, and it will just be able to walk around a house.
Wolf: Yeah. Which it can already do, and they can go up and down stairs. And we’ve seen pretty good progress on that front. Now, I don’t think we actually have the best possible algorithms there. We’re still doing this kind of deeply pre-trained reinforcement learning type stuff on that.
But I’d say, okay, they’re doing pretty well on that. In particular, they had robots that were playing cooperative tennis with humans. That was pretty cool. Okay, the robot running down the track at high speed, great. That can be pretty pre-baked.
It’s a pretty controlled environment, but playing tennis, that’s impressive.
Ben: Yeah. The dealing with chaos and things, and environments that aren’t perfectly set up has been the hard part that they have been slowly grinding through and improving on.
Wolf: Yeah. So basically, in the last 15 years, let’s say it’s 15 years, right? 2012 or around then is when people start using GPUs and deep learning as the paradigm, right? And that paradigm has, after basically 50 or 60 years of some progress but nothing like that, we finally started really cracking those core problems that the original AI conference thought would be interesting and difficult and valuable. And so it seems like we’ve taken a big chunk out of the problem in the last 15 years.
And if we sort of imagine, okay, projecting that forward, well, let’s say there’s maybe about that much left in the tank, and we’re moving at that pace or an accelerating pace or something. That’s a basis for relatively short timelines to the complete picture of AI, where you have AI that actually covers all the capacities of man fully.
Ben: So that’s the part where it just seems like you’re just making a leap.
Wolf: Yeah. So I am making a leap, right. But this is sort of the shape of the argument, right? I’d like to hear—I guess, yeah, how would you pick that apart?
Ben: Yes. I think it’s very straightforward. There’s a premise in the argument where you’re like, okay, so we’ve done this amount, and—then there’s this amount left, and I’m like, why do you think there’s this amount left? Why not this much? Why not that much?
Where is that specifically coming from?
Wolf: Yeah. I guess, okay, so in this case, the why I said, “Oh, we’ve gone this far. Maybe there’s about that amount left,” is because it’s like we tried to project out how hard the problem was, and it basically consisted of a bunch of these things. And it’s often a good bet that the sort of magnitude scale of the problem is related to what you can see of it. And now, obviously, it can have radically large uncertainty on that.
Sometimes it’s much larger than what you can see. Sometimes it’s like what you can see and then a little bit. And or if you look at sort of the Lindy effect in how long things last, historically or otherwise. If it’s lasted this long, often on expectation kind of lasts about that long again in the future, and then that’s its sort of end of life. So it’s just this, I don’t know, kind of general statistical hunch or heuristic that the size of things is about as big as they look.
“AI winter” and the difficulty of predicting technology (12:33)
Ben: Yeah, and this is a heuristic which just performs abysmally in the history of technological development—and specifically in the history of AI development. There is this thing that has happened several times, depending on how zoomed in or zoomed out you get, you can sort of argue about the number of times it’s happened, where there’s this, a bunch of people get really excited about, oh, the scientists are like, “We have got this much space left between where we are now and getting the computer that does everything a human can do.” And we can see the shape of it, we can see how much progress we’ve made, and a bunch of the people who are sort of inside working on it get really excited. And this ends up driving them forward on their research program. They do end up, in fact, then just making a bunch of notable advances and making better software in ways that are quite important.
And then they kind of like, “Well, we’ve got a bunch of new cool capabilities, and it is not thinking like a man. It is not doing everything.” There’s still a ton of really big holes, and we’ve sort of built a bunch of the individual pieces, but it doesn’t really get us any closer to the sort of fully autonomous. And so there was the first one of these, I think that was sort of a really big, was in the ‘70s. And there was this, I don’t know if you’ve ever read “The Lighthill Report” was sort of one of this document sort of throwing cold water on a lot of the people who were like, “Oh, these guys were really excited about they were going to make the human-level machine by...”
I think it was the ‘90s they were saying, and like—
Wolf: This was in the context of expert systems or something like that?
Ben: I forget which one it was specifically. And this was what is now known as the first AI winter. And we have this term, which is older than you and I—for the time when the AI experts, based on their sort of intuitive sense as the leaders in the field, get their vibes about how far is left to go and then kind of realize, “Oh.”
Wolf: They get out over their skis, and then they realize they’re out over their skis, or someone makes them realize, the whole thing cools off for a while, and then that’s an AI winter.
Ben: Right. And before each one of these, they do, on the basis of this enthusiasm, just make a bunch of very important progress on the software. So when we look at that heuristic of like, oh, people have a sense of it’s about this much, it’s about this much. We just know that the people who are at the forefront of the field very frequently have had this sense and just been wrong.
Wolf: Yeah. And so you’ve pointed this out, not just in AI, but in general, that technology in particular is extremely difficult to forecast in that way.
Ben: Yeah. What if—
Wolf: Life is like extremely kind of long-tailed process.
Ben: Yeah. So a couple of historical examples I’d like to bring up here. One is Leonardo da Vinci. Where he has these sketchbooks where he’s sort of spitballing about a bunch of possible types of technology and types of machines that—he’s—
Wolf: He’s got the picture of the guy turning the crank on the helicopter.
Ben: Right. And then—
Wolf: And then it’s got literally a screw to propel it. Yeah
Ben: Right. And so he’s got something, he’s got an intuition that is sort of remarkably far in the right direction of—the mechanics that could power a helicopter. It doesn’t exactly work, but he’s clearly onto something real.
Wolf: You don’t actually have a guy turning a crank.
Ben: Right. And you don’t literally have a screw. It uses much more complicated aerodynamic principles, but the basic idea is sort of in the area. And you could be like, “Oh, wow. Maybe he’s gone 10% of the distance, so it’ll take 10 times as long.”
No, it takes another half a millennium, basically. Another thing that I think about here a bunch of the time is the development of the steam engine. Where if you look through the sort of different iterations of the steam engine, you’ve got the Savery engine and the Newcomen engine, and then eventually the Watt engine. Some of the earlier ones have sort of these very niche uses in just pumping water out of mines, but they’re very inefficient and no one has figured out how to turn them into rotary circular power, which ends up being the separate condenser, which makes it much more efficient and the ability to turn it into turning a wheel ends up being the things that make the Watt engine actually able to kick off the Industrial Revolution. If I imagine myself, it’s like 1698, looking at the engines of the time.
If I imagine myself looking at 1720, looking at the engines of the time and just thinking about how do you use these techniques that people are looking at here to project forward? I do think you could use these and be like, okay, wow, if you’re able to harness the power of heat in a way that can substitute for the things we’re doing with muscle power and horses and water, I think you can correctly reason that that’s going to be a big deal, but if you’re just looking at the details of the technology in 1695 or 1720, I just don’t know how you tell from that position, oh, it’s going to be in 1776 that we get the engine that actually really kicks it off.
Wolf: Yeah. I guess the sort of fundamental principle is very visible at the time, which is, okay, well, you put coal in one end, you get motive power out the other end of this engine that has been produced. Therefore, you can predict fairly reliably that there is this possibility of the Industrial Revolution—where you replace the muscle power in society with fossil power.
Ben: Right. And people have been locked on to this idea with the software that like, “Oh, wow, as the software gets better and better and can do more things, that’s going to be a big source of power.” They’ve—been locked onto that for not quite a century, and I think that in the main, this intuition has proved right.
The cognitive revolution (18:38)
Wolf: Yeah. And if we extrapolate out what’s the equivalent of the Industrial Revolution here is, well, you can get electricity to do cognitive tasks. And so we expect something like a cognitive revolution where you build the right kind of machine, the right kind of engine. We’ve demonstrated the principle that, especially with any particular kind of task that can be measured, you can basically get machines to do it, and do it very well.
Ben: Well, not any task, but certainly a lot of tasks, and their list is expanding.
Wolf: Well, on any task that can be measured, we seem to be getting progress. And so no, certainly not the case that we have solved all tasks. But in principle, it looks like you can take a task that can be well measured and then apply a huge amount of computing power to find a function that does that task measurably better.
So if we just extrapolate that out as a principle like the principle of the Industrial Revolution, then you get something like, well, there is the possibility of a cognitive revolution like this. Now, to your point, the timing of it all is much harder to predict. Back in the ‘50s when they were projecting it out, like I mentioned, they said, “Oh, let’s get a bunch of grad students and we’ll solve this over the course of a summer or two.” And of course, it didn’t turn out like that. It turns out there’s a lot more difficulty to the problem.
There’s been a lot more mathematics results that have been needed to reason about these things. The particular paradigms that we use to do it were not necessarily discovered at the time. The computational power wasn’t there, the industrial base wasn’t there for it. The applications weren’t quite there to make it sufficiently useful for it to take off economically. The data wasn’t there to get the more interesting kinds of talky power that they now have.
It’s based on language, enormous amounts of data. So there’s a bunch of ways that they just didn’t have the prerequisites, and they kind of couldn’t tell. And likewise, if you think back about the Industrial Revolution, well, you’ve guys like Hero of Alexandria had steam power.
Ben: Yeah, that is—
Wolf: It’s different
Ben: people overstate what his little spinning kettle—people talk as though it was much closer to an actual engine than is often the case, but that’s a tangent.
Wolf: It is a tangent, but point being suppose someone... I think some people have claimed that it’s possible that, say, the Romans used something like early steam engines to pump out some of their mines or something like that. And suppose that was real. Could you then project out the like, “Oh, yeah, we’re definitely going to have an industrial revolution in the next few hundred years?” Well, obviously no, they couldn’t.
Ben: Right. And in fact, it was closer to two millennia.
Wolf: Right. And now that’s of course because of the collapse of their civilization, which maybe that happens to us too, right? Maybe we collapse before we get the cognitive revolution or—
Ben: If we are as far from it as the Roman steam prototypes were from the Industrial Revolution, then—
Wolf: It’s very probable
Ben: then that would be the probable ways for things to go. Are we that far? Then...
Wolf: Yeah. And this is the thing, it’s very hard to tell how far away you actually are. The other example people bring up obviously is right before flight or even at the time that the Wright brothers were flying their airplane, you get people publishing, “Man will never fly in another million years. It’s just impossible. It’s too hard. It’s—
Ben: Space flight, you see the similar stuff
Wolf: Yeah. And of course, a lot—
Ben: You know
Wolf: of that is just like, okay, there’s always some overconfident jerk saying something’s impossible or possible or whatever. But you couldn’t really tell at the time whether they were—going to solve the problem.
Ben: Yeah. From this—
Wolf: Because it in fact depended on a bunch of key technologies like their development of propeller technology, their development of the three degrees of freedom balance of how to actually control dynamic flight, the development of the wings, development of light enough structure that you could actually carry a man and an engine and all this stuff. The development of engines that were good enough to propel it. None of that would’ve worked without a bunch of those key innovations. And you don’t know going into it that you’re going to need those.
The parade of apocalypse prophecies (23:17)
Ben: Right. You don’t, yeah. So it’s very hard to tell how far away it is. And then there’s another thing here which I think is very important to bring in. Which is not on, it’s separate from the question of, okay, in reality, how far away is it?
But it’s the way that people tend to evaluate this. Where we’ve seen in many, many areas for quite a long time, there will be some guys, some very sharp intellectuals, will look at something that in principle could end up destroying the human race. And they’ll be like, “Oh, if you extrapolate out, we are using all of the natural resources at an exponentially increasing rate.” And in principle, if you run out of those, if you run out of the farmland or the chromium or—the aluminum. In “The Limits to Growth” book, they did these sort of plot the exponentially increasing amount of this, make some assumption about how much exists in the world and be like, “So clearly we’re going to run out, and the Earth is going to fall apart and most people will die, and then we’ll end up at this more sustainable, much lower population.”
But like you see, so they’ve got a correct argument that, in principle, there is, in fact, a finite amount of matter on the Earth, in the solar system, in the universe, whatever. There’s a sense in which that is accurate. And then from there, they will make this totally, absolutely wrong by many orders of magnitude projection of when that is going to destroy humanity. In the limits to growth case, they were projecting mass famines in the late ‘70s, early ‘80s.
Wolf: I think some of their models actually had the breakdown around now. But yeah, they were making these projections and in particular, they can be vastly wrong by orders of magnitude about the relative size and importance of the various assumptions. And then this means that it can put you in a totally different equilibrium.
Ben: Right. And there’s a persistent, especially since the First World War, you start seeing the first one, which is people being worried that mass air bombing—terror bombing is going to destroy civilization. And they were right that mass air bombing is just possible with the technology that has been achieved between the World Wars and is a big deal. They were vindicated on that. It was not a good time to be in Dresden or Tokyo.
But they were talking about, literally a lot of these people in the air forces of a bunch of the Atlanticist nations in the British Parliament were saying that this will literally destroy civilization. You can find all sorts of—intellectuals, at the time, who think it is going to end civilization. And in fact, we had it. A lot of people died, it sucked, and civilization just sort of kept on trucking. And—
Wolf: And another, I mean, more less hysterical predictions that they made about it were also wrong. They believed that if you terror bombed a city, it would lead to a breakdown of social order there. It would lead to mass panic. The authorities would lose control. And it would break the morale of the enemy.
And in fact, none of that was true.
Ben: Right. If anything—
Wolf: When you bomb people—
Ben: terror bombing seems to have hardened the resolves of the people getting bombed, which—I don’t know how, I don’t think anyone called that correctly. I don’t see how you could. That’s just kind of sociologically speaking, a very weird surprise.
Wolf: Yeah. I mean, it’s not necessarily that surprising in retrospect, but yeah, I don’t know how—
Ben: Yeah. Very few things are surprising when you already know what happened.
Wolf: No, but it’s like, okay, people become determined against exac—an evil enemy, right? But okay, so it’s very difficult to project out these sorts of things. But this is sort of an abstract argument. And I wonder if we can make more concrete projections.
Predicting incremental technology progress (27:19)
Wolf: What in particular does the world look like where we don’t develop, I don’t know, vastly more advanced AI or it doesn’t, I don’t know, break through to being superintelligent or something. What do we think is actually... I guess, to some extent, we can’t really make the prediction. It’s just you’re saying that people are being overconfident—in whatever their prediction is
Ben: I do think there’s some degree—
Wolf: But I wonder, I—
Ben: you can predict. Especially when you come to the current near-term technology. So in general, when I think about technological advances, I separate things into two categories. There is breakthrough technological advances where you have figured out transformer architectures. You have invented the solar panel.
You have invented—fixed wing flight. And there is incremental progress where you have made the engine on your airplane more efficient. You have made your solar panel cheaper. You have made your battery able to store more power. And I think that people can think pretty well, not perfectly, about the incremental improvements.
And then the breakthroughs are the things that are just tend to come in from out of left field in a way that—
Wolf: Is very hard to predict. Yeah.
Ben: And so you see very frequently with the incremental advances, not always, but not infrequently, they end up following mostly a sort of fairly regular advanced pattern. If you look at battery efficiency over the last several decades, it looks like a reasonably smooth curve. Moore’s law is very famous. Not everything works like this, but a lot of things do.
Wolf: Mm-hmm. Yeah. This is sort of advancing under some sort of scaling law—or as we say in AI.
Ben: Yeah. And people often model the adoption of new technologies with a sort of sigmoid curve of it starts slowly, ramps up more quickly, and then sort of saturates and levels off. And if you assume, okay, we’ve got a new, big, important technology, it’s likely going to follow some sort of sigmoid thing like that of—and in terms of a sort of general adoption thing, it looks like to me, the current AI has meant many things at different times. The current sort of LLM-based, transformer-based, the current incarnation of what AI means seems like we are in the steep part of the sigmoid, basically. Where there’s still a huge amount left of just applying this to the world.
Wolf: Yeah. And like—
Ben: In the sense of—
Wolf: Finding new techniques incrementally to make it more efficient in the training data.
Ben: Right. From the perspective of—
Wolf: Make it scale more compute efficient.
Ben: Right. If you compare it to the adoption of the internet, we’re like maybe in the equivalent of the late ‘90s or something. Where the basic idea of what can this do, assuming no further breakthroughs, there will be more breakthroughs at some point in the future, could be very soon, could be very distant, but assuming no further breakthroughs, you can sort of get a general sense of what it can do. And people are figuring out how to apply that. That then takes often billions of dollars of work.
But if you look at what the people were thinking about where the internet is going to go in the late ‘90s, a lot of them were wrong, but most of the big things were predicted by someone, and there were some people who were just reliably having a reasonably good sense of, here’s what it’s going to end up doing. So I do think that for the current stuff, with no further breakthroughs, I think that that is possible. And then that’s very different from the sort of like... And then we imagine that just turning up the dials on the current stuff that we have is going to turn into the sort of like god mind that’s superior to humans in all ways, or a lot of people I will talk to will be like, “Yeah, I think we’re one or two breakthroughs away from that.” And I’m like, “Where are you getting that?”
And—but bracketing all of that stuff and just looking at the current stuff, I do think you can be like, “Oh yes, here’s what it’s going to do.” And like—you know?
Wolf: And if we make that projection, it looks to me like, okay, yeah, we are sort of, I don’t know about in the middle of the curve, but certainly on the steep part of the curve. It’s always hard to tell—how far the curve goes.
Ben: I would unconfidently guess we’re towards the beginning of the steep part, but like—
Wolf: Right. Who knows?
Ben: this is not an area where I think I can make—
Wolf: Who knows?
Ben: very strong, solid projections.
Is incremental progress enough to reach the singularity? (32:05)
Wolf: But basically, if we incrementally keep advancing along the current trajectory, you do get something that’s quite revolutionary—even to the point of being these automated employees that can do all sorts of things, maybe almost fully automated companies, automated research. They can do quite a lot, but there seems to be a few things that are very sort of pesky, kind of missing, right? And that’s this something about the way they kind of... Well, currently, the current systems don’t redevelop their knowledge over time, right? They don’t have internal state.
They’re a stateless kind of function. And the other thing they don’t do is they don’t seem to have this, I don’t know, strategic judgment of a kind, that they seem to always kind of fall off on that one. And maybe that’s something that’ll be fixed, maybe not. But like, these might be—
Ben: People talk about agency. It’s kind of a fuzzy term, but it’s gesturing at the thing that they still don’t really seem to have—in the way that a person or even a cat has.
Wolf: Yeah. And it’s been surprising how far they have gotten. Or surprising from a skeptical perspective. Three years ago, or two years ago, it was hard to say, like, “Oh yeah, these things will definitely end up... Agency will definitely end up being working,” right?
Like, AI agents will definitely work. Everyone knew you can put it in a loop, right, with some kind of harness, and then it’ll do things. Whether that was going to sort of crash and burn or actually work was kind of an open-ended question, whether the architecture could support that. It turns out, yes, you can put them in a loop, and they can more or less run off for a very long time. They do seem to still kind of lose coherence at some point, or maybe they make bad judgments or something.
But—
Ben: They’re visibly better than they were even eight months ago.
Wolf: They’re a lot better, and maybe that just continues to improve. But it does seem like there’s a few things that are gaps between what we have and what people are imagining by AGI or super intelligence or whatever.
Ben: Yeah. There’s a bunch of these terms that people use in a kind of loose and—
Wolf: Yeah, and I think it’s maybe it’s inherently loose because it’s sort of a hand-waving term, right? It’s like we think that there is this possibility of a computer program that can act with all the sort of open-ended agency and intelligence that a human being has, and we’ve got these things that we’ve built that can do a big chunk of that, but maybe there’s some other stuff left over. But we actually don’t even have good theory of what it is that we have left over. And that’s part of the breakthroughs that need to happen, and we just sort of see this vague possibility. And so I guess to your point, it can be very hard to tell.
A lot of these are unknown unknowns. We don’t even know what the breakthroughs needed are. I can’t tell you what’s the gap between what we have and where it could go.
Ben: Right. But we can look around and be like, “Okay, there’s certain things that it seems to be doing better at and certain things that it seems to be having a harder time with.” Anything where it’s in an environment where everything is perfectly fully specified, it seems to do quite well at. If it’s doing a physics simulation where it just fully understands all of the rules, it seems to do quite well. And then if you put it in the actual real world, that seems to be harder because there’s just so much little slipperiness and tiny messinesses.
And—
Wolf: But they’ve been getting a lot better at that.
Ben: They’ve been getting a lot better at that.
Wolf: Largely by just improving the quality of the simulations and the amount of data and so on, and throwing more simulated noise at it and so on. So you get these better reinforcement learning policies and things like this. And so it’s unclear, I guess, how far that will go. But so—
Ben: And it seems like doing that with small amounts of human intervention for now is just substantially better and—there’s a lot of interesting stuff I’m sure they’re going to be building with that. They’ve got the robot cars just straight-up working, and if you’re in the robot Tesla right now, and you’re taking a 30-minute trip, then maybe over that trip you will have to get called in and take the wheel once or twice. But for the most part, that sort of thing just works, which is kind of—nuts.
Wolf: Right. And so even with incremental advances, you kind of get a pretty thorough revolution of a lot of things. But nonetheless, I think most people nonetheless believe there’s still kind of breakthrough advances needed to get to real AGI or something.
Ben: Yes, that’s certainly what I believe.
Wolf: And some people, I think, don’t necessarily believe that. They think we’re on just the incremental path now.
Ben: Yes, I’ve talked to people in both camps, for sure.
Wolf: What do you think, is that the sort of thing that’s possible to know?
How could you tell if we were incrementally advancing towards some completeness where we’re going to cross the threshold and be better than humans at all things versus... I guess in that world, the way to look at it is AI is worse than humans at most tasks, and then it gets better on one task, and the progress is spiky, but it is advancing on all fronts, an incremental advance will basically get there. Versus one where maybe it’s not even defined by tasks and there’s these more fundamental capabilities that the thing is just fundamentally lacking and not advancing in. How do you tell the difference between those kinds of worlds?
Ben: Yeah. It’s quite hard to do because our understanding of intelligence is basically—pre-theoretical.
Wolf: Yeah, it’s happening.
Ben: You can get strong arguments when you’re looking at an area which you understand well, but when it comes to what intelligence even is, like—
Wolf: We don’t have a good definition.
Ben: No, we don’t have a good definition. We don’t have the sort of scaffolding that could produce that. So unfortunately, this seems to be an area where we don’t understand the object well enough to even precisify what it would mean, which, in my eyes, is one thing that makes it quite hard to have affirmative knowledge that we are going to get there. It’s like, how are we going to get there when we don’t really even have that good an idea of where there is? It would certainly not be the first time in the history of technological development for people to be able to get something without really understanding what they had got until later, but it makes it harder.
Wolf: Yeah. And in particular, you tend to get something different than what you thought you were going to get, if you—
Ben: Right, which has certainly been the course of software development.
If you look at what the computers can do, if you look at the shape of what the science fiction was predicting it was going to be good at, the pattern of where it’s been good and where it’s been bad has not really matched what anybody thought it was going to end up doing.
Wolf: Right. Yeah, you get a lot of surprises.
The theology of the singularity (39:40)
Wolf: So this has all been the kind of practical side, forecasting the actual developments and breakthroughs and incremental progress, but there’s this whole other side of this discourse, which is almost metaphysical, right?
Ben: Right. It’s sort of like—
Wolf: It’s about the nature of intelligence. It’s about the nature of value.
Ben: And a lot of it is—
Wolf: The fate of the cosmos.
Ben: Yeah, a lot of it is once we get the super intelligent god mind, what does it do? What is it like? What are its features?
Wolf: The idea of a super intelligent god mind, right? It’s like, this is clearly beyond sober technological forecasting. Even if there’s something there, it’s not—the same type of thing.
Ben: Right. And you’ve been calling it theology, which I think is a pretty good name for the type of activity that is being done. Because you are into this speculative sort of thinking about... You are not doing a sort of engineering type projection.
Wolf: It’s not science. It shouldn’t pretend to be science.
Ben: Yes. And insofar as... So again, I’m coming to the question I ask with a lot of this stuff, similar to the when is this happening is—as it was said, what do you think you know, and how do you think you know it? And when it comes to a lot of this stuff, so this is sort of what is going to happen after the singularity. And so the original coining of the term singularity, of course, was Vinge naming it after the event horizon of a black hole, where when you get too close to it, no one can tell what’s going on in there because the gravity sucks in all of the light, and no information can escape back to you.
So in his original formulation, it’s the idea of the technology has changed so much that you can’t really know what’s going on at that part. It’s just so different from where we are. There’s so many unknown unknowns from these breakthroughs that we can’t really see. We just don’t understand the nature of the phenomena that the advances are being made, and we don’t really know what’s possible and what’s not. Where I just think that that original analysis that he was making is correct.
If you are in 1710, and you’re like, “Oh, okay, so there’s going to be steam engines and mass production.” And they don’t have the concept of almost anything you can do with electricity. The idea of an electronic device, of a motor, that’s not a concept they have. They don’t really understand the physics that would be needed to build one. If you ask them to try to predict what the technology of the year 1890 is going to be like, they don’t have the prerequisites to be able to be making sensible guesses.
And that’s kind of how I feel when people are like, “This is the nature of intelligence. From the nature of intelligence, we can derive what the shape of the super intelligence is going to be, at least in some general ways.” I’m just like I don’t really think... The sort of process you are using to get there just seems to me very unfounded.
Which theories are accessible earlier? (42:40)
Wolf: Well, let me take the other side on that a little bit. I think certainly if we’re trying to sort of predict details like there’s going to be such a thing as an electric motor or a computer or something, and we’re 300 years ahead of time, no, we’re not really going to be able to predict that kind of thing. But let’s say we took someone. Can we imagine someone like Carroll Quigley, like a 20th century historian with a theory of how history works and how civilizations rise and fall? Could they, instead of being born in the 20th century, be born in the 17th century or the 18th century and kind of predict out a set of possibilities for what kinds of things can happen in the world?
And I would say that I would expect that the kind of reasoning that Quigley did to come to his theories, or Spengler for that matter, or any of these guys, the kind of reasoning that they did to project out how they thought history worked in terms of the rise and fall of civilizations could have been done several hundred years earlier. And in fact, a lot of that stuff was done by the Greeks or Ibn Khaldun or like—and so there’s these things that are maybe about the nature of reality or the nature of complex order or the nature of life that seem to be actually accessible at various times in history, where you can have positive knowledge of those things because you have them in front of you, and this constrains what can happen in the future.
Ben: So I think the reason why when it comes to development of civilizations, why you can see people getting a lot of the correct answers much earlier is because they are in contact with the same object. They are in contact with human civilization. And as we’ve got all of these new technologies and many, many more people, the basic fundamental laws haven’t really changed all that much. You can get larger empires because you’ve got fast transport and fast communication. There’s some stuff like that.
But for the most part, you’re dealing with the same sociological laws of units built out of the same apes. And so it makes a lot of sense that—that they would be able to predict things right. And when it comes to technology, I think a closer example here is going to be Benjamin Franklin. Who was not long before the sort of big industrial transition. And there are some parts where he sees the first hot air balloons and starts writing about, like, “Oh my God, this is just going to totally transform the shape of warfare.”
Just instantly he gets it. Takes him maybe minutes. And it’s a long time before flight ends up. It’s over a century, maybe 150 years, between him seeing those balloons and it being correct. But he understands it.
And then he’s also like, “Oh, and maybe this will cause sovereigns to see the futility of war.” And as Clement Attlee could have told him, that is not what happened once they saw it.
Wolf: And this is where I think, I guess this is the difficulty of projecting forward with AI, because what that is an example of is you can see some things very clearly where it’s like here’s this material nature of this thing. The possibility of this opens up some new set of possibilities, and we can see that that’s going to lead to certain effects, or the shape of history and the shape of life in general is such that here are some hard constraints on what can happen. On the other hand, people mix that kind of reasoning in with something much more theological in the pejorative sense. Benjamin Franklin maybe has this idea like, “Oh, we’ll see the futility of war.” Well, does that just come from Isaiah, right, in the Bible?
The theology of the apocalypse (46:34)
Wolf: They’ll beat their swords into plowshares and study war no more. And literally theology.
Ben: And what’s notable is there’s a very sharp transition in the shape of the theology where it’s just on World War I, it turns on a dime. Where before World War I, it’s this very idealistic, “Oh, we’re making progress in understanding the world. We’re making progress in understanding human society. We’re going to figure out better, just as we’re figuring out the laws of dealing with air, we’re going to figure out the laws of how nations work, and we’re going to figure out ways of all getting along. And maybe the heavier than air flight is going to make sovereigns realize the futility of war.”
And then after World War I, it’s the exact opposite. They’re like, “Oh, it is the nature of man to get powerful technology that he can’t control and destroy himself.” And there’s this theological impulse.
Wolf: It becomes apocalyptic.
Ben: And so if you look at what’s driving the discourse around the mass bombing is going to destroy civilization. The nuclear bombs are going to destroy civilization. Oh, it’s not the bombs, it’s that they’re going to irradiate everything. Oh, we’ve learned the radiation isn’t going to be able to kill everyone. It’s going to be the nuclear winter.
Oh, it’s not going to be the nuclear winter. It’s going to be all of the pollution, that the pollution is going to destroy everything, and we’re going to have the mass famines in the ‘70s. Oh, we actually figured out how to clean up the pollution. That turned out to be fine. It’s going to be the carbon.
It’s going to cook the planet. And then this is at the point of people realizing, oh no, the sea levels aren’t rising. The Americans seem to have been quietly moving on from that and sort of putting it back into the memory hole. The Europeans are still going on it. They’ll catch up in five or 10 years or so.
You just see there’s this very theological impulse to take in principle, there’s a coherent way of imagining the thing that man could destroy himself. And so we have this strong psychological need for it to be very soon.
Wolf: Yeah. And so this is the kind of thing that I think is interesting in this whole AI thing, because what we see with AI is another extension of the apocalyptic worldview.
Ben: Right. And a lot of people are jumping onto that. They’re hopping directly from the—
Wolf: The climate emergency
Ben: global warming, we’re cooking the Earth. They’re just hopping directly off of that boat onto the AI boat.
Wolf: Yeah. Or some people were only ever in the AI boat. Like the rationalist crowd.
Ben: Yeah. We were hanging out in that crowd in 2013, and like—today, those guys were always on the like, “No, you should just be building more nuclear plants. We’ll just—solve our way out of the carbon stuff the same way we did all the other—eco stuff.” Like—
Wolf: But this AI apocalypse.
Ben: But now, now we have found the technology that man is going to use to destroy himself unless we have world government.
Wolf: Yeah. And so the reason the theology angle is important here is because I think this is act, it is precisely a theological claim. That it, and I don’t just mean like it’s metaphysical or whatever, like it’s somewhat irrational, predates the, it’s sort of the conclusion predates the argument. But—
Ben: Yeah, because of these very deep ideas about what is the nature of humankind—what is the nature of our society.
Wolf: And not only that, it’s what is the nature of reality and God and the cosmic story. The apocalyptic idea, I think, comes straight out of the Hebrew Bible. It’s like if you look at this sort of the so-called linear history that comes out of the Abrahamic tradition, it’s like history is this procession, a linear procession from event to event, right? Where events are often apocalyptic in nature. There’s some big intervention of God or there’s some big ultimate judgment on mankind or reality or something.
You’ve got the creation, you’ve got the fall, you’ve got the flood—you’ve got the law, and then you’ve got the incarnation or—these various things. And then you’ve got the second coming of Christ, right? And it’s this, I call it the apocalyptic worldview because it postulates the existence of apocalypses as the structuring narrative of reality. And it’s a theological worldview. Everyone knows that this is theology.
Right? And I think the thing that I find interesting about AI is like, so first of all, obviously people are applying the apocalyptic worldview to AI, and in particular, the sort of 20th century incarnation of the apocalyptic worldview, which pretends not to be theological but maybe it actually still is. This thing that you’re pointing out from World War I where people started having these kind of secular apocalypse narratives.
But even prior to that, there’s also the flip side of the apocalyptic narrative is the utopian narrative.
Ben: Right. Which you see in a lot of these. In the Gaian apocalypse, they’re always like, “Oh, we’re going to have the industrial civilization will collapse and most humans will starve. And then there will be like—
Wolf: And then we’ll have the perfect Garden of Eden that’s sustainable
Ben: right. Man will return to nature. There will be the sort of sustainable 100 million people on Earth after the like—after most of you guys are dead.
Wolf: And it’ll be the end of history in some perfect Eden. And then AI has its own version of this, right? We build the friendly AI. It tiles the universe with good things and wonderful gardens and we’re having fun.
Ben: Yeah, all of your machines of loving grace. Which is, of course, very frequently quoted by Dario Amodei, who is—perhaps the person with the most advanced AIs currently.
Theological epistemology (52:51)
Wolf: Yeah. But, okay, this stuff is theology. Nonetheless, we find it useful to criticize it. And we criticize it as theology or criticize it as unfounded or something. But in criticizing it, we’re engaged in theology.
And we necessarily are sort of postulating a different kind of world that we live in that isn’t the apocalyptic world. We’re saying maybe you’re wrong about the apocalyptic world. We actually live in this other kind of world. But it’s ultimately like we’re still engaged in theology, which is why I try to be honest, like yes, I’m interested in the theology of it. I’m interested in AI as a sort of psychologically and practically and philosophically as a sort of approach to these theological questions.
Because it cracks open a bunch of our thinking in new and interesting ways. It allows us to make new angles on theological questions that maybe have been around for a long time. But—
Ben: And so maybe there’s the interesting thing here is the question of theological epistemology.
Wolf: Yeah. What actually is good theological epistemology? And this is—difficult.
Ben: And yeah. And so when I approach this, I try to take, so I was mentioning, when it comes to the rise and fall of civilizations, people can get that because they are in contact with human civilization. When it comes to technological development, we are in contact with technological development. We’ve been doing it for millennia. We’ve been doing it rapidly for several hundred years.
We can look back at that and be like, what was knowable at different times as people have tried to do this? Where were they right? Where were they wrong? What sorts of things could you call? What sorts of things can’t you know?
I try to take that much more, given the track records, what is knowable, what is not knowable, and like—which epistemologies, which ways of looking around at the world you’re in could have worked in the various, like as I back test these, which of these can get correct answers? Which things are knowable and which things are just not knowable if you take the epistemic process you’re using and project it backwards?
Wolf: Yeah. And so let me bring an angle on this, which is that I’ve been reading Nietzsche’s “Zarathustra” with some friends, with some students, actually—some of our folks.
And in recent chapters, he sort of has this railing against theology, actually. He’s saying, “God is a conjecture. You don’t actually have any way of knowing anything about that. You should confine your conjectures to the realm of the humanly conceivable, the things that are actually accessible to you, that you can touch, that are actually part of your reality.” And this feels very related to what you’re saying.
That you are, in fact, able to make progress in the areas where you’re in touch with the real thing. And where you’re in contact with the real thing, you can look at it and say, “Oh, here’s the pattern we notice. This has now expanded our world. We now live in a bigger world because we’ve come into contact with this.” Like in the way that early modern man rediscovering the story of ancient Rome through all these manuscripts and archaeological kind of finds, they start to come into contact with the reality of the rise and fall of civilizations.
And now because they’re in contact with that reality, they’re able to start building a worldview that includes true claims about the rise and fall of civilizations. And that we live in the kind of world where there are civilizations that rise and fall. And I think that maybe is a good basis for, let’s call it, good theological reasoning. But in other words, maybe you’re not going to make progress from a priori synthetic first principles of just making up ideas of what the far future’s going to be like—outside of your domain of contact with reality.
Ben: Right. You can sometimes go one or two steps with this a priority reasoning from first principles, but when you’re stacking 11 layers of it on top of each other, that’s just never worked.
Darwinian theology defeats the apocalypse (57:14)
Wolf: Yeah, you’re done. Yeah. You’re over your skis. Now, what I will claim is that in the late 19th century, we got into contact with new parts of reality, and those new parts of reality are, in particular, geological time and the related long history of biology via the Darwinian kind of—theory and fossil record. And that gives us suddenly this huge vista on the nature of life, on the nature of the world, and we can now—
Ben: It turned a lot of people’s spiritual world upside down.
Wolf: It absolutely did, and I think rightly so, right? It is, in fact, new theological breakthroughs, right? New theological revelation in a way, that we have come into contact with areas of reality previously unknown and now have new knowledge about the type of world that we live in. And I think, so when I try to project forward into what happens after the singularity, so to speak, and try to put bounds on that and try to critique some of the theological claims of the apocalyptic worldview, in particular, what I think is the crux of what I’m doing or what I want people to do is take the geological time worldview and the Darwinian worldview and the astrophysical worldview. We now are in empirical contact with a very broad swath of time and a broad swath of the nature of the universe.
And if you take that worldview and you crash it into the apocalyptic worldview, and in particular into the AI incarnations of the apocalyptic worldview, but also in general, I think it demolishes the apocalyptic worldview, and you end up with a very different worldview. It’s no, there wasn’t a perfect Eden at any point in the past. The world is not defined by these apocalyptic events and utopias or these sorts of possibilities. It doesn’t appear to be moving in a straight line from start to finish in a rote narrative, but there is nonetheless direction to it. You can look at it, you look at the evolution of the universe thermodynamically, biologically, geologically.
We know things about how these things progress. And I think that you end up with a worldview that is much more about the rise and fall of living organisms or sets of living organisms and species, ecosystems, physical systems. And the kinds of projections that I think you can make from that worldview into the future of what AI might do, I think there are some valid projections. And in particular, especially as we come into contact with parts of mathematics that are maybe relevant to this, the projection that I like to make is that this fundamental dynamic of incremental development of life and physical history continues with much the same laws into the future. And that’s often a claim that I think the AI apocalypticism doesn’t like or that runs into it or runs afoul of that.
In particular, people like to imagine that there will be an AI being that’s going to revolutionize the world and create some kind of apocalypse utopia effectively making itself the demiurge. Some material system’s going to promote itself into being the demiurge. You call it a singleton or something. And I think that sort of stuff is just coming out of the apocalyptic worldview, and I think we can actually dispense with it with relatively well-grounded arguments in terms of the kind of world we apparently actually live in according to the kinds of things we are actually in contact with. And where it postulates new breakthroughs in mathematics or in philosophy or something to underpin that singleton worldview, the apocalyptic worldview, or to salvage it, I think we can point out how those breakthroughs haven’t happened and are seemingly contraindicated by what we have seen.
In particular, the claim of the apocalyptic result from AI of promoting itself to be the demiurge singleton requires mathematical breakthroughs that are postulated, in my view, as a result of the apocalyptic worldview. They say, “Oh, we believe in the apocalyptic worldview. Here’s what that would have to look like with AI. Therefore, AI is going to have this perfect self-surveillance capability that relies on this new mathematics that’s going to be developed.”
Ben: Yeah. It’s sort of the back chaining from the psychological void they need to fill into the shape of the thing that they are expecting to find.
Wolf: Yeah. And then when you go and actually study those branches of mathematics, it’s like, no, it doesn’t quite look like that. That actually looks like an impossible problem or an extremely difficult problem. And the Machine Intelligence Research Institute, Eliezer’s thing that tried to actually solve that branch of mathematics, basically failed. In my opinion, because it can’t be solved because there’s nothing there, actually.
The limits of theological reasoning (1:02:51)
Wolf: It was bad reasoning that got them there in the first place.
So this feels to me like, I don’t know, well-grounded theology to engage in that kind of extrapolation from what we are in contact with and critique of, let’s say, false extrapolation. And so I’m curious what you make of that kind of reasoning, because I know you find this stuff a lot less interesting than I do. Possibly because you think it’s not as possible as I do.
Ben: Yeah. So I think that you can come up with reasonably okay guesses. I think you can do better than walking on air. But I don’t think you can be walking on concrete. You can maybe be going through a swamp or something where it’s kind of supporting you, but kind of sinking beneath you.
And so I think—one of the most influential, most important, biggest attempts to take this sort of Darwinian reasoning and apply it to the understanding and projection of society was Malthus, where he was like, “All right, so we’ve got these population dynamics that have been working for everything—up until now. And what happens if we just take that and project that onto what’s going to continue? So if you analyze human societies in terms of these sort of food constraints, resource constraints, then you’ll get dynamics like this. But the premise just didn’t hold because—technology kept advancing, and people were like, “Oh, so we’ve got this new thing, and then we can project what’ll happen when we run out of that,” and then something new comes out. And so, Jevons was like—
Wolf: I think the interesting thing that actually falsified Malthus’ projection is that, as a result of some kind of technological and social developments, humans turned out globally not to be always fertility maximizing. And that’s the interesting falsification, is our carrying capacity of society as a... Never mind resource constraints and so on. The resources we already have, we haven’t made full use of, because we’re not actually fertility maximizing. We could live in a Malthusian world where we’re the things always get bigger.
Ben: If everyone was having 11 kids like the settlers who came over to this country, then like—but
Wolf: Then maybe we could have a debate about whether population growth can be outrun by technological growth. But that’s not even the world we live in. The world we actually live in is one where, for some reason that we don’t entirely understand—
Ben: Right. And so this is where you get swampy, because how many things like this? I don’t think that there was a good way for Malthus to have reasoned to what if that—
Wolf: What if there’s this demographic transition where people—stop having kids for some reason of modernity. We don’t know.
Ben: There’s just so many things outside of this that could come up and hit you upside the head. And so—there are senses in which Malthus is, in fact, right. But those senses do not end up making it an actually useful way of projecting forward. And so there’s just—
Wolf: Right. So it’s like he’s maybe right about some things in theory, but then the actual projection becomes impossible because of these unknown unknowns.
Ben: Right. And so when you start trying to use this logic to project forward to what the super intelligent mind can do, I’m just like, okay, but there’s so many things that if you have to—
Wolf: Right. We can’t actually tightly constrain.
Ben: If there’s 100 of them and you get 98, then your answer is nonsense.
Wolf: Right. And we can’t actually tightly constrain the future into any particular view of what it might be like. I think we can still do good science fiction, where if you get 98 out of 100 on good science fiction, that’s pretty damn good science fiction.
Ben: Right. And I think that doing that is a useful exercise, which is going to give people a better understanding of what’s going to happen than pure ignorance. But I think that one should understand it as relatively hard science fiction rather than as rigorously proved laws of the universe.
Apocalyptic theology, tyranny, and creating the Antichrist (1:07:13)
Wolf: Yeah. To sort of salvage my project here, the part that I find most interesting about it is what the possibilities of the future imply for what we should be worked up about and not worked up about right now. In particular, I was having this discussion with Richard Ngo previously on the show.
One of the things, the biggest sort of impact today of belief in the sort of singleton demiurge as a result of AI is that people are engaged in this no-holds-barred, astronomical stakes competition—for who gets to control it.
Ben: They’re trying to build it.
Wolf: Who gets to control it, right?
Ben: Right, yeah. And if you don’t think it’s real, you’re not going to be in that competition.
Wolf: Right. Or even if you think AI is massively important and massively real, you’re not going to go about it in that desperate arms race, sacrifice all other values way.
Ben: Right. And you see all of these sort of board maneuvers that come from people having this belief about what the thing is going to turn into.
Wolf: Right. And it’s like, okay, if that’s not actually the kind of world we live in, and we can have that argument and reasonably well demonstrate that now, then we can also make the competition around AI and make people’s attitudes around AI less destructive of what we’re doing right now. I think there’s good claims to be made that the way we’re going about this AI stuff is more destructive than it could be because of belief in that apocalyptic power of it. And so I think the interesting thing to do is by challenging the idea of the apocalyptic power, you also challenge the basis for some destructive actions that are happening right now.
Ben: Yeah. And in particular, the sort of, “Oh, we are going to need to like...”
Wolf: We have to accelerate at all costs. Or we have to control it at all costs
Ben: with all of these, going back to as early as the interwar, “Oh, the bombing is going to destroy civilization.” So then you have Clement Attlee, who would later go on to become the Prime Minister of Britain, was saying, “So we need United Nations control of civil aviation.” Every time you get the, “Oh, the technology is going to destroy humanity”, the solution is always, “So we need this sort of—
Wolf: Global government
Ben: international schoolmarm.” “We need the bureaucrats to be in charge so that they can shut it down.”
Wolf: Yeah. To put it in theological terms, it’s apocalypse, therefore Antichrist, right? It’s like there’s going to be an apocalypse, therefore we need to have the Antichrist. And we need to build the world-spanning system that by force kind of enforces peace and safety.
Ben: Yeah. Bostrom is like, “We need all of the chips to have the mark on them so that they can be centrally controlled.”
Wolf: Right. Exactly. And so it’s precisely the same result. We talk about in AI alignment, the pivotal act, right? Well, what is the pivotal act?
The pivotal act is to set up a totalitarian world AI surveillance superstate that prevents anyone from developing dangerous AI. Because it has this apocalyptic potential, it legitimizes and demands a total tyranny.
The leap of faith (1:10:46)
Ben: Yeah. And so to step back from here’s the specific way that this manifests in the AI apocalypse case to your general point about epistemology, yeah, I think that in spite of all the limitations, it’s kind of inevitable to do this at some level anyway. Because you have to figure out what’s important. You have to figure out what world—you’re living in, even with a lot of very critical uncertainty. You can’t just not ask the question.
That’s not really a thing the human mind can do, and it’s not a thing that—you would want to do even if you could.
Wolf: Right. And yeah, we in fact existentially depend on which world we live in.
Ben: Right. And so I think that it’s an important question and—
Wolf: But I guess I like this sort of, to bring it back to the sort of synthesis that I think we are agreeing on about what sort of reasoning is possible here and what sort of reasoning is appropriate, a kind of Nietzschean caution around theology. Right? That it’s like approach theology through the humanly conceivable, the things that are actually accessible to you. And—
Ben: And there are some important questions that in fact you can’t answer from that, which is—unfortunate, but there is nothing about the universe does not have a user manual that promises you—epistemically justified answers to all critical questions.
Wolf: No, and I guess my answer to those aspects is always the leap of faith.
We make some speculations. We necessarily make some speculations beyond the humanly conceivable because you have to kind of complete the picture, right? There is, in fact, something beyond what we have and many of the important questions are answered in the gap between what we have and what is. Right? And so, okay, we’re going to make some speculations about what kind of world we live in and we’re going to act in that way.
Ben: You’re going to sort of look at what’s worked in the past and make an educated guess about what might—work in the future.
Wolf: But we can’t know that we got it right. And so therefore, there’s a leap of faith involved, right? And any civilization is inherently taking a leap of faith. One of the big leaps of faith that I think our civilization took correctly was a leap of faith on natural philosophy. In the middle ages, deep in scholasticism or deep in the dark ages, I don’t think it was knowable that studying mathematics and studying the natural world as natural apart from revelation and authority—
Ben: Right. But this was a very—
Wolf: That wasn’t a knowable thing at the time. But there was a tradition, the philosophical tradition that we’d inherited from the Greeks. I don’t think it was even knowable for them, that they got onto it themselves. But there’s this tradition of studying the world in a particular way with the belief that that’s going to unlock something of great importance. And it turns out that it did.
Ben: Right. And the medieval people who were doing the beginnings of that, and frankly still today, a lot of the people who are best at it—were less of the, “Oh, and if we do this, we will be able to get slightly better technology and have GDP continue increasing at 3%.” They were much more like—
Wolf: It’s a theological project
Ben: “We will come to know the mind of God.” We will be—
Wolf: It’s explicitly a theological project.
Ben: Yeah. James Watt, when he was working on the steam engine, he had a line along the lines of... I’m going to be slightly misquoting it, so take this as a paraphrase. But, “Nature has a weak side, and if we can but find it out, then we can vanquish her.” Or if you go earlier, a lot of the—
Wolf: Right. Which is, I would call that an apocalyptic view, but yeah.
Ben: Yeah. Well, there’s no shortage of those in the tradition. So there’s a lot of this stuff then and now, and I’m kind of in a weird spot because I think that the apocalyptic ideas of a lot of the people currently working on the AI has driven them to make these really cool things that I’m super glad exist because unlike them, I don’t think it’s going to be apocalyptic. And I don’t think they would have made them if they didn’t have these views, which I think are wrong, about what it’s going to do.
Wolf: Yeah. But I think it’s worth the criticism to say, okay, but your apocalyptic view is actually wrong, and it’s leading you to plan the use of this technology, and to attempt to develop this technology, and compete against others in developing this technology in destructive ways that are harmful to your actual interests.
Ben: Right, and because—
Wolf: And harmful to the rest of us as well
Ben: and because I share the leap of faith on natural philosophy, even if I did think in a Machiavellian way, “Oh, I should let them keep going with that,” I can’t really do that. I have to just be like—
Wolf: No, we—
Ben: “No, actually, I think that that’s not grounded. I think it’s there for these other reasons.”
Wolf: Right. Yeah. Because we are partisans of philosophy, we have the faith of philosophy. We feel bound to actually pursue the truth in the matter. What is actually the case, not just what is expedient, not just what is useful socially or whatever.
Ben: Yeah, that is kind of the most fundamental leap of faith behind everything the two of us have been doing.
Wolf: Right. And yeah, again, we’re the students of philosophy, right? To complete the kind of concept there. And yeah, and I guess it’s worth emphasizing, and I like to emphasize that philosophy is ultimately a theological thing. It’s almost useful to attach gods to it.
But, I won’t go there right now. But I guess, yeah, there’s this inescapability of philosophy. You’re always making a leap of faith about the value of things sort of beyond the world that you can actually see. And maybe the most grounded and best such extrapolations are ones where it’s like, as far as you can tell from everything you can see, that might actually be true. And maybe you can’t know it, but it might actually be true.
It’s the kind of thing that could be true. And as opposed to a noble lie where it’s like, okay, here’s the things we can see are true with the best of our possible ability, with all the philosopher’s wisdom that we have on hand. We think this is true, but we’re going to tell people that this other thing is true because it’s useful to us or something. Or because some metaphysical reasoning compels us to think that way. I think that that’s very much in error, but a careful extrapolation from what can be seen can get you to things true and unseen, but it is inherently a leap of faith.
And this is, I think, what the ancient Greeks did with philosophy and inventing philosophy. They made an extrapolation from what could be seen to what couldn’t be seen correctly.
Ben: And when you’re doing this, you have to keep tracking very closely which things are leaps of faith and which things are more empirically proven, because it is very easy to get out over your skis.
Wolf: Yeah. And in “The Republic,” for example, where Plato lays down a lot of the ideology of philosophy, that’s one of the things they do, right? Is this sort of very rigorous kind of categorization of what’s opinion, what’s fact, and what’s knowable by reason and what isn’t, and these sorts of things. And yeah, so I guess to bring it back to the AI question, I think there are extrapolations we can make that are within... Well, there’s the things we can see, which is what we have called kind of the realm of incremental progress, let’s call it.
Right? We can see the things that have been done and the things that are sort of so near at hand that it appears to be pretty baked in. And maybe that can even include relatively long-term projections like that life is going to continue to have this and that competitive aspect or whatever that it’s always had, or it’s always going to continue to have this thermodynamic aspect as it has always had, whatever sort of extrapolation we want to make from geological history or other things. And that’s the sort of immediate. Then there’s the more distant kinds of extrapolation where we make claims like maybe this is more like my xenohumanism type thinking, where it’s like, okay, I’m going to make somewhat more educated guesses about the kinds of minds that can exist in the world and what we might call those and how we might relate to them in terms of value.
And maybe there’s something there. Maybe that’s well-grounded extrapolation, maybe it isn’t. It’s definitely theology, but I think we have to do something there. We have to do some kind of extrapolation to those things because we’re heading in that direction, and we have to make bets about what’s there. And I think my big project is to say that the apocalypticism is a bad bet.
And it’s not well-grounded. It comes straight out of a Bronze Age book.
Many things are actually very wise in Bronze Age books, but I don’t think they had good reason to believe that the world actually worked like that, and I think we have better reason to believe that it doesn’t. And therefore, we should try to ground ourselves as much as we possibly can with AI in the realm of the humanly conceivable and the things that are actually available to us.
Ben: Yeah. And I think I reach a similar conclusion from trying to focus on what seems to me like a more grounded—chunk of the things we can know, but I do—
Wolf: Sure. I’m very interested in the theological side. Right. As theology. But yeah, this apparently is a very different way of thinking about the thing than everyone else.
I think people tend to be either on the very practical, how do I make money with this right now, or what’s immediately going to happen, what’s the sober projection of it, or they’re on the theology side, and they’re talking about utopias and—
Ben: Yeah, trying to make acausal trades and so forth.
Wolf: Yeah, acausal trades. And in some cases, actually, it’s the same people doing both, but they don’t tend to be, I think, relating those two things in the way that we are. And so I always find that interesting when it’s like we’re thinking in a very different way than other people are. Maybe we’re wrong, maybe we’re right, but I think we’re—
Ben: Both of those has happened before.
Wolf: I think we’re right, though. I think the apocalypticism is wrong. I think the historical naturalism, I will call it, is a much more grounded approach. Okay. Well, let’s leave it at that.
This has been a fun conversation. Thanks so much, Ben, for explaining your thoughts on what we can know and not know in the realm of AI prognostication. So—
Ben: Yeah, until next time.
