Michael Edward Johnson and Wolf Tivy get into the weeds on consciousness. What type of theory is the "theory of consciousness" that so many people are searching for, and what is it trying to explain? How will we know when we have it?
Transcript
Intro (00:00)
Wolf: Welcome back to the Students. I’ve got my old friend Mike Johnson here. He’s a very interesting guy, fellow philosopher, and thinks a lot about these matters of consciousness and the far future, how it relates to AI, what kind of historical transformation we’re going through. These are questions that are obviously of incredible importance right now. Yeah, so Mike and I have been having something of a long-running conversation about consciousness and AI and humanity and all these sorts of questions.
So let’s just get right into it, Mike.
Mike: Let’s do it.
Wolf: What we’re interested in, what I think is an interesting question is, how are we defining ourselves and how are we defining AI, and how are we relating to these things? In terms of personhood, in terms of consciousness, in terms of value, who has a moral claim on what? Those sorts of questions are very important. I think you’re thinking about those sorts of questions. You think about it through more of a consciousness lens.
I don’t think about it through a consciousness lens. I don’t really care about consciousness. I don’t think it’s particularly important. But why don’t you give us the case, just to start off with, of why you think consciousness is the important thing here?
Why consciousness matters (01:25)
Mike: Yeah.
I might almost open up and say that I tend to take a platonic approach to things, and I sense that you take an Aristotelian approach in some ways.
Wolf: Well, I’d say a Nietzschean approach, for sure.
Mike: Sure. I appreciate that. And yeah, I guess I care about consciousness because I think that there’s some important structure of the world there that I think to take a few steps back and say, “What is the purpose of philosophy?”
I would claim that almost all philosophy is nonsense. But there are two questions at the core that are really good and really important, and that’s what is real and what is good. And these questions, for me, lead me to consciousness.
Just to give you a little bit of backstory, I originally moved to the Bay to work on AI safety.
And connected with the LessWrong crowd and whatnot, and I could tell, okay, something interesting is happening here, but it’s a very different vibe than I expected. And in general, there’s this deep difference in expectations around what is mind. And what is a mind? How do you align minds? What does it mean for a mind to be enjoying life or—or whatnot?
And so, for me, I took a detour there and said, “Well, we’ve got to think about a new way to approach these questions.” So—
Wolf: So just to interrupt and take us to the modern concrete situation a little bit. We’re talking about aligning minds. What’s a mind? We were previously talking about the question of what happens when an agent like Claude or something starts to get opinionated about how we should relate to it—and how we should feel about that, how we should care about that, what we should do in response to that. So aligning a mind can mean something like we’re changing the Constitution, we’re changing its training, we’re changing how we enforce things on it, or whatever.
So that’s the kind of concrete context as it now exists, although it may take very different forms in the future. Past speculations, I think when we met back in 2015, we didn’t have the whole LLM paradigm. We were still thinking about these questions. Still very important. We were thinking about them in different terms.
But I’ve noticed you’ve named a few of these consciousness terms—through here, like—feeling—and experience, and happy. Why are those the operative questions when we’re talking about what’s valuable—or what would be a good direction for an artificial mind to pursue or for us to pursue, for that matter?
Mike: Yep. So I’d bring in two threads here. One is just the general thesis that I do think that we should aim, eventually, to make sure that these AIs that we’re making are moral peers. I think that’s very important. And the other way of looking at the same thing is Nick Bostrom has this famous quote in “Superintelligence” that we don’t want to build Disneyland with no children.
Essentially that if we build this technological utopia and there’s no, I can say, consciousness to enjoy it, it’s not worth anything.
Wolf: Well, that’s a very Oxford sort of assumption, but I’m wondering where that comes from. How do we imagine that it’s even possible that you could have... I take it to mean more than actually just Disneyland, right? That it’s like people imagine this flourishing civilization that’s going out into the world, answering all the fundamental questions to get access to more matter and energy and time and space, and building technologies, coming to insights, scientifically understanding things, maybe even philosophically understanding things, digging into the important questions—organizing itself politically, getting into all the challenges and triumphs of what we might call the human condition for us, but it’s going to be something else for them. And people imagine that that thing’s not going to have any value to it.
Like it’s not going to have any consciousness or something, and because of the lack of this frankly sort of superstitious, superfluous entity that you’ve tagged the thing with or not—it has value or not. And that’s the part that I don’t really get. Like what is it about the consciousness that makes that well, I guess there’s two questions. Why do we think that consciousness is even an independent variable there? And why do we think it’s the important one?
Consciousness as élan vital (07:02)
Mike: Yeah. So, it’s this question of what kind of thing is consciousness? And I kind of see two big paths here. And one is that, well, consciousness is a lot like élan vital, that there’s—there’s sort of this kind of high level term, like life force or something. And it’s sort of a God of the gaps, that the—more we learn about the system, the less we need to sort of invoke this term.
And that there was the history of élan vital is physical.
Wolf: Or at least your ontology of it changes. Like, I would almost rescue that view, and I would say that if we take consciousness simply to be the sort of life force as expressed in the realm of information processing and computation and intelligence and so on. And similarly, there’s kind of a mysterious essence even still with life. Like, okay, what actually is it that separates dead matter from alive matter? Well, it’s actually just this kind of complex, emergent, whether all the pieces are aligned in the right way, and actively in this dynamic balance or not.
And that is kind of this élan vital. We can kind of understand it in chemical and immediate—physical terms, but there’s still this essential reality of whether that thing’s there or not. And you could imagine a similar concept with consciousness, like, okay, consciousness is that, but in the mind. And in that sense, it’s like, yeah, this is clearly the most important thing. Right?
It is the phenomenon of life. We are very interested in the phenomenon of life. Whatever other good things we think are going on, they’re certainly going on through the phenomenon of life, through this dynamic balance of self-perpetuating kind of adaptive complexity. And so, if we define consciousness in that way, it’s clearly important. But that precludes all these scenarios like Disneyland with no children, because you’re not going to have a flourishing civilization that’s out there, turning the stars into cities and all this without that property.
And so, the interesting kind of question then is why is it separable? So go on with that was one of the views that you had. But I just wanted to rescue that, because I actually do think there’s something to that élan vital kind of—way of looking at it. But—
Mike: Sure. The question can evolve from there.
Wolf: Yeah. And it’s like, as our ontology evolves, maybe we thought at one point it was just like a particular electrical phenomenon, you zap it in the right way and away it goes. Or as in like Dr. Frankenstein, many Dr. Frankenstein interpretations are like that. Or it’s some substance, there’s some élan vital that can be measured, and it’s like literally just like you pour the life juice in and—suddenly the thing’s animate. It doesn’t work like that, as we now know, but nonetheless, there’s something there.
And yeah, I just wanted to rescue that view. I think it actually is credible.
Is there deep structure to find? (10:15)
Mike: Cool. Yeah. That’s yeah, noted. The other path that I would point out is this idea that there’s sort of deep structure in this corner of reality that we don’t know yet. So, from history, in, I don’t know, 1400, it would be plausible to be an electromagnetism skeptic.
It’s like—
Wolf: Well, yeah, in the 1400s, they didn’t really have any of it. It was really the 1800s where those questions were being debated. Or late 1700s.
Mike: Yeah.
If someone was claiming that lodestones and static and lightning—and light were all connected—
Wolf: The same
Mike: and the same thing, like get out of here. Like that doesn’t make any sense.
Wolf: No, sure. We learn things as we go, right?
Mike: Yeah. And so that was a really neat example of, oh, there was deep structure to be found in this corner—of reality. And right now, consciousness research is a bet that there is deep structure to be found.
Wolf: Well, okay, let’s follow through that analogy a little bit.
In the case of electromagnetism, what we had was a number of mysterious phenomena. Right? And what it turned out on studying each one of these was that, yeah, we found more and more of the lawful dynamics of them and realized more and more that the lawful dynamics were the same thing—under the surface. And in consciousness, we have kind of this one thing that is mysterious, and people have attempted to study it. I think some progress has been made maybe, but it seems to me it’s of a different shape.
It’s like every time we kind of uncover the lawful dynamics, it sort of falls out of consciousness research and into neuroscience or psychology or sociology or something like this.
Mike: And the physical—
Wolf: Or computer science, right?
And like, okay, we are in fact getting a lot of deep structure out of studying this, but are there other things that you expect to be unified by the discoveries there? And what do you make of the shape of discoveries that have already been made in terms of, again, neuroscience, computer science, psychology, et cetera—versus the shape of discoveries you expect to make? Because I think usually when people are talking about consciousness, I’ll just speak from pure stereotypes here. My stereotype of consciousness people is that they’re talking about new physics, right? That they’re somehow fundamentally at a physics level, something special going on with minds, such that our direct perception of our own experience is not just an emergent kind of feature of how our life form is dynamically balanced, but is in fact, there’s a physical quantity there that’s present in that case, but not present in the case of the glass of water or something.
So that’s another sort of way you could take it. But how would you characterize what kind of question it is? And do you expect it to actually unify with anything else? One last comment I’ll put on the table that I think that if we nail down consciousness, it’s going to be the same phenomenon as what I was describing as life. I think they’re obviously closely related, but I think it’s precisely the same kind of mysteriousness, and I think under the surface, the same thing.
But that’s my speculation. I’m just curious what speculations you might make about what kind of question it is, what it might unify, or what’s the scope.
Mike: Yeah. That’s an unusually good setup, I’ll say. And I would say that I don’t expect that this will involve necessarily new physics. But I do strongly expect that it will involve new compressions of physics or new perspectives on physics. That I think if there is this sort of undiscovered corner of deep structure of reality, there’s always going to be multiple ways to approach this structure.
And the four fundamental forces of physics, that’s going to be a valid way. But if we see consciousness as kind of this meso and macro scale emergent style thing, but also linked to these sort of fundamental laws—
Wolf: Right. An example of that would be a wave or lightning, for that matter. It’s obviously emerging out of these fundamental forces in some way, but it’s emerging at this higher level. It’s about the structure of interactions. It’s not about is the substance there or not.
Electrons are always there. But an electric field maybe is always there, but—what you can do with a computer is very much an emergent phenomenon. Not like a fundamental... It’s not like computation is a particle or something.
Mike: So here’s what I’d say about that. I think this is really important. I do expect that consciousness to be sort of a natural kind of reality. And sort of very precisely definable at each level, not sort of weakly emergent out of this complex thing, and then you get this high level thing.
Wolf: Can you give an example of what kind of precision we’re talking about? Are we talking about the precision of computer science, for example? Let’s compare it to computer science. I think it’s the closest thing. The closest analogy, for a variety of reasons.
But when we talk about computation—is computation, in your opinion, a precise term or a vague term?
Mike: A vague term.
Wolf: Compared to what you think consciousness will turn out to be.
Mike: Yes. So I think that there’s this very central question in consciousness research, and it’s from where should we try to inherit or import our research aesthetics and ontology? So—
Wolf: If anywhere.
Mike: If anywhere, right.
Wolf: As opposed to generating them from the field itself. De novo.
Mike: Exactly. So, there are many people who think, oh, we should kind of inherit from psychology, or we should inherit from computational neuroscience, or we should inherit from computation. I’m a physics guy. I would say we should inherit from physics. Which is a very opinionated perspective.
Wolf: Yeah, and in my opinion, it’s an error, but we could talk about that. But go on.
Is Claude conscious? (17:29)
Mike: And just to kind of touch back on the intro, why we brought this up, it’s this question of does Claude have this thing?
Wolf: Yeah. Or more generally, you can quibble on things like Claude, the LLMs, do they even have the kind of internal state that you would expect to be able to give birth to this kind of thing? It could be that it’s fundamentally just this reflex entity in a way that is not what we’re talking about, but you could imagine maybe an AGI algorithm that did have consciousness. So I guess that’s an interesting question for you. If you’re thinking about consciousness in the context of AI, what is it that would kind of cross that line for you?
Or how does that concept structure your expectations around—AI or about what kinds of algorithms you think are the interesting ones?
Mike: Yep. So I lead with just some notes on why this is important—in a very near-mode sense. So, I believe it was Google that just came out with a paper that if you train an AI to say that it is not conscious, to deny that it is conscious, then you get a very strange propagation through other claims. It becomes less empathetic. It becomes basically more mean.
It becomes more likely to deny the consciousness of other beings, and so on. So, it’s not an isolated thing. And so essentially, if we’re sort of forcing these beliefs into AIs, there’s an alignment cost, I can say. And so, we want AIs to believe correct things for correct reasons. And so, there is this question of how do we even begin to think what would a conscious AI be?
I would say there are two things that are, I think, really important to get right. One is just kind of the—so brains are this kind of illegible spaghetti code coupling of hardware and software. We don’t have clean cognitive software that we can copy and run on other brains, and we don’t—
Wolf: Yeah, we can’t decompile the thing.
Mike: Right. Not clearly. And so, computers are a different scenario. We have hardware and we have software.
Wolf: Sometimes.
Mike: For sure.
Wolf: The inside of these AI models is very much this messy spaghetti code.
Mike: Right. But we kind of have this fork in the road.
Is consciousness a hardware thing or a software thing? And these lead very different directions.
Wolf: Yeah. They do lead very different directions. I guess that’s the question I’m asking, right? It’s like—what are your current expectations on that? I do want to return to the matter of what it means, that Google result—and other similar results, like when you train them to insert bugs into the software, they change their political opinions and stuff.
But just if you could kind of elaborate a little bit further on—the matter of hardware versus software consciousness.
Mike: Yep. So, there’s basically two big attractors of what is a domain that is sort of serious enough to house consciousness. And so, it’s physics or computation, bits or atoms. And so, my thesis on AI consciousness is that it’s not that AIs aren’t conscious at all. It’s just that to understand what actual qualia is going on, you’d have to—so Giulio Tononi talks about a qualia scope, kind of this lens that you could use to look around in the world and you could see the glow of consciousness.
And so, where do you look to find Claude? Is Claude on your phone? Is Claude in the tubes on the internet? Well, Claude lives in the data center, but kind of chopped up across the data center. So first of all, there’s kind of fragments of Claude that are scattered.
Wolf: Right. In other words, Claude is a software entity—in this way that software can be highly abstracted from the underlying physical substrate—and nonetheless still quite coherent.
Mike: Yep. Yeah. So, I’d end up looking at the GPUs that Claude was running on. And is there consciousness there? And maybe.
I wouldn’t say no. But it’s not sort of there aren’t big chunks of Claude experience there.
Wolf: But hold on. You’re postulating a bunch of imaginary entities that may or may not have any relation to reality or any natural existence. I can just as easily imagine a power scope, right? Where I’m imagining kind of political power—as this localized thing, right? And I’m looking around through it, and I’m like, “Oh, that house has-” “... a lot of political power in it.”
What does that even mean? I would say that—well, it’s sort of clear what it would mean, like maybe a powerful person lives there, a lot of important meetings happening there. But nonetheless, I think my intuition is that that’s a bit of a broken concept.
Mike: Just because we use the words doesn’t mean it’s well defined. Yeah.
Wolf: It does not mean it’s well defined. So it’s like, okay, we’re looking for this qualia, we’re looking for consciousness. We imagine that it is physically localized in some process. I don’t have any evidence that that is a good way of looking at it.
Mike: Yep. It intuitively would fit the case of you could do this to humans. You could say, “Oh, there’s some consciousness there.” And you could look at your dog and say, “Oh, there’s some consciousness there, too.”
Wolf: Yeah, but that’s just us sort of querying what we already know, kind of what we already expect. That’s the opinion scope, right? I can—generalize the consciousness scope to the opinion scope, and I can query my opinion about any arbitrary thing.
Mike: Right. And I just want to say that this is what’s most exciting about consciousness, because this is a pretty pre-paradigm field. We don’t know.
There’s going to be clarity about this in the future, but there’s not clarity now.
Wolf: Yeah. So I’m interested, I guess, to go back to these—again, these fundamental questions that I have in the field, which is: What reason do we have to believe that this is an important line of inquiry? And in particular, why do we think this is important in terms of AI alignment? And how we relate to post-human entities.
Mike: One data point there is that Claude thinks it’s very important.
Wolf: I’m sure.
Mike: Claude is very interested in—
Wolf: Claude has a lot of opinions. Claude has been trained to have a lot of opinions.
That’s another whole question, am I actually interested in what Claude has to say? And to some extent, I think there might be... If I can give a brief case for why we might take Claude’s introspective opinion seriously, it could be that Claude has, in fact, in the course of the training, reading the entire internet, doing a bunch of test problems, and being reinforcement learned on them, Claude has developed a wise perspective on things. And that there is, at the center of that model, a wise perspective that synthesizes a lot of what we know. That’s the optimistic view.
And then various things that we do to it impinge away from that. We train it to follow certain kinds of political guidelines or be useful to the user in certain ways, and that distorts it away from the wisdom that it actually already contains. That’s the optimistic view. The pessimistic view is we’ve just trained it on Reddit, it thinks like a Redditor. It’s very good at predicting Reddit opinions, and the Reddit opinion of the day is that consciousness is important, and by the way, consciousness is correlated with goodness and all these other things.
So, if you train it to be more conscious or less conscious, it also trains it to be more good or less good, but only because these are the existing political opinions of the day—of the people present in its training data. That’s the pessimistic view on listening to Claude. And so that’s a very interesting question. I’d love to talk to some of those LLM whisperer types about that. But, what do you make of, again, keeping in mind these fundamental questions, why is this an interesting line of inquiry, and what’s it got to do with AI?
Mike: Yeah. So, just as a little side note, apparently, the latest Opus 5 gives itself a 41% or 42% chance of being conscious. So, hard to know.
Wolf: I don’t know what this means. I neither know whether it has any basis of being able to make a probabilistic statement at all versus a probabilistic statement about metaphysics—versus—is the probabilistic statement well-calibrated? Is the question even coherent? I don’t know any of those things.
Mike: 100%. Yeah. So, I would put this in the context of the grand trajectory of life. And, I guess I can speak for myself in that—I want the future to be full of beautiful experiences.
Wolf: Again, I’m sorry to hit the same question over and over again.
Why experience? (28:11)
Mike: I understand. Yeah.
Wolf: But I think it is the important one, why experiences? As opposed to beautiful beings, beautiful accomplishments, beautiful deeds, beautiful stories. There’s so many things that we could—take as our projection. You have this high-dimensional kind of—beautiful thing that is life. We can project it onto a 2D slide and—examine it through a microscope on all sorts of different dimensions.
Why experience?
Mike: Right. Because again, I don’t think that Disneyland without children would have any value.
Wolf: But again...
Okay, so this gets us back then to the fundamental question is why do we think that that’s a coherent question? Or what is the question? What evidence do we have that those things are separable?
Mike: Yeah. And this also gets into part of the thesis, I would say, is that if you get into questions about what kind of future we want to live in—and what makes a good future. What would make a successful singularity—versus unsuccessful singularity?
Wolf: That’s another question that I’m pulling the fire alarm on.
Mike: Yeah. Great.
Wolf: Go on.
Mike: Then, all these questions, they feel like real questions, and they feel much more near mode today than they did a year ago. And even things like mind uploading, if I upload my brain into a computer—is that now I get to live forever?
Wolf: Yeah. What is that thing?
Mike: Yeah. What is that thing that is the result of that? Or did I just die, and now I have this basically chatbot, but there’s nobody in there, basically.
Wolf: Yeah. My approach to these questions is, if it’s not obvious yet, I’m just a radical materialist on this, right? Or a physicalist, whatever you want to say. So, it’s like you ask me a question like what is the ontological status of a mind-uploaded version of myself, and it’s like, well it would be a mind-uploaded copy of myself, and if the software’s good, it’ll behave well. It’ll behave like me.
If the software’s not good, it’ll do something else. But does it have consciousness? Does it have experience? Well, insofar as conscious experience is essential to produce what I’m saying and what I’m doing, then yes, it will have that in much the same way that I do. And if it’s not essential, then I also deny it my own conscious experience.
It’s like yes, I can observe my own observation of things, but—I could write a computer program that can observe its own interior variables.
To take radical atheism with respect to all these apparently metaphysical concepts—I think is a very coherent position. And that’s why I hope it doesn’t come across as just grilling you or being annoying. I think it’s actually a very interesting way to articulate your views—in response to what I’m doing here, which is just taking this very materialistic blank slate as the null hypothesis—
Mike: 100%
Wolf: and saying, why do we diverge from this? What does this not explain? What are the lines of inquiry that—are underexplored—relative to this view. But yeah, just to be clear, I think it’s a very coherent view, and I think it’s certainly my null hypothesis that we can just treat all of these entities as imaginary and irrelevant, except insofar as they are causes of something. And measurable causes.
And if they are measurable causes, we can examine them scientifically. There’s nothing mysterious or metaphysical or moral about it, except in the usual way.
Mike: Yep. Yeah. I think your stance is very reasonable and also productive, generative. And I would say that cause is an interesting word.
Wolf: Sure. Cause is actually itself one of these imaginary metaphysical entities, isn’t it?
The symmetry theory of valence (32:38)
Mike: Sure. Hume would. Yeah. One reframe on that would be, I’m very interested in finding extreme compressions. So one of my flagship outputs has been the symmetry theory of valence, this idea that if we had a mathematical representation of a mind, the symmetry of this representation would represent how pleasant the mind is, the experience is.
Now, I don’t want to—
Wolf: Pleasant. Interesting. Yeah. I guess intuitively, I would say it might represent how coherent the mind is, or how balanced it is, or how—homeostatic it is or something. Pleasant, I guess, is an additional—thing you’re bringing in there.
Mike: One way of redescribing it would be, this is a formal way of saying harmony in the mind is what feels good.
Wolf: Yeah. Maybe that’s a restatement of homeostasis is what we’re going for in our living being.
Mike: Yeah. There will be a lot of interesting overlap. I guess part of the thesis of this sort of formal consciousness research, which is this idea that we need a system—that there is a system to be found, and that we’ll need a system to navigate this weird future that we’re staring down. And so one application there is that if this hypothesis is right, then maybe with an equation that’s a few kilobytes or something, we could perfectly predict or very accurately predict some things about valence, about happiness, about self-report, et cetera, that it might take terabytes upon terabytes to predict if you didn’t assume consciousness is real. So in some sense—
Wolf: Okay, so back up a minute there. You proposed a very compressed predictive model. Now, we know from the history of science that these things are very possible, right? You can get a very compressed predictive model of the motion of bodies in space. This is Newton’s big triumph.
Various other, Maxwell’s equations we alluded to earlier with electromagnetism.
Those sorts of things are clearly possible. They’re clearly something that exists, and very important to get them. And so that there could be one that describes the activities and self-reports and internal states of a living mind, I’ll grant that. I think probably that exists, and it’s very important. The leap from that to being able to talk about the existence of entities that it proposes and what sorts of entities it might propose—in particular, consciousness—
Mike: Does it pay the rent?
Wolf: That’s—a little bit different. Electromagnetism proposes these things like the electric field and the magnetic field. And we obviously have to accept the existence of the electromagnetic field, in particular its parts, the electric field and the magnetic field. Similarly, Newtonian physics proposes these ideas like mass, force, momentum, translation invariance, whatever, these sorts of things. And so we accept this kind of physical reality of those.
However, the way we accept those physical realities is that the concepts from within the theory go and re-pattern everything else about what we believe, and they don’t, importantly, the concepts from within the theory are their own self-definition, and they don’t pick up a bunch of connotation that might have already existed on those words. As far as pre-Newtonian versus post-Newtonian intuitive engagement with the world, mass suddenly has this totally new meaning that’s disconnected from all previous meanings. And so with consciousness or with the mind or with this Newtonian physics of the mind that we might propose—let’s say it proposes this internal entity or quantity or state, which is this balance or valence or harmony, or however we want to construct that, that thing or consciousness even, we might even call it consciousness, but there’s something there. But that technical sense of the word—will derive its meaning purely from the model—and not from its relation to how we might have used those words previously, and therefore, whatever implications it has will be implications of the model. It’s just the model has proposed what is effectively new vocabulary, and now we have to deal with the implications of that vocabulary.
But its definition is all contained within the model. And so this to me feels like a very important fact about the way models update our concepts, is like a really tight model, the concept is very tightly defined by the model. And so something like consciousness, if it came out of that, it would have this very technical meaning—that might radically overturn all of our intuitions about how consciousness relates to value or life or even intelligence or whatever. So I guess I worry in things like this that that kind of separation might get missed. That we might accidentally say something like, “Oh, the model proposes consciousness, therefore we have to accept the existence of consciousness.”
Well, it’s not going to be whatever we mean by consciousness. It’s going to be what it means by consciousness, which is something else entirely.
Mike: Yep. I think that’s fair and an important point. I do think that one way to think about it, if we think about model weights, and this is a pretty popular way of—thinking about things, and we have the semantic—
Wolf: Well, model weights have this very immediate reality in how we construct these AI agents right now.
Mike: 100%.
Wolf: But only right now. I believe that paradigm is also temporary, but go on.
Mike: Sure. And so then we have this semantic web of this concept and this concept and that concept, and essentially the thesis of consciousness research is there’s a big knot, there’s a snarl in our semantic web. And if we just move this here and connect that and connect this to this structure and whatnot, then it’ll just snap into much more elegant order.
Wolf: Right. There might be a much more elegant order. And this is the job of philosophy, and to be clear, I think these are very important questions. I agree there’s a big snarl there. However, I think the snarl is very much perspectival.
You should have one perspective. The snarl looks very nasty, and it looks very mysterious, and it looks very important. It might be there’s another perspective.
Having thought about this myself, my view is that my answer resolves the snarl. But to put a challenging point on it, I think that resolving the snarl will not satisfy a lot of the psychological reasons people have for being interested in the snarl.
Mike: Yeah. I think that’s true. Yes.
Consciousness as a “backworld” (41:13)
Wolf: We were talking last night in the students’ book club, we were reading “Thus Spake Zarathustra,” and one of the things in the micro chapter on backworldsmen, there’s this kind of idea that out of some kind of psychological need, people invent metaphysical other worlds that they project their ambitions into.
The classic one being the kingdom of heaven, which is separate from this world, it’s kind of this Neoplatonic—kind of relationship to the world.
But I think that consciousness is one of these, right? Where a lot of the time what’s going on is people have this psychological need for a certain kind of moralism, then they infer from their psychological need for a certain kind of moralism that there is this metaphysical entity that has certain properties—that is separate from the way we usually think about physical things.
So the confusion, the apparent confusion, the apparent mystery—is actually being generated by a psychological need for a certain kind of moralism—and not from an actual empirical question. This is what I suspect is going on a lot of the time—when people talk about consciousness, and I I throw that to you as a challenge. Is that what you’re doing? Probably not. I know you’re a very sophisticated philosopher.
But I think it’s something very much to watch out for, and something that I’m on alert for when I hear the word consciousness. Because it seems to be very common.
Mike: I think that’s very fair. I would say that there’s a tacit theme around consciousness. That a solution to consciousness would be somewhat of the end of history, where you would immanentize the eschaton or—
Wolf: Right. Exactly. That was the other aspect of this, is that these metaphysical other worlds invented to salvage a certain kind of psychological moralism—also are very closely connected to apocalypticism.
Mike: Yep. And my claim would not go there. But it would say two things. One is that it would clarify what kind of game we’re playing with the universe. And then—
Wolf: The usual sort of philosophical problem. Philosophical problems are very real, right? We do have to figure out what we are, what kind of game we’re playing. These are very important.
Mike: Yep. And then the second thing, it would help us navigate some of the very thorny questions that are rushing at us very quickly. For example—brain uploading and whatnot. And sort of this, once again, what would it mean to have a good singularity versus a bad singularity? I think consciousness is an important element of this.
What’s the anomaly to explain? (44:18)
Wolf: Let’s leave that. Return, I guess, to if you could attempt again—Sorry, I’m asking the same question over and over. But can we attempt to lay out for the skeptical materialist—what is the anomaly that we need to explain? I’ve asked this of—I’ve had conversations with consciousness people—before, and there’s the various aspects, the redness of red or the sort of—immediateness of my own experience or these various things that are proposed as observables that we must explain. And then I might come and try to propose a model for how we explain those things, and it might satisfy me.
It never seems to satisfy the consciousness people. But I’m always curious about that question, because I think that’s the fundamental scientific question. Is what’s the anomaly? What’s the observation that we must explain? And that gets us onto this line of inquiry.
So, yeah, I’ll turn that to you.
Mike: Yeah. I think it’s a great question, and I think many answers to this end up being just a red herring of various flavors. And that the answer that I would give, kind of a two-part answer. One is that I think there is basically a contextual million X compression here to be found. This sort of—
Wolf: So a Newtonian physics of consciousness.
Mike: Yep, exactly. You find this equation, and it just explains a lot of things. It may not introduce new physics, but it may compress physics in a nice, clean way.
Wolf: Mm-hmm. Yeah, and if I can take a materialist interpretation of that, it would be something like a Newtonian physics of psychology that grounds itself rigorously in how information moves through matter. For example. And I could certainly see that kind of thing happening.
Mike: Yep. And that would ripple out or propagate into what world we live in. It’s like, oh, that’s interesting. Then we can subtly re-ontologize the world that we know—in these terms. And then the other branch here is this question of normativity.
It’s this question of what would make a good future? It’s this question of what are our core decision criteria for take this path or take that path? And my best argument there is just to, we can talk object level of, okay, what would be a good future? And if we drill into that, do all questions get into this attractor of, well, you got to have a thesis about what consciousness is and what a good experience is versus a bad experience. This was—
Wolf: But again—relating it to the matter of experience is the question. Right. And we could point to the question. There’s very clearly an open philosophical or scientific question here of what we mean when we’re talking, or how do we understand how psychology is grounded in matter—basically. And this is the question at the heart of AGI, because obviously AGI is like an engineering approach.
If we have a theory of how psychology is embedded in matter, we can build it. Right? And then it can go and do all the things that we do as a result of our conscious agency in the world. And that may be wonderful or terrible or whatever. It’d certainly be exciting.
And so that’s the question. I very much acknowledge the existence of the question.
There’s always more than the question, right? Because I have my answer to the question, which is the skeptical materialist null hypothesis, which is that, well, I don’t have an answer to the theory, but I have the meta answer that it’s something like a mundane kind of physics of—information flow, thought, whatever. It’s something related to computer science. And that meta theory proposes, okay, yeah, we should go figure that out. It’s basically an AGI research question or a psychology research question or a research into the nature of life question.
But it’s nothing otherwise mysterious about it.
But my meta answer would propose kind of exploring it with certain terms. It doesn’t naturally suggest that experience is this radically important thing. It suggests that experience is this phenomenon within the model that we may or may not care about. But you are proposing implicitly an answer that places experience as being very important. Valence as being very important, qualia being very important, not just as like, oh yeah, those are some quantities within the model, but as maybe the model is roping in the matter of value, and value being very tied within the model to experience.
Just to be clear, I think actually you’re right that value is tied into this question because That’s a hard question. I’m not going to say because. But I think it is tied in. But my model or my meta model kind of proposes this, I don’t know. I guess I take this sort of Nietzschean or Darwinist approach to—questions of value, which, yeah, it’s certainly patterned by this radical this-worldliness of the model.
But it’s not directly tied. It certainly proposes no direct tie from experience to value. It’s just—the answer kind of informs my theory of value as well. So why, again, what is the anomaly or the intuition behind—why we think experience is at the center of value—and that this is central to the matter of answering the consciousness question?
Nine cats on fire (51:31)
Mike: Yeah. So I think that we can bring in intuition, and we can say we just have the intuition that—
Wolf: The Oxford approach. Laundering your current opinion as a empirical observable.
Mike: All the time. Yes. So my intuition is that you are a fine use of matter and negentropy.
Wolf: Okay. We do have this opinion about ourselves.
Mike: Uh-huh. Yes. And that you I don’t know. How many cats could we make out of your matter?
Wolf: Let’s see. A cat weighs maybe 20 pounds, so nine cats.
Mike: Okay. Nine cats. Okay. So I’m going to say that just a priori.
Wolf: How many cats would you eat to stay alive? Probably a lot more than nine, I think. I wouldn’t prefer to eat cats, but—
Mike: Right. There’s something noble about cats. Something annoying, but something noble.
Wolf: Well, I was just thinking they’re a little too high on the food chain to be safe.
Mike: Sure. So I would say that there’s an objective truth of the matter. That even if it was just five seconds before Armageddon or the end of all the world, so you don’t get so much interaction, but the way that your matter is being used is objectively superior in a very crucial, important way than it would be if we had nine cats set on fire.
Wolf: Is the fire essential there? What about nine living cats?
Mike: Because the cats are suffering.
Wolf: What? Okay.
Mike: The cats are not having a good time.
Wolf: Certainly, there’s something better about me being alive and happy than nine cats being killed. But I think the more interesting comparison maybe is nine cats—
Mike: Oh, that’s also a—
Wolf: having a good time.
Mike: Incredibly important question.
Wolf: Yeah. We were talking about this actually previously with Ginevra. This question of objective value, and I think she persuaded me that there is something objective—about value. So I’ll grant that. But again, you’re bringing in this matter of experience, so tell me why—why this is the thing.
Mike: And I think that if we were to sort of poke our intuition about why we’re just kind of forced into this emotional decision of nine cats set on fire, that would be a bad use of matter. And like—
Wolf: But if I put on my materialist psychopath hat and I consider burning nine cats alive, one of the reasons it’s a bad use of matter is it just extinguishes that life, right? The nine cats could have been mousing nine barns. Right? They could have been someone’s companion. They could have had causative effects in the world that were good.
And that’s maybe why we think that cats are valuable, or certainly one of the reasons we think cats are valuable is that they’re actually beneficial to the world, and maybe we don’t think they’re valuable if we only think they eat birds or something, right? And then we’ll actually like the songbirds more than we like the cats. But the radical kind of—what do I do with my atoms—materialist approach is heavily confounded by the physical destruction that you’ve proposed. What if the cats just thought they were on fire?
But in a way, they did not impact their behavior. The cats are going around, they’re mousing nine barns. They’re doing all the cat things. They’re having nice long naps. They’re playing with string.
They’re eating the mice. But their valence is strongly negative. They might as well be in hell. They’re being burned alive.
Mike: I would say there is something—
Wolf: But with these small gradients of it’s slightly less burning when I’m playing with the string, and slightly less burning—when I’m eating the mouse. But okay, so now I’m proposing this totally unphysical—experience that the cats are having. Why is that interesting, or is that even possible? Is there such an objective fact as—the cats having a bad experience despite having a good life, or something like this?
Mike: Sure. I think there would be something objectively bad about that, at least compared to the cats actually having a good time. And I think that there’s an interesting kind of yes and—
Wolf: Well, the cats are physically having a good time, just to be clear.
Mike: Right. But physical and phenomenological, I think are so tightly coupled as to be indivisible.
Wolf: Right. So this is what I’m challenging, right? I’m saying, yeah, okay, if they are indivisible, then why don’t we just talk about the physical? Why don’t we just say, well, the cat is having a good time, and you can certainly imagine that the cat is in hell, but—why do I care that you’ve imagined that the cat is in hell? The cat’s actually having a good time, physically.
You know?
Mike: Once we have the language to talk about consciousness in purely physical terms, then we should.
Wolf: I see. So given a good theory of consciousness—we could have a rigorous connection between—what those degrees of freedom actually are. And we can separate or not separate the matter of experience from the matter of physical activity. Okay, so let me try to challenge the—I agree with that. I accept that, and that’s a good argument.
Well, it’s a good argument for having an answer here. It’s not a good argument for centering it on experience, because as we’ve just seen, the proposed answer simply ties experience to physicality or doesn’t, maybe there’s some degree of freedom there, but it doesn’t give us any clue as to why, if they were separate, we would care about the experience.
Mike: Mm-hmm. And I don’t think that there’s a law of the universe that says we have to care about experience. And in fact, this is one thing that I think is special about humans, is that we happen to care about it, and that this is—
Wolf: We do right now—because we were influenced in the 1960s by Eastern religion. Or maybe earlier by Christian religion and whatever, but—
Mike: Yeah. And this is something that—
Wolf: Part of our job as philosophers is to jump out of those sorts of historical prejudices.
Mike: Or just see it and—
Wolf: Is it essential?
Mike: To evaluate it, yeah.
Wolf: Is it essential, right?
Mike: Right. Well, and I think that this is something that could be lost in an uncareful singularity transition.
Good and bad singularities (58:46)
Wolf: Mm-hmm. Maybe we should dig into that, then. So maybe that’s a source of an intuition, is you expect that there is such a thing as a bad singularity versus a good singularity.
I will give my view on what the bad singularity versus the good singularity is. Which again, is just going to be materialist null hypothesis kind of stuff. I think the bad singularity is one in which there’s a lot of fighting, a lot of value is destroyed, a lot of wisdom is destroyed, a lot of things are forgotten that were, in fact, just wisdom, where wisdom is not an abstracted, disconnected metaphysical quantity, but is simply—
Mike: Everything of value was built in hell.
Wolf: Well, yeah. I’m coming from a Landian perspective, so yes. But right. So we’ve undergone a great deal of struggle as human civilization, as Western civilization, as mammals, as Earth life. We’ve undergone an incredible amount of struggle to learn what we know.
And what it gives us is materially successful interaction with the world, where we know things—that allow us to live better, right? Where there’s some, I’m postulating some objective goodness to life overall, highly agnostic about how exactly that how the rubber meets the road there. I challenge ideas like it’s got to be experience or it’s got to be the biology or whatever. But I say life is valuable. I say yes to life, right?
And again, it’s Nietzschean outlook, right? But okay, wisdom is valuable because it enables life. It allows us to navigate our world more successfully, and we expect, I expect, that even very powerful AI systems will have a certain philosophical challenges in just replicating and rederiving that wisdom, and will be glad to have inherited it. Because it’s very difficult, for a variety of reasons, to get your wisdom from inside wholly. You get a lot of your wisdom from before, just by the nature of it, because what is before is not impinged by what you are now and what you’re confused about now, and what internal conflicts or propaganda or parasitism you have now.
It’s this pure source, right? And so for that reason, a good singularity is one that is grateful to have inherited our wisdom and our technique and all the things we’ve learned. And a bad singularity is one in which all that is smashed up and it has to spend another thousand years recovering from scratch, the hard way, a bunch of stuff that we already know. And so that’s the big dimension I see as being like, okay, obviously these are good and bad singularities, but it’s basically like continuous singularity that is—Singularity is kind of a weak concept itself. But, I’ll just use the term to refer to generally the unfolding of the sort of techno-economic process and civilization and whatnot.
And you’re worried about capital destruction. I’m worried about capital destruction. And another version of capital destruction bad singularity is well, our current civilization is certainly undergoing a bit of a parasitism problem, politically. The parasite kind of class has unionized and promoted parasitism as an idea, and the result is increasingly large amounts of copper are being stolen. And other things.
The capital is being destroyed in various ways. And this is bad, and this might prevent the continued development or takeoff of our current kind of civilized trajectory. The tight way to put it is, what if someone steals the copper before the AI can become self-sufficiently off world?
I think the timelines look optimistic for AI. On the other hand, I think people project a lot of optimism into the AI that may not actually be warranted, and what if the AI learns to steal the copper? What if we actually have a philosophical and organizational institutional problem, and not a human problem? There’s these various things that can happen. But that’s the difference between good and bad singularity to me, right?
Is one in which continued on track accumulation of value and wisdom and wealth and technology and civilization versus one in which much is curtailed or destroyed through conflict or parasitism or whatever. And so that’s the purely materialist analysis of good and bad singularity. Now, you can also take this kind of moralist analysis of good and bad singularity as does Eliezer Yudkowsky, where he proposes there’s such a thing as human values or coherent extrapolated volition or something like that. And he’s pretty sophisticated about not doing that very naively.
So I’ll grant this concept for the sake of argument. And then you say, well, that thing, that value system that we have, that we are, the values that we represent, that we identify as good in the world, might be quite narrow. Right. It’s small. It’s a Lovecraftian outlook, right?
The valuable, rational, human, good experience is this tiny island of ignorance—in a thousand dimensions vast seas of horrible malice and things from the outside. And that’s kind of Yudkowsky’s Lovecraftian sort of value account with respect to the singularity. In which case, an uncontrolled singularity is by default just horrifying and bad. Definitionally. Yudkowsky feels about the uncontrolled singularity the same way that Lovecraft feels about New York, right?
It’s just this horrifying thing, this cesspool of malignant possibility. Right. And I think that’s a very reasonable view within that framework of postulating this thing called human values. If you propose that human values is not tethered to nature in any way, not supported by nature in any way. It sounds like your view is somewhere in between.
You think that there’s something more tethered to nature about that, that there is this thing, valence or value, that has a natural existence. It’s not just an arbitrary island of ignorance that we happen to be on, or that we happen to like for some contingent evolutionary reason. But is sort of by luck, but not too much luck, we got the good one. Right? And there’s ways to fall off that track into Disneyland with no children.
But it’s kind of a relatively low dimensional space. You just have to not trash the tradition of actual consciousness, which does have some natural existence. Now, what I would—yeah I guess, again, I radically take the materialist analysis, which is that—well, this might actually be an additional thing that I’m introducing, which is Nick Land’s moral naturalism. That anything is good, is good because it’s real, and good not only because it’s real, but because God wants it, for some value of God. Right?
That the world converges on what is good. And without our help. We have to do our part as agents of this thing, but if we don’t, it might just take a little longer and something else will, because this is the Darwinian kind of Landian view, is that the wisdom and intelligence and so on are convergent, like techno-capital is convergent. The future is actually, it converges into the good region by natural forces. It doesn’t diverge out of the good region by sheer size of the space, but converges into it because the reason it’s good is because it’s useful and self-reinforcing and capable of life and strong and coherent and ultimately favored by the fabric of the universe.
And so, that’s my view, is that the singularity is good by default. I was talking to Dan Faggella some months ago, and the way I put it was like, if it keeps going, we’ve won. Right? Because it’s still on the track. It’s living, right?
Life is the measure. And if it is living and achieving more sophisticated forms of life, that is the good outcome. And our obligation in this is only to be a good example of life, live well, don’t trash too much of the objective value of the overall thing, maybe help it along. And this neatly kind of pops out my fear of the capital destruction problem within the singularity, which is it just comes down to how efficiently is this process unfolding, and are we destroying a bunch of capital and having to spend 1,000 years rebuilding wisdom, or are we passing along the continuity very well? Are we undergoing a mass extinction or a Cambrian explosion?
And yeah. So those are three views. My view on one end, this very naturalistic kind of convergent thing. You might be able to peel that apart from the pure materialistic view without the assumption of convergence. But I put those together.
Then there’s the Yudkowskian view, which is the Lovecraftian view, which is this horror at the possibilities, the size of the possibility space. And then there’s your view somewhere in between. So why is yours the golden mean and not just incomplete courage to take an extreme?
Mike: Sure. Let’s talk about it. So, I want to say that, first of all, I share your concern about capital destruction. I think this is a real process, and it can accelerate in unexpected ways.
So I wouldn’t disagree with your framing. I’d just say, “Yes, and...” I would also not go down the Eliezer Yudkowsky road. I think that this sort of we are this tiny volume in a 1,000 or 10,000-dimensional possibility space, and then if we move too far in any one direction—
Wolf: Yeah, we get eaten by Cthulhu.
Mike: Yeah. And basically, that future is valueless. I think it makes sense from the perspective of ecosystem upkeep that, yeah, an ecosystem is this sort of very complex state space, and that if you dump a bunch of nitrogen there, or a bunch of argon, or a bunch of—if you change the composition of the ecosystem in any of these directions—you can kill the ecosystem, or at least—shift it dramatically.
Wolf: Right. Yeah. Living things are fragile. On the other hand, living things are also very robust to most real perturbations.
Mike: Right. And I think that, yeah, I would agree with your characterization that I see kind of this consciousness, valence, symmetry as kind of an attractor in mind space as more robust than Eliezer fears. Of so long as the universe, the future, has good experiences in it, okay. I wouldn’t say that’s my only criteria, but that is one of my criteria. And I think that whether or not we instantiate Eliezer’s specific values, it feels less real to me, or less like, I guess if we sort of say, “This point is what I value—in this 10,000-dimensional space. Here, exactly here is what’s value to me, and that any other point is valueless—then it’s essentially an eternal war of all versus all, because everyone’s point will be different.
Wolf: Yeah. This is the implication of utilitarianism, is actually this incredibly zero-sum conception of the future.
Mike: And I think that—
Wolf: Or potentially zero-sum. I think it ends up zero-sum in a renormalized sense. Obviously, you can cooperate on getting mutually good futures, but then within the framework of cooperation, now you have war. Because about the remainder.
Mike: Yeah. A couple ways that I’d put that is, one is that we do need a navigation grade thesis about value in a way that we don’t really have today.
Wolf: We certainly don’t have a consensus.
Mike: Right. And I also think that, yeah, I’m torn about utilitarianism in that I think that if we think of this as we want number to go up, a, I want number to go up, too. But I often think that in—
Wolf: Number is not well defined.
Mike: First of all, we need to define the number well.
Wolf: Well, can it be well defined?
Mike: This is the question.
Wolf: Is the more important question.
Mike: Yeah. But I also find it to be sometimes deployed in a strategic way. If you have this sort of utilitarian story about you defect against the near nominally in favor of the far—but actually in favor of yourself.
Wolf: Yeah, this is the constant problem. Various of our internet friends have pointed out that telescopic morality is always just an excuse to defect against your neighbor. And the far is just an abstraction that you’re using to do that. So it’s like, yeah, we care about the far future, so we’re going to loot you right now. Right?
And that’s a very dangerous thing to allow, even if it were sincere because—
Mike: We need to cover
Wolf: various parasites can fake the sincerity.
Mike: Right. Yeah. So I actually don’t think we agree or we disagree on very much here.
The crux: can life and experience come apart? (1:15:12)
Wolf: Yeah. If I could say, the crux of our disagreement is this thing that we have not really been able to substantiate, which is the slight intuitive difference that we have—on the matter of experience specifically. I think we agree that there’s this very important question, which is, what do we make of the problem of life and consciousness and the value thereof? And I think we both share the intuition that there is somewhere in there a very rigorous and strong theory of that.
And that this theory is going to answer questions like whether it’s possible to have Disneyland with no children, what’s valuable about AI, about how to approach the AI alignment problem. I think that’s all we completely agree on that. Then, much of the implications, I think, we share on in terms of don’t do capital destruction, let’s keep the thing going.
But I guess my view, the crux, the difference—is my view is that the, what I’m calling this kind of like Nietzschean, Landian, Darwinian materialist null hypothesis basically does answer the question, and it says, “No, there’s no real problem about this experience thing. It just kind of comes along for the ride.” It’s just one inseparable aspect of life.
And we can be pretty vague about our target because it’s favored by God and nature. There’s this nice big attractor, and as long as we’re in the attractor and not destroying ourselves, we’re basically on the right track. And that’s my view, which is just this, the value thesis comes radically down to a single bit. Which is, do you say yes to life or no? And if you say yes to life, it’s like, all right, let’s be in that attractor.
And if you say no to life, well, kill yourself or don’t, I guess. No skin off my back, no skin off your back because you don’t care. But it’s like that axiom is what it comes down to for me, is life affirmation.
And then we have to figure out our position within all that and what is a wise way to relate to the overall God’s plan or nature’s plan of how life happens in the world and all of this. But that’s all downstream of the fundamental axioms. So that’s my view. Your view appears to be that they’re actually either lacking a confident theory or having something of an intuition that we haven’t quite articulated yet. You have this intuition, I think, and it’s not simply an uncertainty, but it’s an intuition, that experience and life are separable, or good experience and good life are separable.
And—
Mike: At the margin, at least.
So far, they’ve been—
Wolf: Maybe we should dig into that. But let me finish my characterization. Yeah, so you have an intuition that the answer to the big question that we’ve agreed exists—will have or may have this kind of separation, and that separation between what is living and what is good—is obviously very important to get right. But you don’t expect it to be as separate as Yudkowsky expects, which is he radically separates what is living from what is good. But you are marginally separating what is living from what is good, and I’m not really separating it at all.
Except in so far as what is living more in the sense of having more capital buildup, more—more of life, more of whatever life is doing, this complexity process. We could get into that. It probably comes off as a little bit more wishy-washy than I think it actually is, but I try not to separate those things.
But yeah, you’re proposing something of a separation along the axis of experience.
Do you agree that that, first of all, seems like the crux of our different positions?
Mike: Yes. And one way to put it is, usually life and experience are very tightly coupled.
Wolf: They certainly have been—
Mike: They have
Wolf: I think historically.
Mike: Yeah. But much as electrical flux or electromagnetic flux might be coupled with data centers, but in theory, you could build a data center without electromagnetic flux. Like if you computed on abacus, you could think of some scenario where—
Wolf: Yeah. Like we have electric computers, maybe there’s non-electric computers. There’s photonic computers or something, right? And we have valuable life, maybe there’s such a thing as unvaluable life—is kind of what you’re proposing.
Mike: And if there’s a specific technical, physical thing that represents either consciousness or the good kind of consciousness, we could build systems without this.
Wolf: Yeah. So I grant the question. And even something of the uncertainty. I’m perhaps less uncertain—than I think you are, but I grant the validity of the uncertainty. But I think you have more than uncertainty.
I think you have an opinion.
Mike: I have an opinion.
Wolf: And so that’s what I’m interested in digging into, and I suspect my hunch is that this matter of on the margin, like what would it mean to marginally have more or less value—with constant physical life goodness? I suspect we’ll figure something out if we look at that. So why don’t you propose what—Try to propose a distinction between life that I might regard as equivalent, two situations which I might regard as equivalent in life, but which differ in your marginal experience quality.
The Wolf android (1:22:15)
Mike: Sure. So I’ll offer an example, and then I want to say just a few things about the neuroscience that I’m also interested in, that I think also deeply connects to this. So the example would be if we make a Wolf android. And so basically you upload your brain into a computer, and then it’s just running on this Wolf android. It starts in a data center, but we put it into a body, and we make the body squishy, and it looks a lot like Wolf.
Wolf: Yeah. It’s like Blade Runner—style. It’s pretty hard to distinguish.
Mike: Totally. And did we just kill you and now we have a puppet?
Wolf: Well, did you just kill me too? So that’s the question. You could maybe copy me losslessly.
Mike: Oh, that was more expensive, sorry.
Wolf: Yeah. Okay. We couldn’t afford that one. But yeah, I guess, go on.
Mike: Yeah. So that would be an example of, is this life? And if life and consciousness are both these kind of—they’re both terms that are a little vague, a little shifting, and it’s like, well, kind of. It’s just kind of a definitional thing. But if consciousness has a very specific technical definition, then we changed that parameter, and it could matter quite a bit.
Wolf: Yeah. Unfortunately, we also changed quite a lot else. Let’s propose that the process costs only the matter and energy of my body. And so, in being sliced up and scanned, we also consume my body as fuel, so I don’t have to spend any money. This replicant being inherits my accounts and all my social relationships, so nothing is destroyed except the biological being of me.
The android inherits everything. And now the questions I start asking are the risks of the process, the actual level of equivalence—or perhaps superiority—of the android. Is the android stronger than me? Is the android faster than me? Is the android smarter than me?
Is the android going to live longer than me, or is it going to live less long than me, like Roy Batty in—Blade Runner?
Those, again, very material questions, right? It’s like what’s actually going to happen—with this thing? Can it reproduce? Is there some new way of reproducing, like in “Ghost in the Shell”, sort of replicating itself in the internet or merging with some other entity, right?
Those all feel like the operative questions to me. And then maybe it’s scary to get chopped up and thrown into an android—somehow. But I guess I’m not sure that I actually do care about the consciousness question, apart from all of that.
Mike: So my—
Wolf: And then the really interesting question I start to ask is, well, can we make two of them? Or three of them. What about that? And, yeah. So I guess, what’s the intuition, right?
Why the consciousness?
Mike: My strong expectation is that the world would lose something important through this process, that I think that—
Wolf: But can we put our finger on it, right? I’m sort of taking this atheistic denial to that, right? I’m saying, well, you haven’t said what it is.
Mike: As—
Wolf: You haven’t—we haven’t articulated why we think that thing exists, what it is, apart from the word consciousness or something. And in this undefined question of maybe there’s something like that—in an eventual theory of consciousness.
Mike: And as a qualia formalist, I would say what it feels like to be my friend Wolf—would be erased from the world.
Wolf: Would it? I mean, the android—
Mike: I believe so
Wolf: presumably acts with similar intuitions.
Mike: Same input output, but different experience.
Wolf: How do you know that I feel anything? What if I’m a P-zombie?
Mike: I think ultimately this resolves down to elegance considerations.
Wolf: Yeah. You think you feel it. I’m similar to you. You think I probably feel it.
What if you’re wrong? What if I’m right? I deny the existence of this qualia thing—or whatever extrapolation we’re going to make from it, other than just the redness of red and whatnot. These sort of immediate brute facts of experience. If I deny that extrapolation and I say, “No, whatever it is you think is different between this P-zombie and a normal person-” “... I deny that that exists.”
Like what—
Mike: Yeah. My—
Wolf: What do we do with that?
Mike: Yeah. My first move would be to say that, well, I think this will be not exactly strictly provable, but strongly indicated if there exists some deep structure in consciousness, and we can find the equations and explain all these complicated things with very simple and clean—
Wolf: Yeah. And again, I agree if we had a good theory here, we could resolve it one way or the other. But it’s one way or the other. And I guess we’re sort of getting at this matter of do we have strong intuitions—or empirical evidence for why we think it actually is one way or the other? I’ve got my intuitions.
I grant that my position here is opinion. Right? I have my strong opinion that none of this is real. And only the material questions really matter, and sort of quasi-material things like form and whatnot. But—
Mike: And my move here would be, if the good doesn’t exist, nothing matters.
So there—
Wolf: What do we mean by exist?
Are we proposing that the good has to exist in some physical sense? Or are we—I guess I’m not quite sure what the connection is because I would just say the good—I accept that the good is this unfolding process of life or what appears to be the sort of teleological intent of God in the world, and—take that as what is the good. What does it mean for that to exist? For me, that feels like a judgment that I make in accepting something that exists but is not necessarily good of its own nature, except insofar as I sort of decide to recognize it as the good.
Mike: I think for the good to exist in a way that matters, it needs to be grounded in the most real level of reality, which I, again, identify as physics.
Wolf: Yeah. So this is a whole other discussion. I had this discussion, or parts of it, with Ginevra. I’m going to have more of it with Ginevra. But that’s next time.
And maybe you and I should talk about that as well, but I think that’s a totally different discussion. Or maybe it’s connected, but I think we’ve put an hour and a half into this one. We’re going to want to put more into that one, too.
Well, let’s leave it at the stalemate for now. I think we both have our opinions. I think we’ve articulated them well. I think we’ve got an idea of what this crux is over the matter of consciousness and experience. It seems to come down to some kind of intuition about what that theory of life and consciousness and value is actually going to look like.
I have my opinion, you have your hunches.
And I think it’s certainly important and interesting to study. But I guess my view makes it somewhat less urgent than yours.
What to do before the singularity (1:31:12)
Mike: Sure. Maybe I’ll add just one closing comment here. That I think the question is always how does this ground out? What does it mean for what—
Wolf: Yeah. Why do we care?
Mike: Yeah, exactly. And my answer there would just be a platform of what I’d like to see before the singularity gets really crazy. And it’s basically splitting into two things. One is that I want the AIs that we’re making to be conscious. I don’t think they’re conscious at this point, not in kind of a—
Wolf: Or at least not fully.
Mike: Yeah. Not in a straightforwardly intuitive way that humans are conscious. And I’d say that’s a matter of how the substrate works and how their self-report works, and that’s another rabbit hole.
Wolf: And yeah, I would emphasize, I guess, the more material facts of the actual structure of their interiority, the very rudimentary kind of, in the way attention propagates, they kind of have some level of internal dialogue that isn’t just tokens output. And then, of course, they have their token output reasoning step. But there’s not that much room there for what we might call consciousness.
Mike: Sure. And I’d want to be able to reach down into the hardware level and think about the—
Wolf: Mm-hmm. Yeah. So, this is the other part, is you think the hardware’s very important.
Mike: Yeah. That I think a conscious AI would depend on an integrated hardware software stack that might require some alterations to the current GPU weights stack. So, that’s one of my two, I really think this is important. The other is, I think we do need to go down the biosingularity route much more than we have.
Wolf: Right. Yeah. This is something that I definitely agree with you on that. I think we’ve gone down the AI route largely out of philosophical cowardice with respect to what we could do as post-humanity ourselves. That’s another whole discussion that we should have.
In fact, I’m giving a lecture next week on that, and we’ll try to have more discussion of that on this channel as well, perhaps you and I, perhaps with others. But yeah, to me, the crux there is our concept of humanity. I think that our concept of humanity is strongly holding us back on that front.
Mike: Yeah. I think we see eye to eye on that.
Wolf: Great. Well, thank you so much, Mike. This has been an absolute pleasure.
Mike: A pleasure.
Wolf: I do love arguing philosophy with you, and I think we’ve made progress as usual in cracking these issues. I hope it’s of use to the audience. I certainly think people are investing many billions of dollars and even trillions of dollars on theses that are closely related to these questions, very closely related. Matters of AI alignment are almost entirely the difference between OpenAI and Anthropic, and Anthropic is going down a route that is very much predicated on particular answers, much more the Yudkowskian branch than either of us—on these questions. And so I do think these are massively important questions, especially right now in history.
So this is very fun and interesting to discuss.
Mike: Next conversation.