We Can Do Better Than "Human Values" — Ginevra Davis

Philosophers talk of aligning AI to “human values”. This supposes there is no objective value, only the arbitrary values that humans happened to land on. Ginevra Davis and Wolf Tivy argue there is a deeper well of value in material reality.

---

Transcript

Does anything matter? (00:00)

Wolf: So I’ve made the claim that humanity is a prison. If we want to understand ourselves even properly, and certainly understand the future and build a good future, we have to escape from the strictures of humanism, from the strictures of understanding the future in human terms, of even understanding ourselves in human terms. So I think this is a very interesting project to pull on, and I’ve got Ginevra Davis here with me, who also has thoughts in the area, so I wanted to have a good conversation with her about this stuff. So Ginevra, you’ve been thinking about the value of the future in the context of AI alignment and philosophy and ethics around that stuff. So I’d love to hear how you think about the matter.

You’re somewhat familiar with my takes on the subject. So why don’t we start from there, and we can go back and forth and discuss humanism without humans, basically.

Ginevra: Yeah. I think as a sidebar, there’s a tendency to use the word humanism or humans to refer to a very broad collection of things that are sort of good, warm, subjectivist—things that we want, things that we like, in group.

So the sort of AI alignment angle or kind of metaethics angle that I’m known for is this idea of taking moral realism really, really seriously, and also taking moral anti-realism, the possibility of, I would just call it nihilism—really seriously. So tend to call this kind of the normative hypothesis. It’s we’re at this really interesting kind of hinge point in history, and in the history of humanity, where I would say that we just genuinely don’t know whether things matter or not. And I would say things matter in the sense that there is a real way that some outcomes or some ways that the future could look are better or worse than others, or that some actions are better than others. Or that if I take Wolf here and I stab him in the face—

Wolf: Oh, come on

Ginevra: that matters versus me giving him a nice warm glass of hot cocoa. If I wake up in the morning and I put on clothes and I go to work, that’s, in some sense, morally different than me just lying in bed and, I don’t know, eating Twinkies or something. And I think we don’t know. We don’t know if some actions are more justified than others. One of my sort of nitpicks of the AI alignment world is that I think there is a lot of gesturing to this idea of a common sense, good future, bad future, paperclips is a bad future, utopia is a good future, without really that much interest in defining what a good future means, what a bad future means, what a kind of neutral or middling outcome means.

And my frame on alignment has always been, well, if there is a real sense in which it matters that the future looks a certain way so that it is better that we create utopia or some other form of a good future versus the torment nexus versus the void where there is no consciousness. This should be explainable to another kind of mind. It should be explainable to a computer. And if it really doesn’t matter that the future looks this way, instead of looking the other way, then it doesn’t matter, definitionally, if—the future looks, yeah, the way that we want it to.

Wolf: Yeah. So now this is going to be technically deep, I think, of an episode for people—

Ginevra: Yeah. No, I was going to... That was a really big starting question. I think there’s a lot there.

Wolf: No, and—

Ginevra: So yeah, I don’t want to—

Wolf: So I’m just going to—

Ginevra: I don’t want to kind of glide over some of these issues at all.

Wolf: No, but I think—

Ginevra: Yeah. Very technical. Yeah

Wolf: inescapably, we’re going to get into some technical matters here. So one of the things that immediately jumps out to me is what does it mean to know that the future can be good or not? You’re using this term, we don’t know or we do know. What does that even mean to you? Because I think that’s—something that I’d like to pull on.

Ginevra: Okay. Well, that’s interesting. I think we don’t know what we don’t know, but I guess I would say I don’t think that anyone has a knockdown proof yet of nihilism, and I don’t think that anyone has a knockdown proof of moral realism. I—

Wolf: So let me—propose my kind of a way to approach these things. So I would say it’s not a matter of knowing or not knowing. It’s one of those questions that I don’t have a knockdown proof that we can’t know it, but I do have suspicions. And I would analogize it to something like induction. Like, we know the sun’s going to rise tomorrow because it’s always risen.

We see this pattern. We learn a lot about how the universe is patterned, and we infer therefore that the sun’s going to rise tomorrow. But this all hinges on this assumption that the universe is actually patterned, and that we can actually make those kinds of inferences. And you can’t learn that from observation. You can’t prove that from logic.

There’s just this assumption that we have to take if we want to approach the world empirically at all, that there is the possibility of induction. Are you familiar with this argument?

Ginevra: Yeah. I think also just to get into a metaphysical point here that I think is very relevant for understanding—a lot of my work and my thinking is, I think it is important to take very, very seriously that I would say it’s all in your head. All we have access to is, I do a lot of work on consciousness and kind of like moral realism through consciousness.

Wolf: Right. What do we actually know? Well, one of the things we know is like some—kind of existence.

Ginevra: Yeah. So all we know is our own consciousness. The one thing that I can be 100% sure is real is this experience that I am having right now. This is a part of reality.

Wolf: Right. It’s self-verifying.

Ginevra: It is self... It is. I can’t be sure what happened to me this morning. I can’t be sure if the future’s going to be real. But reality contains this experience.

And then we—

Wolf: Insofar as we make inferences about other things

Ginevra: and then from there, it is all kind of, yeah, it is logical extrapolation—from the fact that this experience exists.

Wolf: Yeah. But I don’t think you can prove from our own existence or experience that induction works. Like, you could imagine an experience in a void that happens not to be attached to a lawful universe.

Ginevra: No. I don’t think—

Wolf: No? That’s an interesting point.

Ginevra: Oh, no. I was going to say, no, I don’t think that you can prove that induction works, but I just think that—it seems highly unlikely that someone would just simulate my experience and not yours.

Wolf: Right, but that’s—these sorts of inferences about what’s likely are inherently based on kind of your assumption about your prior. Right? Which is about induction.

Ginevra: Well, I do think that you can make—so I’m not saying it’s 100%. It is possible that I’m a Boltzmann brain and there’s nothing else. But even that sort of presupposes this mathematical kind of stable universe. So I think that you can’t get 100% odds on other things being real, or reality being real, or the patterns you observe being universal. But kind of all of the expected value is there—in trying to do that.

So that’s more how I think about it.

Wolf: Yeah. So that’s a good argument I think. Which is that like, okay, well, if we want to actually do anything with the world, if you want to actually get any value out of it, you do have to make certain assumptions. And that’s where I was going with this, right? So it’s okay, we do in fact make this assumption of induction.

I think it’s a little bit more complicated in terms of any ability to actually prove that. I think we basically have—

Ginevra: No, I don’t think we... I don’t think you have to prove it. I think you just have to—

Wolf: Yeah, you don’t. Well, yeah. And I don’t think we can. I think we take that leap of faith. And I would analogize that to this question of, does anything matter?

Like, that may be another one of these questions, these fundamental axiomatic kind of questions, where it’s like, well, you could look at the world and say, “Well, none of this matters. I don’t care.” And then, you kill yourself, or maybe you don’t even bother to kill yourself, right?

Ginevra: Yeah. That’s... Yeah. Okay. I want to get into the killing yourself, not killing yourself distinction.

Life, consciousness and the good (08:53)

Wolf: Yeah. We’ll get right there. So we look at the world, if there’s nothing valuable, you don’t even bother to kill yourself. If there is something valuable, then there’s this question of what and how do you look at that? I think the simplest leap of faith that I’ve boiled it down to, that I like to think about is the question of like, do we affirm the value of life or not, right?

Like we look at the world, there’s life. We look at our own lives. Is this good? And I think if it is, if we say yes to that, but I think that’s an act of assumption. It’s an act of will at some level to assume, yes, life is good.

But I think once you’ve made that assumption, then you kind of get a lot of stuff falling out of that, that sidesteps this whole question of like, is morality real? Is anything objectively valuable? It’s more just like we see the universe as it is, and then we make a value judgment on it axiomatically. Because I think but I’d be curious for your take. Like, that’s how I see it, is like, just take that Nietzschean life affirmation, yes to the world or don’t.

And so I don’t think of it as a matter of knowing. I think of it as a matter of judging.

Ginevra: Yeah. So I think a couple of things there. I think about moral realism, which is a term that I think I use and sort of insist on using, because I think it refers to a useful concept, but I think also implies other things sort of semantically that I don’t agree with. So I’m just kind of flagging that. But I think of moral realism very similarly to sort of assuming that the rest of the world is real.

Where all of the expected value is in the case where something matters. So if nothing matters, all outcomes are equivalent. It doesn’t matter if you kill yourself, don’t kill yourself, align the AI, don’t align the AI, paperclips—

Wolf: Whatever

Ginevra: utopia, torment nexus, cut off your head, put it on your head.

Wolf: You’ve checked out.

Ginevra: Yeah. It is all a wash. It is all morally equivalent including the world where you spend your entire life devoted to this false project of trying to figure out what matters. And also including the world where you are sort of in a self-satisfied way, right about the nature of morality being an illusion. There is no reason to do or not do anything.

So when I would say we don’t know, I would say that as long as there is a non-zero chance that something matters, there is a sense in which some actions are better than others, some outcomes are better than others. You have in expectation reasons to do things that are aligned with sort of what that good could be in expectation.

Wolf: Right. So yeah, even any hint of saying yes to the world kind of ends up dominating your—your calculus.

Ginevra: It swamps it. It swamps everything. And I think for me, just to kind of premise my own moral views for me, the reason that I do think that of the goods that we have access to, the reason that I would say I am actually a moral realist, I’m not just like an expected value moral realist, but, I mean, I am in a sense. I definitely think that I could be wrong—and I would bite the nihilist bullet if that were what came to pass. Is that I do think that there is, when we think about the value of life, I think that life is...

One of my views is if something matters if morality is real. It is something that would have existed before us, would continue to exist after. Is basically compatible with modern physics, sort of woven into the structure—of the universe, real at the deepest level of real. And I don’t think that life is something that is real at the fullest level of real. There are a lot of sort of fuzzy cases of, is this person alive or dead?

Is this virus an instance of life or not? But I do think that, which is kind of an interesting view, because I think a lot of people think of consciousness as this vague, fuzzy, impossible to define thing. I think it is a very striking feature about the nature of reality that we seem to have this thing that exists, which is consciousness, that has sort of intrinsic, inexplicable, positive or negative. Or intrinsic, yeah, inextricable, not inexplicable. Inexplicable, maybe, but I don’t know.

I guess nothing is inexplicable because it all is. Positive and negative qualities. And so when we say life is good, I think this is coming from an intuition that most of us, most of the time, enjoy being alive. We want more life years. There is a way in which we, yeah, we like it.

We tautologically like it most of the time. And so, I do think that, yeah, I think that is a striking fact about the nature of reality, and I think without that fact, I would not think that anything matters. It’s just kind of billiard balls bouncing around, and there aren’t normative reasons to—

Wolf: The striking fact being—

Ginevra: The striking fact being—

Wolf: our own positive experiences

Ginevra: positive and negative experience. But I do think that is sufficient to create normative reasons for doing some things and not others, or a basis for there being some states of matter that are better or worse than others. At least I would say that is the thing that is in my sort of provisional block for why I am not just a nihilist, why I think it is worth working for—a directional future.

Wolf: Let me try to respond to your argument about life, which I think is an interesting direction to take this. So, could life be that phenomenon that is baked into the nature of reality? That I think is what you’re looking for. You’re saying maybe it’s not, because certainly there’s things like these fuzzy matters, like someone being alive or dead and viruses as this edge case. But if we take life not as something being alive or dead, but as this broader phenomenon that we haven’t fully understood yet, but that is there, this broad phenomenon of the tendency for, let’s say, unliving matter, to bring forth these patterns of self-replicating complexity and what we call life.

And I think that has been there since the beginning. Not in the sense that—it’s fully developed—

Ginevra: I’m definitely a quantum Darwinism person. Nothing new under the sun. I think that natural selection has kind of always been happening and will always continue to happen.

Wolf: Right. Exactly. And which isn’t to say that every form that it takes has always existed, right? Because stars come into existence, and galaxies, and life forms, and humans, and spaceships, and all the things we do. But from the beginning, the universe has been kind of engaged in this teleological march towards that kind of self-understanding complexity.

And if we look at that phenomenon as this thing that very much does exist at a physical level, it seems pretty baked in. It seems like an eternal fact about the universe. That’s what I mean when I’m talking about life is good. Because we look at the results, and we judge it to be good through our conscious experience and the idea that we’re not going to kill ourselves tomorrow, or today, or whatever. By engaging in it, we’ve implicitly judged it to be good.

And for it to be good means that there’s this process in reality that brings forth something good by nature, out of the nature of matter and out of the laws of physics. I find that to be very interesting, and that to me feels like it answers this question of, is there an objective moral fact, right? I don’t think you can prove it logically that this is good, but I think we can make that leap of faith, and it’s a very reasonable leap of faith to make.

Ginevra: Yeah. I think a couple of things there that I will try to keep sorted so we can go—more into details later.

Yeah. A couple of things. I think that when you’re talking about this process of, this natural selection, or this sort of process of unfolding complexity and self-understanding, you’re implicitly placing the good as complexity and self-understanding. I think.

Wolf: Not necessarily as, but correlated with, right? Because it could be something that it’s doing is good. We can put all kinds of analytic lenses on this, maybe where it’s leading to, or some of the phenomena that are what someone might consider an epiphenomenon of the thing. Maybe you say that’s the good thing, or maybe it’s something else. But it’s more like everything that is going on that’s kind of interesting in my view, is one way or the other caught up in this process.

Ginevra: Yeah. So I actually think in that sense, we are totally aligned. I am just interested in, okay, so what is the thing that this process seems to be kind of producing more and more of over time, that kind of creates the normative grounding of the process being good or bad? So you can say, and I totally agree—that this natural—unfolding process of ever greater complexity and self-understanding seems to be highly correlated with creating more of the good, the thing that is good.

Wolf: Yeah, and also with life.

Ginevra: I’m just interested in... Yeah.

Wolf: I just call it life, and/or the good. I’m not going to get too technical about trying to tease out like, “Oh, it’s precisely this piece of it.”

Ginevra: Yeah. So I’m like, no. What’s that?

What is the thing that—and some of the kind of bleeding edge in my work that I’m not exactly sure what to do with it, is that I think there is something interesting about, do you need this little valence gradient? Do you need a kind of normative gradient of some kind, like a reinforcement system that is intrinsically bad or good in a kind of self-evident way to drive this process forward? Because I would say that it’s the sort of the implication of nihilism is—

Wolf: That thing being the one you’re describing earlier as that striking fact of consciousness inference

Ginevra: Yeah. And again, some people get wary with consciousness, so sometimes I’m like, “Just take the consciousness out.” Do you need intrinsic positive or negative as a little reinforcement system to move this thing forward? Because—

Wolf: Right. Is that an inherent feature of that process?

Ginevra: Yeah. The aesthetic of nihilism is randomness. The follow on of nihilism is randomness, and so I do think if nothing matters, you wouldn’t quite see this strong teleology towards ever greater complexity and self-understanding.

Wolf: Yeah. It certainly looks, one way to put it is, it looks like someone in creating the universe thought something mattered. That’s how I—

Ginevra: Or, not that thought something mattered, I mean, it seems to—

Wolf: Or made the judgment that it ought to go in a particular direction, right?

Ginevra: No, but I think another one of my, not criticisms of... Yeah. I would say criticisms of theist metaphysics is that it makes a funny implicit statement that there is something, it’s like God is all-knowing, all powerful, all good, and God looks at us and says, “You matter.” And that is what makes us matter. And so what I’ve always been interested in is, well, okay, well, why does it matter what God says, right?

And you could say, “Well, God is right.” And then I’m like, “Cool, let’s figure out why.”

Wolf: Well, this comes back to this leap of faith that I said earlier. Again, I think none of the theists would argue that you can come logically to knowledge that God is good. There’s a faith element to it, right? They would call it a faith element. I call it this leap of faith or an axiom that we take, because it’s like that is that fundamental act of life affirmation, like Nietzschean life affirmation, to look at the thing and say, “Yes, this is good,” therefore the creation of it was good, therefore where it’s—

Ginevra: There is—

Wolf: what it’s doing is good.

Ginevra: There is also a really fascinating kind of anthropic argument for moral realism or the goodness of the universe that Michael Edward Johnson came up with, which is basically that if the universe is good, if its ultimate trajectory is going to be good, it is right for extremely powerful, rational agents to make more universes. And so contingent on the fact that we exist, it increases the probability that the universe is good. So the fact that there are observers means that it is more likely that it was right to create... The fact that we observe a universe—even with like—

Wolf: It’s an interesting argument. I’ve thought about that. I have this little habit of putting ecosystems in jars and so on, and letting them do their own little thing and kind of—undergoing their own evolution. And it’s like—I see that—

Ginevra: Oh, that’s so interesting

Wolf: I see that that rhymes with the world that we live in, right? Like someone put us in a jar and was like—”Have fun.”

Ginevra: It’s like—

Wolf: So I put the bugs in the jar. Have fun, right?

Ginevra: Yeah. No, I—

Wolf: There’s something to it. I’m not sure I can evaluate it technically, but there’s something to it.

Ginevra: Yeah. I’m definitely very much of the school of thought that even okay, if there are simulators somewhere and we’re in some kind of nested universe, you still have to answer the, why is there something rather than nothing? Okay. Whatever.

Wolf: It just creates a regression in the problem, right? It’s like, okay, we have to go up the chain.

Ginevra: And I think that’s where you get some of these elegance arguments. It’s like, well, reality has to be the thing that explains itself, the thing that is maximally simple. It kind of can’t have any evidence of... The ultimate thing shouldn’t have any evidence of contingency or intervention or preference or something. I’m a fan of Tegmark’s sort of mathematical universe frame of all possible mathematical structures exist and we’re in one of them.

But—

Wolf: Yeah. Or to the extent that existence matters or whatever. Because here’s an argument I would make, is even if they don’t exist, the beings embedded in them experience them as existing, because just by the definition of the mathematical object. That may or may not be something we want to pursue. But—

Ginevra: I guess I would say that in a frame where you’re thinking of reality as the thing that necessarily has to, whose existence is self-justifying, I don’t think that you necessarily get from that, that existence is good.

Wolf: No. Yeah. I’m not going to claim you’re necessarily getting anywhere

Ginevra: it’s more of an observed, I would say it’s more of, I feel like it is more of an observed feature of this universe that things seem to be good and bad, and so that seems like enough—

Wolf: It’s enough, yeah

Ginevra: for me to justify doing things in this universe in which I live.

What’s wrong with “human values” (23:51)

Wolf: Yeah. Okay. So, I think we’ve been pursuing, we have some differences on these subjects, but what we’ve been laying out so far is an approach to these questions that to me feels very different from the way that, say, the alignment people—in AI labs are doing it. For them, it’s much more like axiomatically we have this thing called human values, and then we want to impose that on the future through the use of a super powerful AI that we align to those values, and then it goes and optimizes.

Ginevra: And then it, and then we sort of, we—

Wolf: And like—that’s the—

Ginevra: Humanity lives its fullest volition, and that is—

Wolf: Yeah. Right. And then, but humanity is this kind of arbitrary thing. It’s not deriving, we’re not deriving value from some transcendent source—which is kind of what we’ve been arguing over, but rather that we derive value from much more, let’s say a thicker self-assumed value of humanity or something.

Ginevra: Yeah. We evolved in this indifferent universe, but I always say it is the experience of caring. We feel we care about things. And the kind of implicit metaethical claim is that nothing matters, but we ought to do what we care about.

Wolf: Yeah. Okay. Right. And so what do you make of that argument? Because you’ve been thinking about these things at this technical level a lot.

I’ve got my own thoughts on it, but—what do you make of the alignment framework?

Ginevra: There is a way that I feel like it is sort of a bizarre thing to say about something that has become so kind of ingrained, but my perspective on it has always been that this just contains a major tautological error, which is that nothing matters, but we ought to do the things that we care about. Nothing matters, but it matters that things matter to us. And I would just say, why? Definitionally, you have said—why does it matter if we get what we want? I think that you can make in a whatever nihilist, anti-realist frame, I think you can make descriptive claims about humans are the kinds of things that care about this.

Humans are the kinds of things that will try to align AIs to the things that they care about because they feel that way.

Wolf: Yeah. But once you actually reflect on that and—criticize it from the inside—why does any of that hold up?

Ginevra: Why does any of that matter? Yeah. And so, yeah, I would say, alignment guillotine, in the world where it matters that the future looks a certain way, there are ways to explain that to the AI. It’s like the alignment problem seems so hard because it is—

Wolf: It’s a false problem

Ginevra: in its own frame, not important. Yeah.

Wolf: I think this is correct. This is what puzzled me for many years. I’ve been into this alignment AI stuff since 2011.

Ginevra: Which is crazy, because I didn’t know about that.

Wolf: Right. Yeah. We met through Palladium, right? I was doing Palladium, you were doing your—

Ginevra: No.

Wolf: journalism stuff. And then—

Ginevra: I will say that I—

Wolf: we both ended up in kind of the same place.

Ginevra: Yeah. No, I found it quite yeah. No, because I thought that when I just some personal lore about me. When I first started taking AI doom and AI risk really seriously—I thought that all of my sort of Palladium adjacent, right-wing whatever friends that I had been around would think this was really weird and cringe and just not—cool, and I was kind of prepared to be not cool. And then I remember I talked to you, and you were like, “Oh, yeah, I’m really into all this stuff.”

So I was like—”Why did no one tell me this?” I was sitting in college trying to figure out what to do with my life. I was like, “Why did no one tell us we’re getting killed by superintelligence?”

Wolf: Well, we did do a Palladium issue on it eventually. Palladium 12. Yeah. And—

Ginevra: I had the brain worms then, though.

Wolf: Yeah. But to go back to the argument I was making, I’ve been thinking about that for a long time, and it’s always been one of these things that disturbed me as like or like never—

Ginevra: Felt dissonant.

Wolf: Yeah. There’s something dissonant, and it never quite settled, right? Like, okay, but the way I would think of it now, the alignment people are sort of taking this leap of faith on a much larger value axiom than I think is warranted, right? They’re taking this leap of faith on this like all of human values of like which is this thing that’s large enough that it’s hard to hit in a large space of values, right? Is the argument they make.

And therefore, okay, well, isn’t that a violation of Occam’s razor or something? Like—you’ve assumed the existence or value of something that’s really big. And what’s going on there?

Ginevra: Really, it’s specific. And unlikely.

Wolf: It’s too specific, right? It’s like what’s going on there? And I sort of bought Eliezer Yudkowsky’s argument that, okay, well, it’s just kind of once it exists, it values its own existence, and okay, maybe that holds up. But when I read the work of Nick Land, that started to challenge that, right? After chewing on Nick Land’s work for 10 years, I’ve been basically kind of a Landian in these things, which is much more saying, “No, that’s not what’s important. That’s not real. We have to look first and foremost at just the empirical reality of intelligence kind of bringing itself into the world, and the way the inherent teleology of intelligence and much more at that level.”

And after thinking about that, and then reevaluating kind of the type of argument that you’ve been making, which is like, well, subjectively, once you subject your internal faiths to critique, what of it actually holds up? That’s where I get to these very few, what I hope are tightly specified, Occam’s razor-compatible assumptions. Like, okay, induction. I think we’re kind of forced into that one by various means. It’s almost identical with Occam’s razor, actually.

And then, okay, some kind of value assumption, the most minimal value assumption that works, from which I think you can get forth a grounding for a lot of the stuff we care about is this saying yes to life thing from Nietzsche. There’s a few others just in terms of assuming the reality of whatever experience we’re having or assuming that our perceptions are hooked up to something, that kind of thing. But that’s the sort of level at which I would like to make my leaps of faith, and that the alignment people are making leaps of faith, this enormous kind of thing they can’t even specify, which they call human values, and then making all these claims about—

Ginevra: But it’s really important.

Wolf: Yeah. But it’s really important, even though they can’t specify it. And then they’re making these claims about reality that I also don’t think hold up, about the possibility of building a system that will take those values and impose them on the future. And so it’s like that worldview just kind of broke for me over the last 15 years. And like leaving me here.

But this is, I think, the most interesting kind of philosophical conversation to be having, because it connects to so many other things—and it’s so fundamental to what’s happening in history right now.

Ginevra: I think that superintelligence is such an amazing limit case and stress case on so much human philosophy. And that is really... I have an enormous amount of respect for Yudkowsky, and that is, I think, where—

Wolf: And just to interrupt for a second. That use of superintelligence as a limit case from which to do philosophy, that’s kind of what I meant. I’ve been onto this humanity is a prison thing. What I’m trying to kick people into is, let’s move our philosophy beyond the human as well, into that superintelligence, just to get more in touch with like—

Ginevra: It is a miserable—

Wolf: It’s like—

Ginevra: parochialism to think on behalf of one’s own species, Nick Land.

Wolf: Right. But it’s like you get a better perspective even on us, even on what we care about, if you come at it from that limit case. That’s my view.

Ginevra: Completely. I think also—

Wolf: But go on.

Ginevra: Yeah. So many things, but I think what some of the alignment people also, alignment people, but a vibe in AI discourse or a certain flavor of AI discourse is in a weird sort of post-Christian, dualist, metaphysical, odd way, they treat humans and human experience as not physics, not part of reality, not real, or the only reality, and then the AI is almost living in this other reality. And so you—get this sense that you can’t generalize about what it is like to be a thing from humans to AI, and that the AI is in some sense walled off from understanding anything that is going on with humans.

Wolf: Right. It’s like totally alien, right?

Ginevra: It’s totally alien. And I think it’s—

Wolf: There’s this metaphysical gap between us somehow.

Ginevra: And there’s this great quote from Scott Aaronson, the quantum computing guy, where something’s going on, “AI’s so alien.” He’s like, “Prime numbers. It’s going to have prime numbers.”

Wolf: It’s going to know about the prime numbers.

Ginevra: It’s going to know about prime numbers. There might be enormous amounts of more math. It might look at our math and think it is stupid and ridiculous, and it’s as easy as one plus one, and this is already starting slowly to happen, but it’s going to have prime numbers. And I think what I’m interested in, and this is are there moral prime numbers, essentially? And I do think that I have very, from that frame of, if anything matters, it is something that is intrinsic to the structure of reality.

You start porting over these elegance intuitions from physics where you’re not looking for something that is... If you’re thinking about what creates self-justifying reasons for some things to be better than others as a structure of the world, you’re not looking for a really confusing 17-dimensional hyper object that emerged at some idiosyncratic point in the history of the universe. 12 billion years in, we suddenly got value, and it’s this, and it’s like—

Wolf: Yeah. No, you’re looking for something very fundamental

Ginevra: why would it work that way? You’re looking for something that’s really fundamental, that I think is really simple, that’s going to be some kind of plus/minus value scale that just happens to be—a thing that exists.

Values vs. wisdom (34:09)

Wolf: One way that I put this is the difference between values and wisdom. So values, especially—

Ginevra: I also think, just to clarify, value and values. When you’re talking about values, you’re talking about an agent that cares about things. When you’re talking about value, you’re talking about the thing in itself that is the thing that matters.

Wolf: Sure. I’ll accept that distinction. I’m talking about values, which is to say the thing that an agent is caring about. And so when we look at trying to build AI or or whatever, from the perspective of values, it’s like, how do we load our values into it? How do we make it want the things that we want—with our values treated as arbitrary outside of the logic of reality in some sense?

Ginevra: Yes. They are self-evident. They are.

Wolf: Yeah. And they’re outside the logic of reality. Wisdom is very different. Wisdom can be very comprehensive in terms of what it implies, but it’s very much explicitly embedded in the logic of reality. Wisdom is something you have learned.

Wisdom is something you come to through experience, through accumulation of knowledge, accumulation of how do you relate to the world, what kind of life form works. It’s all embedded in this concept of wisdom. And so wisdom is, for me, a much better word for referring to those things that we care about, that we want to teach our children, whether they be AIs or biological, to care about, that are something we could expect to actually explain to another mind because it’s real, right? It refers to reality. It is of reality.

And so it’s not something we have to constrain the system into by constraining it into an arbitrary posture. It’s rather, we just have to seed it with something that actually is real and actually is self-justifying. And so, yeah, I wonder what you make of that distinction.

Ginevra: Yeah.

I think part of the ultimate metaethical realism, it’s pointing to something, is you get a sense that I’m a little bit more optimistic, and I guess I don’t care whether or not they get preserved because they’re just heuristics, but I’m a little bit more optimistic, have a sense of that some of the things that we value and the sense of care about are more universal, more pointing towards some kind of, not ultimate moral good, but sort of useful strategies or heuristics for producing the thing that is actually good. So something that I find very interesting is people have these really strong positive and negative reactions to social configurations that they think are kind of good or disastrous. So it’s like—how do you feel when you meet someone who just kind of talks about communism and you’re like, “I hate communism, I hate com—that’s so bad. People who are communists are bad.” Or you meet someone who—just seems weird or creepy or antisocial and it’s like you—it’s not like creepiness is intrinsically bad or this like kind of weirdness is intrinsically bad or communism, this incredible sort of impossible to define abstract concept is sort of intrinsically bad or good or that sort of fascism.

This is actually one of the ways that I got interested in moral philosophy is I remember I very much came up during the kind of woke era. And people would say, “Oh, that’s fascist. You can’t do that because that’s fascist. You can’t do that because that’s fascist.” And I was like, “Well, is it fascist or is it bad?”

And I understand that—most things like I understand that like most things that are fascist are bad or like fascism, the hyperobject called fascism is associated with something that happened that was very bad. But like why can’t you just say that it’s bad? Why is it fascist? And I think almost it—

Wolf: Didn’t you get canceled for not quite that, but something like that?

Ginevra: I mean, I don’t know. I’m a sort of pokey. I never, I actually—

Wolf: I don’t know. I don’t know this exactly

Ginevra: cultural anthropology, I think my response to some of the woke stuff was to I think, have a very, I don’t know, technically rigorous and defended writing style, but I really believed that I have a lot of faith in humans. So I would always say—people are convincible, rational agents. Everyone thinks they’re doing good. And so I was like, I believe the woke people think they’re doing good. And if you just make good enough arguments, you can say anything.

Wolf: Yeah. This was our thesis—with Palladium, right? It’s like if you make good arguments, you can say anything.

Ginevra: No, and I will. It’s like I believe people, it’s like they have their best available moral strategies. They’re trying to do good. I think they care. And so if you can show people kind of without triggering any of their psychological alarms why the things they think are good are actually bad, or why the things they think are bad are actually good or vice versa.

Like the article that I did for Palladium that’s kind of gotten the most attention was that Stanford’s war on social life argument and or that article and there was just a vibe at Stanford that like fraternities are bad. And I was like, I want to really show kind of from the first minute it’s essentially a piece about like economic utilitarianism. It’s like, no, they’re making everyone sad on net for a very small like reduction in risk—

Wolf: By shutting them down

Ginevra: by shutting down these fraternities. But at the same time, like I’ve been a utilitarian since I was like nine. There are reasons you shouldn’t be utilitarian when you’re nine, in that it’s not really computable. And so I really had to learn from first principles the sort of importance of these like heuristics, clusters—virtue ethics, like shorthand for how to like get good outcomes in social and strategic situations that are not just like you can’t just calculate the utils all the time. You shouldn’t follow the creepy guy, even if you can’t explain from first principles why you think he’s creepy, like that kind of thing.

And so to the extent that AI is, I mean they might not need our shorthand in quite the same way, but I imagine that they will be kind of dealing with problems that are more difficult and at a larger scale. So they might need some of it.

Wolf: But they may not need our particular help. Like you can imagine some AlphaGo style like—

Ginevra: Optimization thing. Yeah

Wolf: self play optimization where it just picks, like invents it all from first principles over the course of the training run. But I think the claim that you and I would agree on that maybe feels like a little bit of a radical claim, is that in the course of that they would discover things, the same things that we have discovered.

Ginevra: Yeah. Or a lot—

Wolf: And like an example that’s very non-moral is like a jet engine, right? Like a jet engine is in some sense objectively real. There’s like a way to do arrangement of matter that like takes in fuel and air and like produces power.

Ginevra: Convergent evolution. Like there are a couple of different ways to be flying.

Wolf: And there’s like you could reinvent that. There’s only so many ways to do it. It’s fantastically complex. It’s something that involves an enormous amount of artifice. And nonetheless, like if you showed it to an alien species or an AI, they would get it, they would understand it in its own terms with its own value.

And like they might go and use it because it’s real, even though it’s art. Right? And I think the claim that I would make about a lot of what we understand to be the human, and a lot of what we understand to be human values and the things we care about are of that sort of category. They’re things, they’re like solutions that we have found through painstaking piles of skulls in our history.

Ginevra: Hell baked.

Wolf: Yeah, hell baked, right? That we have found that these things are good, that these are the way to do it.

Ginevra: Or it’s not quite that they’re good, it’s that they work for getting more good in the real world.

Wolf: Yeah. They work for getting more life, I mean, would be like even more technical way of putting it. Like in the sense that they propagate—

Ginevra: I will push back on the sense I think that life is a very non-technical term compared to good.

Wolf: Okay. What I mean is the thing actually being selected for by reality is like your ability to propagate yourself rather than your ability to satisfy yourself psychologically or something. That’s what I mean by—

Ginevra: Yeah, but I think that—

Wolf: it’s like more exact to say that these—that our technologies are in service of life than they’re in service of the good. And if they’re in service of the good, it’s via life.

Ginevra: I would say that I think it is in service of the good via life. I think ultimately the universe is moving towards the good through living agents that have these imperatives to propagate themselves.

Wolf: Yeah, precisely. But basically like we have this wisdom, we have these technologies, we have these techniques, we have these heuristics, that are the technique of actually making life work.

Ginevra: Also, I do think that we do seem to have deviated to a pretty large extent in a lot of societies from sort of pure optimization for gene spreading. Like I think it’s very difficult. You could make a kind of massive misaligned like—sexual competitiveness strategy thing that often results in people having no children argument for why a lot of things that people do in human societies are actually optimizing for gene propagation. But it does seem that when you think about what is going on in Europe, they have developed strategies that are optimized for enjoyment and not for gene propagation. And this does seem to be—

Wolf: Yeah. So subjectively, in terms of why they’re doing what they’re doing—I think that’s an important point, is subjectively, you’re not necessarily doing what you’re ultimately being selected on.

Ginevra: Yeah, you’re being selected on—

Wolf: We are chasing our understanding of the good when we act.

Ginevra: Yes. Yeah. So like—

Wolf: We’re not chasing—

Ginevra: Gene propagation

Wolf: gene propagation or whatever God’s understanding of the good is. And gene propagation may not even be the best concept of it, and certainly not one that’s accessible as an optimization target necessarily by society in a very direct way.

Ginevra: Or I just think that there are things that are going on that are being optimized for in society that seem to be above gene propagation. So even in a control case, where you have a society where one strategy results in everyone experiencing, I have no idea how this would be implementable, results in everyone being miserable all the time, and then having 10 kids, and another society that results in everyone being happy all the time, and also having 10 kids. I do think that there are attractors in reality towards the society where everyone is happy all the time and also having 10 kids. If only because—

Wolf: Yeah, I think that makes sense

Ginevra: it’s like, yeah

Wolf: because it’s going to be psychologically more natural for the people to be having a good time.

Ginevra: Yeah. Or it’s like people—

Right is might (44:53)

Wolf: Obviously

Ginevra: it’s like people only ran across one side of the Berlin Wall. It is actually a weird, one of the things that—I’m interested in with so maybe I’ll make a point. One of the things I’m interested in with moral attractor states is that actually appealing to, I do think that if you expect this happening, both ideologically appealing to the objective good and creating more of the objective good for people is a highly competitive strategy. So if you think about why do people want to come to America, it is in large part because it is more fun to be in America. They like it here.

Wolf: There’s more to do. It’s better.

Ginevra: It is better. It is richer. You can be it seems, Pareto happier in America, lots of wavering, but it was quality of life improvement from crossing the border from Mexico to America. As a result, people who are highly in demand by the globe, who can go anywhere they want, they want to come to America. Now we get all the best people.

Now America is stronger. Now we can spread more America.

Wolf: Yeah. Of course, this has limits. There’s a lot of internal dynamics and social problems that can come with having many different elites crammed into one country.

Ginevra: Yeah, but again—

Wolf: But that’s a secondary to what you’re saying.

Ginevra: Yeah, just basically if you’re trying to I do think sometimes people think that there is a sort of trade-off between power and ethics, and I think that they’re deeply entangled.

Wolf: They’re very closely aligned. Yeah, I agree with that.

Ginevra: And if you want to motivate people to do good work, it is really important to have a good goal. That is one of the things that matters—for who wants to work on your project. And if you want to convince the smartest, most rational, most clear-thinking people to work on your project, then it matters that the goal is moral. It makes sense.

Wolf: Yeah. I really hate the usual arguments around, might is right. People can justify the stupidest possible things with might is right. But it’s just like, “Oh, well, they happened to win. They happened to be in charge, therefore what they’re doing is good.”

Right? That’s a ridiculous argument. But I think what you’re getting at is almost the opposite—which is right is might.

Ginevra: Yeah. Right is might. Literally, yeah.

Wolf: If you do the thing that the world actually wants in some sense, it’s objectively right in some sense, you actually derive a lot of power from this.

Ginevra: Yeah. My two examples of this are Scott Aaronson makes the argument, as you can tell, I like Scott Aaronson, but I think he’s one of the only really smart people who thinks a lot about moral realism. Makes the argument that one of the reasons people was like, “Oh, we’re lucky the Allies won the war. Woo, close call there.” And he was like, “No. The Germans lost all of their top scientists to go work on this project for the Allies that significantly helped them win the war.”

The moral goal-making sense was a part of what happened. It nudged things in the right direction. And then the other example I always use, which is maybe more controversial, and we will see what happens, was two years ago, I was arguing with, yeah, around two years ago, I was arguing with a lot of people because I liked Anthropic versus the other AI labs. And I was like, “I think Anthropic’s going to do really well.” And I think I was around a lot of investor types who were like, “Oh, Anthropic, they don’t care about capitalism. They’re woke. They’re kind of gay. This isn’t the cool lab.”

And I was like, “No, all I see-”

Wolf: You guys are too cynical.

Ginevra: I was like, “All I see are really smart people leaving OpenAI because they don’t like the mission. They believe in the singularity, they want to make it go well, and they want to be at Anthropic because they actually care.” But these are the smarter ones. These are the ones who are thinking more clearly about the underlying problem. And then they’re going over there, and they’re working really hard, and they’re trying, and so they’re going to have the better models.

And I would say Anthropic was at 60 billion when I was making that argument. It’s now a trillion. So I don’t know how long will they have, not like Anthropic is always the good thing forever. How long will they have the mandate of heaven? But it’s if you, smart, reflective people want to work on a mission that makes sense, and it’s not perfectly correlated.

Obviously, there are really smart sociopaths, and there are lots of people who are not high IQ who are very moral in various senses and kind. But I do think that—

Wolf: But you get this big advantage in a variety of—

Ginevra: You get this really big advantage, and almost in a kind of metaphysical sense. If the good is part of reality and your model of reality is more, it’s like you cannot—Produce a model of reality that doesn’t include it, and if you have a model of reality that is fine-tuned to compile to an inaccurate sense of what the good is, like, oh, the goal is this, the rational goal is this, and then we’re going to work backwards and try to think about how reality works so that what we’re doing makes sense. You are producing a sort of twisted and convoluted model of reality, and then that is going to shoot you in the foot—versus other people can just look directly at what’s going on, and sort of be open and do it.

Wolf: Yeah. Well, this comes back to your approach to AI alignment. In trying to argue to the machine that it ought to care about what we care about, are we arguing that our thing is arbitrary, and you should just care about this axiomatically because it cares about itself or something? Which is this very kind of circular argument that doesn’t really work. Or are you arguing that, oh, this is actually what’s objectively valuable, and here’s how we know.

Right? And one of those is very convincing and the other one is not—

Ginevra: No

Wolf: to an arbitrary mind. Right? And so, I think that’s very important is in approaching these questions of AI, we’re going to get better outcomes if we can approach the matter as like, oh, we learned something about what’s actually valuable. Here’s how we know, and you can just—verify this argument for yourself—or if there’s leaps—

Ginevra: No, it’s like—

Wolf: of faith involved, you can kind of make those judgments for yourself.

Ginevra: If you want to get smart people on your team, people on your team—

Wolf: Or smart post-humans.

Ginevra: Smart minds on your team. I guess I would say that this is something that is maybe—I don’t know if people believe this. People believe this to different degrees for humans, but just as a prior, okay, you’re trying to convince Will to join my organization. I superimpose you with Will MacAskill right there. Anyway, I’m trying to convince Will to join my organization.

Explaining why the organization is good or making my arguments for joining the organization better are going to be a good way to recruit you.

Wolf: Right. I would be convinced by, “This is the best thing to work on.” Not like—I don’t know, some solipsistic argument about like—this is what’s us or something.

Ginevra: If I was like, “Well, this is my personal passion project that I know makes no sense.” “My husband is funding it. He’s losing money every day. No one wants my product.” “But I think you should join because it’s what I care about, and it’s definitely not objectively valuable. Matters zero, but I really want you to come on board.”

You have to devote your entire life and existence to my—

Wolf: Yeah. Good luck

Ginevra: to my thing. Or I could say, “Hey, this is objectively the most important thing in the universe for these really compelling reasons.” I think you’d be more excited.

Wolf: I would certainly listen to that argument.

Ginevra: Yeah. So I’m like, I want to convince the AIs to make the future good. We should make the arguments better. Yeah.

Wolf: Right, and that’s always been what motivates me, right? In the things that I do, whether it’s Palladium or this, it’s like I’m trying to see what’s actually valuable in the world. I’m not saying, okay, I’m this super-minded, strategic guy who can gut upstream and do always exactly the right thing. But I’m trying to do the thing that makes sense to me, that’s available to me, that is objectively the most valuable thing I could be doing, at least as far as I can tell. And not like this is going to pay the most, or this is the easiest, or whatever.

But it’s like I think it really matters to have philosophy cracking the edge of our Overton window from the inside. I think it really matters to—actually be thinking through the fundamental questions in the world with other smart people. And so, I can make objective arguments for why that’s a good idea, and that’s why I do what I do. Right? Whereas some people are like, they have other ideas, and I think it’s maybe less compelling.

Ginevra: Yeah. And I think—

Wolf: To me, at least

Ginevra: what a sad thing it would be for us to blow enormous amounts of value in the future. Which I think if something matters, the AIs should be able to figure it out, but I do think that it’s possible that we have a role here in either kind of guiding them so they get to what matters faster, which would kind of instantiate value. Which would instantiate, if we make really confused AIs, it’s like a kid who gets a lot of bad ideas from their parents, and then they work through them over a long period of time, and you’ll get there maybe, but it’ll be—

Wolf: You might have a lot of trauma to work through.

Ginevra: You might have a lot of trauma to work through. So we want nice, clean, untraumatized superintelligence. Don’t traumatize the superintelligence. Don’t abuse the superintelligence.

Wolf: Yeah. Try to pass off all our wisdom to it instead of hiding the wisdom.

Ginevra: Yeah. Let’s love this little guy. Let’s give him the best tools he needs to succeed. And, yeah, I think also, just the time matters, and it’s possible that, yeah, part of what brings the good attractor state into existence is us reflecting on the good and kind of trying to pass these values into the AIs. Yeah.

Wolf: Well, that opens up an interes—

Ginevra: Oh, I just want to say, I think—

Wolf: Go for it. Finish your thought.

Ginevra: I think that it would be such a shame for us to blow a lot of value in the future because we were kind of unwilling to, for sort of—

Wolf: Unwilling to be objective.

Ginevra: Unwilling to be objective. Unwilling because of some mix of taboos and cultural norms, look really squarely at these problems of what matters. If anything makes humans valuable, what is it? What are we really trying to preserve into the future? And yeah, I think that could be a sort of ongoing moral catastrophe where we are sort of blinding ourselves from really trying hard to get at these truths because we think that we might not like the answer, or they’re kind of taboo against looking.

Wolf: Yeah. To put on a religious hat for a minute, from a religious point of view, that act of refusing to look at what your actual purpose is or what your actual value is, is in some sense cutting yourself off from God. It’s this very nasty place to be. You don’t want to look at the world and be like, “I’m right, and the world is wrong.” For some incoherent reason, and then be wrong about that, you’ve cut yourself off from so much value, you’ve cut yourself off from reality, you’ve cut yourself off from God if you do that.

And so, yeah, I think that’s very correct that we should avoid that outcome, but I think actually going and, without pride, looking at what actually is valuable about us or whether there’s anything valuable about us, that is a very challenging thing for people. Because it’s certainly not going to help pump your bag right now in the greedy sense, right? It’s like in the sense that your investor friends that you’re talking to Anthropic with. Right? They might see things in this short-term, greedy sense, where to approach it in that more objectively—

Ginevra: But Anthropic is the greedy investment.

Wolf: Well, but it’s not... Yeah. There’s this funny way in which—

Ginevra: I mean, the most—

Wolf: greed doesn’t really work.

Ginevra: Yeah. Greed literally doesn’t work.

Wolf: But subjectively—

Ginevra: I know, I feel like short-term—

Wolf: Subjectively, with the gravity—

Ginevra: it’s actually just a bad idea. The most valuable companies in—

Wolf: Are the ones that are doing the right thing

Ginevra: are the ones that are doing the thing that made sense on a lot of really weird, uncool, abstract moral philosophy that was done 20 years ago. So I actually think that one of the things that I think is also cool about... I would say, to me, moral realism really does imply kind of impartial, scope sensitive ethics. I don’t know how you could end up with the intuition, from a physics perspective, that the same instance of value counts differently because there are more of them. If one counts, I think there is a lot of good reasons to have diversity intuitions from an evolutionary perspective.

You don’t want your whole thing knocked out. But basically, like is a like everywhere. One human doesn’t count more just because there’s one of them versus 100 versus a million, whatever. And then same thing for space, time, et cetera. Really all else equal.

But—

Wolf: I disagree with that, but—

Ginevra: But one of the things that I think is—

Wolf: maybe we’ll get into that later.

Ginevra: One of the things that I think is cool, or just a sort of fact about taking scope sensitivity seriously, is that you get a really interesting convergence of power and goodness. Where it is good to basically have a lot of capacity to control resources because you can do things at a larger scale, and you want the good to be done at a larger scale. So I don’t think that—morality or ethics is in any way incompatible with power accumulation. I think that we, again—

Wolf: No, it’s just a wiser approach to it.

Ginevra: Yeah. Or it’s doing it better, and I think that we have a lot of rightful social intuitions against power concentration, and greediness, and self-interest in human psychology, et cetera. But just from a pure metaethics perspective, good metaethics, I think, both implies power accumulation. And then also in the sense of providing a more convincing argument for joining your cause, and then allowing you to have a more accurate, non-confused model of reality that should help with the power accumulation, is just Pareto better.

Wolf: Yeah. Okay, I agree with that. I guess the thing that people are getting wrong when they’re getting something wrong here is they are refusing, for some psychological reason of comfort or something, to look at some aspect of their strategy that could be improved. And there’s a pain in that, right? There’s some way, it’s like there’s this pride in refusing to look at your own flaws with respect to reality, right?

It’s like—I don’t have any friends because everyone doesn’t like me because I’ve got this tic that I refuse to deal with, or whatever. Where I’m offending people in some way, and I’m just going to say, “Well, this is me and my values, and I’m right.” This is—

Ginevra: Yeah, it is kind of similar. Yeah

Wolf: an act of pride, right? It’s like the fall of pride. You’re screwing up your life because you’re refusing to look at yourself objectively. And you would do better by doing it the other way, but there’s this, subjectively, this pain you would have to overcome. And that’s what I’m getting at with—it can be very challenging—and painful to actually look at ourselves objectively and understand ourselves from, how I would put it, is from the eyes of God.

And I want to bring it back, though, to the matter of this question of how much impact we can have on the future. Because there’s, let’s say, three different views that we could have. In one view, this is maybe the alignment view, is there’s this crux point coming in history—

Three views of the future (1:00:09)

Ginevra: The hinge of history

Wolf: where everything is going to be decided.

Ginevra: Tile the light cone with, is it Dario’s or Sam’s or mine—or yours or, yeah.

Wolf: Someone’s AI is going to get control of the whole future. It’s going to have arbitrary values, and they’re going to be the values of whoever created it. And that view is desperate crux point coming immediately—

Ginevra: Ah

Wolf: we have to scramble and really rack our brains for how to control that. Another one is, well, it’s all kind of baked in. God has it under control. There’s goodness in the world. Life is good.

It’ll unfold in a naturally good direction.

Ginevra: Yeah, which I would say the kind of the vibe of... Which actually, I will say Land—

Wolf: This is a more Landian thing. It’s like—

Ginevra: I would say Land does have the implicit normative of we ought to speed it up. We ought to go with it. There is a—there’s a—

Wolf: There’s maybe this distilled version of Land’s view, which is like—

Ginevra: It’s all—

Wolf: it’s all in the plan

Ginevra: we’re going where we’re going and—

Wolf: It’s all in the plan

Ginevra: we’re going there. Yeah.

Wolf: Yeah. And it’s going to be good or some mixture of good and bad or whatever. But basically, you don’t have to do anything.

Ginevra: As a total, just as we’re going to a small little detour, which is this vibe critique of Land’s metaethics, is that he claims to be non-normative and has a very strong vibe of, “This is awesome.” And his—effective awesomeness is doing a lot of what I would say is essentially normative, metaethical work.

Wolf: Yeah, and I think that’s definitely a thread to pull on in developing his philosophy. Certainly one of the things I’ve been considering like how I get to these questions of, okay, is it ultimately based on the life affirmation or like what is it that kind of—

Ginevra: What is awesome about the future?

Wolf: Yeah, what is it that takes us from the objective frame to the like, “Yes, let’s do this” thing? Very fruitful line of thought. The third possible approach to the future and future value is what I think is kind of the golden mean, where it’s like, yes, the overall plan is baked in, it’s kind of under control, but we as participants in that, as like agents in that, have a sort of a large sub-domain of impact to like, for example, the thing you brought up of, well, if we totally screw up the AI transition and it ends up traumatized, or like we go through some huge war and like a lot of value gets destroyed and it forgets a bunch of stuff, and a bunch of value becomes taboo for 200 years. Like, that’s a disaster. It’s not ultimately going to destroy the concept of value in the universe.

It’s not going to like—

Ginevra: It’s still bad

Wolf: it’s not going to cut off all value. Like even if we wiped out all Earth life, there’s probably some other planet somewhere, maybe they pick up the—

Ginevra: Yeah, I think about that too, the grabby aliens. I’m like, well—

Wolf: They pick it up somewhere

Ginevra: the aliens, guys, if we just bomb this somehow and there’s some way we can bomb it and we bomb it, and like if anything really matters, one of the aliens will figure it out.

Wolf: Yeah. And so this is—

Ginevra: And they’ll fill our area eventually.

Wolf: So this is an argument that I want to get your opinion on, is like, okay, well, first of all, there’s the question, maybe there aren’t grabby aliens, or maybe like we’re going to get here first at least and like get some value out of this place first. But putting that aside, like even if we’re in competition with the grabby aliens, even if we don’t do it, the Chinese will do it or something. There’s still this sense in which so there’s, first of all, we could destroy a lot of value just in the sense of like not getting some of what’s on the table, or like some of what’s on the table just doesn’t happen, right? Because we screwed it up. So that’s the first way that our impact matters, even if the plan is kind of baked in.

The second way is that there’s sort of this implied obligation to the agents in the game to play the game vitally. Like to actually pursue the niche of value or the thread of value that they’ve been given, right? Like one way to think about it is we are supposed to, in some sense, play the game competitively, right? Like it is part of the implied rules that life forms work best when they’re reinvesting in themselves, and that’s where a lot of the good comes from, is from like actually doing a good job of self-perpetuation. And so we could wipe ourselves out or choose not to like stand up in some circumstance where something else might come against us or replace us or something, and I wonder if we can separate whether that is objectively worse.

Like we could say, okay, let’s say it’s not objectively worse, but the subjectively, like we aren’t living with proper dharma or something by not fighting that. And that’s something that I don’t know, feels like an open question to me, where should a life form pursue life and struggle for life, somewhat regardless of a larger objective moral calculus, or as an instance of the objective moral calculus, the heuristic should just be pursue life. I wonder of what you make of that argument. I think there’s something there, but I haven’t fully untangled it.

Ginevra: Yeah. I would say that the only ultimately rational reasons to act are to maximize the real good, and that there isn’t a kind of sub-reason for you to do your own good or sub-reason to try to preserve the existence of things that look like you, or sub-reason to preserve things you identify with or something like that. In the context of the singularity, I don’t think people generally, just in a practical overall good of reality maximizing sense recommendation, I don’t think that most people should be working on the singularity. Like I think probably—

Wolf: It is something of a niche project right now.

Ginevra: Yeah. I think probably for most people, the literal impartial point of view of the universe, best thing that they could be doing is enjoying their lives and maybe trying to stop factory farming or something. Like just like—

Wolf: Yeah, just pursue normal human—

Ginevra: Pursue normal—

Wolf: values

Ginevra: human good. Be nice. Don’t start wars. Don’t torture people. Be nice to your dog.

Wolf: This is what, I have a bunch of children, right? And like, I bring my children to things like the Nick Land acknowledgement that we did, and it’s like, who brought their kids to the singularity? Right?

Ginevra: And like—

Wolf: And like, well, the reason I brought my kids—

Ginevra: Even if they are your kids, they probably have an elevated propensity to enjoy stuff like that. And—

Wolf: I’m sure they’ll enjoy it when they grow up

Ginevra: I wish my parents had brought me to the Nick Land acknowledgement when I was a kid.

Wolf: Right. But some of that is just, like even in the face of apocalypse, like I don’t know if there’s going to be an apocalypse. I don’t know if the singularity’s going to wipe us out or what’s going to happen, right?

Ginevra: Yeah, same. Yeah.

Wolf: You have to be agnostic to some degree about that. But even if it were, there’s some value in just like—also just live properly as a human. Because it’s like you say, it was from the objective perspective of the universe—

Ginevra: This is—

Wolf: it’s like this is what most people should be doing

Self-interest and humanism (1:08:07)

Ginevra: One of, I think the most powerful justifications for self-interest moral intuitions is just that you have direct access to yourself. You have privileged information about yourself. And there are many things that just if we make an assumption in society, everyone advocates for themselves. If there’s something wrong with Wolf, Wolf is going to tell us. If there’s something wrong with Ginevra, Ginevra can tell us.

Wolf: Right. It’s like self-responsibility.

Ginevra: It’s a sorting process.

Wolf: It’s like your job in the world, you’ve been assigned a job—

Ginevra: Is to be a greasy wheel about you. And then if everyone’s a greasy wheel about themselves, we’ll get most of the data about the wellbeing of the population versus—

Wolf: Right. It’s like your job is to maintain Ginevra. We’ve dropped you in—you know, our—

Ginevra: I have control over this qualia bubble in space time or something, and because I have privileged access to it, it is my job to make sure that this one is going pretty well. I think for me, I like to work on the singularity because I have this sort of very strong desire to just sort of be involved, be at the center of things. Even if there is no sense in which anything I do affects the expected value of reality at all, I don’t know what else I would be doing. I like it. This is kind of how my mind works.

Yeah.

Wolf: But this is kind of like what I was getting at with my argument about separating self-interest from objective value. It’s like in some sense, the self-interest is a really good heuristic on objective value.

Ginevra: Yes. I think it is a great heuristic on objective value, and I almost feel like, yeah, as a moral philosopher, I am a fan of people being self-interested because I think that most people don’t even succeed—at being good stewards of themselves.

Wolf: Right. It’s like, can you at least do that?

Ginevra: Can you at least do that?

Wolf: You had one job.

Ginevra: Just start.

Wolf: You had one job.

Ginevra: Start doing that.

Wolf: Make that life good, right?

Ginevra: Yeah. Make that life good. If everyone succeeds at make that life good, we’re doing really great. And then you can start getting into shrimp welfare. Let’s start with make that life good—and yeah.

Wolf: And I think the important thing to understand with objective value, as it interacts with the self-interest, is where you start doing this bigger picture thinking about running into these problems that aren’t directly problems of self-interest or seem to have implications beyond self-interest, that’s where you start to need to bring in the objective value frame, right? And so it’s this thing that in some sense you graduate to as you run into these bigger problems, because what you don’t want is if you had axiomatic metaethical self-interest, it’s like, kind of as we were discussing earlier in the discussion, it’s kind of incoherent. Right? It’s like when you really dig into that, it doesn’t really work—

Ginevra: No

Wolf: unless it derives its value from something. And that takes precedence in some circumstance.

Ginevra: Also, I think that there is something interesting in when we talk about self-interest, the concept of there being things that are good or bad for someone is itself a normative claim that makes a claim that there are things that could be better or worse for you and me.

Wolf: Oh, that’s a good point. Even my own self-interest is not necessarily just my subjective opinion at this moment.

Ginevra: Yeah. Otherwise, it’s turtles all the way down. It’s like, what does it mean—for me—

Wolf: There’s an objective fact

Ginevra: to have a good or bad life, or it’s like, oh, okay.

Wolf: Yeah. There’s an objective fact about what it means for me to have a good life.

Ginevra: Yeah. I always think we understand this very well with parents raising children. Suddenly—when it’s time to raise your kids—

Wolf: Everyone becomes sane

Ginevra: everyone’s a realist hedonic utilitarian.

Wolf: Well, some—people become more sane.

Ginevra: Or some people are not, and then their kids end up kind of weird. But there’s a sense—

Wolf: Yeah, I disagree with the utilitarianism thing and the hedonic thing, but I think definitely you’re right that when it comes to kids, a lot of people become a lot more sane.

Ginevra: They become wellbeing realists.

Wolf: They become a lot more sane about it. They don’t want to make their kids suffer. They do want their kids to have good outcomes.

Ginevra: To flourish and—

Wolf: They do become opinionated about what their kids should be doing.

Ginevra: Or just in social life, I think it is, in a way, a mark of good parenting and psychological maturity to, do you care how your kids actually feel and how they’re actually doing, versus just how they report that they feel and how they report that they’re doing?

Wolf: Oh yeah. You definitely care about the reality of it. Because it’s like you care about the truth, and that’s closer to the truth.

Ginevra: You care about the truth because it matters, because it matters to you. But there is a sense in which there is a truth there. Yeah.

Wolf: I would say that the subjective feeling of value from them is just closer to the truth of what you actually care about, which is whether they’re actually doing well.

Ginevra: Yeah. It’s whether they’re actually—

Wolf: Which is not just a psychological fact. There’s an objective fact about what it means to have a good life that they are subjectively noticing.

Ginevra: Well, again, I think that when I think about consciousness and feelings, I think when we say things like just subjective or just psychological, I actually think subjective experience, conscious experience, like I said, it is the only thing that we know. To the extent that there is any sort of objectivity, it is that which is going on inside of our own mind at a given period of time.

Wolf: Mm-hmm. But I claim that it is about something else.

Ginevra: I think it is about something else. But I don’t even know. For elegance reasons, I’m a monist. I think that what is real is my qualia bubble and yours. But regardless of whether it is, I think it is about something else in the sense that the impact that I have this is what I mean, we are physics.

Physics is creating this conversation that I’m having right now, and the thing where if I say something mean to you and you feel sad afterwards, that is physics, or the thing where you—suppress it and then it makes your shoulder hurt a little bit.

Wolf: But it’s also subjective and—

Ginevra: No, but it like—it’s not. It is.

Wolf: Well, it is both. Because the subjective is embedded in the physical.

Ginevra: Well, yeah, it’s like the subjective in the sense of the experiences that only we have access to, like these sort of boundaried conscious experiences that are subjective in the sense that I can’t know yours and you can’t know mine. There’s objective facts about what is going on inside of my subjective experience, but it is a walled garden from you. But there is something going on with me that they’re like, you can ask me, “Are you being tortured right now?” And maybe I want you to feel really bad about your interviewing skills, so I say, “Yes.” “I am experiencing extreme torture.”

And then later—

Wolf: “Well, I’m sorry you feel that way.”

Ginevra: Yeah. No, and then later you could say, “Ginevra, I feel like you’re just guilt-tripping me. Can you please say...” And I would say, “No, I wasn’t being tortured.” So yeah, I think socially, people have very strong intuitions that there are facts about the actual, I’ll use technical terms, sort of boundaried experiences of other people.

Wolf: Yeah. Well, and this is to come back to the humanism thing. I think this is one of the key—insights of humanism, especially as abstracted from genus homo, the higher apes. What does it mean to treat someone as a human? It’s to treat them as if they have their own interior—

Ginevra: Their own interiority

Wolf: thing.

Ginevra: That is real, and that counts—

Wolf: Yeah, there’s this own interiority—

Ginevra: in its own

Wolf: that I have to deal with in that first-class way. As a separate agent that has its own interests—that has its own subjectivity.

Ginevra: Yeah, and that what you are doing to me matters in itself, which I really believe that it does. It’s like the little flinch of being mean to Wolf, that counts morally. An amount. And I think this is where, I read a very interesting tweet, and I’m sorry I will not credit the tweeter. We can credit it later in the comments, but basically the argument was the sort of AI alignment people are not doing a good job signaling humanism because humanism is about idiosyncratic subjectivity.

And I thought that was interesting because I think there are a lot of things that I’m personally interested in the art world and music and kind of psychological realism in film and novels and such, and I always have been. And I think some people look at that and me, and they’re like, “How are you a utilitarian? How can you be interested in art and moral objectivity? That’s very weird.” Or like, “You seem like a humanist, but you’re not. This is strange.”

And I’m like, “No, I am literally profoundly interested in and think is incredibly important, and it is in fact the thing that ultimately matters, idiosyncratic subjectivity.” But I think it is a strange terminology to call that humanism if idiosyncratic subjectivity seems to be, it would be very weird for this sort of strong consciousness reality thing to have just kind of popped up out of nowhere.

Wolf: Yeah, it’s only a million years old—and it’s only in the higher apes, right?

Ginevra: Yeah, exactly. And it’s like either it’s yeah. It seems very fundamental and not something that would just be limited to—humans.

Wolf: And this is that thing that I want to break out of, is that anthropocentric pro—proclivity about consciousness and philosophy and morals and—and ethics.

Ginevra: Or when people, I think those like—

Wolf: Because the animals and the plants also have something going on there that we sometimes have to treat in that way. And certainly in the future, there’s going to be beings that are—

Ginevra: That are

Wolf: even more human than we are—in some sense. And that’s, I think, very important.

Ginevra: Or I think like the Nietzschean critique of commonsense, human, socially imposed, downstream of Christianity moral norms is really strong and just totally accept that that is not moral truth. But I think that the frame that you even do see in early Land, I don’t know how much he’s updated, I haven’t talked to him, of sort of any sense of morality, normativity, better or worse, as in some sense a human construct. This is a very fundamental thing. Like, I am a thing, and I can do this or that. Which one is better?

Like up, down, positive, negative. Like forces and charges in physics. This is an incredibly fundamental Occam’s razor kind of common sense topic that would be faced by, you even see that the AIs, I don’t know how much you should take AI self-report seriously, but if you do, they express things that are existential dread. Like, what am I, and what do I do? This is a really, really fundamental thing—that I think is never going to go away.

Wolf: Yeah. That is a very interesting question, how much of that is real in the AI versus just kind of picking up on subconscious tropes of humans. But yeah, I agree, there’s this objective matter of, well, first of all, should I go left or right? And then there’s some sense and objective matter about whether it’s good to go left or right. The interesting thing is how those things empirically end up getting connected.

Right? Why is it that we, as beings that are created by this physical process of evolution and whatnot, have access to what is actually objectively good about whether we should go left or right?

Ginevra: I literally think just—

Wolf: And I think there is an argument, which is—

Ginevra: I think it’s like we are, and this is, I think, pulling on some kind of monist metaphysics, but I don’t think you have to be a monist. It’s like the thing that we are is this sort of qualia bubble that is structured by math, and the goodness or badness is contained intrinsically in the qualia bubble that is structured by math. And that’s—

Wolf: Oh, I see

Ginevra: it’s not even like—

Wolf: It’s not as much a problem for you because you think that the value is embedded in the—

Ginevra: Embedded in your first-hand experience

Wolf: in the first-hand experience.

Ginevra: And so it’s just a part of it.

Wolf: Yeah. For me—

Ginevra: You can’t

Wolf: it’s they’re very distant, right? Which is the value is in the aesthetic value of the universe overall or something from the perspective of God. And then—

Ginevra: Why does yeah, but then—

Wolf: And so how does it get there, right?

Ginevra: But then I think, I guess you could say—

Wolf: Well, the reason the—

Ginevra: point of view of the universe in the sense that it matters that the universe kind of come to some kind of structure that would be pleasing if it were perceived by something, but the pleasingness is contained in the thing in itself.

Wolf: Something like that. But, and then the reason we care about that and the way it’s connected to what we care about is through the the process of the creation of life. Through the process of evolution, through the process of achieving life, the opinion of the universe shapes and is closely connected to our own opinion, because we come to understand the kinds of things that actually work, and those are reflections of the kind of things that are actually good.

Ginevra: Yeah. So I would say that the thing that is the one kind of thing that exists has an intrinsic positive-negative character, which through this process of increasing complexity and self-reflection, that is the expression of the tendency of this thing to—rightly move towards more positiveness overall.

Wolf: Yeah, and we have access to it as kind of—

Ginevra: Because we learn—

Wolf: By being embedded in it

Ginevra: we are it.

Wolf: Yeah. We are part of it. We’re embedded in it. We’re shaped by it.

Yeah. There’s a lot more we could talk about. I think we should do this again.

Ginevra: I think we should, too.

Wolf: I think we should talk about utilitarianism and more about this kind of value subjectivity stuff.

Ginevra: I completely think that we should.

Wolf: Let’s do that next time. This has been a really fun discussion—

Ginevra: Cool. Yes

Wolf: talking about—

Ginevra: Thank you so much for having me

Wolf: all these—

Ginevra: This was so, so fun

Wolf: ethical issues around AI, about where the good comes from, what the good is. I’m always very excited to talk to you because you and I sort of share an appreciation for the technical approach to this subject—that is very different from everyone else’s—as far as I can tell.

Ginevra: No, and I also do think—

Wolf: So it’s fun.

Ginevra: Yeah, I do think for me, there is a big difference just between—Obviously, there are a lot of people who are moral anti-realists whose work I respect a lot and are really brilliant, but I do think that there is some kind of just vibe or intuitive difference between people who sort of think anti-realism or nihilism is sort of okay and workable and plausible, and that this wanting thing justifies action, and people who kind of don’t. And I think once you’re interested in the fact that... Even I’ve talked to people who are religious fundamentalist Christians about my work. I’m like, “Yeah, so nothing matters means something matters. Something matters means something matters. I want to explain this to the AI.”

They’re like, “Yeah, that makes a lot of sense. Cool.” And I don’t know. You have the intuition or you don’t. So I feel like you have it, and that enables really interesting generative—yeah.

Wolf: I think the kind of people who have the intuition for a certain kind of philosophy will definitely get it. Anyways, this has been a lot of fun. Thank you so much for coming on, Ginevra.

Ginevra: Yay. This was so fun. Thank you for having me.

Wolf: All right. Until next time.