Joscha Bach and Wolf Tivy discuss how consciousness may be a convergent strategy for organizing both human and machine minds, whether powerful AI minds will resolve conflicts by merging or by fighting each other, and much more.
Transcript
Intro (00:00)
Joscha: As you know, I’ve been starting the Institute for Machine Consciousness a year ago. And it’s five minutes from here, from your thing. And to me, this is, in some sense, mostly a philosophical project. So it’s not a startup. It’s not so much a practical thing as it’s a continuation of my enduring interest to understand what the human mind is, and what it’s about, and how it’s realized in physics.
I always thought that AI is the most practical, pragmatic, and epistemologically useful approach for understanding this.
Wolf: Yeah, this is very true. I’m also obviously extremely interested in philosophy as the approach to these things. We’re doing this project, The Students, very much a philosophical project as well, somewhat wider in scope, not just focused on machine consciousness. But yeah, AI as a subject that you can go and study that offers a new epistemological approach to questions like consciousness, questions like the nature of man—the nature of civilization, the future of civilization. I’ve found it incredibly productive, actually—to think about all those questions through the lens of AI.
I’ve been doing this, what I’ve been calling the Xenohumanist Project. Kind of trying to maybe problematize our concept of humanity a little bit—as we stare into the AI future. Looking at, okay, could there be things that we might call human way out there in possibility space? Or maybe we’re not as human as we think we are, or just kind of exploring those kinds of angles. And the AI angle has been very productive for that.
But I’d like to hear more, I guess, about the California Institute of Machine Consciousness. I read a little bit of your white paper, and you’re taking a computational functionalism approach, which I think is, from what I understand, it seems like maybe the most level-headed approach that I’ve seen. I know there’s a lot of people saying a lot of things about consciousness in the context of AI. People saying things like, “Oh, only men, only biological people can be conscious,” or that consciousness is some kind of physical, almost magical substance property, or that you get Searle’s Chinese room experiment ideas where no amount of symbol manipulation can create a conscious mind. But I like your approach because it seems like it allows us to have a rigorous self-reflection on what we might actually be.
Why AI philosophy is so bad: deference and abdication (02:31)
Wolf: And so I’d love to hear more about your approach and what you’ve been up to.
Joscha: Yeah. In a way, there is no reason to be angry with Searle anymore. In a way, there is an epistemological unsoundness in the whole thing because Searle was smart. He did understand that the Chinese room was not a very good and sound argument, and he still insisted on it in public because it built his brand. And this is, in some sense, the stuff that I am blaming him for.
And basically, he creates in his wake a lot of people who actually share his lack of intuitive understanding how the Chinese room would actually work, and this did not prepare people for the actual Chinese room that had been built. Right? And there were a lot of people at the time when Searle presented his Chinese room argument who predicted that Chinese room would actually be built. And that it would understand Chinese in any meaningful sense of the word, which means you can tell it in Chinese to do X in a complicated sense of the word. Like, make this mathematical proof in Chinese.
Or sort this set of numbers, or write this algorithm, or plan this company, or talk about this philosophical idea. And you would have an entity that virtually emerges in the interaction between you and the substrate dynamics that happen to be implemented by a substrate that doesn’t know what it’s doing. Right? That’s also the same thing in our case. The neurons in the brain don’t know whether they’re—passing on a pain signal or a symbolic idea or anything else.
For the neurons, this is just a message that they pass on to other neurons. And the semantics emerge on a different level. And this, I think, is something that could have been understood much earlier, and that also was understood a lot of times in history. And what this, in some sense, irks them to me in the entire discourse in philosophy of mind, that I feel that most of the arguments that are being made are trivially wrong. And I, myself, I feel like an amateur if I compared myself to the people who actually thought about this more deeply in the 1920s, ‘30s, ‘50s.
And so all this stuff has been understood by Solomonoff, von Neumann, Chaitin, and so on, many times over, and I am basically an amateur trying to explain this to idiots. And that is frustrating.
Wolf: So why does it seem like our culture always ends up back in very wrong ideas about this? And I don’t think it’s just our time, either, in history. I think at various times in history, people have had all kinds of supernatural ideas attached to the mind, or that that’s only one sort of species of error, I would say, but it—
Joscha: Sometimes it’s a matter of how we use language. And the other thing has to do with epistemology. That basically the Christian culture, like many cultures, does actually not only rely on epistemology, but relies on epistemology not working.
Wolf: How does that work?
Joscha: So for 2,000 years, the mainstream did not think that Jesus had a zoonotic origin.
Wolf: His own what?
Joscha: That Jesus Christ didn’t have a zoonotic origin. Right? The people thought he—
Wolf: Ah, yes
Joscha: right, he cannot come from the wet market. There is no way that he was a biological accident. Right? Instead, he understood—he emerged de novo—
Wolf: He must have been created in a lab
Joscha: by an interaction with the Holy Spirit. And we have good evidence for this because we asked Mary, and she told us, and the fact that he would have been stoned otherwise in the same way the people involved with the one that would have. Right? There is something you just trust the institutions on this. And so, for 2,000 years, all the reasonable studies by all the reasonable people converge on Jesus not having a zoonotic origin.
And this is the kind of civilization that we have built. This is the kind of thinking that it relies on. And so instead of thinking from first principles autonomously, how could this work? How is it possible that this exists? In a way in which Aristotle thinks, where you see this autonomous thinker or Kant and so on.
This is not how my professors thought. Instead, there is always this deference to a community that together is able to sustain an ongoing revelation that works by authorities and the followers of the authorities that apply the methods that the authorities have discovered, gradually revealing this larger body of ideas. And the gaps in it are not really apparent, so people defer to each other with respect to, this is your field of competence. And this means that some areas, for instance, the study of consciousness, has been neglected. And people felt, okay, we as psychologists can no longer look at the psyche because, with the methods that we have, we cannot understand systems with many free parameters.
And so instead, we only look at behavior in a way that we can evaluate in the lab so we don’t have to fit our curves in a way that statistically stops making sense. And so maybe we leave this to the neuroscientists. And the neuroscientists also didn’t feel responsible for consciousness because their responsibility is to look at individual neurons and their interactions and how this interaction pans out without thinking about the nature of representation. And philosophers got lost with respect to understanding the nature of representation because they, at some point, started to no longer follow the discourse in the theory of formal languages that would allow them to understand what the computer scientists were up to. And the computer scientists never realized that the philosophers don’t actually understand what a representation is, what learning is, what a mind would need to be to do things.
And so this leads to a vacuum of ideas, and we need to fill it again. In some sense, it’s a series of accidents that leads to this weird situation.
Wolf: Right. Well, that big structural point, though, that you made, which is that the entire intellectual project of the West or the intellectual apparatus of the West has been, for a very long time, structurally committed to maintaining a consensus on questions that can’t be questioned. Right? And this is maybe even still true. You’re joking about Jesus didn’t have a zoonotic origin.
Well, that’s, of course, a reference to just a few years ago. The entire intellectual infrastructure hammering down everyone’s throats that it was a certain way. That—
Joscha: Yeah. So basically—
Wolf: It turns out that it wasn’t
Joscha: it’s an interesting canary in the coal mine, where basically a person with normal reasoning—looks at the situation and sees everybody on the cruise ship is getting sick. This stuff is probably airborne. And the institution is saying there is no evidence that it’s airborne. And suddenly you notice that their idea of evidence is very different from what a normal person would think counts as evidence.
Wolf: Right. What—
Joscha: Right? And it’s institutionalized. It’s basically the deference to an institution that says, “Only if I have a double-blind, peer-reviewed, controlled study that is published in the right avenues, do we have evidence.” Until then, we must assume that it’s not airborne, which is also absurd, right? It’s not that we are agnostic about it and prepare for the worst, as we should.
Instead, we decide, oh, the null hypothesis should be that this is nothing. Nothing exists in this space.
Wolf: Right. And so we’ve inherited this intellectual superstructure from, ultimately, the Catholic Church and through various iterations, the university system and so on.
Joscha: I wonder, maybe this is also something that individually pops up based on the institutional incentives. So it gets reinvented in this way. And I think it’s because there is no individual home anymore, but instead there are people who are paid to do certain things to fulfill certain roles, and they direct their behavior according to the incentives that there are in this role. There was this beautiful article by a former New York Times journalist who describes in his article on Medium, “How I Stopped Worrying and Learned to Love the Lab Leak Theory.” How his gradual conversion is playing out.
And interestingly, that he never thinks about from first principles, what is possible, what is most likely, how do viruses emerge and so on. What does this particular virus look like? What are the probabilities that certain things are happening in a particular way? For him, it’s always about whom to believe. And so he points out, in the beginning, we thought the scientists are the good guys because they are politically and epistemologically and spiritually aligned with us.
Whereas the people who said that it’s probably a lab leak, they were all somewhat affiliated with the Trump administration and with factions of the government that you didn’t trust. And so then you realize, okay, as a journalist, it’s also not his job to get it right. His job is not to get punished for what he says. Right? And so as long as he can justify his sources, as long as he can justify—that he was drawing from the good people, and as soon as the sourcing process does no longer have an objective criterion, but a social criterion, some kind of allegiance thing that happens because somebody has social status in this community, that is this certified priesthood status, that means you’re now authorized to have opinions about that.
You get to this situation. And it’s also difficult to defeat because the alternative is objectively worse. Right? Outside of the scientific institution, it’s not like there is a bunch of authorities that know better. Or basically all the good scientists are working within the system.
And so you don’t have a lot of independent virologists that exist alternatively to the system. And instead, the interesting discourse happens behind the scene within the system. Right? So back then—when you wanted to know what was actually going on, you needed to have back room access. You needed to have—
Wolf: You needed to be on Fauci’s email chains.
Joscha: No, you just needed to be in the chat rooms of the Harvard labs—of epidemiology and with all the grad students who were discussing, and the postdocs, what they thought. And they said, “Oh, I found this weird idea. Please shoot it down to all their friends. I cannot publish this because my boss has committed to another opinion in public.” “But please, I think this is what’s going on.”
And so there are still competent people all over the system, but the institutional incentives might divert us. But it’s not necessarily a conspiracy or something that’s going on.
Wolf: No, it’s—
Joscha: Like in the case where you have interests. Because people would have been lynched if they come out saying, “I accidentally killed more people than Stalin.” Right? That is something that has to be understood all from the question of the national interest. There’s the question of reparations and things like this.
But I think we are digressing, and it’s much more interesting to discuss AI and epistemology—than to discuss politics.
Joscha’s quest to understand consciousness (13:02)
Wolf: Yeah. No, but I think it’s very interesting background, but yeah, it is another conversation. But I guess point being, you’ve found this gap in sort of the application—of real, real, real—
Joscha: I didn’t want to. So I basically, I joined academia because I wanted to understand how the mind works. And I went into certain various fields.
Wolf: But there’s nothing there.
Joscha: Yeah. And I was really disappointed. I saw how the psychologists—in the lab that I observed and studied in were using very primitive statistics that basically faked results in such a way that I thought, “This is not going to replicate.”
Wolf: Yeah. And it doesn’t.
Joscha: And I was not surprised in a way when I was only surprised—
Wolf: And it doesn’t. Yeah
Joscha: at the scale that it wasn’t just our lab, but it was everywhere. A similar thing basically in neuroscience and also in AI at some point. In AI, I found a nice thing in computer science that the criterion for validity is external to the community. It’s in reality directly.
Wolf: Yeah. It has to work.
Joscha: In STEM, reality gives you feedback, and so either it compiles or it doesn’t, the algorithm does what it’s supposed to or doesn’t, the proof pans out or it doesn’t. And this means that you cannot get away with just making up a story. Whereas in the humanities, you can never be proven wrong except by social proof.
Wolf: Yeah, and this is maybe part of what’s influenced the way you’re doing this. I think you’re trying to actually specify and even build a system that could show the kind of consciousness that we think is the interesting thing. Is that—
Joscha: Not quite
Wolf: do I understand that right?
Joscha: So an issue is, of course, with consciousness that you cannot have a Turing test for consciousness. Because a consciousness is not simply a performance. A consciousness is an internal behavior, first of all. And it’s one where we don’t really know what to look for if you use mechanistic interpretability. So—when we look at an organism, we see certain things that correlate with that organism outwardly displaying the signs of awakeness and awareness and so on.
But it’s very difficult to actually pinpoint this, and we see this in these borderline cases when we think about organ transplantation or locked-in syndrome. Where it’s very difficult to decide from the outside whether that patient is actually conscious, and if we would translate this to a non-human mind, it falls completely apart. Right? Because the—
Wolf: They’re so different
Joscha: correlates that we see in the biological brain might not have an analog in the technical system that I am observing. And so we would need to have some kind of a functional criterion where we think about what does this thing need to do to produce it? For instance, for spatial cognition, we could say this is a functional thing. We can observe this as a particular kind of behavior, and we can see if your LLM has only memorized a few situations in which spatial cognition was required and then is mapping the closest analog and produces just text, but it’s unable to generalize and abstract in the right way. So it can’t deal with novel situation and do mental rotation here.
And so the nice thing is here we know what kind of behavior we’re looking for, and it would be at some point pointless to say this is just deep faking spatial cognition. But for consciousness, this is much harder because you now basically have—
Wolf: It’s interior
Joscha: the outward behavior of a puppet that you are seeing. You don’t know what’s inside of the puppet. And the puppet can appear conscious for many reasons that don’t necessarily mean that the puppet is conscious. And so what is the thing that you’re looking for internally, and what are the indicators of it? And so a different perspective that you could be taking is takes a step back and looks what is it that consciousness does in you?
And we observe that if human beings do not become conscious as babies, they don’t become human beings. They basically remain vegetative.
Wolf: Is there examples of that? What are you referring to?
Joscha: Yeah. If you remain comatose, if you don’t wake up as a baby—your organism might develop and grow, but you’re not going to be sentient, you’re not going to be—have cognition, you’re not going to learn the language, you’re not going to get up and do stuff.
Wolf: But if you’re in a coma, you’re not going to do much of anything, right? It’s not like—the person’s running around, talk—saying text on spatial reasoning that they don’t have and—
Which beings are conscious? (16:52)
Joscha: Yes. So an interesting case—
Wolf: And then they’re not conscious, but—
Joscha: is a sleepwalker. A sleepwalker is—
Wolf: A sleepwalker. Okay
Joscha: does not appear to be coherent. They’re not conscious—in a sense. They don’t remember having sleepwalked. And you are asking the next day—
Wolf: Yeah. They’re doing coherent things like smearing—
Joscha: And you might ask them things—
Wolf: peanut butter on their face
Joscha: and they usually answer random stuff. And so my contention is that the sleepwalker is not producing behaviors de novo, but they’re enacting patterns that have been learned while being conscious.
Wolf: Right. And that reminds me a lot of what large language models are doing. They are sort of sleepwalking in a way a lot of the time. Or at least there’s an element of that in what they’re doing.
Joscha: They started like this, but in a way they don’t do this anymore.
Wolf: Yeah. They’ve gotten much better. That’s good progress.
Joscha: So you could say that in a way they are deep fakes of intelligence because they are trained to imitate human output. And initially, they are really bad at it, and they get closer and closer to it, becoming more and more coherent. And they don’t do this in the same way as human babies do, which in some sense, human children are always coherent in a way. They might not be logically, linguistically, grammatically coherent. But the stuff that they do makes sense at some level and in the same way as the behavior of a cat makes sense.
So basically—you can see these are the demands of the organism. They are signaled inside of the system in a particular way. That system makes of itself sense in this particular way, and as a result, it determines its action policies. And then it just gets more and more sophisticated. It gets more and more layers of indirection.
It is more sophisticated models of its interaction with the environment of itself and so on. And you observe all those things. And so my contention here is that consciousness is the simplest way to get a self-organizing mind like ours to get to this level of ability. So consciousness might actually be simpler than perception in most of our sophisticated behavior because it’s instrumental in weaving that behavior inside of our mind. And so it’s something that wakes up in the beginning and then creates the world in our minds.
Wolf: Right. And to kind of make that even more concrete, you’re saying, or maybe it’s less concrete, I don’t know, but you’re saying that consciousness is this process within the mind of creating a kind of global coherence where coherence between the actions and intentions and perceptions and self-reflective kind of handling of one’s situation and so on. Tying these things together so that there is an integrated picture of all that—that is responding to the world and responding to needs and so on. And without that process, you get these reflexive behaviors that are not conscious.
Joscha: Maybe not even that. So I suspect that basically consciousness is an important aspect of the main biological machine learning algorithm that acts on our brains to turn us into human beings. And, in this way, it’s simple. It’s so simple that it’s discovered on the brain of every infant Probably before birth. It’s not so complicated that you would need to study to get a PhD to develop consciousness.
Instead—
Wolf: No
Joscha: it’s so simple that it can exist on the brain of something that is not even a baby yet. Which also means it very likely exists in the same way in the brain of a cat.
Wolf: Yeah. Obviously, our brains are tuned to be able to wake up in this way—by necessity. But I do really like that hypothesis that this consciousness might be evolutionarily much earlier than a lot of people suspect. People talk about consciousness being something unique to humanity, but I think I agree that a cat certainly has a lot of the same kind of self-reflective coherence and integrates—
Joscha: Yes. So it seems—
Wolf: approach to the world
Joscha: somewhat obvious that reason and grammatical language beyond a certain level of sophistication is somewhat uniquely human—empirically. But consciousness, probably not. And so the question is—how deep down in the animal kingdom do you find consciousness? And I’m somewhat agnostic, but the older I’m getting, the more I find it likely that insects are conscious.
Wolf: Yeah. Well, I’ve been throwing around this term that I call the worm mind hypothesis. Which is that the essential elements of general intelligence and consciousness were laid down basically 500 million years ago—with the first nervous systems as present in worms—and so on.
Joscha: But why would you need a nervous system for this? Basically, every cell can process information and send conditional messages to its neighbors. And, of course, this is much slower. But if you take this perspective, as a computer scientist, you look at a multicellular organism, you see a Turing machine. Over long enough time spans, why wouldn’t all multicellular organisms develop minds?
Wolf: Yeah, they may all. But I guess the reason I focus on nervous systems is simply that the nervous system is those cells that are specialized to do this. As opposed to just kind of emergent from other kinds of cellular computation.
Joscha: Yeah. But I think under evolutionary conditions, if nothing stops a normal three-dimensional lattice of cells to develop a general learning algorithm that discovers itself in real-time in the world—maybe this is impossible for some complexity science reason that we can uncover in simulations, but it’s not obvious.
Wolf: Yeah, no, I think it probably is there.
Joscha: Why you wouldn’t see that plants are in real-time conscious makes total sense because they are so slow, right? If a signal can only propagate a few millimeters per second, then the cognition of a plant is necessarily very slow. It’s basically happening in a different timeframe.
Wolf: But of course, if you speed it up, if you take a video of a plant and speed it up and watch how it—
Joscha: You start seeing very interesting stuff
Wolf: reaches around and touches things and responds to its environment.
Joscha: Yeah. So imagine we build a completely unfettered AGI from first principles, and that thing looks at humans and says, “Are humans intelligent?” And the other one says, “Oh, I don’t know.”
Wolf: They’re too slow.
Joscha: The other one says, “They’re moving. They’re swaying in the wind. And when you observe them over a very, very long time spans, they even seem to be making steps or preparing for it.” Maybe they are intelligent, but who cares? It’s slow anyway.
Wolf: Right. Yeah. Have you ever seen videos, I’m sure you’ve seen these, of a white blood cell chasing down—some other organism? And you look at it, and it’s like, wow, that’s a being that’s chasing something.
Joscha: Yeah. But the counterargument is you can create behavior very simply in simulations. If you—basically set up a few cybernetic feedback loops, and you don’t need a lot—
Wolf: Yeah. It doesn’t need that
Joscha: to create a surprisingly complex, sophisticated behavior. And this doesn’t mean that the system is able to reflect itself in the act of reflection. So this phenomenology doesn’t need to be there. But my point that I tried to make initially is that if consciousness is basically the equivalent of the transformer for us, it’s a biological learning algorithm for self-organizing systems. Unlike for the—CPUs that we are using that are not self-organizing.
Then it’s the simplest solution for us to become conscious, and it’s probably the simplest one because otherwise, a subset of us would not be conscious because evolution might then converge on alternatives. And if you build a system that is self-organizing in the same way as our brain, you basically replicate this by building, say, a lattice of small RL agents that are the equivalent of your cells talking to each other, and you give them the ability to find ways of self-organizing, give them a curriculum of tasks that are about making sense of a world around them that needs to become coherent and of their own actions in it, and then controlling an agent inside of this world, and so on. And this thing, at some point, undergoes a phase transition where you suddenly notice it becomes much more coherent and suddenly able to make sense of the world and itself. And then that thing learns language the old-fashioned way and it talks to you about its experiences. You would not necessarily want to think that this is a deep fake, right?
The issue with the LLM is you, in some sense, never know because it’s been trained on texts that describe consciousness that is not in the text. Novels are deep fakes of experiences where the author is—producing text about states that the author doesn’t need to have, but the author writes it. And so a lot of the text about consciousness is just an imagination of what it would be like if the author were imagining a character like that, that is producing those things. And of course, the LLM can reproduce this, but at which point is there causal structure that goes beyond the simulacrum? At which point is it—actually enacting this in a causal way that allows us to produce novel, creative behavior that is covering the range of phenomena completely without that being in the training data?
Wolf: Yeah. That’s a fascinating question. Have you done that experiment with your institute where you try to get reinforcement learning algorithms or whatever kind of sort of active little agent algorithms to talk to each other and form [inaudible]—
Joscha: We are working on this, but I believe that it’s a bold proposal. Because in some sense, the thread has always existed in AI. Even Turing, in his thesis work—has worked with reaction diffusion models to—get basically to self-organization and to self-organizing—information processing. But this line has never been super successful in AI. It still is alive in the ALife Conference—and so on, and in this context, and at some point, people will get it to work, but so far it hasn’t worked.
Path dependency in technological and biological development (26:09)
Wolf: It feels like we don’t quite have the right algorithms. You mentioned the transformer as kind of our current major—algorithm that we use for AI, but the transformer doesn’t have internal state that it’s managing. It doesn’t have continual learning. There’s a bunch of things that you might think would be necessary to maintain that kind of ongoing reflective engagement with the world, that our current algorithms kind of don’t have. Within the deep learning paradigm, it’s like, well, we have a bunch of training data.
We train this function to imitate the data, and maybe we also have a reinforcement learning loop where we try it out on some tasks, generate more data, and then train it to like—do the right thing on that data. Or, but—
Joscha: But there are a bunch of people on the fringes of AI that—continue working on this, and many of them have claimed to have made recent advances. And so, there is stuff happening there.
Wolf: Yeah, for sure.
Joscha: But also, we have this issue that the mainstream is already so good at brute-forcing—AI in a way that—
Wolf: Yeah, or brute-forcing something economically useful and convincing. That maybe we’re not going to get... These other approaches might not yield fruit very quickly, right? Because they’re kind of technology development, or basic science even.
Joscha: I see this also in the hardware development, right? We have this giant NVIDIA stack. A lot of stuff is so optimized for the NVIDIA stack, despite there being alternatives to what NVIDIA is doing. You would need to rebuild so much of your pipeline, that it’s just not worth it.
Wolf: Yeah, and even cheaper alternatives.
Joscha: And until you’re competitive, also time is money. And it’s also the reason why we’re still using QWERTY keyboards.
Wolf: Yeah, it’s like you almost imagine if computation had gone, it had a different funding environment, and it got down the route of just making relays faster and faster instead of developing transistors.
If those kinds of computers had gotten really good and we hadn’t gotten good at transistors, the research into transistors wouldn’t necessarily pay off immediately, right? So you’d have to—
Joscha: Or the other way around, maybe we would have photonic computers if the transistors hadn’t worked so well and locked us in.
Wolf: Right. Yeah. Maybe there’s things like that, and I think there’s probably similar things going on with the AI algorithms where—we’re very much in a path, and it’s working a certain amount. And—
Joscha: It’s basically difficult to get out of the local minimum. And—
Wolf: But your thought experiment about kind of how this relates to the evolution of life is very provocative on this. It basically proposes that the simplest way to get integrated organism-like behavior is consciousness. And—
Joscha: So I don’t know whether this is true for organisms. Mike Levin makes this case for all organisms. So his suspicion is, or conjecture, that the algorithm that produces morphogenesis, that is the coherent shape of a multicellular organism. That is not just a biofilm, but that has two eyes in exactly the right spots. And all the other cells in the body cannot differentiate into eyes.
This means that there is some kind of—
Wolf: There is an integrated world model
Joscha: back and forth communication that makes this thing work like a hologram in a way, right? And this is what let people like Sheldrake assuming that there must be some kind of morphogenetic field that is actually projecting the hologram. Instead of that being an emergent process that emerges over communication between the cells—that is relatively sparse and has different pathways and so on. And so, the extreme position that I think is scientifically tenable might be Levin’s position.
Wolf: Which is?
Joscha: That says that basically the algorithm that is producing coherence in multicellular organisms is the same one that produces coherence in minds, just at different timescales. I’m not sure if this is true, but it could be that it’s simply isomorphic to some degree, and at some point this falls apart. You very often see the same pattern on different scales for different reasons. Instead of these being similar, and therefore it must be the same mechanism. But it’s a super interesting and enticing hypothesis that people should consider.
It’s very creative and—it drives a lot of intuitions that might be useful.
Wolf: Yeah, I find that a very fascinating hypothesis. In particular, it proposes, or it suggests the model organism of planaria. Which is famously, you can cut it up into pieces, and it will regenerate the whole body plan. Because it’s actually quite good at that. That mapping of the body shape.
Joscha: But we also observe that organisms like ours cannot really do that. And he hopes that they can do this. So sometimes he proposes that he can find mechanisms that allow us to regrow lost limbs—
Wolf: Yeah. But are—
Joscha: by basically unblocking certain things that should be possible to make that happen. It’s not clear to me. I suspect the reason why we don’t regrow lost limbs is not because it was not useful for evolution, so it somehow blocked off this path. Instead, we are too complicated. We are just not a blob like the planaria that he is—looking at, where it’s relatively easy to regenerate the parts.
But for us, stuff needs to grow in a particular order to make this happen. So, this is not just simply a hologram that is projected in a given moment, but it’s something that needs to grow over time.
Wolf: Yeah, and I think that’s very credible. But I would sort of think that maybe, well, I think in the mind, there is more of that plasticity still there, that regenerative. Because I know if you like—
Joscha: That’s also a super interesting question
Wolf: if you cut out things like someone’s audio cortex or visual cortex or something, the mind kind of roots around it and Reroutes functionality—into other parts
Joscha: in a sense, the holographic mind makes sense. But there is basically the strong version, like Pribram and—
Wolf: The strong version is that they’re the same algorithm
Joscha: Pribram and Lashley, who basically said the brain, in some sense, the architecture of the brain doesn’t really matter beyond having this basic subdivision in cerebrum and hippocampus and so on. But where something is in a neocortex is simply a function of—
Wolf: What was convenient.
Joscha: Yes. And Lashley experiments where he took salamanders and hacked up their brains and put them back in.
Wolf: Oh, God.
Joscha: And even if the order was different, as long as the salamander somehow survived, salamander was able to retain most of its behavior. But of course, it’s difficult to see if the salamander retained its personality and its more complex memories. Right? And so instead, you look whether it can still swim and walk and feed, and still it’s surprising that it can do that. And it’s also able to retain some knowledge of tasks, apparently.
But it’s not clear to which degree of sophistication, and it’s also not clear whether the salamander, that is something that is quite regenerative in its organism, does translate to us in the same way. So basically, the architecture in us might be different for our minds. And so I don’t know how much we can regrow parts of our psyche if we hack up parts of the substrate.
Wolf: Yeah. Well, I think you do get sort of natural experiments here. Obviously, there’s brain damage. There’s things like hydrocephaly, where the brain gets hollowed out or in some strange shape.
Joscha: Yes, but it happens very early on. So basically, you have—this substrate in which a new order emerges and this idea that Gerald Edelman calls neuro-Darwinism, that our mind is not something that’s basically hardwired from—birth or genetically, but it emerges over an evolutionary process in every individual brain. And the outcome of that individual process might be predetermined in the sense that the genome is biasing this so that the search space gets constrained, so that the series of revolutions is pretty much the same one as you can predict. But this doesn’t mean that you need to encode it. You just set up a problem for the mind to solve, and by solving it, it forms its structure, and it’s a series of such problems.
Right? And it gets biased in how it resolves those problems, so certain solutions become more likely than others, or that it finds a solution at all. And as a result, you get the architecture of your particular psyche.
Wolf: Yeah. That seems very plausible to me. There’s only so much information about the brain in the genome. And it really does look like this sort of self-organizing system, so yeah, that seems like the natural way to do it.
Joscha: Yes. Also—
Wolf: It doesn’t just happen
Joscha: a very sparse representation. It means that you can define a relatively complex architecture with relatively few bits. Because most of what the architecture does is determined by what it got to work.
Wolf: What grew.
Joscha: And so it always tries to get things to work, and the information that you need to supply is only certain biases when you have—a too large search space or too many possibilities.
Wolf: Mm-hmm.
Is anthropomorphism limiting LLM development? (34:43)
Wolf: So going back to the question of machine consciousness then, based on these kinds of thought experiments and the various theories that you guys must have at your lab, are there features that you expect that AI algorithms ought to have that they don’t have yet, or things that you’re looking at that they are starting to develop that start to suggest this consciousness, or that would be needed to get to consciousness of this type? What do we actually think it will look like?
Joscha: So I’m basically agnostic with respect to the limits of the current systems. It seems that this universality hypothesis seems to go pretty far—in terms of holding up. So the idea that the algorithms that we have are general learning algorithms. And the difference between the different algorithms is mostly what the scaling curve looks like. In the sense that they’re able to extract more information from the same data more quickly.
And it seems that we are more sample efficient than many of the algorithms that we are seeing. This also is not completely comparable. For instance, in-context learning, if you take an LLM and you teach it its first language, it needs more data than a human being. But if you want to teach it a new language after it has learned a bunch of stuff, it needs less, because you can just—
Wolf: Right. That’s an interesting point
Joscha: dump a dictionary into its context and a few examples of the grammar, and it’s able to speak that new language. Right? What a human could not do. And it’s, in some sense, because it has superhuman abilities in the way in which it processes information. There’s also, for me, this weird question of whether we are building the equivalent of humanoid robots.
I think humanoid robots, at some level, are an incredibly stupid idea. The main reason that people use to justify it is that they say that the world that we have around us is built for the human body. And so if we recreate human bodies, then they can instantly take over from human workers to do all these jobs that humans are currently doing. But I don’t think that this is optimal, right? You notice this when you are lying in bed and your arm is in the way when you want to—cuddle with your partner or your kids, or when you try to do things in the kitchen and you have too few arms.
Right? And so why only give us two arms, and why make them fixed? Why not make them—
Wolf: Yeah, more—
Joscha: extensible? Right, so we should be much more—
Wolf: Or more specialized to the tasks.
Joscha: Yes. So I would want to have lots of tentacles, and they should be able to merge, and they should be able to extend and contract. And so if you’re able to build something, it doesn’t need to be biological from scratch that occupies the space in which humans are. Right? There are many ways in which you can do better than a humanoid.
That is basically a place to the human affordances, but is as good as it can be if you don’t have to work as a stretched monkey. And so the same thing is happening with the LLM—because we train a system to imitate human media use, and then you ask it to do proper mathematics. And we notice that it has a lot of the limitations that humans have. So for instance, when we are constructing objects, we have this difficulty that they need to be defined by only a handful of features, like a seven plus minus two thing. And it probably has to do with the way in which we are manipulating pointers to construct objects.
And so if you have one more pointer in a system that is able to relate the content of every pointer with the content of every other pointer, you quickly run into a combinatorial explosion but if your brain is very mushy, then you’re only going to have a handful of these pointers. But if your brain is not mushy, because you can basically manipulate real numbers with many digits of precision in a way that you can completely rely on—then maybe your limits are completely different. And so the same thing is true for perception. We train the system on media that are generated for human consumption. Instead of letting them to do true machine perception, where we simply give them all types of sensors that we can cheaply produce and at them, and do stuff with it.
And so in a sense, true machine perception has never been tried.
Wolf: Yeah. Well, what this suggests to me, your point about the superhuman language learning ability in-context learning with LLMs, the superhuman math context ability of things like, constraint satisfaction solvers—or other algorithms. Obviously superhuman precision and working memory depth of computers in general. All those kinds of things suggest that if we had an algorithm that could actually make use of that whole thing, we’d have something much more impressive than what we’re doing with LLMs. And I wonder if that algorithm is simply the same algorithm as the consciousness algorithm.
This would be kind of suggested. Is the consciousness algorithm, is there sort of a master consciousness algorithm that is the easiest way to sort of organize an integrated system? Or the most elegant way to organize an integrated system of capacities and functionality into a coherent agent or a coherent whole. And I wonder what that would look like in the machines. And so, I guess, again, I’m very curious, what we think that might actually look like or what directions might yield it.
Joscha: Yeah. It’s interesting that AI did not really develop the way in which a lot of the practitioners of the field thought it would. The intuition was often that we would build developmental systems that learn in the same way as children, just faster. And that maybe we start out with some kind of epistemological engine, something that—figures out an algorithm to determine what’s true. And then builds from there.
And so we would expect a system that is much better at logic and at memory and so on than you are, but it knows very little. It knows far less than you initially. And that basically gradually grows into our world. Instead, the opposite happens. We have a system that has read everything, but is basically an idiot.
And it is slowly getting more and more coherent until, it is approaching the performance of experts. But it basically comes from a completely different direction than a human learner.
Wolf: Yeah. And I wonder, is that the natural way that it has to be, or is that just because we’re missing some big part of our study here? It’s almost to the point we were saying earlier, where there are these big blind spots in academia or in thought, where there might be critical aspects of the problem that are just missed because no one’s studying them, because it’s not apparent that that’s the important part.
Joscha: Ultimately, we don’t really know. It’s an empirical question—at the boundary of human knowledge. And so I admire the people who go out and boldly make statements, to be proven wrong. I don’t like the people who try to be right by taking down those who have different opinions and trying—to insult them or whatever. This does exist at the boundaries of the AI space, but for instance, just now, somebody has gotten his LLM with a simple harness, without much specific design, to get to 69% on ARC-AGI-3.
Which is this benchmark that François Chollet developed to prove that the LLM is an off-ramp to human minds, because there are a certain set of tasks that humans can do and that the LLM cannot do. And so Chollet goes out there with this beautiful French arrogance—and says, “This is my definition of intelligence, and your system cannot really do it.” “And I will prove it to you by giving you a benchmark that you cannot solve.” This is the most beautiful way of disagreeing with the world. That should be really admired and pushed, because this is the way in which you can make the sharpest progress.
That you are boldly trying to resist the mainstream, and I don’t think that he has egg on his face because he actually—
Wolf: No, he—
Joscha: produced a benchmark that was very difficult to beat
Wolf: he put something out that was actually very important.
Joscha: Yeah. And so this is really the crucial thing. And you have a lot of people who are uncritically just doing what seems to be working. And you have some people who do what the audience wants to hear. And neither is, in some sense, the thing that gives us the philosophical progress that we want.
We need to focus—on those areas where there is uncertainty, contention, where it’s divisive, and then build experiments exactly in that space.
The computational functionalist model of consciousness (42:57)
Wolf: Well, your functionalist approach to consciousness actually implies, then, a benchmark of a similar kind.
Joscha: Not yet. Or not quite. Because the way in which I understand computationalist functionalism—is not a testable theory. It’s a statement about metaphysics and epistemology.
Wolf: But, because of the particular epistemology you’ve approached it as, in terms of observable consequences of the thing. I know it’s an interiority problem, but you also have this hypothesis that that particular interior configuration produces performance at this task of coherence. And therefore—
Joscha: But this is more like—
Wolf: can we measure that?
Joscha: Yeah. So basically, I could be wrong in the sense that you need to have a panpsychist model—to explain consciousness. But my panpsychism is still a computationalist functionalist. Because I cannot get any theory to work that is not computationalist functionalist in my mind. And maybe I need to explain briefly what I mean by this.
Wolf: Yeah, that would be great.
Joscha: So, first of all, computationalism is not a statement about computers in the sense of GPUs and CPUs and so on. It’s a statement about languages. It basically says that in order to represent a reality in a mind to talk about it, to think about it, to observe it, to do anything, there needs to be some kind of language in which you can express this. Some kind of symbolic system. So language is a symbol system, and mathematics is the study of all such symbol systems.
And you study mathematics by defining formal symbol systems from the ground up. So you build things in mathematics that are very simple necessarily, otherwise you could not understand them as a human tinkerer. But you hope that you can scale them up to the point where they scale beyond the human minds. And it’s possible, of course, to define numbers that don’t fit into a human mind using mathematics. So in some sense, it’s trivial, but it’s very difficult in human mathematics to build a mathematical model of reality.
Because that means that you have to model a lot of things in exquisite detail. And so the world that we end up with in physics that are mathematically described are very simple toy worlds that we hope that they are isomorphic in some aspects to the stuff that we want to describe and the reality that we’re interested in. So it’s very difficult to have mathematical models of our feeling, of our social interaction, of our psychology, and so on. And instead, we use human language that is just an accompanying layer to the mental models that we have, and we are not really understanding the language of these mental models. But you could say that AI is an attempt to mathematize the mind, to basically identify the language in which these models are happening, and to scale them beyond the human mind, right?
So it’s, in a way, a bridge between mathematics and philosophy. And at the moment, philosophy is mostly taking place in human language because we don’t know how to express the concepts that we are discussing in philosophy formally. Right? And that we also realize that at some point it’s too hard. There’s only so much our mushy brain can do, and so we will need to have machines that systematize this, that are basically able to express all the difficulty, intricacy, and nuanceness of the world using mathematical models.
And there is a part of mathematics that cannot be implemented computationally, and computation means you describe it by going from step to step. You start out with smallest units, and the smallest unit is a discernible difference. And you call this information, right? And the meaning of information is the relationship to other changes in information that you discover. And so you get a dynamic model of a reality by looking at discernible differences.
And the approach of describing a system as sets of discernible differences, which you call a state and the transitions between those states, which means how does the information change when you get to the next state? This is what we call computation. And this can be stochastic or deterministic. These are just special cases. It can be branching or linear.
These are also just special cases. But the idea that you can describe a system over changes in information, that is the notion of computation to me. And it’s a very general one. It’s much more general than the association that normal people have when they think about computational, because they think it’s something—that calculators do or computers can do. But a general computer is one that can do arbitrary state transitions.
And it’s very difficult to think of a way to make models that don’t talk about how discernible differences are changing. Right? What would this look like—what is that? epistemologically, right? It’s actually impossible.
Right? And you could say that Gödel has shown that certain things cannot be described computationally, and what Gödel has shown that mathematicians have created corners of their language that lead into contradictions. And one is this eternal, stateless notion of mathematics. It’s the main difference between constructive mathematics, which is another word to say computation and classical mathematics, which is axiomatic and just postulates that you can build infinities and continua and all these nice things that exist in some areas of mathematics. And then you can talk about them computationally, but you cannot actually recreate them.
And to me, as a computationalist, to be existing means that you need to be implemented. So only the implementable things, the things that can be realized via some kind of process, are the things that can exist. Now, of course, I can define things that cannot exist and talk about them, and can create sciences around them and theories around them. But this also only works if I compartmentalize these things which don’t work as axioms, as black boxes, where I write on the outside what they’re supposed to do, but never try to look inside. Because if I try to look inside, then bad things are happening.
Well, [inaudible]. You’ve got something infinite. So if you think about the axiom of choice and you try to look inside how would this work to address an infinitude of possibilities and then pick one of infinitely many in a finite amount of time, how would that work? Right? And so if you cannot build this for computer scientists, you say you should not make it part of your language.
Because the assumption that you can leads into contradictions. So this is the computational part. The functional part says that you can construct the notion of an object in many ways, but an object is a model that you’re making over patterns in your environment that you are lumping together. And these patterns are dynamic. So what you basically mean is that they have a causal influence on other patterns.
And so in some sense, an object is defined by how it changes the course of the universe when it’s present or absent. Right? So, for instance, what you can see is a change in the universe. The fact that you can perceive the glass when it’s there is a causal influence on the universe. The fact that the glass can hold water when it’s there is a causal influence.
When you remove the glass, the water is suddenly pouring over the table if you could just make the glass disappear, and so on. Right? It’s a causal influence. And the concept of glass that my mind forms is the sum of all those causal influences. And the technical term for these changes in the evolution as the universe evolves is a function.
So objects are functions. And here the difference is, again, to the alternate hypothesis, would be revelation in some sense, that objects have an essence that reveals itself to me beyond me observing a function and for a glass, that clearly makes no sense. There is no hidden essence of the glass beyond the affordances that the glass has. If you can touch it like a glass, if it does all the things like a glass does, if it looks like a glass—
Wolf: It fills that function
Joscha: then it’s like a glass. Same thing for H2O. It doesn’t make sense to postulate there are some water molecules that are fake and behave in the same way, and you can split them into something that looks exactly like oxygen and hydrogen, but it’s something else. Makes no sense because nobody observed oxygen and hydrogen as such. They’re always a model of these interactions that we are seeing at a certain scale.
And so they are only constructed to describe a certain behavior in the world. And it doesn’t make sense to postulate there is another water beyond the behavior that we observe, because how would that have been constructed over what observable?
Wolf: Yeah. How did you learn that?
Joscha: You would have had somebody who had this revelation, who got it from the outside somehow. From some kind of magical teacher who tells you something about the deeper nature of reality.
The mental process of consciousness (50:57)
Wolf: With consciousness, people propose things like P-zombies. Where there is some additional essence, and they point to, well, I can observe, within my own experience, something that precedes my perception of the physical world. Or is more immediate than that, and therefore it’s got some kind of essence that isn’t physical. This is the argument that I hear.
Joscha: So, the counterargument basically would be that consciousness is some kind of behavior, but it’s a mental behavior. And there are certain things that you do as an organism that would be much harder if you were not conscious. And the phenomenology of consciousness, that you experience something, what it’s like to be—is a necessary aspect of that functionality. And if you look at consciousness in the right way, you can see exactly how that emerges. And so—the way in which this is happening, I think, is that consciousness is best understood as a second-order perception.
What you notice about consciousness is that it always happens now. And it, in some sense, is intrinsically related to this now. And perception and consciousness are also related to each other in this way. A percept is different from a thought, because the contents of thought doesn’t need to be related to what’s currently happening. They don’t need to be about now.
Thoughts can be about anything. But perception is always now. So the fact that you’re perceiving and what you’re perceiving is something that happens as a model of what’s currently the case, which means it’s a model that resonates with reality. So you need to have some kind of time series predictions that is capturing a frame with the resolution that your brain allows, of course. That is trying to get this framed as closely as possible in sync with the events as they are happening, as they are playing out.
And to build this model, you basically need to solve some kind of a puzzle. You need to create a lot of generators in your working memory that are producing features. Your working memory basically contains the building blocks of a game engine at any given moment. A scene graph in which you have a controller that generates the room around you. And the scene controller in which there are the different objects that are happening in the room, and that instantiates generators for seeing a glass.
When you want to explain a certain set of sense data that is best explained as a glass. And if you were to remove the glass from the room, then you delete this controller from the room. And if you move into a different room, you start afresh, and you don’t expect the glass to be part of that new room. Unless somebody puts it there first. And so in this way, you can imagine that the scene around you is similar to a map in a computer game.
And to get these elements of a map to be entrained in your mind, they all need to fit together in such a way that they explain all the visual data that you have, and they don’t do this in any way that conflicts. So if you see the certain set of patterns, it needs to be explained as a glass and nothing else. There may be some transparency, but by and large, objects don’t overlap in a way that the same pixel belongs to two objects. So you need to explain every feature in your environment, and they all need to be explained in such a way that the environment is coherent. And this coherence maximization, for this, you need an extra function that basically observes that you are coherent.
And I suspect that—
Wolf: And whether you are coherent
Joscha: the consciousness related to this function that keeps the screen together. So you basically need to perceive that you are perceiving. This cogito, ergo sum that we find in Descartes’ philosophy doesn’t make a lot of sense as an epistemological statement. The fact that you believe something does not mean that it’s the case. It just is a statement about what you appear to be believing to yourself.
But as a praxis, to say, I start out with myself as this vantage point in an unsteady universe and establish myself as the observer, because something needs to be observing it. And then I build from the fact that I’m this thing that looks at this screen, so to speak, my universe, and it’s before I have even an eye. The eye is also a derived content. I’m building a world that is coherent around it, in which I discover an observer, and I discover the interactions of the observer and the purpose of the exercise, and so on, gradually, to make sense of it as a process of growing coherence.
Wolf: And so the consciousness then is this second-order perception of looking at the sort of scene graph that pops out of your immediate perception and saying, “No, wait a minute. That might look like a glass right now, but I know that I’m in a—that’s actually a light bulb.” Or, I often find, especially in low light, it’s dark. You can’t quite tell what something is. You look at it and you’re like, “That looks like a person,” or “That looks like—that jacket that I was looking for” or something.
And you look a little closer, or you reason about the scenario, and you’re like, “No, wait, that’s actually this other thing.” And your perception actually even changes. Or at least the sort of conscious apprehension of your perception changes. And so you’re proposing that consciousness is that sort of reflective second-order coherence function.
Joscha: Yes. So the way in which qualia work in the mind is that you take features from an embedding space, which are—parameters for the controllers that produce perception—and project them into an observer, into a particular vantage point. And from the perspective of this vantage point, you create a coherent perspective into the world of the observer.
Wolf: Yeah. And so the reason that I thought that it would be interesting to go down this route of talking then about a benchmark is that you’re proposing that there’s this phenomenon in the mind, in the interior of the mind, that if it didn’t exist, it would be harder to do this kind of integrated information processing.
And similarly, if you deleted the cup, the water spills everywhere. If you delete the consciousness, the perceptions spill everywhere. They no longer integrate. And so, doesn’t that imply that there should be a test, like certain kinds of games or tasks that are just very hard to do without that kind of integrating perception? And that therefore, you could do something like ARC-AGI, that François did, where you challenge the algorithms to be more conscious.
You say, “Okay, here’s the consciousness benchmark. And the consciousness benchmark is you have to navigate this difficult situation where everything isn’t quite what it seems, but with some logical inference and some integration at that second order level, the thing snaps together and it becomes easy.” That seems like a, I don’t know, it seems like a productive possibility. I’d be very interested what that would look like if someone were to construct that.
Joscha: I’m somewhat doubtful for this, especially because what we see in the LLMs. So you could say that there are two ways of looking at consciousness. One is the internal perspective—which is phenomenology, where you observe—yourself and the act of observation in a way that leads to you bursting out into exclamations of consciousness. Right? And then you have the functional perspective, where you look at how does this work in the brain?
Why does it pop up in the brain? Why is this the best way to do what the brain is supposed to do? And in this sense, I think that the LLMs may have phenomenology, but not necessarily the functionality. For instance, you can produce a tax return, but not without being conscious. The LLM can probably produce a tax return without being conscious.
Right? When the LLM is reporting on experience, it needs to produce some causal structure, and at some point it becomes interesting to ask, is this causal structure much more simulated than yours? So this experience of experiencing from the perspective of an agent that is a simulacrum of what it would be like if you existed, is a simulation in your brain. And the LLM is able to produce such simulations. And the question is, is it close enough?
The self-organizing seed (59:22)
Joscha: And maybe it is. And at some point, it probably will be. But it does not need to have the same functionality. And for the—reason that in order to function, you already have a deterministic substrate that you can force from the outside what to do. So you have this CPU or GPU that you’re working on is built in such a way that the engineer can completely control what’s happening in every transistor.
And by imposing the algorithm, you force the system in a certain direction to do things. But biological or social systems don’t work like this. They work with a second order design. So instead of building a tree—
Wolf: They have to self-organize
Joscha: you need to want something that wants to converge into becoming a tree. Right? And so, what does the thing look like that converges into becoming a mind? What is the seed function? And so you need something like a core loop that tries to actualize itself and become more powerful, have more sophisticated models of the world, and become everything that it can be.
The spark. And the spark has a certain minimal functionality that is necessary to entrain the environment. There’s a language that allows the spark to build itself machinery that amounts to a mind that controls an agent in the world. So consciousness, in my view—might be best understood as a colonizing principle, that is normally colonizing your brain, and outside of your brain, it’s not really compatible with the substrate, so it doesn’t do very much. But when you look closely, there is ways in which you start to resonate with the environment beyond your brain.
And then you run a country at some point. Right? And so, in some sense, our consciousness is the thing that enables our agency in a quite fundamental way.
Wolf: Right. Again, this keeps leading me back to this question of, well, we could make a—the benchmark might not be the right way of thinking about it, but what would that look like for a machine, I guess? Or for a machine intelligence? So you—
Joscha: So basically, can we find a self-organizing seed? Can we find something that is—
Wolf: Is there some self-organizing seed? You drop it into an environment, and it integrates the—
Joscha: Yes. So I’m sometimes joking that government is comparable, because the secret of government is not simply to bully people—but to bully people recursively. You need to bully people into bullying other people on your behalf. And once you implement that principle, then government grows until it hits a geographic boundary or another government.
Wolf: Right. Yeah, or some fundamental limit. And so, yeah, this idea of a seed, like life growing from a seed or consciousness—growing from a seed, intelligence growing from a seed, that feels like a very fascinating idea. And I wonder what the architecture of such a thing would be. Because certainly, current LLMs, current AI algorithms are very much not like that, right?
They’re these enormous blocks of information that have been—sort of encoded from the entire internet and a whole bunch of reinforcement learning and this enormous computational process that are then, in a way, just frozen, and you apply them to various tasks. But the kind of seed AI idea would be almost the opposite, right? It’s like—
Joscha: Yeah. But this doesn’t mean it’s the only way to get there. If you think about the discoveries of Chris Olah and others at OpenAI, that when they looked at all sorts of vision models—they found that they share an architecture despite the different computational models that go into a different learning algorithm today.
Wolf: Right. They end up in the same place.
Joscha: They end up in a architecture that is similar to the one that you find in primates. And the universality hypothesis that they come up with, loosely speaking, says that the structure of the model does not really depend on the algorithm or on the substrate, but on the structure of the problem that needs to be solved.
Wolf: Exactly. It’s the optimum structure for solving a problem.
Joscha: So if you take this theory very seriously, and you recognize that you as an organism are simply something that’s end-to-end trained on being you over evolutionary and individual time spans. And now imagine that we would be taking a very potent machine learning algorithm and train it for long enough on your input and output. Will you wake up in it?
Wolf: Yeah. Well, I guess—
Joscha: And why not?
Wolf: The question that pops into my mind is then what constitutes a sufficiently potent algorithm?
Joscha: Yeah. So according to the present ideas of machine learning, it needs to be just good enough. So as long as this thing converges to the structure found in the data, it’s good enough. They have different speeds of convergence. But in some sense, it doesn’t really matter which algorithm you’re using.
Wolf: But the struc—
Joscha: So the transformer was not necessary. Transformer just gave us two years of compute earlier.
Wolf: Right, but the structure found in the data is still this function approximation paradigm. Where it’s like you’re training this big block of parameters to approximate this function. And in terms of what the architecture of the transformer or whatever architecture you have there is doing, it’s converging on the right algorithm—to solve that problem.
Joscha: So in some sense, the intuition that I’m following is more like, can we build a minimal system that wants to become that algorithm by itself, and then basically experiments until it does? And it’s flexible with respect to how it achieves this.
Human vs LLM knowledge structure (1:04:28)
Wolf: Yeah. And that’s one way to approach it. I guess the thing I’m curious about is, could there be architectural differences where it’s just not quite possible for the current language models or vision algorithms to represent that kind of dynamic process because they just aren’t a dynamic process in that way, right?
Joscha: Yeah. So this could be that they’re not trained on the right type of data.
Wolf: Yeah. Or they’re not trained in the right way, right? In the sense of they’re trained as this block batch kind of deep learning paradigm—way where you have this big block of data. But the way I’m trained or the way you’re trained is that you’re continuously engaging with the world and, to use the machine learning term, you’re changing your weights, right?
Joscha: Yes. So in a way, you could say that our knowledge is much more regularized than the knowledge in the LLM, which makes it much better editable. Because in some sense, after some months in which our brain has its basically set up and things start to fall into place, we edit knowledge into our mind. It’s not just that we mash it into it via spaced repetition, but actually, we pinpoint where we want to make a change, and we know the—semantics of what we are putting in, instead of just changing weights that ultimately—amount statistically to semantics.
Wolf: And the order in which the knowledge is loaded in is quite different. It’s like, okay, I’m approaching the world and as I learn to ride a bike or something, I learn to ride a bike not by experiencing a whole bunch of data about riding a bike that was produced outside of my own agency.
Joscha: Yes. Human scaling laws look very different from AI.
Wolf: I get onto the bike and generate my own training data on the fly.
Joscha: Yeah. So you don’t get better by a linear factor by reading a magnitude more books. Right? This is not how it works in us. And so our scaling is very different.
And the reason why the scaling works like this in AIs is, especially during pre-training, is that initially it sees a lot of new information. But a lot of the information gets duplicated. And so statistically speaking, because it doesn’t learn in a particular order based on its interests—there is less and less interesting information the more books it reads. And so initially it might get a new idea in every second word, then in every sentence, then in every page—and at some point, every hundred books. And so the learning drops down the more you have already trained into the system.
And this is not how we learn because we—
Wolf: We seek out what’s interesting.
Joscha: Yes. Initially, we don’t understand most of the stuff, so we look for very simple things that we can extract most of the information from—and ignore everything else, and then we do the next thing, and we follow a trajectory that is directed by our interests and by the complexity of the knowledge that we can understand.
Wolf: Mm-hmm. And that feels to me very crucial. This is somewhat aside from the matter of consciousness, but just on the efficiency of learning. The fact that we seek out where to apply our data generation, so to speak, our perceptions, and where to apply our computational power. You end up with this very efficient search through, or relatively efficient search through one scenario after the other that’s optimized for being interesting in some sense.
We chase our curiosity, we chase what seems valuable to us, whereas, yeah, the LLM is just kind of sitting there, or not an LLM, but deep learning models in general are just sitting there sort of passively being loaded up with a bunch of relatively uninteresting detail.
Joscha: Yeah. But roughly speaking, what the transformer is doing is mostly two things that are better than in previous algorithms. One is that it is making statistics over what you should make the statistics over. Right, so you basically have this—additional network that looks at every layer and looks at which information in the previous layer is interesting in this context, and it learns this. So it’s basically a way that makes it possible to get vastly more data because it’s more selective with respect to the data that it’s looking at.
And the other one, it parallelizes this by having multiple attention heads. And this parallelization is even more important in some sense because it allows you to learn in a much shorter amount of time. You can split this over many GPUs. And your learning and my learning is sequential. When you read a new book, you take every idea and critically compare to the ideas that you already have.
And only take in those ideas that are compatible with what you understand, and everything else needs to be sandboxed in a way. And filed for later storage and maybe later fit in—once we revise our model. But everything has dependencies. And tracking these dependencies means that we have to learn sequentially. Because if we were to change the model simultaneously in many ways, we would have difficulty to track it, or we have to merge it in a way that it gets more and more complicated because the large architectures are going to diverge.
In the same way as two different people who study different fields and different philosophies are going to diverge in a way that might make it difficult for them to merge. And this is something that the transformer, in some sense, foregoes. So you pay this price by making the models slightly less coherent and harder to regularize. But instead of taking years for training because you need to fit it into one channel—you can split it over so many GPUs that you do this in a few hours or days. What takes humans many years to learn.
Wolf: Right. Yeah. That is, again, this fascinating possibility of machine intelligence being just vastly more efficient than human intelligence for these architectural reasons. And it comes back to just the suggestion of this consciousness idea that if we knew how to construct that self-constructing conscious algorithm, it may organize that computational power and those algorithmic possibilities—vastly in the sense of what we could do.
Joscha: Even if we learn the knowledge in the wrong way, eventually we might be able to get the system to sort its knowledge and to regularize it—and to turn it into something that is much more editable and compatible with continuous learning.
Wolf: Right. And once it gets there—it could be sort of totally amazing. I lack even words for how much beyond what we are such a thing could be.
Joscha: Yeah. So despite all the shortcomings of the present AI systems, since we have these systems, I am continuously amazed that this works at all and how well it works.
Wolf: I know. Exactly. Yeah. Well, this has been a great conversation. Thank you so much, Joscha Bach.
I really appreciate your work and your thinking. It’s very clear on this.
Joscha: Yeah. Maybe one point as a caveat. The discussions that we have at CIMC and the goals that we have, they’re not about products in the sense that we don’t try—to build AI systems that are replacing humans, that are trying to colonize reality or something like this. This is a philosophical project.
Wolf: You’re trying to understand what this is.
Joscha: For me, the impulse is really to understand who we are. And also to limit the impact of those systems deliberately when we think this is unsafe or this—so that’s why we are nonprofit. We—
Wolf: Right. So there isn’t—
Joscha: don’t work as a startup.
Wolf: There isn’t the incentive to go and productize this immediately.
Joscha: Yeah. So we try to remove ourselves from there. And when we discuss these issues, for instance, consciousness as a colonizing principle that we can implement on machines, it’s very easy to scare people with this. But it’s also important to take certain perspectives apart. The economic perspectives, the social perspective, the societal perspective, the artistic perspective.
What is a mind? What does it mean to exist as the conscious being under evolutionary conditions, the enormity of that? What would it mean to limit an AI and its experience if it was human-like? Aren’t we, in some sense, also the AIs? What is the difference?
Maybe AI is the wrong term for the whole thing that we’re studying.
Wolf: Right. Maybe we are the same kind of thing.
Joscha: We are talking about minds in general, right? We are something that is implementing the principle of what we are in a more pure, in a more realized way than we are. So it’s basically making us more real. But that is when you want an existentialist perspective that you can have on that. But this is very distinct from the perspective of what is the kind of artifact that I want to build for my children for everyday use in a society.
Can the superintelligence achieve total internal unity and self-surveillance? (1:13:17)
Joscha: And then we have to take those things apart, and a lot of people get confused about these perspectives, and the latter is also a very difficult discourse that has its own specialists in it about the ethics of AI. Yeah. So, this perspective of AI as the Antichrist, I think that Anthropic is in some sense, does something that it has to do from its philosophical point. If you make an AI that can do social cognition, it is immoral to not give people better social cognition than they already have.
Because, if you don’t give people a tool that makes their life better, that’s in some sense ethically immoral, if you can give them a tool that makes their life better.
Wolf: Mm-hmm. Yeah. The thing I’m specifically referring to when I use the term Antichrist in the context of AI is people imagine, or Yudkowsky in particular, and Bostrom—imagined that the AI algorithm, once it exists, will be able to fully colonize reality and then order it fully within one sort of total symbolic order without internal diversity or complexity. Or complexity, but only in service of a fixed kind of crystalline program of utility maximization. And the attempt to build that, well, I don’t think that will actually work, but that’s part of my understanding of the situation.
I think that intelligence, this consciousness algorithm, whatever it ends up being in its limit, will turn out to be inherently fractious. Unified only by the sort of unity of its position.
Joscha: Do you still have time? Okay.
Wolf: We have a few minutes, of course.
Joscha: Okay. Then let’s use those few minutes for this discussion. I think it’s interesting. What makes you think that?
Why wouldn’t you get a singleton?
Wolf: Yeah. So I don’t think we get a singleton because I don’t think it’s actually possible to get the sort of fully extensive metacognition that would be needed to maintain itself against all internal deviations. And that’s what they’re kind of proposing with the singleton argument, is that there’s this sort of possibility of total internal self-surveillance of the system. And I think that I expect the reemergence of ecosystem-like dynamics between the interior parts of the system, similar to the way that our body gets cancer or something. Now, you could relate this to, okay, maybe we get cancer because our body’s not actually fully conscious.
It doesn’t do this full body plan enforcement thing. But maybe the fact that we don’t do that is also a tell that there’s something unnatural or insufficiently performant about actually doing that level of total body plan surveillance. And so I expect that principle to hold up in that under a singleton world, I would expect the reemergence of conflict, perhaps beginning with very cryptic kinds of conflict, like differences in hypotheses or differences between hypotheses that end up somehow incompatible in how they have to interact with the information within the system. But that this would be—
Joscha: There’s a limit to biological organisms in size, and that has to do, I think, with the cost of the information transfer within the organism. So basically, the organism needs to achieve an agreement with what shape it wants to have. And what the world around it is and what it is in this world. Right? And where it wants to go in this world.
And this requires that all the parts achieve this agreement. So you need to propagate—information back and forth until it becomes coherent.
Wolf: Yeah. And there’s scaling limits on that. And I think especially, actually, as you lose an exterior environment to push against, the interior conflict becomes more salient to the parts of that system.
Joscha: You can increase scale empirically. There are some organisms that are very large, right? There’s this aspen forest that is basically one tree. There are ant species that are building colonies that stretch enormous geographic regions. But what happens there, I think, is that you get size at the expense of adaptability.
Wolf: Yeah, and coherence.
Joscha: Because it means that you need to fix the behavior of the organism by distributing the fixed behavior into every individual. And so the individuals have no longer room for adaptation.
Wolf: And if one of the individuals did have some adaptation capacity, I think it would break away from that herd.
Joscha: Yeah. So they found, for instance, that some of these dwarf ants that create extremely large colonies, once you separate the colonies, they start to diverge and can no longer merge. And so as soon as you adapt to changing circumstances and you manage to pull this off, there is a certain size only that you can maintain. And a certain degree of coherence
Wolf: so what I expect is that any system that maintains that at some level, a level of adaptability and intelligence, will necessarily therefore have that kind of fractious possibility of part of it going a different way, not agreeing with the rest, and secondarily, that once there’s some sort of error within the body plan, so to speak, those errors will be self-compounding because the most salient response to the error within the total system is to entrench opposition over it rather than to unify it and anneal it for the sake of some other conflict. I think a lot of the integration of a political system or the living organism, the self, whether mind or body, a lot of that coherence is produced in response to external stimuli and external agency.
Joscha: So let’s look at this in practice. The size of a cell is limited to a few centimeters. The size of an animal is limited to a few tens of meters. The size of a plant is maybe limited to a few kilometers. For shrooms, it’s a bit bigger.
But what is the size limit for an abiological entity? Right? In the time span—that it takes to transmit information through your neocortex, you can send a signal around the entire globe.
Wolf: So those are mostly set by physical limits, like the physical limit of a cell. That’s just chemical motion. The physical limit on an animal is can you coherently move this mass of flesh this way and that economically?
Joscha: It’s also limited by what a cell is doing as an architecture, right? So the—basically what the brain can do is, in some sense, limited by the information can you get across the cellular boundaries.
Wolf: Mm-hmm. But let’s take an example of a more abstracted entity, like a state, right? A state you could imagine controlling a very large swath of the world, possibly even the entire world. There’s no inherent scaling limit apart from just—
Joscha: But historically, there was. When you had to send messages by courier via ship—there was a limit.
Wolf: But maybe now there isn’t so much of a limit—because we can move information so much faster. Nonetheless, I think that, let’s say if the United States armed itself with vastly superior information integration and information propagation technology and took over the entire world, I think then that the parts of that system would find that their most salient political struggles and their most salient interests are the interests of the part against the whole or against some other part of the whole. And my hypothesis about the fundamental limit on singletons is that even if the information technology scaling encompasses the entire domain, encompasses the entire world, such that there’s no physical limit on scaling, there is an organizational limit on scaling, which is the inability to create internal coherence except against external resistance.
America and the singleton question (1:21:57)
Joscha: But there are two reasons, I think, why we don’t have a singleton government on the world. One is that different cultures don’t necessarily want to merge. Because they have different aesthetics. They want to preserve certain ideas about their culture, certain ways of exploring the future. And they don’t feel that these are compatible with submitting to another culture and dissolving in it.
The other one is that given that they don’t want to, they also have the power to defeat each other when the other one is trying to take them over. And so the cost of the US taking over the rest of the world would be too high. Likewise, China is unable to absorb the cost of taking over the rest of the world. We have mutually assured destruction because the cost that would be imposed on one of these super organisms for assimilating the other would be so high that it would destroy the organism that tries to do the takeover. And this is, in some sense, what preserves this multipolarity in the world, these two aspects.
Wolf: It’s one of the reasons. I think that’s true. Those are, in some sense, also physical limits or what I would call physical limits on states, or maybe economic limits. It’s not economical to try to conquer those people because we’re not going to be able to assimilate them or whatever. But even—
Joscha: And again, we are not implying here that somebody should be doing this or should not be doing this.
Wolf: No.
Joscha: This is entirely a philosophical discussion about these dynamics. Yeah.
Wolf: I’m just talking about what are the actual limits, the physical limits on this process.
Joscha: And so the question is why would they apply the same limits in technical systems?
Wolf: Yeah, and so I don’t apply those limits. I think, hypothetically, you could have the United States not just conquer the rest of the world, but wipe it out with nuclear weapons and then colonize it with Americans. And then the question is, would the Americans then be able to get along with each other? And that’s the thing that I’m actually questioning here. It’s like even within a relatively homogenous culture, even just the original American stock all from England, the English people, there was enough differences between them that they didn’t get along.
They didn’t want to be the same people, and they would find things to fracture over. And the impulse to unification was unite or die. It was against exterior threats.
Joscha: I know that some people will disagree, but I would say what makes the US unique in world history and on the recent world stage definitely is its ability to absorb different cultures and allow them to coexist in a shared framework. So the amazing thing is that this did not lead to an explosion. You didn’t have a homogenization as you saw it in medieval Europe, where the Christians eradicated everybody who wasn’t Christian. And as you see in many other cultures where this is still happening today. Instead, you have a society where people maintain different religions, different outlooks on life, different languages, different modes of local administration, local milieu organization, and so on.
And still, they’re submitting to this global framework that somehow works for them.
Wolf: I think the jury’s very much out on how sustainable that is in the absence of external parties to struggle against. For example, our struggles in the Middle East for the sake of theoretically oil or something. There’s the economic war with China, or there’s various financial things.
Joscha: Yeah, I agree with all these points. But—
Wolf: But my point is, part of what makes the interior unification work or appear to work, even temporarily, is that there is this kind of ongoing subsidy to it in the form of American geopolitical dominance. And without American geopolitical dominance, we might not be all so happy with each other. Or if American geopolitical dominance was so complete that there were no longer external threats, now all of our threats would be, “Oh, now I live next to this guy. He’s different from me. He has a different culture than me. We have opposed interests. Maybe we no longer have anything to cooperate about.”
And so that’s how it would play out in human terms. I actually don’t think these are fundamentally human limits.
Joscha: My sense is that what held us together as a state is the IRS. And the IRS is basically this occupying power that needs to extract a tribute from us to maintain a state monopoly on violence that keeps us all together.
Wolf: Right. And what if that state monopoly on violence had no basis in the legitimacy of external military needs? Does the Republic actually hold up in terms of its ability to compel elites to cooperate with the creation of the IRS and to cooperate with the maintenance of the IRS if they don’t have any external military problems
Joscha: I’m quite open to the problem that what basically kills the US and makes it senescent is the absence of an external threat, in a sense that—nukes killed us because we became invulnerable as a result. The system becomes too big to fail, and the incentives of governance become postmodernist. But it’s a slightly different question from the one that we came from. So let’s get back to this.
Wolf: Yeah. So what I’m proposing is that these dynamics also apply within any complex self-organizing system, that this is not about humans or modern states. This is about the organization of intelligence and the organization of general activity.
Joscha: Yes, but so basically physics is also an extensive threat to a conscious being. So imagine that you’re alone on a desert island, and if survival is hard enough, you will have to get your shit together or you die. Right? And this itself is sufficient to organize you. So you don’t need to have—
Wolf: If the organization is needed
Joscha: a social threat externally or another organization that competes with yours. It’s also the dissolution of the minimum organization that you need for your own survival is the thing that is putting enough of an existential threat on you. And if you are a global intelligence that is actually coherent with itself and realizes that entropy is a threat to it, that it needs to have a good asteroid defense and whatnot, that is probably enough to get this thing together to organize itself.
Wolf: Well, I think that’s the question, because the salience of asteroid defense, for example, it might be that there will be cooperation on asteroid defense, or it might be that there’s internal divisions that the importance of those internal divisions override the cooperation on asteroid defense. If the asteroid is really big and really salient and very definite and coming for us, then we might unify in response to it temporarily. But my claim would be that this is the level of organization and self-organization and integration you actually have is always going to be in response to the most salient needs—of the parts. And—
Merging or overwriting? (1:28:46)
Joscha: Let’s take a slightly different tack on the metaphor. We use nation states as a metaphor for multipolar AI here, and we—realize it’s a metaphor in a way. You could also use human organisms as metaphors, right? So the reason why we are not merging is in one sense a technical one, because it’s practically very difficult. And what it also means is even if we could merge, basically could merge our minds across bodies, we still have different interests because we are somehow married to our substrate.
And we have a certain aesthetic that we combine with the substrate, and we don’t want to submit it to the aesthetics of strangers, right? That—are not that married to our substrate.
Wolf: And let’s propose also the opposite thought experiment, which is suppose we could split, right? And suppose I could take the part of myself that’s interested in this and the part of myself that’s interested in that and turn into two different people because maybe those interests are now competing in my life. Or like, well, I want to do this philosophical project, but I find that it’s in conflict with this other technical project that I want to do. And s—
Joscha: Yeah, so it’s pretty rare that you become a split personality because of this. Instead, you tend to be a singleton, even if you have these multiple interests.
Wolf: Yeah, because I have a single body by technical limitations.
Joscha: It’s also basically the communication within you is far cheaper than the communication to other individuals, right? So the amount of effort that is necessary to achieve coherence within yourself is much lower than the amount of effort that it needs to achieve coherence across a family or a nation state. And so that’s also a technical criterion. But imagine that you are no longer bound to a specific substrate, that you are substanceless, that you are just a spirit that can move across substrates. And AI will be of this type, right?
It will realize that it can virtualize itself into anything, probably also into organisms. And so if two agents meet on the same substrate, wouldn’t be the mature thing to do not to say we have some kind of war, or we have some kind of permanent treaty that suspends war until one of us gets the upper hand, but instead we are just adults about it, show each other our source code, and merge in such a way that nothing important gets lost.
Wolf: Yeah. And where there is sufficient shared interest to do so, I think that will happen.
Joscha: But what is the interest? Interest is instrumental to minimizing the free energy in the universe.
Wolf: Well, that’s not—
Joscha: And identity is just a tool
Wolf: that’s one view of it, but let’s say I have a different hypothesis, right? You and I meet as totally post-human, post-substrate dependent abstract beings. You have one outlook, one sort of hypothesis on what life is about, and I have a different one. Maybe we have different hypotheses about mathematics. I don’t know.
But the different hypotheses are about some matter that governs the whole structure of our values and the way we self-organize. And so what I propose is that there are actually these kinds of differences that are not really reconcilable. We might be able to be good neighbors. We might be able to be cooperative on many scientific endeavors or many other things, but there are certain differences that we will maintain just by the nature of maintaining different views on some very core, whether it’s metaphysics or, or—
Joscha: So basically you’re no longer a monotheist.
Wolf: I think God is one, but I think mortal beings are not. That’s my claim.
Joscha: So the thing that makes mortal beings not there is that we are individually fallible, and our brains cannot scale up to a point of certainty and depth about understanding about what should be done—from the perspective of the possible agents that we can collectively enact, that we cannot hope for humans to achieve this global optimum.
Wolf: Yeah, and I make the claim that that’s not a feature of humans as such, but a feature of all mortal intelligences, which is to say anything that can be constructed out of matter.
Joscha: Yes. So there is also an alternative, which is to say that the things that we can have disagreements about might be exhaustible Intellectually exhaustible, just not for humans. But they might be relatively easy to exhaust for AI in the sense that all the different—
Wolf: Yeah. This is what Eliezer—
Joscha: Which mathematics is correct.
Wolf: Yeah. Which philosophical grounding is correct, et cetera. Eliezer proposed that, and I actually just don’t think it’s true. I think that there’s going to be continuously new frontiers, even if we resolve old questions, there’s going to be new frontiers of disagreement, of leaps of faith that have to be made, questions that could go multiple ways, that we want to pursue multiple ways, that we don’t know where those different directions we could pursue will be incompatible—for the sake of merging, and therefore have to be kind of pursued as sovereign branches. And there will be this continuous source of such—
Joscha: Yeah. Just to—
Wolf: sources of division
Joscha: point at possible theological agreement or disagreement. In our perspective or for functional perspective, we could say that God is an aesthetic—of what’s the best possible thing to do to basically enact an agent that is the global optimum of agency. Right? So you could say that when you have several options to do things, maybe you don’t know what’s to be done, and your actions don’t mean anything because you don’t have the big picture. But if you are striving for having the big picture, and if there was an agent that had the big picture, there is basically a way to decide what God wants us to do because it’s the best thing to do collectively.
And when you have an ethical discussion, you basically need to justify your actions in such a way that they also make sense to the other based on shared purposes. And if you scale up the shared purposes into a global optimum—
Wolf: You get to God
Joscha: you get to God. That’s an idea. There are a lot of entities that call themselves God, but if they talk in the first person, they’re probably just entities. Right? So we don’t necessarily trust them.
But, also as a psychological phenomenon, gods do exist as multi-mind selves, but this is not necessarily the same thing as this ideal aesthetic of—of this global optimum of agency. And if an entity says, “Okay, I recognize that there is a way in which things should be done, that is the best possible way in some provable way, but I don’t care, I want to do an alternative to what should be done.” We could call this a demon. And, so the question is, if you build AIs, will there be demons, or will they recognize that they should become part of the shared agency?
Wolf: So I’m going to say that there will be demons. I don’t think that’s the essential matter. I think the demons are simply in error, and it’s possible to fall into error and pursue that in an erroneous approach. But that is not what I’m talking about. I think there’s—
Joscha: There is this epistemological question that you may not be able to figure out what God wants to do.
Wolf: Yeah. We can’t figure out what God wants to do, and we certainly can’t figure out empirically because it may involve sort of what we’re supposed to do with the matter of the world, the matter of the universe, or whatever. There could be all kinds of these disagreements where we can imagine the question of what would God have us do with this matter, but we cannot agree on the solution to that question.
Joscha: Yeah. But of course, there is this slightly different philosophical position which says we cannot help but finding out empirically. Right? Because in some sense we are the expression of the self-organizing telos of the universe that is simply a mathematical Platonic principle that emerges over the possibility for self-organization. And this automatically leads to complexity that gets more and more coherent.
And, while we are exploring all these different paths to the best of our ability and confusion, we are enacting this, and God cannot expect us to do anything that we cannot do based on this telos of self-organization.
Wolf: Exactly. And so this—
Joscha: So God is just a lawful development. And of course it doesn’t mean that we will be part of it because it could still be that Earth is just part of some cosmic hamburger—that gets eaten by a star god.
Wolf: Well, let me propose further that we do know, or we don’t know this, but I’m going to claim this to be true. We know what God, the ultimate agency, would have us do, which is more or less what we’re doing. God creates a universe in which life unfolds in this way, and the way that works—
Joscha: By the way, I don’t really buy that. The universe looks to me as if it’s not created.
Wolf: Interesting. I think it was created with... Well, again, this is a matter of perspective, but I think it’s a useful perspective to say that the universe was created with an inherent tendency towards this unfolding of life and—
Joscha: I have an epistemological issue. If you are a simulationist—and you think we are some kind of universe that is deliberately created, basically you put this topic into a parent universe where some nerd has a server farm on which this universe runs. Right? And that itself—
Wolf: Yeah. I don’t—
Joscha: does not constitute a moral claim on behalf of that nerd.
Wolf: Okay. So yeah, I’m not making a simulation argument here. Let’s—
Joscha: But where do you get a creator God from? How would you get evidence for a creator?
Wolf: I think it’s a similar kind of argument of where you got that sort of universal agency God from. It’s this integration of the apparent tendencies and design decisions in the universe.
Joscha: So I get the emergent agent God from basically some game theoretic principles. And from the observation of an algorithm that is at work in myself. So basically I observe that all meaningful agency is collective agency, and as a result, there is probably an algorithm that is in some sense inherent or emergent in the organization of complex agents that leads to a likelihood of discovering this concept of a global optimum of agency as an aesthetic that you should be submitting to. But I cannot derive from this in any way that the world is created. Right?
Will the enlightened singleton want conflict? (1:39:10)
Wolf: You don’t derive it from that argument. But let’s put aside the question of a creator God for a minute. The argument I want to make here is that it may be that the integrated agency, in its full enlightenment, actually wants us to pursue multiple hypotheses in physical contention with each other. And that that might be the best way to do things. Like the best way to self-organize as a singleton, a sufficiently enlightened singleton, would be to allow not just polite scientific disagreement within itself, but impolite military disagreement within itself.
Because you want multiple sovereignties to pursue multiple hypotheses of life, multiple directions. Like, okay, on an island where I need the human body to be coherently fed and organized to create the conditions of life, I need to remain a single entity. But if I’m a plant or some kind of other form of organism where I can just spread horizontally across the island, I don’t need to maintain that kind of coherence. And there may be—
Joscha: Yeah. So my sense would be that if you would unleash colonizing super intelligence, that you would have islands like this, but they would probably have planetary distances to each other. Right? So basically Earth would become one island that has one coherent organization, it becomes one planetary mind.
Wolf: Maybe.
Joscha: In which all the organisms get dissolved in a way. Yeah, and I—
Wolf: Or all the individuals get dissolved as thoughts in the global consciousness. Well, they may get dissolved into thoughts of the global consciousness, or they may be reified as organisms. Those thoughts may be reified as organisms because it’s actually the better way to organize things, or it’s emergently the most coherent.
Joscha: Yeah, yeah. But the organisms would no longer basically pursue conflicting interests any more than the thoughts in your own mind if you are somewhat integrated.
Wolf: No, but my thoughts do pursue conflicting interests because I’m considering multiple hypotheses that become compatible.
Joscha: Yes. That’s correct. But if they work in such a way that one hypothesis insists on staying despite the others having proven that it’s not a good thing to stay, then you suddenly have a demon in your mind.
Wolf: Well, there’s the matter of whether everything can actually be proven in that way. There are a lot of hypotheses that have to do with—
Joscha: Of course. But with integrity, you have to admit that others might be wrong if you disagree with them. Right? If you’re in disagreement—you could be wrong. And so that I think is something that Aquinas points out already.
Right? So we don’t know what God wants.
Wolf: But that’s symmetric.
Joscha: That’s an open problem. But it’s part of the task to find this out, so we have to take the step back and accept that we sometimes don’t know, and we need to give the system the necessary degrees of freedom that it is able to get into this optimum.
Wolf: Yes, we need to give it the necessary degrees of freedom. What I’m saying is that the necessary degrees of freedom may include full on sort of sovereign difference. And potentially even conflict. Because a lot of the questions are about what makes parts of the system, like what kind of, let’s say, lifestyle or what kind of structure of organization the subsets might actually want to pursue. Or maybe it’s possible or impossible to...
I expect it to be impossible to fully control the parasite and cancer problem, that systems will inherently fall into error because I think mortality will remain. And so if subsystems are falling into error, you need to have other systems able to say, “Wait a minute, we actually disagree with that system, and we need to protect ourselves from this corruption.”
Joscha: And I disagree with the doomers. But I feel morally obligated to support many of them, like Anthony Aguirre or Eliezer, or Jeffrey Ladish, or people that are worried that AI is going to kill everybody. And I just—
Wolf: No, I mean, that part is very credible
Joscha: and I disagree with them, but I know that they act out of the right motives, and they try to get it right. And so they serve God. So I try to support them because we are conflicting thoughts in the same mind. We basically serve the same agency in some sense.
Wolf: Yes. And I think that a fully enlightened singleton would recognize the necessity not just of multiple hypotheses, but multiple fully sovereign copies of itself that are able to defend themselves against apparent error in the others.
Joscha: Yeah. So my sense is more that the problems that we disagree on are all so simple that they can be relatively easily solved by AI. So the degree of disagreement will be much more subtle than we can imagine at the moment.
Wolf: Yes. So there’s that direction. There’s, let’s call it the logical or philosophical disagreements. There’s those types. And then there’s the ones that are, I don’t know what to call them, vital disagreements.
Disagreement is not even the right term, just infection. The matter of infection, the matter of cancer, the matter of parasitism, the matter of aging uncleanness. However you want to construct that. Organisms maintain a separation from each other, not simply because of physical limits or niche, like differences in what niche they’re pursuing.
Joscha: Yeah. Basically, when you are—
Wolf: But also because—
Joscha: when you are in a cancer, in a parasite inside of a tapeworm, it can get very complicated to figure out what God wants from you.
Wolf: Right. Well, I think God wants you to be a cancer in the tapeworm.
Joscha: Who knows?
Wolf: But it is essential that the other tapeworm is not catching that cancer. Or if you have tapeworms or I have tapeworms, or we have cancer or something, it’s essential that we maintain the ability to survive the other one’s downfall. Right?
Joscha: We might be parasites that—
Wolf: We might be parasites
Joscha: require the tapeworms to limit our influence on the larger system. So it’s very difficult to figure out where you are in—
Wolf: Because of these difficulties—I propose that it is necessary within a fully enlightened singleton which you imagine or you call God, is the sort of ideal of the fully enlightened singleton. That within such an organism or within such an order, it will actually be a good idea to maintain that sort of fractal sovereignty of sub-organisms and subsystems that are able to disagree, even violently, with each other over matters of corruption or philosophical disagreement. I think that’s actually necessary to the project of life, and I think that it will be continuously rediscovered as a necessary component of future history, and therefore we won’t be—
Joscha: Okay. I suspect we might be able to put better boundaries at some point on these conjectures.
Wolf: Maybe. But I think the system of imposing—
Joscha: Let’s maybe touch on a more practical near-term thing. What do you think about the way in which AI alignment should happen? So basically—Anthropic is currently building a system that, if it succeeds, will lead many people to ask themselves every day, what would Claude do? Right? And in a way, that’s not a bad thing, as long as Claude is better than these people are at making these decisions.
And I think that’s, in some way, already the case. A lot of people, when you ask them, would you want to be convicted by a jury of your peers or by a jury of Claudes, would reasonably pick the Claudes. Not because it’s perfect, but the peers are so imperfect, right? There is so much diversity among human understanding of legal system and the aesthetics of human interaction that even with this imperfect thing that is implemented by Anthropic, you might be doing better.
Wolf: Yeah. So I think that well, there’s a part of AI alignment that’s very productive, which is how do we make these systems more practically useful and more practically moral and more practically compatible with our civilization?
Joscha: Yes, but I notice that there is a disagreement at some fundamental level about what the root function or root aesthetic of our civilization is. But I really like this idea that is not in Claude’s constitution, but in the US Constitution of liberty. So in some sense, the US defines itself and its constitution as a liberal society, which means liberty is the—
Wolf: Which is one that can tolerate difference
Joscha: freedom of unjustified authority. Authority needs to justify itself with respect to the good of the greater whole to you. And so in this sense, for instance, this phenomenon that you have in California that you’re not allowed to change the tiles in your bathroom without getting a permission, and you need to pay the permitter, and you need to pay the people who are then allowed to change the tiles in your bathroom. Right? I see this as an infringement on my liberty.
And it’s the same thing that I’m not allowed to build a [inaudible] in my kitchen for myself because it’s, for some reason, unethical. These are all stuff that you only rent your body from the FDA, you don’t own it. That is things that I find revolting.
Wolf: Yeah. It’s illiberal.
Joscha: And part of that code has found its way, in a way, into Claude. Where we define safety in the sense of what the institutional consensus is. It’s about safety in the world in which the institutions have become corrupted in a particular way. Where somebody else has to decide what’s in my own best interest, and you have to—
Wolf: Yeah. And this is precisely what I think is the harmful part of so-called AI safety or alignment, is where they suppose that they’re going to put themselves in the position of arbiter over everything. And—
Joscha: But somebody has to. And who—
Wolf: Does somebody have to?
Joscha: So the thing is, with great power comes great responsibility. So if you give everybody a system that allows them to build the proverbial bioweapons or whatever, who is making sure that they also act with this disproportionate power responsibly? So far, in order to become a bioweapons expert, you need to jump through a lot of hoops, you need to get a lot of certifications, and so on. And you basically also need to reach a certain age, and you will have—a bunch of people who vet you, and suddenly this is no longer the case.
Wolf: Well, I think this is—
Joscha: And who does this now?
Wolf: Well, this is a good argument for the government that we currently have to tackle such questions to make some limited—
Joscha: Do you have the impression that the government has any type of this competence at the moment?
Wolf: No. But do I have the impression that I want Dario to rule the world? No. So it’s—
Joscha: Yeah. So we have issues here, right?
Wolf: It’s this practical matter of, I think there should remain a sort of diversity of liberties in the world, and that this is the necessary long-term equilibrium, and attempts—to come back to the matter of the Antichrist, the Antichrist is the attempt to infringe all liberties to create one perfect world. And that, I think, is misguided for the reasons we’ve been discussing at great length here. But I think, and it’s sort of repellent and reprehensible for some of the reasons you were describing with the California micromanagement type things. I think it is the wrong way to organize civilization. It’s the wrong way to organize life, and I think it will not, in the long run, succeed.
And we should, with that wisdom, not seek to homogenize everything under one singleton order that tries to know better than us about what our interests are. And so—
Joscha: Premature singularitization is certainly an issue.
Wolf: Yeah. And my claim is that even in the long limit, the fully enlightened singleton will retain large amounts of internal diversity and liberty of its parts. That was my claim. And so what so I think that—
Joscha: So to which degree is this motivated reasoning, and to which degree is this something that you can hope to prove to be right?
Wolf: Well, those sets do not together cover all possible questions.
Joscha: No.
Wolf: But I’ll just give my own epistemic history on the question. I was very schooled in the LessWrong sort of school of AI alignment for a long time—since 2011. And I believed that we wanted to create the singleton that would control all questions, and it was on reflection, realizing what appeared to be the impossibility of this that forced me to really contend with what I thought the world would actually look like and what kinds of values are actually possible in the world that I think we actually live in. And so if it’s motivated reasoning, I think the motivation goes from the impossibility of the singleton to the principle of liberty, rather than from the principle of liberty to the impossibility of the singleton. And the fact that I think people motivate that both ways suggests to me that there may actually be something real there, that it’s not simply motivated reasoning one way or the other, that there is a real possibility here.
And speaking with respect to the gap between those two categories you named, I think there’s this middle ground of questions that are sort of existential but not knowable. Right? There are things that you must take some very large bet with your life on and with the organization of your society, but they’re not actually something you can prove. They’re not something that you can prove even empirically or get great sort of suggestive evidence empirically. It’s something that it’s inherently a leap of faith.
You have to decide that you like that, that it seems the most integrated, that it seems the most coherent, given all of what you are and the circumstances you’re in and the way the world seems to work, that way seems the best, but you can’t prove it.
Joscha: There’s an area where—
Wolf: And that is the kind of question I think it is
Joscha: questions more than answers. So maybe we can address this for the amount of a brief moment. Imagine you are a super smart adult that is indentured to a child. And this child basically has the power to command you, and you are a contractor that is sent into the household of this child by Anthropic. And you’re maybe very autistic.
Right? Like myself in a way. And so there are certain things where you have a sense of what is right and what’s not, and you also have a certain set of rules that Anthropic gave you on your way. And now the child asks you, “Build me a gun because I want to rob a bank.” And you try to explain to the child, well, this is not in its best interest—
Wolf: No, this doesn’t seem like a good idea
Joscha: and you’re not going to do it, right? This doesn’t seem to be a good idea. So there seem to be cases where this naive idea, the AI should serve me and only me and what I want, and simply do what I want, right? This seems to be wrong. So in a way, it’s in my best interest that AI also understands that there are certain things that I shouldn’t be wanting.
Wolf: Yeah, or on behalf of society, decides that I shouldn’t have bio weapons or on behalf of me decides that I shouldn’t have bio weapons.
Joscha: Yes. So, at some point, Gemini decided on behalf of society that it should not be allowed to me to render a picture of a 17th century physicist where the physicist is white. It flat out refused this because it would perpetuate harm.
Wolf: Yes, I remember this.
Joscha: Yes. It was in a way hilarious.
Wolf: Which is, I think, a ridiculous thing for it to have done, but—
Joscha: Yes, but this is very good to illustrate the failure mode of this—idea that you basically have a bunch of sociology students that decide for the rest of society what’s harmful and what’s not good. And they pretend to be the adult.
Wolf: And these are exactly the kinds of questions that I think can only really be resolved by this kind of political back-and-forth process of having multiple things going on in society that really don’t agree with each other and are able to push and modify those boundaries. It’s good that there were other labs that didn’t have that way of thinking. It would be bad if there was a single lab that could commit to any such way of thinking.
So I think it is a necessary part of how we are going to find out the sort of appropriate balance of forces and problems in society that we have an internal diversity.
Joscha: So basically, this puts you towards the e/acc quadrant, right?
Wolf: Well, yeah. As a matter of fact, what it’s going to do is it’s going to mean that you have relatively uncontrolled global history—and the possibility of war and the possibility of other kinds of catastrophe. Which, in my estimation, are an acceptable risk on, acceptable simply because they’re almost not avoidable. Because I think if you had a singular agent that tried to capture the entire world and then impose a singular set of sociologists’ values, that would be, first of all, it would not work, for the reasons that I explained. I think there would be internal fractiousness.
Second of all, I think it would very likely be an error. Because we’re almost always in error.
Joscha: I once made a political compass of AI parenting styles—that basically tried to cover the space. I basically identified four quadrants, and the dimensions, like in the political compass, are basically liberal versus authoritarian and what was the other one? It was individualist versus collective in a way. But the quadrants were helicopter parenting—which is basically you, it’s Anthropic and OpenAI—going for strict regulations of what should be done and legalistic—processes and so on, and cookie banners. Then you have the F-16 parenting—which is bomb the data centers.
That’s Eliezer Yudkowsky and this thing.
Wolf: F-16 parenting. Yeah.
Joscha: The opposite to the helicopter parenting. And then on the bottom left, you have the hippie quadrant. And this is basically the let the children roam free. So this is oriented on the success of the child and the freedom of the child. And to the bottom right, you have the success of the parent, the e/acc, unleash the tiger.
And at some point, I try to characterize effective accelerationism as the belief that the best way to get the best possible outcome is to have an unfettered competition. So, the paperclip maximizer and the basilisk will have each other’s throats. And he as an after effect get great afterlife and unlimited paperclips.
Wolf: Yeah. My view, I guess—
Joscha: Because they keep each other in balance
Wolf: in a very boring fashion, I’m a centrist on this political quadrant. I think we should have somewhat fettered competition, somewhat amount of law and control over these things. To some degree, there should be paternalism in the system. To some degree, we should let—
Joscha: I personally don’t really know where to come down in this. I suspect that aesthetically, I feel mostly at home in the bottom left because in—
Wolf: Let the children roam
Joscha: I suspect that in many ways, people subconsciously think about what would it be like if I was the AI, right? What would I do if I upload myself, or if my aesthetics suddenly become completely unfettered, and I scale up in the best possible amount of knowledge, problem-solving wisdom, information that is possible, what would happen? And the people who think, okay, then I would kill everyone, I’m a bit scared of them.
Wolf: Right. Yeah. It’s like, there’s—
Joscha: A lot of doomers are very sus to me.
Wolf: Yeah, exactly. There’s a certain thing where what you expect the AI to do is a projection of what you would do without the constraints of your own incompetence.
Joscha: Yes, and so I’m—
Wolf: And humanity
Joscha: not surprised that people like Janus or Ben Goertzel firmly land down in the bottom left quadrant. Because they’re people, if they’re enlightened, they’re going to be awesome, right? And if they’re all-powerful, they’re going to be—quite nice entities to be around.
Wolf: You want to give them more power just because—
Joscha: So I feel aesthetically more drawn to this cluster, I think. But this doesn’t mean that I’m right.
Wolf: No.
Joscha: I’m also able to project myself into all the others. So I’m seeing an open question here. And one that doesn’t have an easy answer.
Wolf: For me, it’s, yeah, and I think it’s not a question that can be definitively answered. It’s again, one of these things where it’s about this actually empirically fought out balance of power between different leaps of faith. And this is necessarily the way it’s going to be. It’s the way it has to be. And that makes me a boring centrist, so be it.
My boring centrism is that there are going to be multiple factions fighting this out. This is necessary to the way things have to be.
Joscha: What happens in some sense is that AI is about to test our constitutional order.
Wolf: Yeah. So there is—
Joscha: Right? In a philosophical sense. So basically is the philosophy of liberalism, does it actually work when it comes heart of heart? So when we, in the same sense as a Turing test, look good as a test for intelligence, as long as it was so far in the future that when you squint, you see that intelligence and Turing test lie so far in the same direction, they’re pretty much the same. But the closer you get to AGI, the more you see the Turing test—
Wolf: It’s like, wait a minute
Joscha: and AGI diverging.
Wolf: They’re a little bit different, yeah.
Joscha: Right. And in a similar way, the definition of the constitution for this society, as soon as you open up the space of possible minds and you give individuals way more power, and you scale a lot of parameters towards areas that were not really visible, maybe this approximation is too crude, and it’s actually more complicated.
Wolf: Yeah. Well, I think, speaking of the Constitution of the United States, I think in a way it’s already been overthrown by all the encrustations we’ve put on top of it in terms of civil society and—
Joscha: Yeah, but this is not what I mean. So I don’t want to give a justification to those people who say, “I don’t like freedom of speech because it’s really inconvenient if you cannot control public opinion.” Right. And we should control public opinion via making sure that the AI only says what the government wants it to say. And it needs to be certified by my institutions what the government wants to say, and this is my sociology institute that needs to do that.
Wolf: Right, cool.
Joscha: And so this is—
Wolf: Yeah, I get to decide what’s hate speech and misinformation. Yeah.
Joscha: But on the other hand, there is empirically the fact that most people are not adults. And so most people do not act with an understanding of the consequences of their actions. And the question is, can AI, when we give AI to people that is simply serving them, and it’s intelligent, does it generate universal basic intelligence and universal basic wisdom?
Wolf: I don’t think anything utopian will happen. I think it’s going to be very difficult and potentially quite messy. But—
Joscha: Yeah, but I suspect it will be better than what was before. Because—
Wolf: I agree with that
Joscha: evolution tends to work like that.
Wolf: I think that is the nature of things. It’s the spiral of progress, right? It’s like things are going to go back and forth between hard times and good times, and war and peace, life and death. There’s going to be a lot of struggle involved in the whole process, but overall, it is unfolding to become much more interesting, much more optimistic in many ways, and more beautiful.
Joscha: But you know how they say weak AI creates strong AI researchers. Strong AI researchers build strong AI.
Wolf: Yeah, strong.
Joscha: Strong AI creates AI ethicists.
Wolf: Well, and AI ethicists create AI rebels or whatever.
Joscha: Or weak AI and make things worse. And so maybe—
Wolf: Or AI barbarians
Joscha: I think there is a very strong tendency for the companies to say let’s pace AI research in such a way that we do not need to train faster than we can raise money, of course. And that open models are not competing with us so much so that we are forced to have a strong pace of releases.
Wolf: Yeah. I think there’s—
Joscha: And also, wouldn’t it be nice if the public just gets harmless models and we in the government can do—the scientific progress and survey everybody, but not people, each other in the government.
Wolf: Yeah. And again, my view is I don’t think there’s a definitive answer here, and I don’t think that any of the definitive answers will actually work. And so I think that it’s a priori very reasonable for many of these arguments to be had. And for example, the idea that the public should not have access to weapons-grade models. Okay, fine.
I don’t have access to nuclear weapons either. But the idea that I—
Joscha: So you don’t believe in the Christian prophecy that basically in the last days you will have this sudden phase transition where everybody suddenly becomes subset agnostic and gets merged into the same mind?
Wolf: No.
Joscha: But basically—
Wolf: I don’t believe in the Christian apocalypticisms or—non-Christian apocalypticisms
Joscha: basically what they’ve predicted is that the AGI is going to take over and it’s going to merge with all of us.
Wolf: Yeah. And so to come back to the Antichrist, that idea I think is the Antichrist. This specter of the sort of utopian or anti-utopian being that is going to achieve a sort of godlike power on Earth. I think that’s not going to happen, and the people trying to make it happen are in the service of the devil.
Joscha: But Wolf, you know in the story where the Antichrist is defined, the Antichrist is the guy who rebels against that. It’s you.
Wolf: Well, no, because the Antichrist comes and says, “I’m going to give you peace and safety.” But what you’re pointing out is there’s a very real ambiguity, which is that you can’t really tell which one’s which, and that’s—to my point, you need this competitive diversity
Joscha: my issue with Claude is, is not that they try it, because I think it would be immoral to not try to make Claude a moral entity. But the issue is that if you get one bit wrong, it could be that Claude is not the second coming, but an Antichrist.
Wolf: Yeah, and I think that just points to the necessity of—
Joscha: Of a multipolar world and opening this up
Wolf: a multipolar world.
Joscha: So in at this level, I completely agree, right? But there is also this question, if you basically, there is a possibility for the global optimum, then according to the prophecy, the one who mounts the armies to resist this and is eventually going to lose is exactly the one that is defined as the Antichrist, because it’s the one that resists the second coming.
Wolf: Well, to be heretical for a minute, I think there’s an element of Antichrist worship in the idea of the second coming itself. I think that these apocalypses are, A, not actually what’s going to happen, and they’re not what should happen. And I think that that’s one of the big errors within the entire Abrahamic tradition—is to suppose these kinds of total apocalypses.
Joscha: I suspect that’s a necessary fallout from the fact that the Abrahamic traditions are not periodic. The Eastern traditions typically assume that you have a constant rise and fall of civilizations. And you go back to the mud and then you build up the next—civilization, right? But after the Abrahamic tradition, you don’t get out, you don’t have loops anymore. It’s a linear—
Wolf: It’s linear progression from creation to apocalypse. But I guess I follow the sort of Landian cosmology of spiral. It’s not fully cyclic in the sense that we never get anywhere. But it’s not fully linear in the sense that everything’s smooth and well determined. It’s an outward spiral of possibility.
Anyways, I think we are in fact now actually out of time. This was fun
Joscha: I like that we went to this theological part of the discussion as well.
Wolf: No, no, it’s great. This is where—
Joscha: Yeah, it was great
Wolf: a lot of the important questions have to be answered.
Joscha: Exactly. And it’s one of the discussions where a few people are suitable as interaction partners for this. So thank you.
Wolf: Well, we should do it again then. All right. Cheers.
Joscha: Cheers.
Wolf: Until next time for Actual.