Oliver Habryka joins Wolf Tivy to tell how he revived LessWrong after it nearly died, built the Lighthaven campus, and launched his $35m/quarter project to reform grantmaking.
Transcript
Intro (00:00)
Wolf: Welcome back to The Students. I’m here with Oliver Habryka, who’s these days running LessWrong, I think—
Oliver: That’s right
Wolf: and running Lighthaven.
Oliver: That’s right.
Wolf: Both projects that are very interesting, at the cutting edge of certainly AI discourse, rationality discourse. A lot of very interesting things going on over there. This is a community that I’ve been involved in peripherally throughout the years. I think I first got involved in 2011. And you rebooted the whole thing in, what, 2017?
Oliver: Yeah, 2016, 2017.
Wolf: 2016, 2017. And yeah, it’s been going very well for you over there. So I wanted to talk to you about the problem of building high-quality intellectual community. This is something we’re obviously very interested in with The Students, but you’ve done a great job of it.
Oliver: Thank you.
Wolf: Obviously, not solely. There’s many other very—
Oliver: No, definitely
Wolf: very intelligent, talented people over—
Oliver: I luckily got to—
Wolf: over there
Oliver: inherit a lot of stuff that other people have built—
Wolf: Yeah, no, it’s—
Oliver: Eliezer built. Yeah
Wolf: I think Eliezer obviously is the final credit on the matter, but yeah. So I’d love to just hear more about what got you onto the problem. Let’s go back to 2015, maybe—right? So when I was there in LessWrong 2011, I read the stuff, it was amazing. It’s kind of this huge, eye-opening new worldview for an ambitious young man who’s—intellectually inclined.
I think many people have gone through that experience. And very interesting stuff. But maybe by 2013, I’d gotten a little bored. I think maybe many people had gotten a little bored. The community had started to go into something of a decline in posting, at least.
And by 2015, 2016, were you thinking about rebooting this, or how did that actually happen, I guess?
Oliver: Yeah. I think the big thing that you’re missing here, and that’s of course the weirdest part of this whole story. Well, there are many weird parts of the story, but one of the weird parts of the whole story is, of course, “Harry Potter and the Methods of Rationality.”
Wolf: Ah, yes.
Oliver: And it was actually in between those dates. 2011 is, I think, when kind of Eliezer wrote a lot of these essays to make the core of LessWrong. And then around 2013, ‘14, ‘15 is when he started publishing “Harry Potter and the Methods of Rationality.” And that, a kind of hilariously, enormously successful, if you showed up at MIT or any of the other top universities—in the US in the technical degrees, would just obviously be the most popular book, and was easily my—
Wolf: Which is, I know. It’s just absurd, right?
Oliver: It’s absurd.
Wolf: It’s a Harry Potter fan fiction book.
Oliver: Fan fiction book.
Wolf: This is very much Eliezer’s style.
Oliver: That’s right. And so in some sense, actually, the most active years up until then much later, kind of the last two years, the most active years of LessWrong’s history were 2015, because that was when Eliezer was kind of publishing the final chapters of “Harry Potter and the Methods of Rationality”—
Wolf: Ah, right
Oliver: and then tons of the internet showed up and just like... This was around the time when I found all of this, was like—2014.
Wolf: That’s when you got in.
Oliver: Yeah. Well, actually for me, at something like 2013 is when I kind of found it—via “Harry Potter and the Methods of Rationality.” And then that was the peak, and then actually around that time, related to, God of all things, Roko’s Basilisk.
Wolf: Right. Yes, our old friend Roko.
Oliver: He got tired of moderating LessWrong, didn’t really want to handle the online mobs that were constantly out for blood, and then took a step back, and around the same time, there were just a bunch of people who had gotten really triggered and just really tried to destroy the site. So it was a specific user called Eugine Nier.
Wolf: Right. Well, actually I remember at that time in general, 2015, you’re talking about online mobs trying to like—
Oliver: That’s right. It was also this peak, like—
Wolf: This was, I remember being on Twitter at that time with my friends and we’re sitting around and it’s like, “This is completely unsustainable. We need to get off of this hellscape of Twitter where everyone’s insane, there’s crazy people from 8chan trying to come after us.” There’s all kinds of weird stuff going on at that time. And then, yeah, so we sort of ended up retreating into private Slacks and stuff, right? And some of the intellectual crowds that later became things like Palladium.
But it sounds like you had a similar experience then at that time, just this kind of sense of siege and craziness.
Oliver: That’s right. And then kind of LessWrong just solidly died between 2015 and early 2017, of just like—interactivity levels went to approximately zero.
Wolf: Oh, so it didn’t taper off from 2012. It—
Oliver: That’s right. It was like—it peaked in 2015.
Wolf: There was a peak.
Oliver: And it was of course a large distributional shift. There’s many arguments on LessWrong about how do we feel about all the fan fiction fans who are here. But I think it ultimately succeeded at just attracting a lot of really good people combined with a bunch of quite weird fan fiction fans.
What defines the Rationality community? (04:48)
Wolf: Yeah. Well, actually, before we go further—why don’t we, many of the audience are going to know what LessWrong is. But why don’t we give a little bit of a perspective on from our perspective, especially from your perspective, what is LessWrong? I remember when I showed up, it described itself as a community dedicated to human rationality.
Oliver: That’s right.
Wolf: How do you see it? What is LessWrong?
Oliver: LessWrong is like, one of the ways to think about it is it’s a place where the rationality community got started. It’s the place where like—
Wolf: What’s rationality? Not to—
Oliver: That’s what I was going to go into next
Wolf: contract the whole thing.
Oliver: Is like, rationality community got started, and it did so by Eliezer writing originally—kind of a long series of essays, originally on overcoming bias together with Robin Hanson—
Wolf: Yeah, at Robin Hanson’s blog
Oliver: kind of the GMU economists and a bunch of them. And then eventually they told him to get his own apartment and get his own place because he was—outgrowing Overcoming Bias. And then he founded LessWrong and populated it with the essays that he had written, but then kind of opened it up to lots of other people. And kind of what is the rationality community is a question always very difficult. It’s now a many tens of thousands of people community all across the world with a pretty complicated and thick ideology and worldview that’s very hard to summarize in—a few things.
But I think like as I’ve been thinking more about this and more and more people have been asking me this question of what is the rationality community? I think if I had to break it down to two core attributes of what defines the rationality community—I would be like, number one, it’s a bunch of people who are just really kind of pretty obsessed with the truth. Pretty obsessed with—
Wolf: And how to get there
Oliver: yeah, how to get there. It’s like epistemology is a term that gets thrown around every day. And fundamental epistemology. And people are very deeply obsessed with honesty of just like—what does it mean to be honest? How are we honest to each other?
And then a second thing, which I think often gets forgotten, but is actually really important if you want to understand most of the dynamics in the community, is that it’s trying to be scope sensitive. It’s really obsessed with trying to figure out what are the biggest things that matter in the world. And just people who are trying to take a perspective on the world and being like, “What are the things that will matter?” And having a sense that most things probably don’t matter. There’s only—a few things to pay attention to that will determine the course of history, that will determine the success of an individual’s life.
And so those two things, I think, explain most of what is going on. It’s like it’s a community that are deeply obsessed with the truth and deeply obsessed with—trying to identify the big things that really matter.
Wolf: Yeah. And then, of course, there’s all the particular heritage of it coming out of, I don’t know, internet transhumanist discourse at the whole AI current—
Oliver: That’s right
Wolf: obviously.
Oliver: But I think both of those are very well explained by those two core things. It’s like, why is it AI? Why is the community so obsessed about AI? And you’re just like, well, number one, AI is just really interesting if you’re trying to understand how to better arrive at true beliefs. Because it’s just like mechanistically the domain that you would want to study.
It’s like—how do you quantify and operationalize truth-seeking? But also it’s just the big thing. If I think almost any futurist who was worth their salt anytime in the 2000s, if they were trying to predict which things will matter, I think they would’ve been catastrophically dumb if they hadn’t put AI into their top five. And so just obviously that will show up and just like—
Wolf: Yeah. And just to make that even sharper, I understand that Eliezer got onto rationality and created the rationality community as part of his AI efforts. He was pursuing these AI questions, trying to study how do you build a being who can think.
Oliver: That’s right.
Wolf: And then, that obviously leads to a lot of engagement with the problem of how do you actually come to the truth and all this.
Oliver: That’s right.
Wolf: And then also he was trying to get other people to think about this, who thought, “Okay, how can I make them think better?” And so this is the original genesis of the rationality project.
Oliver: That’s right. Yeah. And so that’s the rationality community, and then LessWrong was on the internet, the place where it formed and—
Wolf: Yeah, the place where that happened
Oliver: kind of Eliezer opening up, other people posting. And early Scott Alexander and Slate Star Codex, that’s where it got born and Scott started posting under his username, Yvain. And then that gave rise to that whole branch of the community. And now one of the... Also, would you describe it quite exactly the same as the rationality community?
Unclear, but definitely one of the biggest intellectual influences and movements that exists out there right now.
The (temporary) death of LessWrong (09:21)
Wolf: Yeah. It’s definitely influential. So yeah, let’s go back then to the matter of that drop-off, 2015 to 2017. The arrival of the eternal September of fan fiction types killed the thing off dead.
Oliver: Well, that’s—
Wolf: Maybe I’m editorializing.
Oliver: That’s—
Wolf: What actually happened there?
Oliver: Yeah. I think definitely it wasn’t the fan fiction people. Fan fiction people were great. It was mostly just the trolls. There was this one guy called Eugine Nier—
Wolf: Just... Right
Oliver: who was just creating thousands of sock puppets account every day, and just like—
Wolf: Really?
Oliver: downvoting everyone he didn’t like. And he had a specific thing. It’s like he was kind of slightly alt-right vibe. He really hated anyone who was social justice-y or had anything remotely to do... He tried to identify any user that might possibly be trans and just tried to mass down-vote them and a bunch of other stuff.
Wolf: Oh my God. Yeah.
Oliver: He was super triggered.
Wolf: So, just the general online culture war apocalypse of that time.
Oliver: That’s right. But combined with just specific technical problems. It was an old fork of the Reddit code base. And it just didn’t have any good moderation tools, and you just like—
Wolf: It had vulnerabilities to that
Oliver: And then you had some moderators. Eliezer took a step back, and then you had some part-time moderators, and they didn’t want to deal with clicking delete on 500 new accounts every day. Where the only option to moderate the space. And so I think it died out of a mixture of technical reasons and the fact that just there wasn’t a good... There was no admins who were actually putting in all the effort to moderate and cultivate a culture.
Wolf: Mm-hmm. So then at some point in there, as I understand it, you and maybe others, I don’t know the details of the story, came to the conclusion that this was not something that could be accepted.
Oliver: That’s right.
Wolf: That this community or this place was too important to let it just die to trolls and chaos. And so you undertook a project to kind of reboot it.
Oliver: That’s right.
Wolf: How did that happen? Who else sort of had that idea? What was that conversation?
Oliver: Yeah. There was a passing down through a few hands of moderators, and then eventually Matthew Graves, Vaniver—was kind of declared benevolent dictator for life. I think he had been an active user on the site and had done a bit of moderation, but I think largely people were like, “Well, look, I think someone should be authorized and given the power to do something.” But he was mostly halftime, didn’t really do very much, was pretty skeptical that the site was really going to survive in the long run, was thinking of turning into a read-only archive. And then I think it was like New Year’s Eve, 2016, where I approached him and was like, “I think this is unacceptable. What are we doing? We’re just letting the central hub of our ecosystem community die. What—idiots are we?”
And then he was like, “Okay, yeah, I think I want to give you a shot. I want to give you the option of like—Creating an alternative, and then I got his blessing, and then I, for the next few months, just built the whole... I was a terrible programmer. I was still enrolled in my computer science degree at UC Berkeley. I had done some programming, like I’d founded a web design and web development company in Germany where we, I had done some simple front-end web development. But that was a much bigger project that I was taking on, just building a whole modern forum software from scratch. And then I did that.
Wolf: There’s actually a lot to it.
Oliver: That’s right. And then I built that over the next six to nine months. I wouldn’t say my code was great. And then we had a big vote where we started the LessWrong 2.0 beta. People could join.
And then we had a big vote where MIRI, a machine intelligence research institute, which Eliezer founded—they kind of owned LessWrong as the platform owners. And they were like, “Well, either we can transfer ownership to Ollie, or we can shut it down and turn it into a read-only archive.” And then we put it up to a vote—on which of the two we want to do to the high karma users. And that was then when the official passing of the torch happened. And then—
Wolf: And then this is the genesis, or did you incorporate at that time as, what was it, Lightcone?
Oliver: So we actually were fiscally sponsored by the Center for Applied Rationality up until 2023. Like—
Wolf: So until quite recently
Oliver: long time. Just like, yep. That was really nice of them. And they had a charity, and they were just like, “Yep, this is compatible with our charitable objectives.”
Wolf: Right. Of course. Yeah, it makes sense.
Oliver: And none of us paid ourselves any real salaries until like 2021 or something. So it’s like surprising—
Wolf: Right. It was just a labor of love.
Oliver: That’s right.
The software question (13:41)
Wolf: Yeah, one of the big questions that people often ask in makeovers like that is, why new software? Why not fix the old Reddit code base or—use something else, use something off the shelf? What was your thinking process in that? I know certainly new software has the opportunity for many adjustments of making it just right for the particular epistemic community and so on. I wonder how much you took advantage of that.
How much was that on your mind?
Oliver: The very key thing is just that there didn’t exist anything on the internet that was oriented around long-form collaborative writing. And just like, where would you have gone? There are tons and tons of places that are online forums, and those were like all—deeply oriented around this ephemeral kind of content. And I think this is just what you saw in the whole internet across that period. That was the death of the blog and the rise of the social media network.
And with the rise of the social media network, everything by default became ephemeral and short. And only with the rise of Substack did you actually start seeing anyone investing at an infrastructure level into long-form content again.
Wolf: Yeah. Or Medium, I think also, but—
Oliver: Yeah, Medium—
Wolf: kind of more—
Oliver: was at the time, of course, I couldn’t have used Medium. That was a proprietary closed—system. But also, I think that was around the time when Medium was already obviously becoming corrupt. It started out with being this great platform for blogging. And then I think at some point it just started, like I had multiple friends who had all their content on Medium and then Medium just paywalled their content.
And we’re like, “You now have to pay to access the essays that you wrote here.” And just like it ran into a giant monetization issue. And so I think that just that wasn’t a business model that anyone was pursuing, and definitely not in an open source fashion. And so we did technically start from a code base that was some random open source project that a guy from Japan called Sacha had started, that was like an open source forum code base. It was a terrible code base.
Shouldn’t have started with it, honestly. It hurt me more than it benefited me, even given my bad programming skills at the time. But yeah, it was just nothing that you could have done. And you could have started with the old Reddit code base, but that one was just like Reddit started as open source and then became closed source. So you would’ve already started with an eight-year out-of-date code base.
And that was just a terrible choice.
Reviving LessWrong (16:09)
Wolf: Uh-huh. Okay, so in the course of revitalizing it, obviously there’s this whole software half of it, but there’s also a very social component to that.
Oliver: That’s right. Mm-hmm.
Wolf: How do you take something that has run out of steam? I assume it had run out of steam not just because of the trolling and the difficulties with moderation, but also the community had sort of scattered.
Oliver: Yeah. That’s right. The community—
Wolf: The energy had gone.
Oliver: That’s right.
Wolf: How did you get that back? How do you actually take something that had existed and make it anew—or regather everyone? What was that process like?
Oliver: A lot of it, a huge amount of effort went into just consistently signaling that somebody will be there to take care of anything you invest into this community and invest into its ecosystem. And will curate it for you.
Wolf: It’s not—
Oliver: And so a big commitment that I made, that was just one that I fought for many months about, but about a few months into the project, I started reaching out to old core contributors and I was like—”I will do this for five years.” Like, “I already care so much about the content that is already on LessWrong, and what is there and what has been written in its archives, that I will promise that I will take care of anything that is posted on LessWrong in terms of not letting it get destroyed by trolls, in terms of doing moderation on it, and not letting the culture become toxic—for the next five years. So I’m just like, even if that means that it’s just me and a person with no money and low activity levels on the site. Maybe I will at some point start taking a different job, but I will make sure that if there’s content that I will preserve it and make sure that it continues to stay readable and well presented and as—the internet shifts, it can adopt to those changes.
Wolf: So just sending that signal of—
Oliver: That’s it
Wolf: someone is actually home, this place is actually maintained.
Oliver: That’s right.
Wolf: Your investment is safe here, basically.
Oliver: And one of the big components of this was design. I think just one of the things that LessWrong 2.0, I think did to the whole rationality community, which I’m very proud of—is I do think we kind of—I don’t know. I don’t want to be too self-aggrandizing here, but I do think we taught the rationalist community style. There’s a sense in which I think I’m just very proud of LessWrong’s design, and a lot of effort went into just making the website feel nice and making it feel cared for, and making it be a place that communicates a whole visual aesthetic. Like everything was—
Wolf: Yeah. No, the LessWrong 2.0 definitely has a style. It looks better than the original. It looked very polished, very nice.
Oliver: And it also just tried to kind of define a whole literary genre. I’ve become very—obsessed over the years with an aesthetic defines, in many ways, a whole way of relating to the world and what literary genre do you fall into. And I’ve fought a lot—over the years of trying to refine this aesthetic that’s some way at a mixture of... When it came to stuff like fonts, I looked into what are fonts that are most interestingly evocative of a mixture of modern scientific notation and Renaissance writing—and things that feels like it kind of evokes the correct kind of scholarly spirit, and at the same time trying to invoke some of the more modern components around AI, and trying to create—this mixture of this is a community that is rooted in deep scholarship. This is a community that is also, at the same time, pursuing ambitious, bold technology and trying to merge with that.
So just a lot of time went into that aesthetic, and I think that just contributed quite a lot to just people being excited about it because then they could get a better intuitive sense what it would mean to participate and be on there.
Wolf: Right. And that’s you just sitting there kind of researching these things and thinking about them, talking to people about them. And so a lot of the design was you? Amazing. Yeah.
And I remember even just the logo, right? The compass logo.
Oliver: The compass, yeah.
Wolf: It’s kind of like, what is this thing about?
Oliver: That’s right.
Wolf: It’s about navigation, right?
Oliver: That’s right.
Wolf: And, the whole map kind of motif—
Oliver: That’s right
Wolf: that gets used a lot. You’ve used the map motif quite a lot.
Oliver: That’s right.
Wolf: Even just in the publications you did of the old essays—and so on, but also just aesthetically throughout it, and obviously it’s conceptually very core to the rationality thing, map and territory—
Oliver: That’s right
Wolf: kind of idea. Yeah. So, okay, so you kind of pulled people back on just with this commitment that, look, this place exists. Someone’s investing in it. It’s gonna be in competent hands.
It’s safe to be here. How much of it is just that, versus did you have to really prod people to post again?
Oliver: Oh, totally.
Wolf: Or like—
Oliver: I mean, there was like, we made a—
Wolf: What was the whole process?
Oliver: And early on, we were just hustling everything. Just the whole kind of ecosystem had dispersed over this diaspora, so I would—almost every day be like, somebody I respect who was previously posting on LessWrong was now posting on a random blog. And it’s just be like, finding the post, link posting on LessWrong, copy-pasting the whole content into the website, sending them a DM being like, “Hey, by the way, I copy-pasted your whole content onto LessWrong. Let me know if you didn’t want that.” And just lots of...
I think that was good. Mostly people liked that, but asking for forgiveness instead of permission, and just consolidating—and pulling back all the content
Wolf: not getting them to post it themselves, but just posting it—
Oliver: That’s right
Wolf: on their behalf and then giving them the keys.
Oliver: And then just being like, “Hey, can I just like...” One of the first features that I shipped that was automatic RSS cross-posting, which was you can, as an admin, I can just make it so whenever you post something on your blog, you also post it to LessWrong with its full content.
Wolf: Right. Yeah, you just make that automatically.
Oliver: Make it so that everything gets consolidated into one place again. And that was just a huge amount of hustling and just messaging people and encouraging them to post on the site again, and doing lots and lots of talking and user interviewing and figuring out how we can help them.
The downvote problem (21:48)
Wolf: Mm-hmm. Through that process, and since then, how much, I assume you’ve thought a lot about the epistemic norms. I know you’ve added things like the second kind of component of upvoting—
Oliver: That’s right
Wolf: and down voting, like for agreement and disagreement, as opposed to quality. So, there’s various things you’ve done, obviously, in thinking through how the epistemic norms kind of—how the rubber meets the road on those. Can you say a bit more about that? Was it just kind of continuing the old sort of actually existing rationalist norms and LessWrong norms? Did you have to do a lot of innovation there?
Did you have to revamp anything? How do you think about it in terms of actually curating the intellectual community, not just kind of getting people to post on the platform?
Oliver: Yeah. One of the big ones, and that was actually kind of a thing that was true about every other social media platform at the time, where we had to build some new software is, for example, up voting and down voting. Really, really important tool. And in particular, just the whole Internet at the time, I think for reasons I could go into, it’s like a really interesting sociological study, but everything was moving towards up vote only. Like—
Wolf: Yeah, well, because you’d get brigaded down votes.
Oliver: You would get brigaded down votes, but also—
Wolf: And people don’t like to—
Oliver: It’s just a—
Wolf: They don’t like negative feedback.
Oliver: That’s right. People don’t like negative feedback, and it marked a shift in communities. If you have lower trust communities, allowing negative voting now suddenly produces very different dynamics. But allowing both up voting and down voting allows a certain kind of coordinating around... In particular, it allows you to punish bad behavior, which is like you can’t otherwise do.
Wolf: In relatively subtle ways.
Oliver: That’s right.
Wolf: Yeah. Of course, the stereotype of the Redditor is like they just down vote anything they disagree with.
Oliver: That’s right.
Wolf: And you end up with this hive mind consensus—
Oliver: That’s right
Wolf: kind of thing.
Oliver: That’s right.
Wolf: And so I assume you thought about that and kind of adding new dimensions.
Oliver: That’s right.
Wolf: And how did you overcome that problem?
Oliver: So one of the big ones that I think is, like Reddit, it really sucks to try to maintain a culture on Reddit. Because the whole system is set up that whenever anyone in your community produces something good, that is the context in which you will have least control over your culture. Because you have this Reddit front page, like the way a subreddit works, is you have to subreddit. That like—most of the time, its activity is only visible to the members of the subreddit. But then when anyone produces something popular, it shows up on a Reddit front page.
Wolf: Ah, right.
Oliver: And now all of these people who are from a completely different community, just get shown your content. They show up and their votes matter just as much as the votes from you. Community And so whenever there’s this high visibility event where in some sense you might expect norms and praise and various things to get set, in that context, actually, the members of your community can’t do anything because they get completely drowned out—whatever the Reddit mob wants.
Wolf: Swamped by the masses. Yeah.
Oliver: That’s right. And so one of the key things that was one of the earliest things we did was we just gave vote weights to different users. So one of the earliest things that we did is just we’d split the vote into you can vote on something in a weak way, it’s a weak up vote, or you can vote on something in a strong way, which is strong up vote. When you just hold and then you kind of have to hold for a while. And then your strong up vote just has a huge range of how powerful it can be.
Where just like, a new user shows up, has a strong up vote weight of one, makes no difference whether they small up vote or strong up vote something, because they don’t really have the discretion to do anything different. But if you’re someone who has been on the site for many years, your weight is going to be eight, nine, 10. And so just like that creates much more of a sense of even as things get popular and things get up high up on Hacker News, the voting continues. It’s influenced because it’s a really valuable form of engagement and it’s good to see that people can influence how things work a bit, but the established users and the people who have built kind of some trust on the site get to continue to have a lot of influence.
Wolf: Sort of entrenching the inside community actually is—
Oliver: That’s right
Wolf: is an important thing.
Oliver: You have to on the wide open internet, and occasionally the hordes—start arriving.
Wolf: Right. It’s this defensive problem, right?
Oliver: That’s it.
Wolf: Against the Eternal September—
Oliver: That’s it
Wolf: so to speak.
Oliver: Yeah. So that was definitely a big one. And then definitely, more recently, I’ve just been extremely happy about us having split karma, kind of the votes on LessWrong into approval and agreement. And that just like really felt to me, just like at that point I felt like that even just already communicates like the whole philosophy like not the whole philosophy, like a major philosophy of the site.
Wolf: Right. It’s like you’re getting fine-grained about what kinds of epistemic feedback you’re getting.
Oliver: But also just like the quadrant of up voted strong... Like up vote disagree. It’s just such a great quadrant that shows what we care about. We’re just like—yeah, this is a place where we expect many people to disagree with each other. And we celebrate the disagreement by, and we still try to meaningfully arrive at the truth, but we understand that somebody can say something that many people disagree with, that nevertheless I’m very, very glad about them having said.
And like in the last week I’m currently in the middle of a comment thread, where I’m arguing with some people about Richard Ngo’s writing.
Wolf: Oh, yes.
Oliver: And I have some really solid plus 20 karma, negative 70 agreement karma. And I think that’s like, there’s something cool about just like, yeah, I can take pride in having written something that a lot of people are really glad to have read and also disagree.
Wolf: Right. Yeah.
Well, I could ask about Richard—but that might be a bit of a rabbit hole. He’s a great guy. I love him. But, that’s like software mechanisms—
Engineering good epistemics (27:19)
Oliver: That’s it
Wolf: on the platform itself. Was there anything you did socially—around the norms of—agreement, disagreement, kind of epistemic progress?
Oliver: So one of the big things we did that had some software component, but was largely a social component—is just like, as I said earlier, the whole internet kind of especially over that time period, was trending towards extreme ephemerality. And that’s also just a big attractor. One of the standard things that happens as you have niche communities is they became very oriented around a marginal piece of work. And I think Chris Olah all the way back in 2015 around that time, was writing about distillation debt and research debt. Where just like you have these ideas that everyone, in most research communities and scientific communities, people are always encouraged to push forward the frontier of knowledge.
But nobody’s encouraged to reconsolidate your knowledge. Like come up with the correct name for something. Long-term incentives are often quite off. And so in order to push back that push towards ephemerality, we kind of instituted a annual review process. And one of the things that was like, and this is still confusing to this day, is at the end of every year, we have an annual review where we review all posts that are at least one year old.
So at the end of 2025, you review all the posts from 2024. And so it’s kind of this experience of just like we look at posts and we look at the ideas that they contributed over a longer time horizon. We don’t just look did they—right now chase a fad? Did they say something that was very fun in the community? Did they successfully comment on individual news like events that were happening?
But just like, did they contribute to something lasting in a substantial way? To the frontier consolidation.
Wolf: And you go back and engage with what you said. It’s like, oh, maybe we forgot we had that conversation. Actually, it was really important and it connects to something we’re doing now.
Oliver: That’s right.
Wolf: Or that guy did that.
Oliver: Or being like, “Oh, man, everyone was super excited about that and that did not age well.”
Wolf: It just totally didn’t pan out.
Oliver: That’s right.
Wolf: Yeah. Someone made a bad prediction. Someone made a great prediction. Yeah, that’s a really interesting—
Oliver: That’s right
Wolf: aspect.
Oliver: And then kind of in that context, there was just like a huge amount of social hustling of like, we got people to write reviews of just like how well did this work age? And just like tons of people, we gave out big prizes like every year. We gave like 20, $30,000 to just like across the reviewers to as prizes of who wrote the best reviews. And then we had a vote and we thought a lot about a voting system and we dug deep through the mechanism design literature and we built a quadratic voting system where people have points—and they have to assign those points across all nominated posts that year. And you can vote on a post, but the more you want to up vote that post, the more you have to pay.
So it’s like if you want to give it plus one, it costs one. If you want to give it plus two, it costs three. If you want to give it plus three, that costs six. Just like every marginal point costs linearly more. And it proves quadratic voting.
And that’s just like we thought a lot about a mechanism design and that’s like then we have a way of aggregating, which brings out a popular post, like the well aged post in a year. And then we put a huge amount of effort into making literal physical books and—
Wolf: Yeah, I remember that
Oliver: designing those books. And I think that was really when, I definitely remember when we first put out those books. That just like to so many people, it’s like, oh, now it feels real. Like now I believe that we can make real intellectual progress on the site, which sounds so silly. But I do think there’s something—
Wolf: No, it’s something real. I mean, this is why we did it with Palladium, right? Put it into this physical format.
Oliver: That’s right.
Wolf: It’s this beautiful artifact, and it’s like, okay, now there’s something to show for it, right?
Oliver: That’s right.
Wolf: Otherwise, it’s just a blog.
Oliver: That’s right.
Wolf: And actually, I think you guys used the same printer.
Oliver: The same printers. That’s right, yeah. Not the same printers, but the same design team for the second year post-mortem.
Wolf: Yeah, Asimov.
Oliver: Asimov, yeah.
Wolf: Yeah. I think you did a great job on that. Very interesting. I remember this was the early generative AI—
Oliver: That’s right. Yeah
Wolf: period, and doing these generative—
Oliver: That’s right
Wolf: sort of rehashes of these maps that you had.
Oliver: Definitely. Like style transfers where we—combined the slight Renaissance watercolor aesthetic with the scientific maps that we were using and—still very proud of the design of those year’s posts.
Wolf: Yeah. I thought that turned out really well.
Funding LessWrong; the funding landscape for unusual projects (31:23)
Wolf: You’ve mentioned kind of this commitment to sort of do it even from a basement.
Oliver: That’s right.
Wolf: But then you’re also mentioning handing out prizes. So at some point in here, we have this all-important matter of funding. And so now I understand that, at some point in the rationalist community, certain members of the community became very successful. But why don’t you tell the story about how this project ended up actually sort of paying back in financial terms or getting a budget to be able to do this work?
Oliver: Yeah. I think early on, I think the first grant that we got was—I worked at the Centre for Effective Altruism between 2014 and 2016. Kind of started the EA Global Conference after Leverage had started EA Summit. And then, they started a new project called EA Funds, and we were actually one of the first people who got a grant from that.
Wolf: Okay, cool.
Oliver: So kind of a direct—there were some people who I had a long-term professional relationship with who are now in a position to just—they attracted a bunch of retail donors who just were excited about—I think that was from the Long-Term Future Fund already. I’m not quite sure. It was a fund that was dedicated to figuring out how to improve the future, and they gave us a grant. And I remember being—I think they gave us an $80,000 grant—for, I think there was a whole year of work for me and, I think Ben Pace was already working with me back in the day. And there were so many people who left comments being like, “How can you spend $80,000 on a website?”
Wolf: I know. I’m thinking now, like, “Only $80,000?”
Oliver: That’s right.
Wolf: Two guys working on this full time.
Oliver: That’s right. And it’s just like every time, it’s like, “Ah, yeah, man, how do you spend $80,000 on a website? Ask Mark Zuckerberg.” This is a lot of stuff to do on a website, really.
Building the best modern internet online forum software, turns out, costs more than $80,000—especially if you want to be anywhere close to industry salaries, which we did not. I paid myself $40,000 that year, if anything at all. I don’t remember.
Wolf: Yeah. And that’s actually always been one of the strengths of the rationalist and EA communities—is that willingness to eat dirt for a while—
Oliver: That’s right
Wolf: for the cause, right? I think so many other movements and tendencies in society are very held back by that unwillingness to make personal sacrifice. And rationality and effective altruism really did put that at the center of the ethos, and that’s very commendable.
Oliver: And I feel like, I think some of it is also just kind of the Silicon Valley startup spirit.
Wolf: Yeah, of course. There’s the startup thing. That’s part of why the startup—
Oliver: That’s right
Wolf: community is successful, right?
Oliver: That’s right.
Wolf: It’s because they actually are willing to do that.
Oliver: In the first two years, you’re just like, “Yep, you’re going to live off of ramen. That’s life.”
Wolf: Yeah. Funny, it’s not like that anymore, but
Oliver: Yeah. Now, definitely, you can get valuations really quickly and—but I still know some startup people who are expecting to not really make anything in the first two years.
Wolf: Yeah. But I guess, back to the funding question—how critical do you think that was to being able to do it?
Oliver: I do think that early grant was just really valuable and really important.
I would definitely now pay. If I could send money back in time, I would easily incinerate $500,000 to make that $80,000 happen. Maybe a million dollars. Yeah. That was a huge multiplier and extremely important for anything in my life at the time.
Wolf: Right. And I guess I’m interested partially just from the dynamics of how these things get going overall. If you’re going to build new communities, build new infrastructure, there really has to be that element of—money getting injected at some point because it can be very hard before it’s really taken off to sort of get that, I don’t know, market return or whatever. I think nowadays probably you guys have more funding.
Oliver: That’s right.
Wolf: But in those early days, it’s just this—it’s like credit in the economy, right? Credit is this incredibly overpowered thing if you can get access to it—
Oliver: That’s right
Wolf: early in the development. But it seems like one of the things that kind of goes on still in a matter of American culture and intellectual culture outside of directly for-profit things is there’s still this sense in which there isn’t really money available for new things. It’s like once it’s built up and become part of an existing patronage network or something, it’s like, okay, people will fund it. But those early days are very difficult, and I just would love to hear more thoughts from you on that difficulty and the generality of that problem.
Oliver: Definitely, I do think—and that’s kind of a development that I’ve just seen play out over the following decade in effective altruism. The thing that I’ve seen in society tends to be that somebody—and science funding, I think, is an extremely strong example of this. You look at the history of early science funding and it’s a lot of enormous amount of judgment-driven people who get to come in, and they’re like, “I want to make a bunch of research happen.” And I think Patrick Collison recently released... Oh, no.
Maybe there was a big recent letter by a bunch of people in the Trump administration about reforming industry and science. And they had a bunch of quotes about historical approval processes for scientific grants. And they would be like, you would have a bunch of people who are just empowered to give out large amounts of scientific funding, and they would review a submission by someone for funding on Monday. They would come to an agreement on Tuesday. By Wednesday, the grant agreements were fully drafted.
Wolf: That’s not how it works anymore.
Oliver: By Thursday, the money is being sent out. And you had this early, high discretion, in some sense high trust, relying a lot on personal relationships funding—that then enabled these early projects to just get money quickly out the door.
Wolf: Yeah, personal relationships, personal judgment. Even the story of that $80,000 that you got—
Oliver: That’s it
Wolf: that’s not because the people with the money saw the value in LessWrong, it’s because the people who they trusted to empower to make that decisions knew from personal experience—
Oliver: That’s it
Wolf: the importance of this project.
Oliver: That’s it. And so that early funding really relied on that early judgment-driven thing. And then a thing that I’ve just seen is, I’ve even seen just over the last decade, tons of grant makers do that for a while, and then due to a mixture of adversarial pressures of people will then try to get into your social graces, like—networking. They try to just sycophant, start surrounding you if you give a lot of resources away with high amounts of discretion. There’s just risk aversion that sets in of just you have a grant maker, the grant maker makes one controversial grant.
Somewhere in the process, somebody just gets really freaked out, and then the organization and the grant maker accumulates scar tissue of just like, “We have to have processes. We need to have more review. We need to be more conservative how we do things.” “We want to make fewer, bigger grants to organizations that can give us all the information up ahead.” Grant timelines go from three days to one week to a month, six months, a year plus.
And I’ve just seen all of those things happen over time. And so just, I do think there is an enormous amount of value that people get to do when they tap into a network for the first time and distribute a bunch of money and distribute a bunch of things.
Wolf: Does that almost speak to the value of the MacKenzie Bezos approach of just dropping a ton of money at once?
Oliver: I mean, like—
Wolf: Obviously, her priorities—
Oliver: That’s right
Wolf: are not the same, but—
Oliver: The key thing that doesn’t work about that at all is I don’t think she gave money to things she trusted. I think she maybe had—at most, $50 million of things to give that she had any reason to trust. And after that it was just blatantly obvious—
Wolf: The rest is just fashionable stuff. Yeah
Oliver: she would just be fascinated by the domain, just like Google, which charity was active in that domain, and then just dump—$20 million on an organization that just was the best at marketing and SEO, and you’re like, now you’re in very bad adverse selection territory.
Wolf: Yeah. Yeah, so this matter of funding, obviously you guys managed to succeed at that. The Rationalist community is very successful in many ways, having obviously been early and right on AI, on even COVID—a number of things. And that obviously enabled the next big project—which is Lighthaven.
Building Lighthaven (40:10)
Oliver: That’s right.
Wolf: Where I remember back in Berkeley, there was the Rose Garden Inn—
Oliver: That’s right
Wolf: and then somehow the Rose Garden Inn became Lighthaven. You guys basically purchased it with help from your friends—who had by this point become quite wealthy. Why don’t you give us maybe more of the story of how that happened. How did that get conceived?
The idea of a physical campus.
Oliver: That’s right.
Wolf: It’s almost this crazy idea, or it might have seemed like a crazy idea.
Oliver: It is crazy.
Wolf: It’s like taking an internet forum, and then we’re going to give it a physical campus.
Oliver: And also a pile of software engineers who are like you. Why are you suddenly getting into Berkeley real estate, developing $20 million properties?
Wolf: Right. Yeah, so how did that happen?
Oliver: Yeah. I think the key thing is, between 2016, 2017, and that was 2021, we had developed LessWrong more and more. We had started also developing the AI Alignment Forum, which is kind of a sub-forum of LessWrong that was focused on AI.
Wolf: Yeah, using the same software.
Oliver: That’s right. And then we just did user interviews, kind of in classical Silicon Valley style. We reached out to our top authors basically every week. We tried to talk to them. We tried to understand what we could do to make them more productive.
And just over time, at the beginning, they were like, “Well, look, man, I want to get more people online to read my stuff. I want to have better tools, like why the hell do you not support LaTeX in these correct ways? I want my comments on my stuff to be better. I want more long-term reward in terms of things like the annual review.” But then over time, they shifted towards this other thing where they’re just like, “Well, what I need are collaborators. What I need are co-founders. I want to myself start an organization that has a research project that just requires me to meet other people.”
There were people who were like, well, at this point, the issues that people had and that seemed to be holding them back from producing really good intellectual output were stuff that just were now in a physical realm, around meeting other people, around having really good work environments. About being part of a culture that isn’t just about sitting in your room and writing comments to the internet—but wanting other people to just get high bandwidth feedback about all of this stuff. And so then roughly around the time when we noticed that, the pandemic happened. That, I think in some sense made all of that worse, but actually during the pandemic, we did a huge amount of infrastructure work to try to, in some sense, it kind of bridged the gap between us as a group of people that was facilitating a lot of online connection, which up until the pandemic was largely limited to low bandwidth connection. And then during the pandemic, all the options that people had for high bandwidth communication were gone.
And so they started seeking online high bandwidth connection. And so a lot of what we started doing in the pandemic is we started—there was this app called Gather Town, which was kind of a Pokemon clone, but for online meetings—where you would have a tiny 2D map, and you would have a tiny character, and the character would just look like a Pokemon character, and you could make custom maps of where you would walk around on, and then you would have proximity chat. So you would just have a tiny camera, and it would be embedded in a tiny window in the bottom right corner, and you would have your mic on. And so you would have this virtual space that you could just walk around in as you’re stuck in your house and can’t leave, and just have proximity chat with other people.
Wolf: I never heard of this.
Oliver: And so we started organizing these parties, and debates, and events—in our visual, tiny Gather Town world. Me and Ray started going really wild, and I think Ray did us a lot of credit and just—we built this really cool, amazing virtual world where just people got their own offices, and then they had their own offices that they could hang out with. And we built all kinds of Easter eggs and puzzles into the space. Now, actually, there’s kind of two senses. People would show up in our Gather Town world, and then they would go on a half-hour walk as they explore all the cool things that we built and the non-physical, digital space.
In that context, we just got a bunch of experience of what it was like to facilitate higher-context things in the infrastructure we built, and I also had some experience running EA Global and things like that.
Wolf: So learning in simulation, so to speak.
Oliver: And then at the end of the pandemic, there was actually a huge opportunity. Because before the pandemic, kind of the rationality and EA community and AI safety community, they had some spread. There were, of course, a few hubs, but even in places like Berkeley, people were spread all across Berkeley in different places, and people were spread all across SF and the Bay Area and things like that, and people had moved down. But actually, man, before the pandemic, the Berkeley in-person social scene for the rationality community was really very atrocious.
Wolf: Really?
Oliver: And just there were occasional house parties.
Wolf: I remember there was group houses and stuff.
Oliver: That’s right. There were group houses, and the group houses—
Wolf: But was that a thing?
Oliver: they had some real culture. But if you were someone who would show up, there was a single LessWrong meetup that would happen every week. And it was just the most evaporative cooled of—the thing that would happen is the people would show up and the people who were here for serious reasons, they wanted to contribute to the community, they wanted to make real progress on AI safety and stuff like that they would show up in a community, and then they would find private pockets, and then they would successfully no longer attend a meetup, and instead go to private pockets. They would sort of silo themselves off. But so it meant over a decade of cruft, just the only people who would still be attending the meetups were the people who hadn’t successfully found a community anywhere else in the Bay Area.
And that just produced actually just a huge barrier to entry where you’d show up and be like, “Man, all of these people are obnoxious.”
Wolf: Well, it’s adverse selection again.
Oliver: That’s right. You have this huge adverse selection on the central entry point in some of the things where people show up and try to find community.
Wolf: Yeah, that’s terrible.
Oliver: And at the time, I was running group houses. I was doing some amount of in-person community work, but it was just honestly very difficult to create something. And you had occasional parties, but everything, it just felt very hard to onboard and very difficult for people to feel like they had a community to build something on top of and things like that. And things were very spread out. And then actually, at the end of the pandemic, most people who had been living in the Bay Area or even other places in the world, they had moved out into places with usually a cheaper cost of living or just nicer to be in than—SF and Berkeley.
Wolf: Yeah, I also moved out of Berkeley at this time.
Oliver: That’s right. But that meant that as the pandemic was ending, there was the highest mobility of any time in history for our extended community. And so we were like, “Man, as the pandemic ends, we really need to be ready to just actually build a physical hub.”
Wolf: Where are we going to land?
Oliver: Where are we going to land? And then we can actually just pull everyone together. We can start dozens of new group houses. We can have an in-person co-working space and a place where everyone comes together and meets. And so we ran this event called Triple S Sanity and Survival Summit, just kind of the earliest time when we could get the people to accept being at a large in-person event after the pandemic.
And then we kind of brought everyone together, and they were like, “Oh, man, it’s so amazing to be surrounded by 70, 60 other people who actually I get to talk in person again and I really obviously share culture with.” And then at the end of that, we were like, “Okay, I think we’re going to build a campus. We’re going to do something—it has to be done after this that brings everyone together.” And so we started by building what we call the Lightcone Offices. We announced that kind of right after the retreat, after we got buy-in from everyone, where we just built an in-person co-working space in Berkeley.
We actually just rented one floor of a WeWork and eventually another one, where we rented out a whole floor and just invited the people that we respected and trusted. And they had a process by which you could get to join the space, and then that just pulled a lot of things together. Constellation, which now is still an active co-working space in Berkeley, did roughly the same around the same time. We helped them get started, had a bunch of collaboration with them early on, and then that’s kind of how we moved into the in-person space. First noticing that just there weren’t that many good marginal web development features that would make a big difference.
Noticing getting kind of our feet wet in the pandemic with facilitating these high-bandwidth spaces. Then building a co-working space, and then just very quickly after that being like, well, first of all, this doesn’t make any economic sense. Turns out office space in Berkeley, especially if you’re doing something that is more on a month-to-month basis or short-term rental, it’s just more expensive than paying interest on a mortgage. And so we started looking into the economics. And then I think co-working was just always kind of lame in a sense of just like, well, I want to have a space where I can run conferences, where I can sleep people, where people can become permanent research scholars, and things can feel a bit more like you have a campus that you can aggregate on.
And this crucial aspect of—
Wolf: These curated in-person spaces where serious work is happening, that becomes a social hub that isn’t adversely selected.
Oliver: That’s right.
Wolf: And so instead, as opposed to the meetup model where it’s just whoever is kind of no longer has anything going on, or they’re new, they concentrate in no’s. But if you have an actual campus where the interesting work is happening, then the entry point can be—
Oliver: That’s right
Wolf: the high-value stuff.
Oliver: That’s right. Yeah. And so we built the offices, and then we kind of started looking at properties of what we could do something cooler. And so we first of all, actually, we considered moving to SF. We looked at a bunch of properties in SF.
We also looked at a really, really cool property in the Presidio. And then we’d kind of a whole team retreat in the Presidio. We booked out a fraction of the place that we were considering moving things to. Or we had a hotel right next door, and then we build a fake website of what people could sign up for to get space at that campus. We had dozens of user interviews with people.
But it was just like, during those user interviews, we discovered that people didn’t want to move to SF. It’s just—it’s just too hard at the time.
Wolf: Berkeley is somehow still rooted, despite all—
Oliver: That’s right
Wolf: the pandemic mobility.
Oliver: That’s right. About 50% of people or something moved away. But the other 50% were just in Berkeley. And it just was too hard. People had partners.
People had relationships there. People had—just kind of grown roots. And it was just pretty difficult to move that. And my favorite crazy idea, which we explored for a bit at a time, was to just make a long-term plan to take over Treasure Island. It’s just so cool.
You’re just like, “I can’t quite get people to move to SF. What if we get them halfway?” And then you have that whole island that is completely underdeveloped. I do think there was a cool alternative future.
Wolf: Yeah. I mean, that’s maybe another 10 years down the line if you get that time.
Oliver: That’s right.
Wolf: Because existing in an already existing city is—
Oliver: That’s right
Wolf: like Berkeley, much easier than—
Oliver: That’s right
Wolf: having to deal with—
Oliver: Trying to develop an island
Wolf: with the particulars of a place that’s underdeveloped and does no one live there and—
Oliver: That’s right
Wolf: and so on.
Oliver: That’s indeed why we didn’t do it.
Wolf: Yeah. So it ended up making sense, get the Rose Garden Inn. I guess, this might be too much in the weeds, but how did that particular property happen?
Oliver: Yeah. So the Center for Applied Rationality, which we were still fiscally sponsored with at the time, and I had—volunteered with many years prior, they had actually run workshops, rationality workshops, in the Rose Garden Inn, like all the way back—
Wolf: Okay. They had rented it out
Oliver: in 2013, 2014. And so I had known the property. But then actually, I did the most boring, mind-numbing work. And so many great things are the result of extremely boring, mind-numbing work, where I did a complete grid search of the East Bay. I just went to Google Maps and I just started at a street, and I just went on Google Maps, just looked at every building.
Wolf: Yeah, that’s interesting.
Oliver: And that’s just left to right, because we knew really that we wanted a bunch of outdoor space. We wanted it to feel—like a physical campus, ideally with multiple structures, that just in Berkeley really seemed to me like it would make the vibe much, much better than what we had seen—so far. And then that meant that doing pre-screening on Google Maps became pretty feasible, and then we ended up with, I think, four candidates.
Wolf: Ah.
Oliver: One of which was the Rose Garden Inn because it just had this whole courtyard structure. But it was also there was a school of theology up on the hill in Berkeley that they were thinking of selling to a retirement home company to develop into a big retirement home. And so we considered—
Wolf: Was that that big empty campus, the Clark Kerr campus?
Oliver: No. Clark Kerr campus. That one, that’s so big.
Wolf: That’s too big.
Oliver: But no, that’s UC Berkeley. That stone thing. But that was just a specific school of theology. It’s just north of UC Berkeley.
Wolf: Okay. Yeah, I think I know the area. Yeah.
Oliver: Yeah. And then we considered trying buying that. We reached out to a lot of architects who had experience with spaces. And then all things considered, we came back to Rose Garden Inn. But to be clear, we were extremely startup-like, so it’s not like we then made an offer for them.
We were like, “Should we do something here?” And then we just one day were like, “Wait, every single room at this hotel is currently available.” So we just booked every room in the whole hotel. And then just ran a workshop with three days notice. That we did three days later.
It’s just completely hilarious. We were thinking of buying the space and we wanted to try it out, and we just—unilaterally, without, just booked out all the rooms. And then we were sitting there in a courtyard discussing whether we should buy this place. And then just the owner, who was the head of a hotel-owning family, Amish Patel, who owns like 15 hotels across the Bay Area—just shows up in the courtyard and is like, “What the fuck is going on? Why?”
“Who are you? What are you booking out my hotel? What’s going on?” And I was like, “Oh, yeah, we actually would love to buy this hotel.” And he was like, “What? I’m so confused. What’s going on?”
And we had talked to a real estate agent, and she was just like, “Let me handle everything.” She really—was like, “Make sure that you go through me or whatever.” And we were just like, “Fuck that.” Just booked out his whole hotel, met a man in his courtyard. He was very confused.
And then, we were just like, “No, man. I think we would love to potentially buy this place. I think we don’t know yet whether we want to buy this place.” And then we negotiated that we would rent out the place. And then not only did we rented it out for three months, and not only did we successfully get him to rent it out for us for three months—we got him to agree that we could start doing renovations in his hotel so we could get evidence about how difficult it would be to renovate his place—in the next three months.
Which I think was obviously a good deal for him because we were starting to renovate his hotel, but it was also just such a crazy thing to be like—
Wolf: Yeah. It’s like, “Well, what are these guys going to do?”
Oliver: these random 25-year-olds suddenly show up in my courtyard, tell me they want to buy my hotel, and then start bringing in construction workers to revamp my floors and replace my bathrooms. So that was extremely hilarious. Yeah. And then we learned. During the three-month test period, we ran multiple workshops with other people just to test it out.
We ran a big Slate Star Codex meetup organizer retreat. We just renovated some spaces and just actually did the work of getting contractors in, seeing what it would be like. And then using that, kind of came up with a plan for what it would be like to buy the whole space, negotiated a price, and then found Jaan Tallinn ended up backing a large mortgage for us. He provided us with the cash to buy the property and do a good chunk of the renovations, and then he would have a secured claim on the property, and that seemed like a good idea. And then I think he was pretty interested in just having a more diversified portfolio because just a lot of his money ended up, was in a few AI companies and in a huge pile of cryptocurrency.
Wolf: Right. Yeah, get some real estate in there.
Oliver: That’s right.
Wolf: Yeah, it makes sense. And then all your friends can make it an intellectual center.
Oliver: That’s right.
Wolf: By now, I think, actually, how do the finances work out now?
Oliver: I think we are now breaking even this year. I think we’re probably going to be within $100,000 of breaking even. And that’s doing a good amount of subsidies of just events that we think we like and we are doing the best we can.
Wolf: Yeah. You’re subsidizing stuff and nonetheless still—
Oliver: That’s right
Wolf: still quite close to breaking even.
Oliver: It’s still breaking even. That’s right. Yeah. And I’m just like, I think now it is solidly in the self-sustaining category.
Wolf: Nice. Yeah. So, that’s a pretty epic story, from kind of early LessWrong, kind of this almost failure of LessWrong, to deciding to save it, revamping the online presence, building a new website, and then through the pandemic, kind of realizing that you could and should build this—
Oliver: Space
Wolf: this in-person space. And now there’s Lighthaven. Lighthaven is—the new name of the Rose Garden Inn.
Oliver: That’s right.
Wolf: And hosts all these great conferences, and I’ve been to events there. It’s a really nice place.
Oliver: And I think it is, it has become even just amongst Silicon Valley, one of the places in the Bay Area. It’s just really—it’s just succeeded at being like—
Wolf: Yeah. It’s one of the centers.
Yeah. What’s that like? That’s less than 10 years of action there and quite a growth trajectory.
Are there other parts that you have planned or other things that you’ve already done that I’m not mentioning?
Oliver: That’s right.
Wolf: Like—
Reforming grantmaking with Lightcone Commons (57:43)
Oliver: The big thing is we just—
Wolf: Where does the story go from here?
Oliver: I think we’ve just launched our, which kind of almost brings your whole question about funding full circle. Because we just launched our, in some sense, I expect it to become the third big project of our legacy—which is Lightcone Commons, which is a funding coordination platform where we’ve been working with the Survival & Flourishing Fund for a decade, where we’ve been working with distributing a lot of Jaan Tallinn’s philanthropy. And in the context of that, we’ve been doing—a huge amount of development of algorithms and tools and websites and stuff like that to allow just people who want to use their money in the most effective ways towards their goals, to find people that they can trust to find good opportunities for them, tap into their personal networks, just distribute really serious amounts of money to make the world better according to their own values.
Wolf: Yeah. This is the S-process—
Oliver: That’s right
Wolf: system that Jaan uses to de-conflict. As I understand it, the intention is basically that a bunch of trusted allocators can basically allocate money unilaterally without too much paperwork. But the process kind of de-conflicts it, so it’s like if everyone thinks that this one project should definitely get $200,000 but not much more than that, the process makes sure that that happens. So even though they’re giving the money unilaterally, it doesn’t stack up or something.
Oliver: Doesn’t.
Wolf: Yeah. But you’re kind of taking that sort of idea, just like a software platform for allocating money efficiently and effectively and opening that up to a wider—
Oliver: Approximately
Wolf: wider community.
Oliver: Yeah. First of all, I do want to be like, “Look, man, I feel annoyed at the Survival & Flourishing Fund, calling this thing the S-process.” Such a complicated mathematical term, and I think Scott Alexander also in a recent blog post was like, “And they aggregate everyone’s preferences using a complicated mathematical process.” Like the simplest goddamn algorithm.
Wolf: No, but it actually comes down to a bunch of unilateral allocators.
Oliver: It is. The algorithm is the simplest one ever. It’s just like you have two people. They want to fund a project. Let’s say you have two people.
They both want to give away a million dollars. They both want to fund project a for $500,000. Then they want to fund project B and C, and they don’t care. They don’t have the same preferences. What it just means is just that if two people want to fund a project, they split the bill proportional to how much money they’re giving away.
So just like if you have two funders who are giving away the same money, they split the bill 50/50 on anything they both want to fund. If you have two funders where one funder is two times bigger than the other one, they split the bill two to one. That’s it. That’s the whole algorithm. It’s nothing else.
Wolf: And there’s some math to make it work in the edge cases, but basically that’s it.
Oliver: There really aren’t any edge cases. Basically, you have two philanthropists who want to give away a million dollars. What would they give away money to if nobody else was giving money to? Then you just split the bill. You refund them the money because the other party kind of paid them—paid half of it.
Then you just do it again. That’s it. You do that until it converges.
Wolf: Yeah. And a lot of the time, problems that seem like they ought to be solved by some fancy pants system—
Oliver: That’s right
Wolf: actually, the best approach is just some dead simple—
Oliver: That’s right
Wolf: like—
Oliver: And it’s just like a very—
Wolf: simple thing could possibly work
Oliver: dead simple algorithm of just like—if two people want to fund something, they fund it proportional to how much money they’re giving away. This is not even necessarily just. Is it fair? I don’t know whether it’s just and fair, but the answer is that it’s simple—
Wolf: But it’s simple and it works
Oliver: and easy to understand, and everyone can agree that that’s kind of a reasonable-ish way of distributing money. That’s important for getting these coordination problems solved. It’s not trying to be like, “Oh, we calculated all the ideal fairness that we would distribute, that you should pay of your fair share for this.” Your fair share of a project is just how much money your fraction of how much you want to fund it multiplied by how big you are, this rent. But yeah, so we brought that algorithm, and now we just started kind of a big allowing other funders to try to join that platform very easily, recruiting evaluators who just are looking at a huge amount of different applications and organizations and projects and trying to figure out how to distribute that money.
And I think it’s already been—we have definitely received an overwhelming amount of interest in joining. I think we’re going to be giving away $35 million in our first round. And that’s intended to be a quarterly round.
Wolf: Quarterly.
Oliver: So right now, by default, we would be at $140 million distributed if that repeats every round. And I think by default, we’re probably going to be at more than that. I think by default, we’re probably—
Wolf: You think it’ll grow
Oliver: going to be like grow. And so I think if we don’t mess up and we do a good job at actually really implementing the lessons that we’ve learned in this branch of our work over the last decade, I think we will be distributing $300 million or $400 million a year over 2027.
Wolf: Amazing. So you mentioned earlier, though, there’s these problems with persistently—kind of giving out money—
Oliver: That’s right
Wolf: where everybody comes out of the woodwork and starts trying to squeeze themselves into the shape that they think is going to get—the money from your project. Do you have thoughts about how you’re going to manage that kind of adverse selection problem or the spoofing and all that?
Oliver: Yeah. One of the key things is to kind of really allow funders to have an easy time bringing on new evaluators to empower. So just like you have these people who have their personal networks, and you have these people who know what good projects are in a domain, and the whole process is oriented around a funder empowering them very strongly to be like, “If it was just you, I’m asking for your own personal take here. How would you distribute money? How would you just give away your first million dollars, your second million dollars—up until $20 million?”
And then that’s trying to really tap into a bunch of people’s networks. And so we’re going kind of pretty broad on just we’re going to have 30. Over time, maybe up. I think at 100, things start breaking, and we have to do some clever things. But we’re going to have 30, 40 evaluators on this platform who are just all going to be tapping into networks, being empowered to make these pretty substantial grants.
And just like we’re trying to make it so that instead of ending up in a world where just like you have these big foundations, and these big foundations will often then try to build these centralized, anti-unilateral systems where just like—
Wolf: Yeah. It’s like the foundation
Oliver: the foundation needs to approve the grant, and it goes through 15 different boards and whatever. It’s like everything of what we designed is trying to counteract that. We try to make it so that just you have a single evaluator, and those evaluators, most importantly, we are actively trying to recruit people who are going to be doing this part-time because we’re trying to maintain the people—
Wolf: Yeah. You don’t want a professional—
Oliver: That’s right
Wolf: grant maker.
Oliver: I think a professional grant maker is a mistake career. I think it is the grant making space, especially in the domain that I tend to work in, which is trying to figure out how to make consequences of AI be good and trying to figure out how to raise the sanity waterline. Those are very low feedback domains. And so if you’re a grant maker, you’re another layer removed from the people who are even trying to solve these problems.
Wolf: Right. I guess what you actually want to do is grab people from the domain—
Oliver: That’s right
Wolf: who are trusted. It’s like, “Oh, that guy, I know he’s got the good network there. I know he’s doing a lot of really good thinking about—
Oliver: That’s right
Wolf: what needs to be done.”
Oliver: And it’s like I know he’s well-respected.
Wolf: Let’s let him allocate 5 million, right?
Oliver: That’s right. I know he’s well-respected in the field. I can read his output, and he’s spending most of his time trying to make progress in the field and producing things that I can evaluate—and see how those things pay off. And then you can easily bring in those people and empower them to make a good number of grants. But trying to do so in a context where they’re also being given enough guidance by people who have time to investigate, is this organization for real?
Are these people just lying to you? There’s definitely a skill of being a grant maker that you need to learn, and that’s just the other side of this coin of just everyone learns how to not get taken for a ride in their first year in grant making. You show up, and it’s just like you understand what does it mean for a reference to be good, and what does it mean for a reference to be bad? The reference will, unless you’re in a set of very narrow circumstances, never tell you that you shouldn’t give someone money. The question is, how do you figure out how to elicit the correct information from them?
How do you just figure out when people are lying to you? How do you figure out how they exaggerate and try to pitch you on things? And so there’s kind of a middle ground, and we’re really trying to make it so that you can get these people with a strong track record, and you put them in a context where both we provide support, and we provide them with enough other people who have done enough grant making over time that they just don’t make these really dumb mistakes. And I think that’s a system that has worked, for example, reasonably well in things like DARPA. When I’m thinking about successful frontier science funding, I do think the DARPA special projects model of just you recruit someone highly respected in a field, you give them a four-year tenure, you give them a large budget, you give them large amounts of discretion about how to distribute that funding.
It’s just quite good, and that’s a decent amount of inspiration for how I would be going about—trying to do all of that.
Wolf: And what about the sort of third branch of the problem? So there’s—sort of this three-way market problem, right? There’s the funders, there’s the allocators, and then, of course, there’s the projects.
Oliver: That’s it.
Wolf: How do they end up in the system? Are they recruited by the allocators, or are they putting themselves into some black hole of grant process, like speculative kind of—
Oliver: That’s right
Wolf: grant application?
Oliver: One of the key things—
Wolf: How does it work?
Oliver: One of the key, and just we just have by far the simplest application process of any place you can apply for funding, I think, in history. You sign up. There’s an application page. It’s just like, “What’s your name? What country do you live in? What kind of organization are you?”
It’s like, “Are you a 501(c)(3) or an individual—or a for-profit?” Those are the only three required questions. Then what you get is you just, here is a Google Doc. Basically, we have our own Google Docs-like platform. And just like you can live edit this anytime.
We don’t care. Here are three suggested questions that you can answer that we have found to be helpful. But just tell us why your project is worth funding in this Google Doc.
Wolf: Yeah, give us the pitch.
Oliver: And then we use AI to extract all the standardized data that we need from the data you have.
Wolf: Ah, cool.
Oliver: So we then just have AI systems that read, figure out, okay, how much money are you asking for in different scenarios? Who is involved with this project and what’s their rough track record? All of this stuff that I think you end up with these 15 required questions that you have to fill out, and you have to figure out how to put your pitch into the right shape to fit into the right questions.
Wolf: Yeah. As a grant-seeking organization, I’ve had that position occasionally in the past. It was always one of the most annoying things about grant applications is just, okay, now I have to spend a week or two understanding—
Oliver: That’s right
Wolf: your particular ontology of what you’re asking for and fit our project into it. And that’s just ton of work.
Oliver: That’s right.
Wolf: And often for very uncertain payoff.
Oliver: And another key thing of the whole system is just applications are just fully optional in the system. So any evaluator in the system can just be like, “And then I recommend that you give money to these guys.” They can just specify the three required questions.
Wolf: Yeah. We know who these people are. Yeah.
Oliver: That’s right. And if maybe it’s an established charity, or maybe an established for-profit or anything, they can just add that unilaterally. And then we’ll show up at your door and be like, “Hello, we have $300,000 for you,” and you can reject that if you want.
Wolf: Right. Amazing.
Oliver: But it just doesn’t make sense to have these standardized processes. What’s the point? I think a lot of the point of it tends to be this organizational scar tissue of something went wrong in the past.
Wolf: Yeah. No.
Oliver: And now the organization willingness to ask about is one thing, because at one time when we tried to give a grant to someone in India, it was a giant pain. So now we need to ask everyone to do that. And then at a time, our lawyers complained about giving someone a scholarship and then figuring out how to do that in a tax-deductible way. And so now we need to ask everyone whether they’re doing anything that is scholarship. And then another time, one of our donors got really angry about doing anything that had anything to do with high school students.
So now we need to ask every applicant whether we’re going to do anything with high school students—or anything who are minors. And it’s like, look man, these questions do not matter for most people. They’re not important for the evaluators. They’re just this organizational scar tissue that just keeps growing—as a result of these risk-averse grant-making organizations. And we’re just trying to remove all the potential ways where the risk aversion can seep in.
Wolf: Right. And so you’re trying to close the loop on the whole problem of funding to enable, hopefully, more good things to come out of the community you’ve been building, by applying a lot of those same rigor of epistemic norms and institutional design norms and mechanism design norms to create a new kind of fundraising process.
Oliver: That’s right.
Wolf: Interesting. Very cool.
Oliver: And another big thing that’s extremely, I think that’s been among the more controversial things that we’ve done is just every evaluator just gets 2% of the money that is distributed through them. Just a strict fee. Just like if you successfully distribute $10 million on behalf of any of the funders, you walk away with $200,000.
Wolf: Right. Makes sense, yeah.
Oliver: And grant evaluation and figuring out how to give away substantial amounts of money in a cost-effective way is just a really difficult job, and it’s been a domain in which just there are approximately zero performance-based incentives. It’s just like it’s the nonprofit sector. Everyone gets paid nothing. And maybe if you’re the executive, the CEO of Coefficient Giving or something, you can get a medium-higher salary. But you’re never going to be in a position where you have obviously demonstrated good enough judgment that you can walk home with some serious money.
Maybe enough to just actually hire your own staff, make your own decisions, fund your own projects. And so just that strict fee model. We’ll see how it plays out. I think that’s one of the diciest incentive settings things that we’re doing. But I just think that one of the worst parts of all of philanthropy and all of the nonprofit sector is cost plus.
Just everything being based on you run your foundation and as someone who’s interested in building a grant maker, your incentive is always to hire more staff, bring on more people, figure out how to spend more money. Because the only way in which you can expand your influence over the world is to—
Wolf: Be bigger
Oliver: spend more. Because nobody will ever give you an endowment that then the executives of the company can spend unilaterally, and that’s just a terrible situation.
Wolf: Right. Hmm.
So that’s the next big—the next step of your master plan, so to speak. I guess one question I’m very curious about then is having done all this—you can answer this two ways. What would you do differently if you were starting from scratch today—with a new intellectual community project or something like that? Or alternatively, what advice would you give to someone who’s trying to start up a new intellectual community or something productive—along these lines, maybe with different focus, different emphases? Would love to hear your thoughts on just—how do we do more of this?
Because I think we need more of it.
Oliver: Yeah.
Leadership, courage, and unilateralism (1:13:09)
Oliver: I think the biggest mistake that I made, especially in the early years of the whole work on LessWrong and Lighthaven, was just I didn’t have the courage to be a leader, and I think this is extremely common in the social organizing and nonprofit space in particular. Where I think first of all, it just first showed itself in just how I ran my organization originally, where I didn’t think of it as my organization. I think it was correct to bring people on board and be like, “You’re co-founders of this.” But I was just hesitant to call myself, to be CEO. I was hesitant to make decisions on behalf of the organizations.
I approximately, in the early years, tried to do everything by consensus. Making decisions as an organization, even if you’re trying to bring—or maybe especially if you’re trying to bring very mission-aligned and passionate people onto it, consensus is just a really, really terrible way to lead an organization. And so even internally in LessWrong, the early experience was just that over the first two to three years, everyone just became very tired of doing anything because trying to get the organization to do anything felt like you needed to get everyone else on board. Then when you needed to get everyone else on board, people felt like every project proposal, every direction for the organization to go would be an opportunity for horse trading. And that’s what you see a lot in consensus-based systems.
It’s just suddenly every decision becomes an opportunity for someone to be a bit hesitant to agree with the decision in order to potentially get something else that they also—care about getting done.
Wolf: And it becomes this n squared negotiation problem.
Oliver: That’s right. It becomes this whole n squared negotiation problem. But even just beyond that, just this feeling of just everything you’re trying to do just becomes this thing that you need to get buy-in for. And so just a few years into the organization, we just had a break and I was like, “Okay, I actually just think we should run this organization hierarchically.” I’m just like, “They’re the CEO. I make the decisions. It is my job to take into account everyone’s preferences.”
And we kind of discussed, nonprofits unfortunately don’t have equity, but kind of spiritual equity. And be like, “I think everyone should get roughly equal equity,” but I’m just like, “I’m going to be taking the reins.” And I’m just like, “I’m going to be making decisions on behalf of us, and I’m going to be doing that unilaterally without checking in with everyone.” And that was just essential. That was the only time when the organization started working and getting to do anything on purpose.
But I think that actually extended much further than just the organization. At the same time, kind of when I look at the failures of most intellectual communities, the success of, I think both Lighthaven and in some sense LessWrong itself, and also things like the Lightcone Offices, is often downstream of just you notice that you’re in a domain where you would like to create a certain culture, create a certain community, and just everyone is hesitant to stick out as advocating or doing something that it can clearly be blamed for. And I think that was ultimately a lot of the motivation for why I didn’t want to act like I was the CEO of my organization because if I do things that other people disagree with, then I can be blamed for it. And there’s like—
Wolf: Right. But there’s also this, who are you to put yourself in charge—
Oliver: Exactly. Who are you to—
Wolf: fear, right?
Oliver: put yourself in charge? And I think this is particularly true in just communities that are oriented around scientific inquiry and intellectual development, which is like the rationality community is a giant pile of contrarians. It’s very hard to be meaningfully a leader of the rationality community, but just having the courage to do it anyways. And then the biggest thing that I think that took me a really long time, and that was kind of the next step, was just that you then therefore need to be out there, and you need to make judgments and decisions all the time where they are visible by other people. And you need to be able to take the shit that other people will give you, and you really want to maintain—you want to start—you want to be in conflict with other people all the time, and you want to show to other people that you can successfully navigate that conflict in a fair and just way.
And so just the correct thing to do if you’re trying to build an intellectual community is to be out there, put yourself in a position of authority over anything that you can govern, which in my case, because I kind of due to the vote, I was able to govern and had the authority to govern LessWrong. But then just govern actively, govern with your own judgment, record your decisions, make your decisions transparent to other people. But just make them a lot, and make them fully on your own conscience of just, “I’m doing this because I think this is a good idea.” This will over time get you to build trust, and then that gives you the leeway to actually start coordinating more and more ambitious projects with more and more different people. And then you just see this with successful CEOs in lots of other domains.
I have my own deep disagreements with Sam Altman, but it’s very clear that he has a long history of successfully pulling the deals together. And just that history of pulling the deals together just then gives you more momentum to do more ambitious things, which then you can use to get—more momentum.
Wolf: Building that track record.
Oliver: And just when I started, I didn’t write many of my moderation decisions. I was kind of waffley and tried to make my moderation decisions hiding behind abstract procedures and things that didn’t kind of make myself a locus of judgment that could meaningfully be blamed. And just when I started doing that, I just noticed how just coordinating things in my ecosystem got better. And over time, I became more trusted and was given more leeway.
Wolf: Started doing what exactly? Just more directly explaining your—
Oliver: Both internally to my organization, but then also kind of on LessWrong, it’s like there is no—and I kind of had even a long essay that I wrote about this the other day. There is no committee that makes moderation decisions on LessWrong. And this is actually quite rare in online forums. Almost every online forum tends to be like, “Oh, it’s really important that if you ban someone, that gets looked at by five different other people because you don’t want a conflict of interest.” And you want to make sure that you give them due process and a lot of different things like this.
And just instead, LessWrong has maintained a very strict, like I am the head moderator of LessWrong. I will make decisions about who to ban, who to rate limit, who to ban from certain posts, who to tell that they will be banned if they take a certain action in the future again. And just that is all done directly accountably to me. And being comfortable doing that took me a really long time. And just we had people, and I mean, God, moderating online internet forums.
We just had people. There were many people I respected who tried working with LessWrong in the early days. Max Harms, someone who I deeply respect, he trialed with us for two weeks. It was just like, I doubt those were maybe among the most stressful two weeks of my life. He was just like, he felt constantly everything he was doing, he would just be about to be torn apart by—the LessWrong community.
And just having the courage to put yourself out there and stand up to the mob.
Wolf: Yeah, definitely a nest of contrarians.
Oliver: That’s right.
Wolf: It’s impressive to have pulled it together—and become a leader of it. We’ve been going for quite a while now. And this has been awesome.
Oliver: Thank you.
Wolf: I feel much more like I have an inside story on the LessWrong story, the Lighthaven story, and excited to see where this fundraising or this funding venture goes. And thank you so much for your thoughts on kind of how people—the problem of courage, basically, in leadership. I completely agree that that is an essential component. Would love to see more projects like this. So yeah, thanks so much, Oliver.
Oliver: Thanks so much.