Yeah, it’s big brain time. This week we’re reading ‘Understand’ from Ted Chiang’s 2002 collection Stories of Your Life and Others.
what is the ceiling on human intelligence? can we jooice it up? did Chiang inspire the whole AI doomer movement? would superintelligence beings have to annihilate each other instead of cooperating? Do we buy the orthogonality thesis?
Also: introducing David Deutsch’s ‘universal explainer’ theory of intelligence, which gives radically different answers to all of the above. Is the dumbest guy you know really capable of making novel advances in quantum physics? The answer may surprise you.
On abstractions and ‘chunking’: how important is working memory? Should we expect our high-level explanations to converge on a theory of everything? Would super-smart people really communicate in short series of grunts? Could they hack their own autonomic nervous systems or incept a linguistic killshot?
tl;dr: gestalt gestalt gestalt gestalt gestalt gestalt. gestalt gestalt? gestalt gestalt, gestalt.
intro and synopsis
Benny: Welcome to Do You Even Lit? I’m Ben with the boys Rich and Cam. Today we’re discussing Ted Chiang’s Understand from the collection Stories of Your Life and Others. We talk about the idea of the intelligence explosion, whether Ted Chiang called it, the continuum view of intelligence versus the view of universal explainers, intelligence and interoception, and gestalts.
Rich: I was going to reply to you on the group chat when you said, I’ve started losing weight now, I look kind of scrawny. It’s like, yeah, the key is to feel disgust about your body, no matter what state it’s in.
Cam: That’s what I’m going to tell my daughters. Make sure they’re nervous.
Benny: When people tell you don’t look good, you internalize that shit and you take it to the gym.
Cam: And you count those calories.
Rich: Anyway, to the topic at hand — have you guys read this story before?
Benny: Yeah.
Cam: Yeah, but like five days ago. And then I re-read it. I hadn’t read heaps of Chiang. I read more from Exhalation. And of course, the main story, Story of Your Life.
Benny: Have you not gone through his catalog?
Cam: Well, like half of it, maybe.
Rich: It feels psycho to read one of his stories and not read the others, if you have the book.
Cam: I love this title of his collection, though. Because it’s kind of a double entendre, right? Story of Your Life and Others. For ages, I just thought that was the name of the story. But then I realized, oh, it’s “and Others”. And other stories.
Rich: Oh, man, I never even got that.
Cam: And then the story, the story of your life. I.e. Arrival.
Rich: And then they dumbed it down by renaming it Arrival after the Hollywood film came out.
Cam: He must take ages to write his stories, eh? Because when was this published? Like 2000? And then he published Exhalation in 2020, roughly? Or slightly before, maybe?
Benny: Something like that, yeah.
Rich: Yeah, this is it, I think. There’s two collections. And he was even working for like Microsoft or some software company as a technical writer. This is his — not his day job, or maybe it is now, but it didn’t used to be. I mean, I assume he got a decent payout from Arrival, and maybe he’s self-sustaining now, or maybe he’s even retired or whatever. But he was in the software industry, which makes sense.
Benny: Yeah, I didn’t know that. He’s quite famous now, I would imagine. He’s probably self-sustaining with his writing. I mean, I hope he is. I hope he’s just trying to write more books, to be honest. That would be cool. That’d be good for the world.
Benny: Okay, so let me summarize this, and then we’ll get into it. So a man named Leon, who originally has some catastrophic accident where he’s trapped under ice, his brain shuts down, he experiences severe brain damage. And I think with current technology, he would have died, but we’re in a universe where they’ve recently developed some sort of neural generative technology where they can sort of regenerate the neurons that had died. And so they give him this treatment. It turns out it actually makes him smarter. It takes him a little while to realize this, but it increases his brain functionality from baseline. And then he loves those side effects. He agrees to be part of a study that will issue more of the treatment and then see what the effects are. So he takes more of it. He just gets even smarter. And then there’s basically a feedback loop. So he realizes at one point he’s becoming too smart and they might want to control him. So he basically vanishes off the grid, confuses the FBI and CIA as to his whereabouts, and travels around the country, basically making money off the stock market, manipulating whatever he needs to manipulate, giving himself the treatment and just recursively getting smarter and smarter.
Benny: And then near the end of the story, he realizes that there was one other person that this happened to. And he hadn’t found them earlier because they too had deleted all traces of themselves from the hospital records and from FBI’s database and stuff.
Cam: Over a certain IQ, you just get off the grid. Everybody does it. It’s just what you do.
Benny: Below a certain IQ, you’re off the grid. Above a certain IQ, you’re off the grid. This is the true midwit meme.
Benny: And so the final scene, which I think I’m going to wait to reveal until a little later, is basically these two super intelligent, for lack of a better word, humans discussing possible options with each other about how to move forward and navigate through the world.
Cam: So one quick thing just around how the vaccine — well, it’s not a vax, I suppose — how the treatment works is it doesn’t work on normal people. So they try it on healthy people and there’s no gains in intelligence. And it only seems to work on brain damaged or injured patients.
Can you jooice up human intelligence
Rich: So wouldn’t it work extra good on you, Cam?
Cam: And that’s why it works on this guy.
Rich: ‘Cause you were saying it wouldn’t work on you.
Cam: I know, that’s why I was surprised.
Benny: But then why do additional treatments with this medicine work, right?
Cam: Yeah, that felt like a potential plot hole. I wondered that. So yeah, he even realizes, because they do it on Alzheimer’s patients and maybe someone else, and nothing was as good as on him.
Benny: That seems like an odd constraint to place in the novel, actually, in hindsight. Why at the beginning say that we know it doesn’t work on normal people? Like what was the point of that plot detail?
Cam: Well wouldn’t everyone just be jacked then? Like, why he’s the only guy.
Rich: Yeah, to make him special.
Cam: I think they just needed a one-liner around — he needs to think, he needs to realize, well, why does it still work if I’m not brain damaged? And it must work in conjunction with having it in the system or something like that.
Rich: Yeah, but that plot hole is just sort of part of this broader possible plot hole, which is part of the broader question of whether you could plausibly just ratchet up intelligence beyond the current ceiling of human intelligence over and over again, right? And then the supposed mechanism is something like increasing synaptic density or cortical density. Like you break a bone, and when it repairs I think it’s technically stronger because it’s knitted back together with more tissue or something.
Cam: So what doesn’t kill you does actually make you stronger, right?
Rich: In certain cases, yeah.
Benny: Nietzsche the physiologist.
Rich: Yeah, so do you guys buy it, I guess, is — can you keep ratcheting up human, like what are the constraints on intelligence?
Cam: I thought it was a cool idea. It was certainly a cool idea for a story. And even kind of prescient in a sense of, like, that seems to be the prevailing view amongst AI researchers and AI doomers and accelerationists. And yeah, this is written like 1991. It’s just an interesting idea that if you up the quantity of something, what would happen? Like what would the threshold be? Would it have an impact on qualitative things, or would there just be more of it?
Benny: Everyone seems to have this view to some extent, right? Like this is what differentiates our brains from brains of monkeys and chimps, for instance, and dogs and rats lower down, etc.
Cam: I think brain size is — I don’t know, Deutsch wouldn’t like to use the word “explain” for this, but explains like 10% of variance of IQ, or correlates with that.
Rich: Yeah, if you think about the biological constraints on human intelligence, there are some obvious things. There’s brain size, which trades off against infant and maternal mortality, right? Because humans’ heads are so big that we’ve reached the efficient frontier. And maybe we can transcend it through caesareans and stuff like that — it’d be interesting to see if head size trends upwards. But that is a hard limit, insofar as a bigger brain, with a bunch of caveats, can have more potential to be a smarter brain. And then the other one is, the brain uses a lot of energy. I think it uses like 20% of your total, I can’t remember the number, but it uses a ton of glucose, and that would be a big constraint in the ancestral environment. But it’s not a constraint today. Like, calories are so cheap. You could feed the brain a shitload of energy and that would not be a problem at all. It would easily pay for itself.
Benny: So did Chiang call the worries about intelligence explosions coming from the AI community? I’m not sure exactly when Yudkowsky started writing about those sort of concerns, but I don’t think it was as early as the early 90s, and this story came out in ‘91. So does this basically predict the AI doomer concern that is now super common, where you can get these recursive feedback loops and someone can just make themselves smarter and smarter and then they’ll just go?
Cam: And is not sort of trustworthy as well. Will lie to controllers and then want to potentially destroy them.
Benny: Yeah, will deceive everyone around them, exactly.
Rich: I have a different take on this, Benny, which is that similar to what you said before — that this was more like the status quo belief, not so much the recursive self-improvement, superintelligence, but the idea that there are different levels of intelligence and that there’s no obvious upper bound. That you could have humans are smarter than chimps, are smarter than rats, are smarter than bugs, and there’ll be other things that are smarter than humans, going up to like God or an alien or something. I reckon that’s basically what everyone’s intuition is, even if they haven’t actually expressed it that way. And it definitely was what my intuition was when I first read this story several years ago. So yeah, I was like, oh, this is so cool, this is such a great portrayal of what the next level of intelligence above humans would look like.
Cam: Which I think it is cool. If you were trying to think, what would that look like in a sci-fi sense? And he just kind of leans on meta relationships of understanding and abstractions. I think it was a really cool way of doing it. An intelligent way.
Rich: With the paradox being that you have to be a certain level of smartness to portray a certain level of smartness, or to convincingly trick everyone.
Cam: And a pun paradox there. So let’s just spell out that paradox: that it’s hard to write about super intelligent beings because—
Benny: You’re trying to get in the mind of something. That’s what I love.
Cam: Yeah, if you actually demonstrate what it’s like to be them, you kind of have to be them. So you’re always bluffing to some extent. And it could just come across as woo. This didn’t — this came across as cool and elegant, and you kind of understand it. Yeah, I thought he did well.
Rich: Which suggests that Chiang is super smart, right? Like just to be able to hoodwink us like that.
Benny: He’s gotten at least one dose of this.
Benny: Yeah, as you were talking, I realized I was sort of conflating just improving intelligence with recursive self-improvement. So you’re right. There’s actually no recursive self-improvement in this story. And the recursive self-improvement is what at least modern AI doomers are worried about, where you get these systems that can be their own AI researchers and can therefore improve themselves and that’ll make them better. AI researchers don’t improve themselves ad infinitum.
Cam: Well, in some scenes he did. Like he learns how to—
Benny: But here he needed — no, but the whole point was that he needed to get more of the medicine to keep injecting himself. It was the medicine that was actually making him smarter, right?
Cam: Yeah, but he learns how to steal it, and then you could think he might be able to learn how to produce it as well.
Rich: Yeah, you could analogize it to — you know, the analog to the current AI trend would be if you had an AI that just invested in scaling itself. So it didn’t redesign its algorithm, it just put all its resources into capturing more data and more compute and then retraining itself. Which is kind of like the magic medicine equivalent, or just like more brain juice type equivalent, rather than some recursive pulling oneself up by one’s own bootstraps kind of thing. And in both cases the question is, what are the limits to the magic juice? It seems like it does actually help. We’re at an equilibrium, a level of ability, but there’s no reason to think that we’ve fully tapped it out. But then we’ll probably get into arguing about why there probably is some ceiling somewhere.
Cam: I’m surprised Chiang didn’t do the whole self-recursive thing, where he didn’t sort of say — the character Leon kind of realized just how to do it internally. Because he often will manipulate people just by kind of thinking, right? He kind of has this mind control because he can tell with people’s pheromones or body language and his own stuff. And that’s how he tries to influence others. I’m surprised that wasn’t just the next step, where he realized kind of how to produce the effects of hormone K internally.
Rich: Yeah, that could be another plot hole, actually. If he has full control over his biochemistry, then why couldn’t he just biochemically engineer more of it? I don’t know, maybe there’s some—
Cam: But maybe it’s mined from somewhere else or some shit.
Rich: Yeah. Exogenous chemical that he can’t synthesize, or—
Benny: I guess closer to the end he’s somewhat self-improving insofar as he’s improving language and then using that to realize there’s more and more problems with his perceptions of the world, correcting those perceptions, then developing a more and more precise language.
Cam: Imagine having the best language and just not being able to talk to anyone, because no one else can speak it.
How would super-smart people communicate?
Benny: Yeah. What do you guys actually think about the linguistic representation here? Like, do you think that’s an accurate portrayal of intelligence? It does capture something about intelligence — intelligent people, if you look at — I mean, humans are the only thing that uses any sort of language, right, in any sort of coherent way. But it does seem like as humans have gotten smarter we’ve refined our language, right?
Cam: Yeah, well, you imagine like cavemen grunting and stuff, it would be — but then maybe language is universal, so maybe it was always. Because linguists talk — when you compare languages to other languages and other cultures, people say like this tribal language only has like “one”, “two”, and “many”. But then a lot of linguists will say, well, actually every language is kind of as good as others, because you just add words to it and you can always add concepts to it. Which to me sounds like the current status quo, where some languages are more useful than others because you have the word for the internet and you don’t need to introduce it. But then other people would say, like, one reason people like English is because it’s influenced by Latin and German, and so you have multiple words for a single concept with slightly different shades of meaning. I think that’s why Borges liked English so much.
Benny: But also, isn’t there something here where language is actually getting less complex over time, right? Like McWhorter’s whole line is that language is actually becoming less complex. And so I’m not actually sure how to resolve that sort of paradox. Maybe just different time scales? Like if you’re looking at the history of the species as a whole, yeah, of course language has gotten exponentially more complex. But then in the last couple hundred years, that complexity is starting to decrease as we homogenize language across countries more and more. That does sort of cut against this idea that as we get smarter and more knowledgeable, we develop more and more abstract and precise languages.
Rich: If you map technical jargon, surely that would be a massive increase rather than decrease over time. And then if the homogenization is actually weeding out redundancies or weeding out conflicting syntax or something, which creates confusion, then that would still overall be a good thing. This is something I wanted to talk about in this story — the way that he gets smarter through the subsequent injections is through coming up with simpler and simpler patterns, or, you know, Ted Chiang’s favorite word is gestalt, right?
Benny: So often that you have to wonder if he’s doing it on purpose somehow. You think an editor would have pointed out, you seem to be using this word quite a bit, so there must have been a rhyme and a reason to it, I would imagine, right?
Cam: I can imagine someone reading it that doesn’t really know what it means and like—
Rich: Or doesn’t know how to pronounce it.
Cam: Doesn’t know how to pronounce it. And just trying to—
Benny: Yeah, what does it mean, Cam, just for the audience?
Cam: Well, my understanding was it’s kind of like the shape of something, right, that takes on a form — sometimes a form of components that build up to it. But I didn’t actually look up what it means. I just kind of thought that’s what — I might be totally wrong and embarrassed right now. What does it mean? What does gestalt mean?
Benny: I thought it tied into anti-reductionist type of ideas, like complexity science sort of stuff. Like, the — what’s the phrase that’s on this? The whole is—
Rich: Greater than the sum of its parts.
Cam: Yeah, that phrase often goes with it.
Benny: Greater than the sum of its parts.
Cam: Benny, your definition is what Google says. What a coincidence. “An organized whole that is perceived as more than the sum of—”
Benny: I must be right then. Google got it from me, I think. They emailed me last week. How should we define gestalt?
Cam: I think it’s “guest-shalt”, guys, by the way.
Benny: I think hanging over this conversation—
Rich: Wait, are you serious?
Cam: I think I’m right here. The first time in my life.
Rich: Why didn’t you just fucking Google it? Don’t leave us hanging.
Cam: I just Googled it now as I was checking Benny’s definition.
Rich: I’m going to Google it. I don’t believe you.
Cam: You can even throw in a bit of a German in there. Gestalt.
Benny: Gestalt. Pronunciation good. Sorry, what are you saying, Cam? You’re saying hard g or soft g — “gestalt”? Yeah, I think you’re right, that’s what the American pronunciation is.
Rich: Oh no, I’ve been owned by Cam.
Benny: Both American and British. Gotta shut down the podcast.
Cam: Maybe it is gibberish as well.
Rich: No, now you’re fucking with us.
Cam: Maybe it’s archetype. I wonder how many listeners we lost when I said archetype.
Benny: Maybe it’s German.
Rich: Remember that talk we had about how mispronouncing words on purpose is only cute when you’re five years old?
Cam: My partner thinks I mispronounce words, mis-say words on purpose — like get words mixed up because it happens so often — and she thinks I do it to invoke laughter from others.
Rich: To invoke laughter.
Cam: So that’s my hormone K.
Benny: That was a Leon sentence right there. Like, “I invoked laughter in these humans to distract them from the fact that I was emptying their bank accounts.”
Cam: Yeah, well, I mean, to go to what you were saying before — there’s this thing where people sometimes think that what’s smart is this complicated, obfuscatory kind of language, right? Which often is not the case. Good science writing would be making it simple. And I think there’s a Dolly Parton quote which I quite like, which is, “you don’t know how much it costs to look this cheap.” Like with good plain writing, it’s very easy to understand, but it’s hard to realize how difficult that was to do. But that said, like some stuff is super complicated and abstract and is very hard to understand and write about, and it will kind of sound like gobbledygook to others as well. So it’s not just — everything needs to be monosyllabic Germanic words to be good writing. Because some things are hard to talk about, and I don’t completely throw out all that postmodern stuff as I kind of did when I was an angsty teenager. I thought it was all rubbish. I think there are some sort of hard content that’s difficult to write about, and it’s almost implicit knowledge, and starts to sound a bit crazy or obfuscatory.
Cam: Anyway, so back to the — you were about to talk about gestalts, Rich.
‘chunking’ abstractions towards a theory of everything
Rich: Well, I was just going to say, it’s similar to the concept of emergence, right? And so one thing that I’m not clear on is, on which level of abstraction is it helpful to talk about things? I guess it’s very context specific. But there are two versions of emergence. One is just the idea that you have macroscopic theories that are more useful to talk about than whatever the microscopic equivalent is, the bouncing atoms. And the other is that there’s something in the macroscopic theory which can’t be captured by the microscopic — which has quite different implications. So which is the most fine-grained level, the most interesting or important? Which level should you pay attention to, these kinds of questions. And here it’s interesting that he ends up communicating in short millisecond grunts, which implies an incredibly compressible level of knowledge. And I don’t know what I’m saying, I’m just confused about whether it makes sense that knowledge would be chunked together like that, or whether actually it would get more and more detailed with tons of fine stuff underneath of it.
Cam: Or like a bit of both as well. Yeah, I know what you mean. So like, first of all, what is chunking? Chunking, I think, was first talked about in regards to memory. George Miller was a rock star psychologist, I think in the early 20th century, and he had his rule of seven: we can only remember seven things. So if I say a list of letters or numbers, you can’t really remember them, but if they were like seven chunks, maybe you can. So chunking was kind of used in regards to memory. But in some sense, chunking is — maybe what it is to be intelligent is to use — so you kind of have this first idea of like, I give Benny a cookie, and that’s giving. But then what if Benny gives me a cookie as well, and then, do you have to remember two things, or like, suddenly that’s one thing. If that’s selling, that’s trading. Or if you have money there, that’s like this other one concept of selling, and then you can remember that. Or that’s a market of cookies, and then multiple markets if you aggregate them all up, that’s like one concept, that’s the economy. And not being able to do that would totally constrain you in terms of thinking about these things, because we just can’t remember them all. But then it also—
Benny: Sorry, just one question while you’re describing this. What’s the difference between this and abstraction?
Cam: Yeah, I mean — I don’t know, maybe there’s no difference. I suppose it is by definition some form of abstraction. I’m sort of talking about essentially chunking multiple parts into one. So it’s like one concept, but concepts kind of have to be abstract really.
Benny: Sure.
Rich: Because you could chunk things that aren’t abstractions, right? Like you could—
Cam: Yeah, maybe you could chunk things that are concrete.
Rich: Brute memorize random facts by associating them together. Like a mnemonic example is if you’re memorizing a deck of cards, instead of memorizing each one card, you associate the suit and the number with a person and an object and a location, and then you’re like, it’s Homer Simpson juggling a ball standing in my childhood bedroom or whatever.
Cam: Yeah, Einstein moonwalking on the moon or something.
Benny: I’ve been walking on Earth.
Cam: Yeah, here’s also an example for you guys right now. So if I just listed out some letters — like, let’s just say M, D, P, H, D, R, S, V, P, C, E, O, I, H, O, P, right? That’s hard to memorize. But I actually listed letters that were MD, PhD, RSVP, CEO, IHOP. So then suddenly — but that’s still kind of hard to remember. That’s like six or so chunks. But then if you make that one chunk of, the MD PhD RSVP’d to the CEO of IHOP, then you’ll probably — but if you try to remember that, you could remember that. And suddenly that’s one chunk. And that’s as concrete as the letters. And that’s kind of used for memory. But then also chunking enables us to, as we were talking about, things like the economy, where it’s very similar to abstraction as well. I’m not sure if abstraction always entails chunking. But it seems a very important part of intelligence. But then, kind of as Rich was saying, if we only talked about it at the level of the economy or aggregate demand, that seems wrong as well. And it also kind of feels anti-Deutschian — like you can only talk about people with the aggregate, and it’s kind of like game theory sense rather than individuals being creative.
Cam: So I think you wouldn’t only talk in little grunts, right?
Benny: The moral of the story is, you wouldn’t simply grunt.
Cam: If you were super smart, you wouldn’t just grunt.
Benny: Well, but okay, but that was less about his perceptions of the world and more about his ability to communicate with this other super intelligent person, right? So the grunting was less — there’s a difference between abstracting concepts into other concepts and chunking — that’s just how you think about the world and how you might memorize things or process information — and then there’s actually how you communicate that. And the grunting was about the communication, not about the actual information processing side of that equation. But as you were talking, I realized there is some interesting, maybe not a paradox, but a riddle here, where we all seem to recognize the importance of working memory just for navigating the world and being smart in general. And I think we’d all say having more working memory is better, because you can just hold more information in your head and then manipulate that information more—
Cam: Yeah. And that’s like a quantity sense, right? Like a kind of Chiangian, just rank that up, ramp that up. That’d be good for you.
Benny: Yeah, right, just like raw information — it’s there, it’s accessible, and now you can manipulate it. But there’s sort of this pressure that comes from having only a limited amount of memory, right? It makes you chunk things up earlier, because otherwise you can’t hold too many things in your mind at once. You can’t hold 17 totally unrelated bits of information. So you have to figure out some way to relate it, some way in which they connect with each other, some way to abstract it to some meta concept. And then you can keep that in your head and then descend into the details as necessary, right? And so the more limited your working memory — up to a point, presumably; if it’s just one bit of information, this won’t work — but to some extent, the more limited your working memory, the more you’re going to have to chunk and abstract. And so you’re going to form these connections, which are good. Connections are a very useful way to represent the world and to understand how things are related.
Cam: Wittgenstein has this great quote around the wheel that’s not connected to the mechanism — I’m fucking it up, but the wheel that isn’t part of the mechanism. I need to memorize that. But it’s funny, it’s like if someone gets good enough working memory, maybe they become dumb, because they never need to chunk and abstract stuff, because they just remember each detail.
Rich: I don’t think that’s right. I want to push back on that, because no matter how much working memory you had, you’d still need to use compression unless you had the same amount of bits as there are bits in the universe. And then you don’t need any abstraction, because you can just calculate everything from the motion of particles. But anything below that level, which is everything, it’ll be necessary to be able to compress things and use abstractions, and abstract away details of the molecular level or whatever lower-level abstractions are there. So I think it would still just be better to have more working memory and be able to do more complex manipulations of concepts or symbols.
Cam: It would probably be inefficient just to use it on memorizing things, rather than using it to do multiple calculations kind of at the same time while remembering previous ones.
Rich: I just don’t see why more isn’t better. You chunk things up, but you can’t chunk things up indefinitely. For instance, in your example of going up from — I really like that chunking example — going up to the economy. Is there something higher level than the economy, for instance? And then is there something higher level than that? Where do these gestalts top out?
Cam: Like that South Park episode, have you seen that? During the recession and everyone’s like, “it’s the economy” — that becomes like the mythological kind of spoke. No, it’s nobody’s fault, it’s the economy.
Rich: Because assuming they do top out somewhere, which I think they do, then you do need more chunks of working memory to manipulate them. You can’t just keep making them more and more efficient. Like, what would it mean to have a theory that’s higher than the economy? This is my big question. Why should we expect there to be a theory of everything, other than the very extremely fine-grained level theory of everything, which is like, the universe started in this state and the particles have been bouncing around ever since? Which is famously unhelpful, right? Like Deutsch’s great example of trying to explain the position of a copper atom in the tip of the statue of Winston Churchill’s nose. It’s extremely unhelpful to say, well, we could track the motion of these atoms back to the Big Bang, and that’s how the atom got there. Like, okay, great, that is maybe some theory of everything, but it’s not useful at all unless you’re Laplace’s demon. So yeah, why would we expect there to be very high-level theories of everything, as opposed to very fine-grained layers of everything?
Cam: Well, that just seems to be how things work, doesn’t it? Like, our current theories — instead of remembering all the separated or disparate facts, we have our theories unify things. We have an understanding of phenomena that bring things together. Which — I’m now wondering, as you mentioned Laplace’s demon, like, is this idea of Leon being super smart — it feels maybe anti-Hayekian, where his famous use of knowledge in society, no one person or government can have all the knowledge bits in the world, and that’s why something like a fully planned economy wouldn’t work. But an AI potentially solves that problem, right? And Leon feels like he might be able to solve that because he knows everything. But is that even plausible? I haven’t thought about that enough. Is it plausible that AI would know everything, or is there just too much complexity, and too much stuff, and too much dynamic stuff that changes every minute of people’s preferences, and localized stuff, as Thomas Sowell emphasizes?
Rich: No, it’s logically impossible. You can’t build a model of the universe because it has to contain your model of the universe. I don’t know if that’s the level of fidelity that you want, but no, you can’t. I don’t think you can do anything even close to it. Because the AI’s mind has to be contained within its model of the universe, which includes its mind, and so on.
Cam: Which seems like Leon got to, right? Going back to this book — it was just kind of said, that’s what it was kind of hinting at. He understood how he thinks, and he understood that. But yeah, it kind of feels like this infinite regress, right?
Rich: Yeah, even just to compute — maybe I’m getting too literal and not understanding you, but even just to compute everything, you wouldn’t be able to have enough compute to do it. Like all the atoms in the universe — you couldn’t do a tiny, tiny fraction of that. There’s no way around it.
Cam: Sure. Then going back, then we chunk things up. Maybe.
Rich: Yeah. Is there something higher than economics? Like he’s getting smarter by tapping into gestalts higher than economics. What would it mean to unify economics with cell biology or music?
Cam: Well, then we’re getting kind of glass-bead-game-y, right? Which I thought this touched on. It’s like unifying these separate fields, which happens somewhat. You know, Dawkins — John Maynard Smith and others took a bunch of economic theories which Dawkins popularized in evolution and applied them to evolution. But could you do the whole thing?
Benny: Maybe just — yeah, one thing to inject into the conversation here is that abstraction and higher levels of emergence aren’t useful just in and of themselves. They’re useful to solve certain problems or understand certain things. So talking at the level of the economy is useful for some things. If you’re going to talk about the effect of the Fed increasing interest rates, then to not have the concept of the American economy as a whole is going to really hamper your ability to both understand those effects and to communicate them. But it’s not as if the notion of the economy is a better, more comprehensive notion than — Cam, your example of two people just trading a stick for bread or whatever it is, right?
Cam: The idea of selling.
Benny: Depending on the context, depending on the scenario, you need to also be able to just talk about people trading things with one another, right? And whatever problem—
Cam: You need both, right? You need the lower-level chunks.
Benny: Yeah, yeah. So, like, and maybe that’s one criticism I have of the book — it sort of implies there’s this hierarchy where it’s always better to be higher and higher on this emergent ladder. But I don’t really think this sort of—
Cam: Yeah, I think it’s always better to access the higher levels, to have that access within your domain, as well as the lower levels.
Benny: Like the possibility of going there.
Cam: Yeah, but it seems like he stays there, right? Because he just grunts. And it’s just like “economy” and the guy’s like “America” and it’s like, that’s it. Which I don’t know — I just want to say, I think it’s a funny idea. Like, one reading is that this guy’s just deluded and paranoid. He’s super — he takes this drug and it’s just ramping up the paranoia, and he’s like, CIA are on him, FBI on him. And then, is he even smart? And he’s just kind of grunting, right? And he meets this other guy and he’s just like, “he’s the same”, in the mental asylum.
Rich: Just in his bedroom grunting.
Cam: Which, I think that’s not the reading. Because I think this is science fiction, and it’s not a literary exploration of going insane. And funnily enough, that’s what Always Sunny — I think I told you guys — has an episode based on Flowers for Algernon, which is a very similar idea of a dumb person becoming super smart, where Charlie in Always Sunny, he does that. And the doctor’s just monitoring him as someone who thinks he’s smart. I think that’s maybe where the meme comes from. Have you seen the meme of all the newspapers on the wall and drawings everywhere, and it’s just getting all conspiratorial?
Rich: Oh, is that what that’s from? Nice.
Cam: Yeah. It might not be that episode, but that’s kind of what’s going on. He’s becoming nuts. And he’s saying all this stuff that dumb people might think seems smart, you know, like the idea of a smart person.
Rich: It’s kind of interesting to think that if you did reach this level, to a normal person, no, another person wouldn’t be able to distinguish between — they wouldn’t be able to arbitrate your claim to be super intelligent.
Cam: Yeah, I struggle with that with you guys, getting across these ideas.
Rich: Yeah, I guess you could just prove it by doing super impressive things, like solving some novel proofs or, I don’t know.
Cam: Yeah, and so that’s what I was wondering — were there external things that he did? And like his memory was one, and then IQ test—
Rich: There were, right. I mean, you could just say that he’s lying about everything, but then that probably kind of ruins the story, because he like rigs the whole stock market to make infinite money.
Cam: Yeah, yeah, yeah, “I remember 20 letters”. But yeah, then seeing his name — remember, like he saw his name in the stock market, that some other super intelligent person is communicating with him? But that’s classic schizo shit, man.
Rich: That’s classic schizo shit.
Cam: Yeah, that’s what set me off. I was like, okay, this guy’s nuts.
Rich: No, it was a Jeesh style. You wouldn’t understand.
Cam: Yeah, just saying — there’s this thing, this crazy guy just on the street saying “gestalt”, like, fucking 20 words out of 100.
Rich: Benny, did you have a thought to finish about the gestalt thing?
Benny: No, I was actually just gonna explain the ending, so that we can start talking about interoception and bring in all elements of the story. So like we said earlier, he is sort of hiding out in a hotel, making a bunch of money, trying to get more of this drug and throwing off the scent of the FBI. And he’s making money on the stock market, checks his balances, his portfolio one day, and notices a certain pattern that in some way spells the name Greco. And he is made to understand that this is another super intelligent being who’s signaling, trying to get his attention. I’m not quite sure why he chose Greco as the name. I didn’t actually catch that. Do either of you know?
Cam: Yeah, I wanted to say, where the fuck did Greco come from?
Rich: Yeah, maybe a tribute to the cradle of Western civilization or something. He is an aesthetics nerd too — he’s more into poetry and writing and art and music, right?
Benny: Yeah. So that’s how Leon becomes aware of this other person, and he discovers this other person’s name is Reynolds. They’re both sort of chasing each other around the internet trying to find hints of one another. I think we’re sort of made to think they’re at the exact same level — they’ve taken the exact same amount of the medicine. The current level at which Leon is at, he calls — oh, what does he call it? Hyper — super sonic? No—
Rich: Critical, maybe.
Cam: It’s super critical.
Benny: Super critical. And so once you’ve taken enough of these doses, you go super critical, which is sort of the maybe the final level of intelligence or something. And he realizes that Reynolds, the other super intelligence, had gone super critical a couple days before he did. Anyway, they find traces of each other on the internet, eventually track each other down, meet. He talks about how it was pointless to set traps for each other because they all would have seen it coming, and it just would have dissolved into this total zero-sum game. So they meet. He walks into Reynolds’s apartment. It becomes clear that Reynolds wants to use his powers to further humanity, basically, and Leon is out for himself — he basically doesn’t care about helping humanity. So their ends are orthogonal, and they realize they sort of have to battle it out. Before they battle it out, they have this epic repartee of grunting which Cam keeps talking about, where they’re communicating purely by grunts and exchanging all this information, learning about each other’s worldviews and what each other have learned, and they’ve realized they sort of focused on different things. And then the final battle comes, which basically is two magicians dueling each other, where they’re trying to control each other’s bodily reactions by mechanisms that I still don’t totally understand.
Cam: I’m sort of imagining like eyebrow raises and—
Benny: Increasing — exactly. What did that mean? Like, increasing blood flow to certain areas, and then triggering certain hormones to start releasing things. And so they’re both trying to control and defend their body.
Cam: And multi-layered, eh? Like there’s all these tracks. It’s one off, and then it sets all the stuff off.
Benny: Yeah, there’s attacks and sub-modules and sub-attacks and decoys and all this stuff.
behavioral priming gone WILD (Greco vs Reynolds grunt battle)
Rich: It reminded me of those priming experiments which didn’t replicate, where they’re like, I’ll talk about being old, and then you’ll walk down the hallway more slowly afterwards or something like that. And it’s like, you’ve implanted these ideas in their mind, and it’s like, “I’ve implanted the idea. I’ve incepted an idea that your heart’s just going to stop beating” or something. Like priming on steroids — all of which is fake.
Cam: Yeah, and that’s what he was doing before he meets the super smart guy. He was doing that in the bar to people and stuff. Or he could do that. Just like giving someone a hot drink.
Rich: So, it’s crucial for his later levels of going super critical that he has access to all his own physiological states, which obviously we don’t. So that’s another thing — it’s an interesting question of whether, if you become intelligent enough, you can gain control over your autonomic nervous system and how your organs and viscera function and control your heart—
Cam: Finally, the erection becomes voluntary.
Rich: Right, true. Yeah, you don’t need Viagra.
Cam: That’s super critical.
Rich: Camera’s the first thing you use your powers for.
Cam: Call my wife!
Benny: Up, down. Up, down.
Rich: It was kind of a cool idea. I don’t know — I mean, probably a neurologist would have a good insight into this, but I think by design the autonomic stuff is not within the domain of control. And then yeah, even the model of how it works is that it’s basically this predictive processing idea, where actually you think that your brain would have a good understanding of what your organs are up to and stuff, but it doesn’t. It’s still beyond the veil, and your brain is just doing its best guesses as to what’s happening with all of your various organs. And it’s collecting the raw sense data, updating its predictions, etc. It’s not even like this tightly controlled thing that you might expect, where the brain knows exactly what’s going on and then tweaks your temperature a little bit or tweaks your heart rate down or whatever. It’s just this allostatic feedback loop of predictions and data meeting in the middle and constantly updating. So that mechanistically to me suggests that you literally couldn’t control these things. And I think there’s no examples of people being able to control their own heart rate or whatever. But it does all originate in the brain, so I don’t know, maybe there’s some way.
Benny: But also different parts of the brain, right? I think there are physiological constraints here, where your prefrontal cortex is sort of what’s associated with consciousness and your narrative experience and stuff. And I think a lot of that allostatic regulation is going on in the more amphibian parts of your brain and stuff. So I think even the layout of your brain — somehow you’d have to override the status quo there. You’d have to give your prefrontal cortex — you’d basically have to develop new neurons in some way to attach certain parts of your prefrontal cortex to other parts of your brain.
Cam: Rewire the gestalt of the brain.
Rich: Yeah, not software, but hardware changes, right? Which you couldn’t do short of doing like brain surgery on yourself.
Cam: Yeah, he should have done a bit of that. Got the scalpel out. “Just have I flick this one wire?”
Cam: That kind of freaks me out, the idea of control on the heart. Like I remember hearing stories of people going to these rooms that are created that lowered the sound volume so you can hear your blood rushing and shit like that. People can’t handle it. They’re in there for 10 seconds before they start going insane. Then there’s this one guy who used to — I don’t know, maybe he’s former army or something — but he had this issue of too much loudness in his past. He apparently just went in the room for an hour and he’s like, “oh, finally a bit of silence.”
Cam: But yeah, that stuff kind of gives me the heebie-jeebies.
Rich: Yeah, I want to pay even less attention to that stuff. Like you know how the classic advice is if you’re feeling stressed or anxious, you do breathing techniques like box breathing? Like, you transfer your attention to your breathing. When I do that, I get more anxious. I’m like, “oh, god, like now I’m—”
Cam: I’m not breathing properly, yeah.
Rich: Now I have to take over the breathing.
Cam: What if I don’t breathe?
Rich: Like when it was running in the background by itself, I’m happy, and when I’m thinking about it—
Benny: That’s one more task.
Rich: Yeah, if I was in charge of my own heartbeat as well, I’d be like, come on, man, I want to outsource this. I don’t want to think about this.
Cam: I got a mate that got kind of fucked up with that, because he was focusing and then he notices his heart doesn’t actually — I think the doctor confirms, so that is a bit of a skip or something — and like it’s all he can think about and he’s just like having trouble sleeping.
Benny: Every time he breathes — “please beat, please beat, please beat”, just waiting for it. Yikes. Okay, so just to finish it off, the way the story ends is Reynolds introduces the idea to Leon of some sort of trigger, which is basically something you could say or some sort of impression you could leave someone with that would totally destroy them, for lack of a better word. It would—
Rich: Just scramble their brain.
Cam: One word. One word that will destroy someone.
Benny: Yeah, one word that would just implode their brain. So like a trigger.
Rich: Donald J. Trump.
Cam: True, good.
Benny: First day in office, half the country will just die. And so this freaks Leon out, and he’s searching his memory, searching his brain for this trigger and where it might have been — where the sequence of words might have been implanted by Reynolds. And Reynolds lets him do this calmly, and Leon can’t find anything. And then Reynolds divulges that the trigger was not actually a sequence of words. It was a sequence of impressions, and some of those impressions included things that had already happened, like him walking up the stairs to his apartment. And so it’s this unspecified sequence of impressions that, as soon as Leon starts thinking of them, each one leads inexorably to the next one.
Cam: That’s the trigger win.
Benny: And this basically shuts down Leon’s brain, and he implodes basically.
Cam: I can’t stop thinking of like the whole priming-on-steroids thing now. Like, “when you walked up the stairs earlier, we had someone hand you confetti”, and it’s just like all this BS. I think there was — who’s that magician who’s kind of a skeptic, he’s like an illusionist — Derren Brown with the M. I think he had some early videos on priming, which is kind of like obviously BS, where they’d prime people, and then they’d be on the TV, and they’re like, “we show you the images, and when you come in” — and it’s like, you know, some old man standing there, and like, “we walked you past an old man”, and stuff like that. Just showing you all the stuff.
Rich: Yeah, this bit of the story gets kind of goofy, right? I mean, it didn’t bother me too much, but it’s very hard to suspend disbelief for this kind of stuff. And also the really goofy bit is that Reynolds raises his forefinger like a “but actually” guy, and then he goes, “Understand.” And “Understand” is the linguistic kill shot, right, that just triggers it all and scrambles his brain into mush.
Cam: Which does make you wonder about info hazards. I found the idea of learning and understanding things too much to kind of destroy yourself, or at least to harm yourself, seems possible, plausible.
Rich: This is not really an info hazard, though, right? It’s just some generic self-destruct code, more than it is an idea or concept that — or actually maybe I’m wrong about that. Like, what’s the implication that—
Cam: It’s like he understands himself, like finally, and like there used to be the whole thing of, you know, robots like in Futurama or something — “I, you know, self-destruct” — as soon as you try to understand yourself.
Rich: Yeah, yeah. Do you guys want to talk about why they had to kill each other? I’m not exactly clear on—
Cam: Yeah, definitely. You go.
why can’t we all just get along??
Rich: Well, Reynolds’s goal was to help humanity. Greco’s goal was to be left alone and just work on his cool one-man projects that only he could comprehend. And I wasn’t clear on why there’d be a conflict between them. And also, if they’re extremely intelligent — so Chiang’s implied philosophy here is that he’s buying something like the orthogonality thesis and the AI research stuff that values of very intelligent beings do not necessarily come together. They could have totally distinct values that they can’t cooperate on or compromise on. Which I think is probably wrong. But also it didn’t seem like they even tried.
Cam: Yeah, why can’t they figure it out? Two smartest guys in the room. But it was almost — maybe they didn’t unpack it, because I think at one point Leon said he realized they were incompatible, for some reason. Which he didn’t say why they were. He kind of got there on the sixth level when he kind of realized, and he realizes that Reynolds must know that as well.
Rich: He heard a certain grunt and he was like, “oh no, it’s never going to work out”. We grunted at each other for four seconds and we’re all out of ideas.
Benny: I mean, isn’t it just that they’re both so hyper agentic in the world, and would be requiring to manipulate so many things to put their plans into action, that those things would necessarily conflict with one another?
Cam: Yeah, I suppose. It’s just — do you think it’s kind of going back to like, problem solvable? Or would there inevitably be trade-offs? I suppose, given certain value inputs, that’s possible. But then could those values be persuaded, and some more right and some more wrong, so you get some realism here? I mean, there’s also this other thing I thought about, like, where — they kind of want to do stuff by themselves. For me, collaboration is very auspicious for creativity. Like, just getting into Nozick’s experience machine, that seems kind of scary, because you’re just by yourself, and you kind of lose not only your loved ones, but you also lose the creative chance to engage with other minds. And I thought that would probably be amazing for them, right, to collaborate? But also, yeah, why couldn’t they solve the issue? It’s just like, man, I just want to do my poetry, you save the people, like, why did it conflict?
Rich: Like they must have hacked loneliness, right? Because they’d be completely isolated, obviously, from any normal human. So they must — I guess if they can control their own physiology, then they can also control their own psychology or something. But they certainly could learn heaps from each other, because we got that flurry of exchanges where they quickly updated each other on all of the ideas that they’d come up with. So that’s the proof that positive-sum interactions are possible, right? And then it’s also interesting that they do that, and then they’ve already predetermined that they’re going to defect in the cooperation game, or however you want to put it — that one of them just has to kill the other.
Cam: Yeah, it seemed like the one thing they can’t change was their goals at the start, which does feel a bit paradoxical, overriding those.
Rich: Speaking of prescience, it is interesting that this is the orthogonality thesis, right? But I’m pretty sure this is — yeah, so whether or not Chiang inspired people, or it was already—
Cam: Yeah, straight from Bostrom.
Rich: Floating around out there. Would this be a good time to talk about whether or not we buy any of this stuff? The orthogonality thesis, the concept of super intelligence, the possibility to have agents that have massively different abilities to comprehend things?
Benny: Yeah, let’s do it.
reconciling David Deutsch’s ‘universal explainer’ theory with IQ
Benny: Yeah, I think we should. I think we should also maybe just give a quick overview of Deutsch’s view, because I think it’s going to be hanging over the conversation whether we explicitly mention it or not. And so we should just talk about the concept of universal explainers, just to get it out there, and then we refer to it. Who wants — does anyone volunteer to be tribute?
Cam: Who is Deutsch, Benny? Oh, Rich, Rich can.
Rich: I’ll have a crack at it. And then, I mean, Benny, you’re far and away the expert. So yeah, David Deutsch is this—
Cam: Give him a call, Benny.
Rich: Yeah, as Benny’s close personal friend.
Cam: Get him up.
Benny: My best bud now. Hey, David.
Rich: His adoptive granddad. He’s all of our adoptive granddads, really. He’s this really cool physicist, and he’s an expert in the philosophy of knowledge and a sort of fan of Karl Popper. And he has this really fascinating theory about where is it that knowledge comes from, which is like one of the big central questions in the study of knowledge. And what is it that makes humans different to other animals, and what is it that might make a true general intelligence — artificial intelligence — different to, for instance, the types of artificial intelligences that exist today. And the central idea is this idea of being a universal explainer, which means that instead of thinking of intelligence or creativity as a continuum, it’s probably something more like a binary, where if something is able to be understood, then a universal explainer can understand it, or can explain it. And so you don’t have different levels of things which can be understood by greater or lesser minds. You simply cross over a threshold, and once you cross that threshold, you can, in principle, explain anything that can be explained. And humans have crossed over that threshold and chimpanzees have not, for instance. Artificial intelligences have not crossed over that threshold, and at the point where they did, they would be people like us. They would be equal to us in any way. So it’s a very controversial view. It makes a ton of sense to us. And we probably internal— or, I don’t actually — does it make a ton of sense to you guys?
Cam: It doesn’t make heaps of sense to me. I’m confused by — I’m confused by both sides. I contain multitudes.
Benny: Of all the — Cam can’t explain this. He’s falsified the universal explainer hypothesis with the universal explainer hypothesis. It’s impossible to grok. I mean, I think — yeah, Rich, I think I basically buy your resolution here, which is focusing really hard on hardware differences, and basically laying—
Cam: Well, before you do that, it’s worth just saying — there’s this distinction that Deutsch invokes and other people invoke, and everyone knows in computer science, of hardware and software. And software is the program that runs on your computer, obviously. And then hardware is stuff like the specs of working memory and speed — like how fast that program runs, how much can be stored in memory. And so if humans are computers, then we probably have that too. And we have — we obviously have memory, and working memory gets talked about in psychology, and there obviously seems to be differences in speed of calculating things. But then do we also have this kind of analogy to software? Do we have programs that get run, which maybe are ideas influenced by culture, but also ideas influenced by our genes, perhaps or perhaps not? Certainly the hardware things seem to be influenced by genes. But this distinction seems pretty core to our understanding now of intelligence. So yeah, what were you saying about the reconciliation, Benny?
Benny: Yeah, I was just gonna say I basically buy what I think is Rich’s view — Rich, correct me if I’m wrong — which is to lay apparent differences in something like IQ, which basically tracks what we think of as smarter and less smart people, at the feet of hardware. And to say we can basically account for these differences by saying people have different processing speeds, one, and two, working memories. But if we were able, in theory, to augment those to give everyone the same processing speed and working memory, then the fact that we’re universal explainers would start to kick in, and we all could explain and understand the same class of things. You know, I think that is probably maybe the only coherent way to reconcile the two ideas. The two ideas being universal explainers and this continuum view of intelligence. I don’t think Deutsch actually buys that sort of reconciliation, but that’s sort of where I am right now, I think.
Rich: Yeah, and I just want to throw in a quick note so that people know how batshit crazy this idea is, which is that Deutsch thinks that, in principle, the dumbest guy you know is capable of making novel advances in his field of quantum mechanics, and that the only thing that gets in the way is that person’s interests, that person’s cultural knowledge and background and modules, and possibly any literal hardware problems — as in, if they have some kind of brain damage or had some accident or something like that. So his theory is massively in violation of common sense, and the empirical view of what intelligence is. Like, we all know what a dumb person or a smart person is. We’ve all met plenty of them.
Cam: That’s not me, right? The dumbest person you know? I’m not. Someone else, right?
Benny: Depends how you pronounce gestalt. That’s what it comes down to.
Cam: The archetypical dumbest person you know.
Rich: It depends what room we’re in at the time, but—
Cam: Yeah. That’s funny. If you’re in and out of it. So, it doesn’t change. So, I mean, that’s kind of why I don’t buy it, to be honest. Well, it’s why I’m confused by it. But it seems so batshit and obviously wrong, right? It seems like humans have variance in intelligence. And I’m not even sure if it’s just memory and speed. Because when you have these IQ tests, you have verbal as this big component, and then spatial reasoning as this big component, and then you also have memory tests, which are related to verbal. Let’s just say it is speed and working memory. What I feel like Deutsch is trying to smuggle in is that’s not the intelligence that we care about. And he’s got this one little quip that does quite a lot of good work towards that direction — of like, the difference between Einstein and the village idiot isn’t just like the village idiot needs more time to crank out the calculations. It’s like there’s this difference of creativity, or the spark. But maybe creativity requires this ability to crank out calculations quickly in a short space of time, and that’s how it builds on each other. I think that was Arthur Jensen, who was an intelligence researcher who’s big in biological explanations for that — like, he thought that was the big underlying thing for g, was just kind of like, well, either processing speed or working memory, or both maybe.
Rich: But yeah, it might help to lay out a couple of quick points of evidence in favor of Deutsch so that it doesn’t seem totally ridiculous off the bat. So the obvious one that comes to mind is that chimpanzees have incredible working memory. They have much better working memory than most humans do. Go on YouTube and look at these videos of them memorizing nine-digit sequences of random numbers. And there’s even, I think, like “are you smarter than a chimp.com” or something — you can test yourself against chimps, and they’ll beat you on a battery of various cognitive tests, or reflex tests, things like that.
Cam: Yeah, I’ve seen them. And even just remembering stuff, yeah, it’s crazy.
Rich: Yes. But chimps can’t do very simple things—
Cam: It depends if I’m in the room with a chimp whether I’m the dumbest, because then, you know—
Rich: Yeah, yeah, maybe with a really dumb chimp you’ve got to hope. But yeah, there’s clearly some sharply defined thing separating humans from other animals, for instance. So the fact that you can crank up working memory or computational speed is not of central importance to Deutsch’s underlying point, which we’d have to get more theoretical to flesh out. Which may not be that interesting. But basically it’s just the concept that humans are Turing complete, meaning that they can run any finite sequence of steps that can be run on any other type of computer. In the case of a human, that needs external memory like a bit of paper and a pencil, for instance. But they could walk through any algorithm in principle. I think that is technically true, but maybe not pragmatically as interesting as Deutsch thinks, or people in that camp think. In that, clearly it matters a lot. The way that I think of it is, the computer in my watch — well, that’s probably not even a great example — the computer in my alarm clock is also Turing complete, perhaps, but it’s not useful to say therefore it is the same type of thing as a supercomputer. Technically that’s true — if you had until the end of time, you could step through any arbitrary program, in principle. It makes a dramatic difference, to the point where it seems kind of silly to focus on that thing being the unifying attribute at the expense of the others. So yeah. Cam, would you buy Benny and I’s modified version of it, which is that universal explainership could be true, but there are important variations in the hardware which just manifests, leads to different outcomes?
Cam: Yeah, I mean, that was my original reconciliation as well — that IQ tests are just measuring hardware differences. And maybe where I differ to Deutsch is, like, that matters hugely in practice. But when I get a bit confused, maybe where I then draw away from it is — it just seems like, you know, the dumbest person you don’t know, it just seems like they’re not going to understand Einstein’s theory of relativity. It’s sort of common sense. And not just due to cultural reasons — there kind of seems to be this barrier. And perhaps we don’t know the answer to that question, so I can’t claim that. And it also — it seems largely, not completely, but significantly genetic as well. When you test people’s relations and people’s non-relations that they were raised with or raised by, like, it just pops up hugely. And some of the estimates are massive, like, you can quibble about them, but the high end of the estimate is like 80% genetic, right?
Rich: But the genetic thing, doesn’t that get answered by the hardware argument? Because the genetic thing is explaining hardware differences, whereas everyone still has this binary universal explainer attribute.
Cam: Yep. So yeah, but I suppose what I’m saying is, like, these really dumb people and really smart people, it just — I don’t think they could understand the same thing, given more time or the right culture or the right idea that just suddenly unlocks it. It seems like there’s this barrier of, like, calculation. And that describes intelligence — not completely, I think it’s almost this combination of, what you were kind of saying before, this raw processing chimp versus this kind of thing that humans have. I think that’s true. And both matter. Obviously both matter, but — variance of our intelligence. Yeah, sorry, I’ll stop. Go around in circles here.
Benny: One variable we should put into play here is just that of interest, which Deutsch often appeals to to explain people’s differences of abilities, quote unquote. So he’ll say like, yeah, well, most people aren’t interested in understanding Einstein’s theory of relativity, and that explains a large part of why most people don’t go out of their way to actually understand the relevant mathematics. I think he attributes too much to interest, but I just wanted to put that on the table as a giant—
Cam: Yeah, interest obviously plays a role. I mean, and interest also is arguably genetically influenced. But it seems almost — like, if it didn’t come from someone who I respect so much, it seems almost laughable to say, like, just imagine an incredibly far left of the bell curve person, which we probably haven’t even rubbed shoulders with much, and then very far right end of the bell curve, and just say, “well, one’s just interesting, one just happens to be interested in quantum physics”. Like, of course that’s not the reason why he can do it, and then the other — what’s the other one interested in? Counting rocks? And he’s struggling with that. It’s like, he just wasn’t so interested in counting rocks, whatever it was.
Rich: One too many. All right. Let me try and expand the definition of interest a little bit. Because when you put it like that, it sounds fucking crazy. But a good example I think Brett Hall or someone has come up with is, if I say to you, do you think you could learn to speak fluent Mandarin? You’d say, oh no, that’s so hard. I mean, maybe not to you, but to a random person, that would probably say that’s super hard. You’d have to be really smart to learn languages. It’s simply not true at all. The dumbest guy in China, the absolute dropkick of China, speaks perfect Chinese, because his interests are heavily aligned with learning Chinese. And then, you know, Deutsch I think talks about how — he said something like, when you’re an immigrant in a foreign country, you’re not incentivized to learn a different language. Then when you move, you’re heavily incentivized to learn a different language, and so you do. You might want something, but without actually following through on it. So there’s a deeper sense of interest where it’s actually talking about what is going to motivate you to do something as well.
Cam: Yeah, like your higher-level kind of motives aligning, which is a good point. But it potentially doesn’t hold with language, because as Pinker and Chomsky kind of argue, and Paul Bloom, language is potentially this kind of evolved instinct that everyone learns effortlessly, and is different to other kinds of understanding.
Rich: Yeah, the point is less about that and more on just reframing your sense of what it means to be interested in something.
Benny: Cam, have you never been in a situation, like in a classroom or something, maybe where you’re faced with some subject material you absolutely hate it, and maybe when you’re young you even just convince yourself you’re not good at it, and later in life you realize that it’s actually very interesting, and you might even have what you think to be a natural knack for it? Like, have you ever had an experience like that? When I reflect on those sorts of moments in my life—
Cam: Oh no, definitely. I used to think I was bad at English or something, and I was more of a maths guy. But now if anything I’m probably more of a word cell. And it’s largely interest. Like, imagine reading Shakespeare — we read Shakespeare together, right, and I actually kind of had fun. I know Rich had the wrong translation. But like, Dostoevsky — imagine being forced to read Dostoevsky when I was 15. Like, I couldn’t think of anything worse, and I just wouldn’t — I wouldn’t have learned it. Yeah. Someone trying to learn something that they’re not interested in, they don’t learn it. I don’t think it’s completely true that they don’t learn it at all, because we all kind of remember Pythagoras’s theorem and stuff like that and multiplications. But like, you don’t learn it as well, and often not at all. And suddenly when you like it, it becomes effortless. It doesn’t even feel like learning sometimes. Just like browsing Wikipedia or something like that, or reading about someone, and you kind of feel like, maybe I should do some of the real work, you know, and read the Iliad, when you’ve actually been learning a bunch and you just happen to be interested in it and it feels like—
Rich: So yeah, the second part of my reconciliation attempt is that interest is downstream of hardware. So if you have pretty good working memory and good computational speed, then things like maths, or just handling abstractions and playing intellectual games, become lower friction and lower effort, and have less aversion associated with them. And then you can build up those cultural modules and climb your way through the tech stack or whatever it is. And if you have hardware that doesn’t run the algorithm very smoothly, then you find that a bit more aversive, and maybe you go towards more system one type activities — instinct driven, proprioception driven, bodily physical things, hands-on things. And that manifests as what you like. But I think what you like is downstream of your capabilities.
Cam: It’s potentially bi-directional. There’s definitely that phenomenon of, like — which we probably all fell into — but at school, the kid who’s good at maths starts to like maths kind of because they were good at it, and potentially doesn’t have this intrinsic interest in maths. They get status from it, and all the adults and the teachers are saying this is a very good thing. And then often, when they’ve left school, they might come across some beautiful explanation, or some YouTube — 3Blue1Brown or Mathologer or something — and then you’re like, “oh, I like maths. I actually like maths for maths. It’s fascinating. It’s beautiful.” But the interest earlier was more of this kind of — because they were good at it, and they did well, and that felt good. So yeah, I think that happens.
Benny: I was just going to say, from Deutsch’s perspective, I agree that that’s sort of a reconciliation, to view interest as at least partially downstream from genetics and hardware. But I don’t think Deutsch would like that reconciliation. I think he would still say we’re leaning too hard on genetics. And I think for all practical purposes for him, basically everyone’s the same, with minor fluctuations. And you can basically put down Nobel Prize winners and the differences between Nobel Prize winners and village idiots to luck and interest.
Cam: Deutsch is essentially a blank-slater. He refused—
Rich: Yeah, I just think he’s wrong about that. I’ve never been able to — he’s never really given reasons that I’ve been able to find. The only one that I’ve heard is this assertion that, because from Deutsch’s point of view all problems are in principle soluble, that you can make up for any hardware damage or hardware deficits by tacking on additional memory, brain-computer interface, something like that, and therefore you can solve those problems. But that doesn’t deny that those differences exist in the first place. And even the fact that he says that suggests that he does agree that those differences exist in the first place. But you’re probably more of a scholar of precisely what he said about this.
Cam: But I think he thinks you could do that with a chimp, right? You could as well.
Rich: No, that’s the other way around, where he says you could give the chimp UE. But what I’m talking about is whether you give someone who has UE more hardware, you know, better hardware capabilities, so that they can keep up with the AI or the space alien or—
Cam: Yeah, yeah. Well, I think he definitely thinks you can do that. But yeah, I think regardless of what he thinks, where society’s in this position of trying to understand how to reconcile this stuff, how intelligence exactly works, and what level does genetics play, what level does this quantitative hardware stuff play, and then what level does this kind of software ideas and culture play — and it is kind of hard to join it all up. Gestalt.
unresolved AI safety concerns
Benny: Yeah. Also, one thing that our reconciliation leaves the door open to is some sort of fear of super intelligence and possibly bad consequences thereof. So if you’re on the pure Deutschian view, he often says there’s absolutely nothing to worry about. Because for him, like Rich said, there’s just this binary between things that are universal explainers and things that aren’t. And once you’re a universal explainer, you’re capable of understanding everything. So we’ll, you know, if we invent other things that are universal explainers, they’re just like more humans. So we’ll just commune with them as we commune with regular infants and teenagers, and we’ll raise them in society, and it’ll all be fine. But if you adopt our sort of view, then you have to take the differences in something like processing speed very seriously. And just by virtue of the fact that these super intelligences, if they’re built on the same sort of trajectory that we’re on right now, they’ll be built out of silicon and be able to think just much faster. And if you’re ready to acknowledge that just being able to think at near light speed, right — whereas our neurotransmitters are based on chemicals, it’s pretty slow in there, our brains are relatively slow — it’s much faster than just electrical signals. So yeah, if your brain is based on electrical signals and you’re just able to think even the same category of thoughts that humans can, in principle, but like, way faster, then that’s a difference you might have to take seriously. And that could give rise to fear about, okay, what’s going to happen when we start introducing these things in society? I’m still not concerned, but for other reasons, I think, right? I think you have to take that possibility seriously.
Rich: So this is where orthogonality becomes really important, right? Because another one of Deutsch’s views is that morality is sort of non-random, that there are objective moral truths, and that it tracks in line with coming up with better and better explanations. So if you invent a super intelligent AI, it won’t, in fact, tile the universe with paperclips, because it will be capable of coming up with very sophisticated explanations about the world, which includes moral explanations. And to the extent that we have good moral explanations, it will converge with us, and we won’t have this problem of alien goals that are totally inexplicable to us. They will be — they might be alien, but they will be explicable. And so yeah, if our modified theory is right, and AI can think much faster than us and pragmatically speaking have different capabilities than us, if it’s willing to help us out and bridge the gap, then no worries at all, we get basically the Deutsch outcome with extra steps. But if it does have orthogonal values, or if morality isn’t objective, then we are totally screwed in the same way that the doomers keep talking about. Even if they’re wrong about — on a technical level — about universal explainership, they’re missing the universal explainer piece. So like, yeah, I guess it’s probably way too big of a can of worms to open to talk about orthogonality. But yeah, that’s the second — maybe we can talk about that some other time or something.
Cam: We’ll let Leon and Reynolds hash it out.
Rich: Yeah, I mean they did. They didn’t, right? That’s kind of grim. That’s Ted Chiang’s opinion.
Benny: Yeah, yeah. I would like to talk about this at some point, though, because I do go back and forth about this. Like, on the one hand, I do like Deutsch’s argument that, yeah, moral knowledge is just another sort of knowledge insofar as you’re some sort of moral realist, right? It’s just things there to be discovered about how you and society can live better lives. So as you get smarter, it seems like you should also — if you get more knowledgeable in general, it seems like moral knowledge to go hand in hand with that. On the other hand, there are counter examples, so—
Rich: If the AIs fuck up in the manner that we’ve fucked up, then we’re in huge trouble. Like, we’re universal explainers, and we’ve still genocided entire — or, you know, destroyed entire species and committed perhaps incredible acts of evil, like factory farming, things like that. And we are universal explainers. So what if an AI does that as well? It can still make mistakes just like we do. So I don’t, again, understand why Deutsch isn’t concerned about that. Even if the AI ultimately converges to having really great enlightened moral values, it could still wipe us out when it’s in the learning phase.
Cam: Well, he might be concerned about it, but he’s a very non-authoritarian. So I think he says that we have no right to enslave this other person that we’re worried about. Why couldn’t they do that to us? And if we were doing that, they will fight out, and so to. Right? So like, we have only a moral weapon — kind of argument and persuasion. And same for — well, that said, you know, war is not a pacifist either.
Benny: No, he’s certainly not a pacifist.
Cam: Speaking of genocide — no, I’m just joking. So anyway, yeah, I probably need to get to work pretty soon. That’s a big can of worms. Maybe we jump on Benny’s other podcast, check out increments.com.us.
Benny: Oh, we could do that. That’d be fun, actually. Also, that’s territory we haven’t really covered. So I mean, we could try and shoehorn it into a book, but it would also just be fun to talk about that from first principles or something.
Rich: Yeah, we might as well. I doubt we’re going to find any book, now I think about it, that’s any closer than this is to the orthogonality piece. Cool. That was good. We’ve solved it. Big brain time.
