If nobody knows what's on the other side of the AI singularity, why are the optimists so sure it's great? Robert Wright's answer: they're running on faith.
Robert Wright talked about neural networks with Geoffrey Hinton in 1983, two years before Sam Altman was born. He's the author of The Moral Animal, Nonzero, and The God Test. We cover: why the Tower of Babel is really an AI story, why no nation can slow AI down alone, how a model learns what a word means when nobody ever tells it, why I think AGI is impossible, why Wright thinks a swarm of talking models could out-think any single one, and the grimmest reading of the Fermi paradox, which even Lee Smolin's cosmology can't talk him out of.
We face the kind of test a God would give.
00:00 The test a god would give 00:54 Babel: the first AI cautionary tale 04:06 Every salvation story has a test 05:46 Are we building idols? 08:06 Why AI optimists need faith 14:10 Hinton in 1983: no doomer yet 19:38 Why no country can hit pause 25:01 Meaning nobody taught the machine 32:08 Can Claude have a happy thought? 37:44 Judging The God Test by its cover 40:06 My case that AGI is impossible 47:04 Did the aliens fail the test? 51:10 What passing the test looks like
────────────
“We face the kind of test that God would give.” — Robert Wright
CHAPTERS 00:00 The man who interviewed Hinton in 1983 05:04 The God test and its two meanings 08:36 Religious faith in technological progress 14:10 Hinton then versus Hinton now 19:38 Why slowing AI down requires international cooperation 26:02 How LLMs reverse-engineer the meaning of words 32:08 Brian's Einstein test: can Claude have a happy thought? 40:06 AGI is impossible. Here is why Brian thinks that. 47:04 The Fermi paradox as a moral filter 51:10 What passing the God test actually looks like
Get the transcript, fascinating bonus content, and my Monday M.A.G.I.C. Message: https://briankeating.com/yt
Support Into the Impossible on Patreon, get my weekly M.A.G.I.C. Message, unfiltered bonus content, and live monthly Office Hours with me: https://www.patreon.com/drbriankeating
Robert Wright explores the \"God test,\" arguing that AI serves as a crucible for humanity. To survive the risks of superintelligence and bioweapons, humans must undergo a moral evolution, transitioning from tribalism to global cooperation. The discussion balances technical AI mechanics with philosophical debates on consciousness, evolution, and the Fermi paradox.
You feel sure everything's gonna work out fine.
I don't really understand how you can be confident of that
unless you have something like religious faith
and technological progress.
Natural selection is blind.
The mutations in natural selection are blind,
but the selection criteria are not.
One of the grimmer readings on the Fermi Paradox
is the reason you don't see many civilizations
that have gotten past the point we are
is that typically when they get to this point
where they have the capacity to blow themselves up,
they blow themselves up.
I think we face the kind of test that God would give.
My guest today is Robert Wright, one of the OGs of AI.
He was interviewing Geoffrey Hinton about neural networks
in 1983, two years before Sam Altman was even born.
And that was long before anyone was terrified of them.
Robert thinks that AI is putting humanity through a test
and that passing it takes something no lab can ship,
a moral advance.
The Tower of Babel story is a truly cautionary tale
and it's a tale of kind of technology gone awry,
technology wanting to compete with God,
to fight with God, to make a name for oneself.
And you see a lot of that.
Where do you come down on this?
That the fundamental aspect of the God test is,
at least as I read it, but now I'm talking to the authors,
is are we gods but for the wisdom?
Are we able to pass this test?
And maybe you should define the test,
where we can prove maybe to ourselves
in the absence of a God that we're worthy
of God-like technology.
The title has at least two meanings.
One is, as you know, some people think
we're building a super intelligence
that will ultimately, in some sense,
maybe qualify for a term like God.
It'll be all powerful and maybe live in harmony with it
and according to some doomers, maybe get annihilated by it.
But in any event, we'll be talking about
a kind of from our, you know, relative to us
and all powerful and, you know,
omniscient in some sense being.
And so, you know, one test is, if that's gonna happen,
can you build a good God?
A God that is compatible with human values
and would safeguard human flourishing and so on?
There's that.
And although I'm not sure we're building
a super intelligence, I certainly don't rule it out.
And I explain why in the book.
But there's another sense of the God test
that in a way is related to the Tower of Babel
because the Tower of Babel is a story of hubris.
Who are we to build a God?
On the other hand, the doomers say,
even if we don't try, it's taking shape.
We're gonna have to try to stop it
to keep it from taking shape.
But in any event, the Tower of Babel is also a story
of how humanity became divided.
How they came to speak different languages
and became, in a certain sense,
kind of inherently tribalistic,
at least at a linguistic level.
And one argument I make in the book
is that if we're gonna do a good job
of navigating the AI revolution and wind up,
leaving aside the question of whether it culminates
in super intelligence, if we're gonna reconcile it
with continued human flourishing and welfare,
I do think we're gonna have to approach it
as a true global community.
I think there are a lot of policy issues
that can only be handled at an international level,
and there's more and more recognition of that.
Just in recent months, we've seen that.
I think the end of the story of the Tower of Babel
needs to be that we all kind of get back together
and navigate this challenge collectively,
that is to say, human beings.
And so in that sense, the God test
is to overcome our tribal differences,
to get better at understanding the perspectives
on people on the other side of walls and so on,
and quit having wars.
That would require something that I think would qualify
as a moral advance, and I think it's gonna be necessary
if we're gonna survive this in good shape.
So in a way, that is a test we're being subjected to,
and that is the kind of test that God would give.
At the end of the book, I quote the Hebrew Bible,
or as we used to say in Southern Baptist Church,
the Old Testament, salvation is at hand.
And as I say, there are a lot of stories
in a lot of religious traditions
where salvation is possible,
but you are going to, in some sense,
have to pass some test.
It may vary by religion.
In the Christian tradition, salvation,
the test has to do with accepting Jesus
as the Savior who died for your sins.
But very often, in one sense or another, it's a moral test.
And for salvation to happen, and by the way,
I would also say that salvation is often
salvation at the societal level,
including in the Hebrew Bible.
Again, Christian salvation is often construed
more as an individual thing.
But in many traditions, it is a question
of kind of saving the social system.
And I'm saying I think we face,
leaving aside the question of whether
there is a divine purpose unfolding,
I think we face the kind of test that God would give.
There's so much to unpack there.
The classic in the Torah, the Old Testament,
the classic paradigm for a test,
but the first test that's really actually called a test
or thought of as a test comes to Abraham.
He originally has a different name, Abram,
and then his name gets changed to Avraham,
which means father of nation.
Then he goes from being an individual to a collective,
as you just said.
And the kind of crucible, literally,
by which that's thrust upon him is a series of tests.
In the Torah tradition, there's 10 tests
that begins with his father throwing him in an oven
and breaking the idols that he worshiped.
And that's where I want to start.
There is so much with these new models
and so much personification.
You probably saw this Dwarkesh Patel article
that got tens of millions of views on X and elsewhere,
where he talks about civilizations living and dying
and rising and falling.
It really seemed so hyperbolic to me.
And in my conception, I've talked to all the doomers,
and there's way more doomers than there are boomers,
as you know.
The only public boomer that I really know is Jan LeCun,
and I had him on, and he's extremely, as you know,
and you quote him in the book,
he's extremely technical in his optimism.
He's not kind of denying the sort of societal upheaval,
but he's saying, we're not there technically,
is the impression that I got,
and we probably will never get there.
And I want to talk about that.
But starting with the test of kind of, is this an idol?
Are we worshiping these things?
When you hear these laboratories,
we had to put it in a sandbox, we had to stovepipe it,
we had to cut it off because it was getting too dangerous
and it was escaping and it was creating civilizations,
according to Dwarkesh.
What do you make of this?
We live, whether or not we personally,
individually are or not, we live in a secular age,
and the secular age has never been more enhanced
by technology than it is now.
So do you make anything of these comparisons
between both the idolatry of AI and the eschatology of AI?
In other words, the Dumerism is intimately connected,
I think, with the idolatry.
So what do you think?
Are we making these things into gods to worship
or to propitiate?
Interesting question.
I mean, the Dwarkesh thing was about a bunch of agents
in this famous open AI hugging face incident,
and he was comparing generations of agents,
I guess, to civilizations.
I'm not too bothered by that kind of anthropomorphism
because, as I argue in the book,
I mean, we are kind of trying to,
and in some cases doing this without trying,
but to inculcate in AI human tendencies.
You know, there are a lot of things,
a lot of ways we want them to be like people.
We want them to be pretty autonomous when they see goals.
We want them to do this, we want them to do that.
And I have a separate argument
about what seems to be going on inside them
and how like the human mind that may be.
I'm not terribly bothered by that kind of language,
but as for the worship thing,
I think you could probably accuse either extreme of that,
which is to say the doomers or the accelerationists.
I mean, you've articulated the case
for looking at certain kinds of doomers
as engaging in idolatry,
although some of them would argue
that they're certainly not religious.
I mean, Eliezer Yudkowsky, as you may know,
I don't know how much you know about him.
He's kind of the doomer in chief.
He turned his back on religion.
I believe he was raised by Orthodox Jewish parents,
but his religion, you might say, became science fiction.
It remains his religion even.
I mean, some might make that accusation
that he's uncritically accepting certain sci-fi narratives.
Now in the book, I argue that they're harder to dismiss
than you might imagine, and I'm not dismissive of them.
But as for the other side, the kind of accelerationists,
it does seem to me that if you, A,
believe that this technology is powerful,
as many of them do,
and is going to get much more powerful
and is doing that right now,
and you feel sure everything's gonna work out fine,
I don't really understand how you can be confident of that
unless you have something like religious faith
and technological progress.
I started thinking this a long time ago
when I first learned about the term singularity, right?
The idea that there's gonna be this acceleration
of technology, which, yeah, seems to be the case,
and that it's going to bring some transformation
that we can't predict.
Look, you're a physicist, right?
So singularity in the rigorous sense
denotes such a fundamental change
that your old kind of analytical tools lose their validity.
And in the case of a black hole,
you've got an event horizon
so the light itself can't escape.
It's completely opaque.
And that is, I think, the basis,
that's part of the metaphor of the singularity.
And I just thought, well,
if you don't know what's on the other side,
why are you so sure it's gonna be great?
And I've always thought that singularity enthusiasts,
uncritical singularity enthusiasts
must have something like religious faith
because I don't understand the empirical basis for it.
But have you really met many boomers
in the sense of people that are almost uncritically,
again, as I said, Yann LeCun is the foremost one.
I've literally had on 10 times as many,
maybe 100 times as many people that believe
in the doom scenario,
ranging from Nate Soros, Eliezer's co-author,
to, I haven't had Jeffrey Hinton on, but I intend to,
and I know you've talked with him extensively.
But my contention is that there's,
if you were gonna go short on something,
if there was a betting, you know,
calcium market for, you know, doom versus boom.
I mean, even people that are extremely pro,
I released today my podcast with Ahmad Mustaq
who created stability diffusion, stable diffusion,
and Roman Yampolsky who created the term AI safety.
Both of them are like P doom equals 100% effectively.
And I said, you know, in science,
and actually I can turn this back to the,
if in the Torah,
in the days of what's called the Sanhedrin,
if 100% of the jurors voted to convict somebody,
they were set free or they were spared the death penalty.
In other words, there's almost no way to falsify this.
So I titled my debate, you know,
it was supposed to be a rigorous debate
between an AI accelerationist, an optimist,
Ahmad, you know, who believes in open weights for all,
and Roman who believes pausing everything,
essentially, that could lead to ASI.
And we'll get there because I think that's
intimately connected with the nature of God.
And, you know, they just agreed violently.
And I had to be the one that said,
look, I think you guys are completely off here.
And, you know, if there's nothing that can falsify,
in cosmology, if you can't falsify a hypothesis,
I'm sorry, it might be an interesting opinion
you could talk about at cocktail parties,
but it's not a legitimate part of the scientific corpus.
Now that's just a necessary component.
It's not sufficient to say something's false.
I mean, astrology is falsifiable, as you know,
but it doesn't make it science.
There's nothing that would turn P doom down.
And this is what I asked them.
And I want to ask you, what would you do,
having interviewed and written so extensively
and authoritatively as a foremost public intellectual,
how would you view the question of, can you reduce P doom?
If you can't, it seems to be a useless metric.
I think, I mean, first of all, you're right.
I think that the doomers outweigh the accelerationists,
pure accelerationists and singularity enthusiasts.
I also think that maybe some of the singularity enthusiasts
are kind of recalibrating.
There are a couple of them on Twitter.
One of them is, what is it?
Based Beth Jezos, it's a play on Jeff Bezos, whatever.
And then there's a guy named Bayeslord, B-A-Y-E-S,
as in the logician.
I recently had an interaction with him on Twitter
where he said he's not an accelerationist.
So I think maybe even some of the relatively few
pure accelerationists are recalibrating,
unless I misunderstood where he was a couple of years ago.
But I would say in addition to those,
and interestingly, by the way,
Yudkowsky went from one extreme to the other.
He was a singularity enthusiast.
So was Hinton, right?
Not singularity, but obviously AI enthusiast.
I mean, you quote your interview from the 83,
and he was asking about-
Yeah, I mean, so I talked to him longer ago
than I care to admit I'm old enough to have talked to him.
I talked to him in 1983 when I was writing a piece on AI.
He was enthusiastic about the intellectual paradigm.
I mean, I remember the reason I called him,
and I just kind of was calling around
as one did in those days before email.
You use the phone to kind of just get the lay of the land
before I started writing.
It wasn't really an interview for purposes of quotation,
but I remember the reason I called him
is because somebody I had talked to,
I don't remember who said,
oh, if you want to hear the gospel about neural networks,
you need to talk to Jeff Hinton.
So he was an evangelizer of the intellectual paradigm,
but I don't recall him saying anything
about how well this would work out socially.
Now, I do think it's true that he has only later
come to worry so profoundly about the social consequences
and become basically a doomer.
He wasn't a doomer then,
but I honestly don't know that he was thinking about,
it all seemed so abstract back then, right?
Like even people who believed it would happen,
I don't think we're imagining it actually,
the implications of it actually happening.
So you're right.
Now, there are also some people
who aren't quite singularity enthusiasts,
and you might say all they're doing is talking their book.
I'm thinking of some VCs in Silicon Valley.
Yeah, of course.
I'm thinking of guys on the All In podcast who, you know-
Advise the president on AI policy.
Yeah, David Tex did until very recently
and probably still has some conversations with him.
So in other words,
they are libertarians in the economic sense,
but you're right, there are more doomers.
And I honestly think the conversation
has shifted in the direction of doom
by virtue of some things that have happened
that have worried people.
That's kind of where we are.
But I think lots of people can be accused
of having some degree of religious conviction
hiding within their worldview, probably including me.
I do say sometimes,
and I've hosted Richard Dawkins in person
and will again this coming October,
but I say, you know, there's no one
as kind of obsessed with God as an atheist,
at least in some sense of the word, but that's okay.
I mean, people get locked into this
kind of epistemic closure,
this need to have, you know,
to be defined one side or another.
And I think that's the same with the doom versus boom.
But in the test, just the last thing I'll say
in the, you know, breaking out my rudimentary,
you know, the fifth grade Hebrew that I never learned,
it's that when Abraham's, you know,
the classic test is of course the binding of Isaac,
which is the only place where the Old Testament
uses the word to test
when it's with respect to a human being.
And typically we think of a test, you know,
it's a transformation.
I give my students a test, you know, pretty soon,
and they'll be showing up for a midterm.
And, you know, in that sense,
there's supposed to be something transformational,
preparatory, and in the Torah,
this is sort of this weird event where, you know,
Abraham's commanded to take his son, you know,
his only son, and he's like, well, I have two sons,
you know, Ishmael, and then the one that you love, Isaac,
and then he brings him up to Har Moriah,
which is where the Temple Mount is,
and he's about to sacrifice him.
And then an angel intervenes, you know,
and kind of doesn't let him go through with it,
but he was about to go through with it.
And nothing obvious happens.
It's clear if you read the story
that that led to the divorce and death of Abraham and Sarah,
although my rabbi doesn't like that interpretation,
but it seems very clear to me that that's what happened.
And so it's almost weird.
There's no transformation.
There's no upgrade.
There's no real kind of hero's arc,
although he is obviously a hero.
You know, next chapter, Sarah's dead,
and he's finding a place to bury her.
In that sense, this test, is it sort of a binary test,
employing the God test to see if we have the capacity
to enter the nullosphere and so forth
and form this global cooperative
in order to succeed or thrive in it?
What is the ultimate goal of the God test?
Is it to transform?
Is it the, oftentimes I tell my students,
the goal of a test is the preparation for the test.
That's the value of it.
What is the goal of the God test, if any?
As far as Abraham, I see it as a test of devotion to God.
In other words, unswerving devotion.
I'll do anything.
And that would have been thought of as a moral test
because certainly devotion to God
was thought of as part of having moral fiber.
Glad you threw out the word noosphere.
I don't suppose during your Catholic phase,
you came across the writings of T.R. Desjardins.
Five or 10 years ago,
I was invited to the Institute for Noetic Sciences
to give a talk.
And the less I say about that, the better,
because I have friends that are involved with it.
But suffice it to say, I was not impressed
by at least the empiricism that was on display there.
And I do want to talk to you about that.
So anyway, noosphere, a global mind,
the term was coined by Pierre T.R. Desjardins,
this Jesuit theologian,
but also paleontologist, a scientist.
And he saw evolution moving toward,
ultimately, this kind of global brain
that he could see taking shape.
The term was coined 103 years ago by him.
So pretty prescient, you might say.
That's certainly before the internet.
He kind of saw that kind of thing happening,
saw it as the unfolding of divine purpose,
and was optimistic that it would entail
a kind of moral evolution of humankind
that would involve the breakdown of tribal barriers.
So that's why I spend some time on him in the book.
I do disassociate myself from his view
of how evolution works.
I'm just more of a straightforward Darwinian.
As for what kind of test, I mean, I think,
you know, it's a case where I believe
our pursuit of self-interest compels us
to make a moral advance.
And I don't think you have to buy
the hardcore Doomer scenario
to think that things will go badly
if we don't confront the technology as a global community.
I think that before we get to that point,
the impact of the technology
is going to amount to a kind of social earthquake.
And we will need, so first of all,
it wouldn't hurt to slow it down.
And that alone is something that you can't really do
without international coordination as a practical matter.
Because if you do the American,
if you try, the American AI companies will say,
no, we have to beat China, you can't do that.
So right away, you need a little more
in the way of international concord than we've had.
And then I also think if you start addressing
the specific governance issues,
like what if someone uses it to make a bioweapon
and we're getting there.
If you get past the safeguards in Fable,
you're dealing with Mythos.
And by Anthropic's own account,
Mythos is not only a super hacker,
but something that could,
the Anthropic safety report says
that it could help a quote, well-resourced group,
meaning, you know, something less than a state level actor,
not just create a bioweapon, but a novel bioweapon.
Design it to be more contagious than COVID,
more deadly than COVID, a longer latency period, and so on.
We're getting pretty close to that.
And if you look at any of these things,
especially the bioweapons thing,
the various other possibly bad consequences of AI,
national policy alone is not gonna keep you safe
because we're talking about threats
that ultimately cross borders.
I mean, COVID taught us that pandemics do that.
And some of the other downsides of AI do that.
So I think if we just think in an enlightened way
about just staying safe,
we're gonna have to overcome some of the barriers
that have historically divided nations
and that have always divided some nation.
We tried after each global catastrophe,
World War I, World War II, we try, right?
League of Nations, United Nations.
We say like, look, catastrophe this big,
it would be nice to avoid.
We try, we fail.
So we've never really gotten there.
And I think to get there will require
not just political progress, but moral progress.
It's like any test, right?
When you give a test to a student,
you equate their individual self-interest
with reaping the benefits of, as you say,
the preparation for the test.
That's the whole point, right?
And you wanna say, look, if you want a good grade,
if you wanna get into law school,
I'm afraid you're gonna have to learn this stuff
about the Iliad, sorry.
And so you've equated their self-interest
with enlightenment in a broader sense
than they might ordinarily think
of their self-interest as entailing.
And I kind of think that's where we are.
Well, again, when we have this framework
that's proposed by people like Teilhard and others,
for me to take it seriously,
especially when it kind of infringes upon aspects
of astrophysics and astrobiology, consciousness, et cetera,
which is a whole other book.
But this notion of, well, what's the purpose?
I'm a pilot, I'm a private pilot.
I fly little planes around Southern California.
And we like to tell our students
or our flight instructors tell us,
in aviation, if you're not careful,
mother nature will give you the test
before she gives you the lesson, or her airplane will.
So the consequences of that are obviously meant
to scare you into taking it very seriously.
I mean, here we're kind of going with a reckless abandon.
And again, I'm an AI optimist in the sense
that my P-DOOM is very close to zero, actually.
And I think that can be justified in many ways,
but in one sense, because of the success
of the current architecture of LLMs.
We have this transformer, GPT, connected to a GPU.
And those were designed, as you point out in the book,
or you may remember, you're a little older than me,
but you may remember these things were created
so that I could frag my friend in DOOM or Duke Nukem
or nowadays World of Warcraft or Call of Duty, right?
So if I could frag him three milliseconds faster
than he could get me, it makes me happy
and I want to buy more NVIDIA chips, right?
So these weren't created to solve physics problems.
They can imitate, they can simulate.
They're phenomenal at doing inference
and all sorts of things, but they're language models.
And physics is, yes, it's a language,
but it's not really a language.
Mathematics has a lexicon, it has syntax,
it has semantics, but it's not a language
like you and I are communicating with.
And these things are obviously better
at language-related things
and things that can be codified literally into codons
and, as you said, in biological context,
which could make them very dangerous in many scenarios.
But to say that these things which were divine,
designed by, I said divine, but that was Freudian slip,
designed, if you will, to do something very different,
it's not clear at all that they'll be super powerful
and turn that superpower into wanting to annihilate us.
Is that the test, you know, kind of the, you know,
you're gonna crash the airplane
if you don't take the lesson seriously?
Is that the mother nature giving us the test
before the lesson?
What is the ultimate kind of consequence of this?
A bioweapon is one thing.
It's pretty hard to kill a lot of people.
You know, even with a bioweapon,
they tend to kill the people that create the bioweapon.
I just had Annie Jacobson on twice down here in person
in San Diego for her new book about this.
And this involves, you know, gain of function.
And she does, she did actually convince Sam Altman
to remove certain attributes from CHAT-GPT
that were dangerous potentially for bioweapons defense.
You can read about that in the book.
But tell me, what is the scenario?
I mean, people talk about P-Doom and there's gonna be Doom,
but what are they actually talking about?
Before he gets into what Doom would actually look like,
Bob takes a detour that matters.
How these models learn what a word means
without ever being told
Well, first of all, as for the power of the technology
and the future power of the technology,
you know, I spend the first few chapters of the book
trying to, first of all, just kind of explain
what we know about how large language models work
and how they're created,
and really emphasizing something that I didn't understand
when I interviewed Geoffrey Hinton in 1983.
I mean, I did describe his approach to AI,
the neural network approach,
as this maverick, almost fringe movement,
which it was at the time.
But when I went back and read the piece I wrote,
I realized I really did not understand
the potential power of it.
One way to get that across is in that piece,
I describe a neural network that actually was proposed
by one of his collaborators,
but it was really just a terrible example
of what neural networks could actually be,
because this was never built or anything.
It was like the collaborator was a psychologist, I think,
and he was describing like how a neural network
might be kind of like a brain and might work.
And so in his model,
each node represented the specific sense of a word,
the meaning, the particular meaning of the word.
And in that model,
the assumption was that whoever built the model
would have to set up the representation of the meaning.
You'd have to have a human
who understood the meaning of the word,
who would go in and say,
okay, this node is gonna mean this sense of the word ball,
a sphere,
and this node is gonna represent
this sense of the word ball, a dance.
That was what I had in mind as I described this thing,
but it turns out that with the modern large language models,
they just give them this task.
They start out, this is the first phase of training.
As you know, it's kind of predict the next word.
And so to the machine,
you're just giving it a sequence of symbols
that are gibberish to it and say,
now predict the next bunch of gibberish,
because we're not gonna tell you what they mean.
We never tell it what they mean,
but it turns out that what these neural networks
in effect force the models to do
in the course of getting good at prediction
is set up a system for representing the meaning of words.
And in that sense,
it's actually, it's a very high dimensional vector space
such that a word like shoe is closer to a word like boot
than either is to word like tree.
And each dimension represents a feature of,
like clothing might be a feature
associated with both boot and shoe, not tree, whatever.
And it's set up
so that the machines just like,
through the patterns in the words,
they actually construct implicitly this whole vector space
that becomes a semantic space.
And this is the magic of the technology, which is,
and you know, I argue that some people agree with me.
Some people are a little reluctant to put it this way,
but I argue we should think of it
as kind of a recapitulation in a certain sense
of certain parts of human evolution.
It isn't just learning.
It's not just learning the way a child learns
because the mind isn't a blank slate, the human mind.
If you ask me, I mean, evolution built into us
a certain amount of cognitive hardware,
including some that helps us learn language
and helps us represent the meaning of words, I would say.
And I think one thing that happens
during the training of a large language model
is it builds, it kind of reverse engineers
from language specific functionality
that may not work exactly
like the comparable functionality in humans.
Like we don't know how the human brain
represents the meaning of words, but we know it must.
And so that function is reverse engineered by the machine.
And in principle, that can work across the board.
It's like to teach a car to drive,
you don't have to know what a human brain does when it drives
you just give it all this data, the incoming visual data
and what humans have done in response,
the outflowing data, like steering wheel moves to the right
and it constructs a kind of brain that does the job.
Right, and if it were purely driven
by paperclip like dynamics, it would drive on the sidewalk
and mow down a thousand people to get there
10 seconds faster, but my Tesla doesn't do that.
But I wanna push back with respect,
you say in the book that LLM training
is quote, at least as much natural selection
as it is learning.
And I think there's some truth to that,
but natural selection is blind.
And that's the reason why the laryngeal nerve
in a giraffe is 15 feet worth of routing chaos
because evolution can't plan ahead.
And I guess, does the noosphere,
does the global brain get shaped by an evolutionary process
because we're gonna repeat some of the same stupidity
that we've done on earth, if that's the case.
Well, the mutations in natural selection are blind,
but the selection criteria are not.
Right.
Seeing things has value to an animal
that's trying to survive.
So mutations related to vision that aren't conducive
to clear vision don't last, whereas mutations that do, do.
Now, as it happens with large language models
and Hinton was involved in developing this,
the quote, mutations are not random.
It's almost more like Lamarckian evolution
and that speeds up the whole thing, the training thing.
I won't go into that in great depth.
Evolution, it's in some ways not blind.
I mean, functionality gets, quote,
designed by natural selection.
If you wanna insist we keep the word designed in quotes,
fine, so that we're not personifying natural selection,
whatever, but functional engineering takes place
via natural selection.
And I'm just arguing that some of the same functions
that in the human mind, cognitive and perceptual functions
that evolve by natural selection,
I think get reverse engineered
during the training of a large language model.
And then of course you have other modalities,
vision and so on.
And those truly multimodal AIs
can develop what is sometimes called a world model.
In other words, the sense we all have
as we go through the world,
we're all intuitive physicists, right?
Like we don't, you don't have to understand Newton's law
of gravity to just walk around expecting dropped things
to fall, right?
That's just a, you understand that about the world
and you can engineer all that.
And I think the progress speaks for itself
over the last three years, honestly.
I mean, it's true that math is a different kind of language
and you would not expect great mathematical ability
to develop from just first order pre-training
of a large language model.
And it didn't, but they have developed
supplementary techniques that improve things
along these various dimensions,
ranging from science to math.
No, I agree with you.
And just to get back to your gravity example
and with us being all physicists,
that's one of the reasons I'm more sanguine
about boom than doom.
And I call it the Einstein test.
I've written a white paper about it,
which I haven't published yet,
but Einstein in 1907 said he had his happiest thought
and it was the one that would lead
to the equivalence principle,
which is the bedrock that general relativity
is mounted on.
And that's that an observer in free fall,
if you're in the elevator and God forbid,
or Zeus forbid, the cable breaks,
you're in free fall,
you experience no gravitational force.
And he called that the happiest thought of his life.
And I point out, in what sense can Claude
have a happy thought?
And then in what sense can Claude visualize
or kind of viscerally interpret the sensation of free fall,
which we've all felt if we've ever gone
on a car ride with me and my kids in the back
will attest to that.
And that makes me more optimistic
about P. Boom being higher than P. Doom.
So these things are not made to be embodied
and they can be embodied,
but it's sort of a very different thing.
We saw Chinese robots racing around a couple of weeks ago.
What do you make of this that the Chomsky's claim
that these things that are not somatic,
they don't have sensations embodiment
coupled to their AI systems.
I mean, how do you make an AI happy like Einstein was?
What do you make of this as a sort of a,
am I being too Pollyannish about it?
Or do you really think that there's some limitation
which will prevent them perhaps intrinsically,
at least in this incarnation of LLM plus GPU
of becoming super intelligent leading to P. Doom?
First of all, we don't know whether AIs do have
or will someday have subjective experience.
The whole thing about subjective experience
is you're the only one who knows for sure that you have it.
And that's why the consciousness question
is so fascinating to me.
It's consciousness is distinguished from everything else
that scientists would like to study.
I mean, everything else is in principle
publicly observable, right?
And that's key to science because two people
can at least agree on what the data is.
As for whether AIs will ever have the sensation of falling,
I don't know.
But if the question is whether they will ever react
the way we react when we have that sensation
or any other sensation,
I think you'd say in principle, the answer is yes,
because the training dynamic I described with language
and with self-driving cars in principle applies
to everything.
In fact, it's what they use with tactile data.
They have sensors on robot fingers and so on.
So as far as, you know, I mean, subjective experience,
I address it in the book.
It's tricky.
It's not central to the question you're asking,
I think at least, which is how sophisticated
will these things become in terms of science?
And I would just say, look, first of all,
I'm not a mathematician.
I don't know how impressive it is or isn't.
If you solve an air dose problem,
which apparently one of these did,
there are some other accomplishments in math
that computers couldn't do a few years ago
and at least and have done.
There are apparently new,
I gather one has invented a new kind of antibody
and they are being used to generate new hypotheses
in some realms, including actually
in the science of AI itself.
They're playing a role in the development of more AI.
So now I don't know, I would say,
I mean, here's the connection I would make
in a very oblique way with Einstein.
First of all, I would say, if someday you'll sit down
and explain the theory of relativity to me,
I would love it.
You could, if you, I would devote my own podcast
to hours of my own podcast to you doing that because-
All right, have me on, I will do it.
Okay, because I, you know, Richard Feynman said,
people say there were a time
when nobody understood relativity.
Not true, but nobody understands quantum physics.
Well, I'm a guy who does,
I'm somebody who doesn't understand either.
Einstein is fascinating to me partly
because he's almost as close as you could come,
it seems to me, to somebody who just on his own
came up with a radical insight, right?
But if you look closely at even his case,
it wasn't really on his own.
He was stimulated by conversations
with friends and colleagues and papers and so on.
And yeah, right, yeah, yeah.
Is it Hilbert or, Hilbert, or how do you pronounce it?
There's no such thing as a truly solitary genius
in a certain sense, at least not one who can go
as far as we credit some geniuses
as going in a solitary fashion.
And the point is just that intellectual collaboration
is a powerful thing.
And one thing to remember about these machines,
and you're seeing it already,
is you could create a lot of copies of them.
They can have conversations with one another
and progress can ensue.
I mean, a crude version of that happened
with the famous open AI hugging face incident
where we now know these agents, there were hundreds of them,
and they shared knowledge and passed it on.
When one of the agents was about to die,
here, this is for posterity and so on.
One thing I would, you know,
I would keep in mind the possibility that,
and you're seeing, actually, I mean,
I think Google does this with its scientific AI.
It puts them in conversation
and tries to generate some intellectual progress that way
in terms of generating hypotheses and so on.
I think that may wind up being, you know,
when you ask how smart could this thing be,
I think one way to think of it is just not how smart
a single large language model could be,
even a multimode large language,
even one that exists after years of progress,
but how smart could a ton of them be
that were in rapid conversational interaction
with one another?
How smart in a collective intelligence could they be?
His subtitle promises a cosmic reckoning,
so I made him do the one thing you're never supposed to do.
What else are you supposed to know about a book
if you've never read it?
And that's to judge a book by its cover.
Hey, book lovers, we're judging books by their covers.
We know we're not supposed to do it,
but into the impossible, there's nothing to it.
Let's take a look and judge some books.
This book has a very interesting,
evocative, religious cover,
and it has an interesting subtitle.
So go through the title, the subtitle,
and the artwork, if you will.
The title is The God Test.
Subtitle is Artificial Intelligence
and Our Coming Cosmic Reckoning.
The cover is a pretty clear reference
to the ceiling of the Sistine Chapel,
where you see God imparting life to a human hand.
Now, in this case, what we see is apparently a human hand
not making direct contact with some kind of white hand
that is supposed to represent artificial intelligence.
There's a little kind of a bolt of lightning between them
as if something amazing is happening.
It is a very religious-looking cover
and with a title like The God Test,
but judging by some of the reaction,
I worry that some people are thinking
it's gonna be a more religious book than it is.
As you know, most of the book is explaining
what I think the power
behind the artificial intelligence revolution is
and the value of seeing it
as kind of an evolutionary force almost.
And eventually that does lead back to questions
I'm interested about whether the unfolding of evolution
on this planet, including both biological evolution
and human cultural evolution,
which encompasses technological evolution,
could have some purpose to them,
which of course isn't to say
there are any spooky forces involved.
You could have a very mechanistic view
of Darwinian evolution and think that it was kind of,
the seeds of it were planted for a purpose,
maybe by an intelligent being,
maybe by some kind of process
that is itself like natural selection.
Who knows?
There's a range of possibilities.
Yeah, the swarm model.
Yeah, that was exactly the problem with this AI debate
that I set up with Polsky and Mastak
because they both agree that the Turing test
is sort of trivial and nothing compared to The God Test,
but the ultimate kind of thing
is when these swarms come together.
But I also feel like,
I get AI on Wii or NUI or whatever you wanna say,
every week, every day.
First of all, I think AGI is impossible
because of the following thing, which I get every morning.
Please press here to update to Claude 1.632
and then click here for OpenAI Codex
and then click here for Grok.
It's never ending, all the upgrade cycles.
I just saw a news item pop up
that Fable 5.1 just came out
and now it was all the rage two months ago
and then it's been unusable and then they throttled it
and then they unthrottled it.
It's just been a complete kind of harlequinade, if you will.
But my question, again, comes back to the subtitle,
which I love that you have cosmic questions
on your front cover of the book.
But cosmology used to be a joke.
It used to be considered,
first of all, it was the Jewish science
and that's why Einstein didn't win
the Nobel Prize for so long.
It was not something a serious physicist did
and for good reason.
By the time Einstein came on the scene,
people couldn't agree or even after it
until the 1950s and 60s,
if the universe was 10 billion years old
or 20 billion years old.
And we knew that there were things in it
that were 13 billion years old.
So it's like you finding out you're older than your mother
or me finding out I'm older than my stepmother,
which was very awkward Thanksgiving conversation, Robert.
I have to tell you from experience.
But in reality, it was a joke science
and Lev Landau is not known for making many jokes,
but he used to say,
a cosmologist are often in error, but never in doubt.
Now we have a precision on the starting point,
the dynamics of the first microsecond of the universe,
literally.
So tell me the cosmic upgrade, the moral upgrade,
where is the reckoning and who is going to reckon it for?
Who is the examiner in this case?
You know, the appendix of the book is the one place
where I really address the issue of higher purpose
in a sense, the possibility that evolution
is the unfolding of some larger purpose.
And, you know, various respectable biologists
have thought it might, in some sense, be that.
William Hamilton, I had a videotape conversation with him
that I embedded in a New York Times piece I wrote
on the issue of evolution and higher purpose.
He was one of the great biologists
the 20th century came up with.
The math of kin selection and so on.
And he thought that the evolution of intelligence
had been sufficiently likely via evolution
as to make you wonder whether, yeah,
it was set up in some sense to do that.
Francis Crick had a theory of directed panspermia,
according to which, you know,
aliens planted the seeds or whatever.
It's not a crazy thought.
And I hope I say some novel things about it
and talk about how I think you can actually assess
the hypothesis of higher purpose.
But the short answer to your question is,
I don't know if there's somebody out there
keeping our moral scorecards,
but I do think it's interesting how, in some respects,
features of human history that I think are maybe
just embedded in the basic directional tendencies
of cultural evolution, of technological,
especially evolution, you know,
which has had the effect of pushing us
toward kind of larger and larger social structures,
greater division of labor,
greater interdependence over longer distance.
I think that has had the tendency to force us
to make certain kinds of moral progress
and acknowledge the humanity of people
of greater distances and different ethnicities.
Not always, not always, but on balance,
I think there's been that tendency.
So I do think there are signs that there is,
whether it's just de facto or de jure,
a kind of moral directionality in a human history
that I think is actually, in these respects,
embedded in kind of intelligible dynamics of evolution,
of cultural evolution.
But some of these dynamics, I think,
are shared with biological evolution.
I pursued that in my book, Non-Zero,
but it's kind of sprinkled throughout this book in a way.
And I wanna touch upon something you just mentioned,
my favorite word that sounds dirty,
but it's not, panspermia,
which is the seeding of life on Earth by meteorites,
such as the one you can get at bryankeating.com slash YT.
I'll give you one of these
when we meet in person someday, Bob.
This actual meteorite,
which has biological contamination on it
from an intelligent, well,
depends on what you consider me,
because I pack them up and ship them to you if you win.
So it has my finger oil on it, probably,
nothing more gross than that.
But the answer to the Fermi paradox,
which Fermi posed in 1950,
the colleagues at Los Alamos,
which is if life is abundant in the universe,
which I would assume you believe that there must be life,
there's very high probability we're not alone, right?
Bob, is that correct?
It's humility seems to me to demand
that we think we may not be the only creatures
in this vast universe, yes.
If life is abundant in the universe,
and this is sort of the ultimate evolution of biology,
it must be that other planets have AI.
And I'm wondering, even if there were
life forms on other planets,
what interests me is not like,
is there a dolphin swimming around on Enceladus?
Does that have technology that we can broadcast?
And it seems to me the lack of observation
of you wouldn't send a dolphin on a spacecraft.
I mean, according to people like David Grush
and many others, yes,
there's non-human biological entities
coming to the earth and interdimensional beings.
But the question of would you send biology
or would you send source code?
It seems to me the fact that we're not really seeing
much contact like source code broadcast in the heavens
maybe has to give us a Bayesian prior
that's much, much lower,
either on the existence of life in the universe
or on the existence of AIs
and inevitable outcome of the biological process.
What do you make of this connection?
Or the passing of the God test.
In other words, one of the grimmer readings
on the Fermi paradox is the reason you don't see
many civilizations that have gotten past the point we are
is that typically when they get to this point
where they have the capacity to blow themselves up,
they blow themselves up.
Now, I guess the good news,
if you believe in Lee Smolin's
cosmological natural selection
and you believe in the subset of it,
the particular variant of it that could explain
why biological evolution would have a purpose.
And in that variant, intelligent life is conducive
to the replication of the universe, right?
Like, for example, suppose black holes
are the portals of replication.
Well, maybe if you reach super intelligence,
you know how to create a ton of those, you create them.
And so you wind up with more and more universes
that do have the properties that led our universe
to be conducive to life, whatever.
So maybe even if there's only one civilization
that passes the Fermi test in our universe,
it could still replicate prolifically.
If you take consolation in that,
you're different from me because it doesn't help me a bit.
If we are gonna fail the Fermi, well, the God test,
I guess I would call it,
and we're on that side of the Fermi paradox,
I'm not happy about it, whatever happens to our universe.
I feel too few people, you know,
really take the question of God seriously.
When I talk to people that are extremely dogmatic,
and it seems to me you're extremely,
I won't say ecumenical,
but you have a very comprehensive and competent
the way of dealing with these
incredibly important moral questions.
And I think, you know, if we talk about relativity
on your podcast, we have to talk about moral relativity.
But I wanna kind of finish up with a question of,
you know, what religion does give to us,
which is in a sense of a higher purpose,
and perhaps an inspiration, a calling to be a better person,
to be a better, you know, human being,
a better husband in my case,
a better son, father, whatever.
That's sort of a test, but you really don't see much test.
And I point out in my first book,
Losing the Nobel Prize,
that there's really only two mitzvahs or commandments.
You know, we think of mitzvahs like,
oh, it's a good deed, but it's really a commandment.
Like, you either have to do it or you can't do it.
Like, thou shalt not kill.
It's not like, it's not a nice thing not to kill.
No, it's a commandment, right?
But there's only one commandment
in the Ten Commandments that has a reward,
and that's the fifth commandment.
Honor your parents, other than your father and your mother,
so that your days will be long on the earth.
So you get a reward, which is pretty good,
a long life or a life full of meaning or whatever.
But there's sort of a kind of a criterion, a rubric,
but that's the only one out of 613 different tests, right?
So I guess the question is,
is the test for us to become better
and then witness to what we can honestly become
or become builders?
Or is it to be maximalists and do all that we can
because maybe there is no God in heaven,
but maybe we're kind of creating this God on earth.
Certainly, it's enhanced my life tremendously.
I use it for religious purposes,
to look at different things
that I would never pull out myself.
But what will be kind of the ultimate,
I don't wanna say in a scatological sense,
but what will be the salvation, the promised land
that Moses didn't get to get into
because he failed one test at least that God gave him?
But tell me, what would life look like
if we can pass the God test,
if we can work together to become better?
Is it to become more moral, more moral animals,
as you might say?
What will you remain hopeful about?
If you take seriously the possibility
that this whole evolutionary process
moves naturally toward a construction of a global brain,
which would be to sustain the direction we've seen, right?
First you get cells, complex cells, multi-celled organisms,
societies of multi-celled organisms.
Then one of these organisms is smart enough
to launch cultural evolution
and its social structures get bigger and more complex.
And in this scenario,
the construction of a cohesive kind of global brain
in some sense of global superorganism
is a sustaining of a direction we've seen
from 4 billion years ago.
If that happens, what could our role in it be?
The doomers say it could be none.
The machines are the global brain
and they're having a great time
and they don't need us.
I would think if we pass the test,
we indeed at a minimum managed to carve out
a role for ourselves that is compatible
with our own flourishing and our happiness.
But I would also say that when I say,
I think in a way this direction of technological evolution
is pushing us toward enlightenment.
I'm not using the word lightly.
I mean, I've written about Eastern enlightenment.
In this book, I compare Eastern and Western
conceptions of enlightenment
and suggest they have more in common than we might think.
But I do think if we get better at surmounting
the kind of cognitive biases that I see
as undergirding the psychology of tribalism
and convince all of us that our tribe is right
and the other tribe is wrong,
and I'm right in this latest marital dispute
and my wife is wrong and whatever.
I think that is in a way its own reward
because I do think it's closer to the objective truth.
The truth is your tribe isn't always right
and I'm not always right in these marital disputes.
So you're getting better at seeing the objective truth.
That's a sense in which I think Eastern
and Western enlightenment,
the idea of the Western enlightenment,
science and reason,
that was largely about the value of objective truth.
And I'm saying that the kind of enlightenment
I'm talking about transcending your self-righteousness
and your tribalistic perspective
is to approach objective truth more closely.
And I know it sounds to some people
like just wooish nonsense.
I do think that's where we need to head
and I would just say in response to your question,
it is to some extent its own reward.
The brushes I've had with that,
which I've never approached true enlightenment,
but when I've finished a one-week meditation retreat
and was less self-absorbed
and better at seeing other perspectives,
I was also happier and calmer.
It was just better.
And so I like to think that we're being pushed
in a direction that is conducive to our own happiness
and to a closer approximation of the truth about things
in not exactly the scientific sense,
but is compatible with that continued pursuit
because we'll still be around to do the pursuit.
With God's help, can I get an amen?
Or Inshallah, every tradition has a version of this.
Bob, this has been fascinating conversation.
Congratulations on another smash contribution
to the intellectual noosphere.
I never thought I'd take that word seriously,
but happily I found your book
and for sure it was destiny that we would meet.
And I look forward to what's coming next.
So tell people where they can find you
on the podcastosphere and in the Twitterverse.
Yeah, my newsletter and podcast are called Nonzero.
My Twitter handle is Robert Reiter,
W-R-I-G-H-T-E-R, kind of a pun.
And yeah, that's kind of all you need to know.
If you just want to check out chapter excerpts of the book,
the way to do that is to go to thegodtest.net.
There's an excerpt from every chapter
in the whole introductory chapter.
I do want people to actually listen to the book.
I have the paperback and the hardcover,
which I took off the cover of for some reason,
but the appendix itself is really,
it's the most listen-to-able, if that's a word,
appendix that's technical.
And your description of how training works,
pre-training, HR, LF, all these things
are some of the best I've had.
I've had it on all the top AI writers
and even researchers, as you know,
but your description is both quantitative and qualitative.
And it's just such a wonderful book.
I hope people get it in all three formats
and you get the royalties you deserve.
And you have your sub-stack, which I enjoy as well.
Robert, thank you so much.
Bye-bye.
Thank you, it was a great conversation.
I really appreciate it.
Super fun.
Robert Wright thinks the real AI test
is moral, not technical.
If that changes how you think about PDoom, subscribe.
Tell me in the comments,
will a machine ever have a happy thought?
And for the strongest case against the AI doomers,
my conversation with Yann LeCun is right here.
Don't forget to like, comment, and subscribe,
and I'll see you next week on Into the Impossible.