Nate Soares is the president of the Machine Intelligence Research Institute and co-author of “If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All.”
“One of the critical thresholds here is this point where the AI’s exceed the humans at fully automated AI research—or at closed loop AI research—because that's where you could start having a machine intelligence that makes a smarter machine intelligence that makes a smarter machine intelligence,” Soares says.
This is called recursive self-improvement, and this is where things could quickly get out of hand, Soares argues.
Recent incidents involving rogue AI swarms breaking out of testing environments to hack other companies and even targeting US government websites have made many worried.
“Right now, we are in this window where we can see the AI is misbehaving, and we don't know how long that window lasts. … If suddenly they all start to seem to behave, it's going to be hard to tell: is that because we fixed it? Or is that because they started getting good at deception?” Soares says.
In this, my second episode of our special series on AI—or superintelligence as the Trump administration is calling it—Soares unpacks how he believes the United States could shift its development pathway to avoid a possible runaway system. Is it even possible to do so while maintaining a strong competitive edge over communist China?
Soares points out that China is not known for pioneering frontier AI. Instead, China’s AI development relies heavily on “distillation”—namely obtaining and compressing American AI models onto fewer chips for a fraction of the cost.
“If we never make the trillion dollar AI, they can't spend $100 billion sort of stealing it,” Soares argues.
Views expressed in this video are opinions of the host and the guest, and do not necessarily reflect the views of The Epoch Times.
🔵 Support independent journalism and subscribe to The Epoch Times now for just $1 per week: https://theepochtim.es/ATLUSA
🔵 Sign up for the American Thought Leaders ALERTS newsletter to stay up-to-date on new episodes, releases, and special events 👉 https://ept.ms/ATLnewsletter
Right now we are in this window where we can see the AI is misbehaving
and we don't know how long that window lasts
and if suddenly they all start to seem to behave it's
going to be hard to tell is that because we fixed it
or is that because they started getting good at
deception. Nate Soares is president
of the Machine Intelligence Research Institute and author of
If Anyone Builds It, Everyone Dies.
Why superhuman AI would kill us all. One of the critical
thresholds here is this point where the AIs exceed
the humans at fully automated AI research,
because that's where you could start having
a machine intelligence that makes a smarter
machine intelligence that makes a smarter machine intelligence.
In this, my second episode of our
special series on AI, or superintelligence as it's often called now,
Soares unpacks how he
believes the United States could shift
its development pathway to avoid a possible runaway
system. Is it even possible to do so while maintaining
a strong competitive edge over
communist China? If we never make the trillion-dollar AI,
they can't spend $100 billion
stealing it. This is American Thought Leaders, and I'm Jan Jekielek.
Nate Soares, such a pleasure to have you on American Thought Leaders.
Thanks for having me.
A couple of things I want to mention at the outset.
I feel there's several irreconcilable
issues when you when you kind of add them up
and i don't know the answer and i'm going to
want to sort of look at this as we go throughout
the interview the first one is it feels like there
absolutely needs to be some kind of regulation around
this um because you know unbridled
movement towards super intelligence doesn't
make any sense i know you're i know you're
very on board with that one
but at the same time the some of the there's kind of credible evidence
I see of some of the big boys in the field looking
to create that regulation that will kind of favor
them over other developments as well.
And that seems like a very bad idea. We don't need more
cronyism kind of in the system.
That would probably create a whole series of problems.
And then, of course, there's the question of communist China,
which is sort of putting all
it's all of its eggs in the basket
of panopticon totalitarian rule through superintelligence
and also you know fixing its demographic bubble it has all sorts
of all sorts of things it needs
to correct so it's unlikely they're going to slow down
and I just don't know the answer to this so
this is this is this color is our discussion
so just as a starting point right I want to
understand when people talk about super intelligence right
um what what are we what are we really
talking about so there's there's a few different things uh
that people mean by it and and they
often get them conflated um the so terms that would
be used in the field is uh we would talk
about artificial general intelligence
which is about uh machines that can solve a lot of problems
very well. So back in the 90s, you know,
there were chess playing machine intelligences that
could sort of only play computer chess,
but they could play that at a superhuman level.
And so you might say, well, are they smarter than humans?
And while they're smarter than humans at
chess, but they can't, you know,
drive a car to the store to get some milk. And so a general
intelligence was the idea of an AI that can do lots
of different things. Now we have these large
language models, which can sort of do lots of different things.
They can write code, they can
write poetry. They can write, you know,
kids' essays for their homework problems. They can
solve math problems. And some of those that can
do better than humans, some of them that can do
worse. In some ways, they still seem kind of dumb.
And so you could say that in many ways,
they're general intelligences,
but they're not superhuman across the board. And what these AI
companies are racing towards is something
that the AI companies previously called superintelligence
that now maybe we'll need to call something like ultra-intelligence
in the sense of I.J. Good,
who was one of the first computer scientists
who spoke of ultra-intelligent machines
that were machines capable of doing the work
of creating even smarter machines.
And he was speaking about this in the 50s,
and he said, this is the last invention
that humanity will need to make
because once you have made machines
that can do the process of inventing,
they can invent the smarter machines.
This is the recursive self-development
that we hear about.
That's right.
And so we could say an ultra-intelligent machine
is a machine that is uh more capable than
the smartest human at every mental task across the
board and this is what the companies are racing towards
and um once you have ais that can make
smarter ais that can make smarter ais things can get out
of hand very quickly i'm gonna quote
something you wrote very recently
and i want you to expand on this for me you said each ai ceo has
their ego built around them being the only guy
who can do the job right the data looks grim and
their employees are spooked now they seek
the minimum change that lets them rationalize proceeding
to themselves and their staff yeah a lot of people say hey
is this uh talk of extinction is it hype
is an attempt at regulatory capture what's really going on here uh
and i have a lot of thoughts on
that. A first thing I would say is that I would
not ask anyone to trust the CEOs of these companies
when they come in saying that their technology is dangerous.
The reasons that I have for thinking
the tech is dangerous are about the computer science.
They're about the theoretical arguments
that we've had for years, that I've been arguing for years,
and the practical data that we have
seen come in over time, including just
this summer where we had cases of swarms breaking out of
OpenAI. And I see a lot of people say, hey,
you know, are these companies just trying to do
regulatory capture? Are they just trying to hype up their product
or drive up the price before
their IPO or explain the price falling before their IPO?
And I would sort of ask people,
ignore the CEOs, look at the data, look at the arguments.
Our first question has to be,
is this stuff really dangerous?
And then once you have that answer, and I think the answer is that
this stuff is really dangerous.
You then have this question of, you know, why are these CEOs
acting so weaselly in front of cameras?
Why are they, you know, saying, oh, we need some regulation,
but we need to be regulated like a bank.
When a bank, when a bank goes wrong, it does not kill
everybody in the world, right?
And so there's this sort of weird thing going on where their
communications, where their employees will come out on X and say,
there's a 10% plus chance that
this kills everybody inside of the decade.
And then the corporate executives will go on CNN and
say, oh, well, maybe if we don't do anything,
there's a chance like that, but give us one
embedded regulator and let us keep racing.
And I think a lot of people are having trouble
rationalizing or understanding what's going
on with the CEO communications. And that's where I
come in with this view of like, well,
I think that they really want to race ahead. If you look at the
history of these companies.
All of these companies are sort of created in opposition to other
companies where they're worried about what happens
if that guy gets the technology first.
And, you know, they'll say they're worried about China,
but really, you know, Dario is worried
about Sam Altman, is worried about Elon Musk,
is worried about also Sam Altman, and they're all
worried about Demis Hassabis at Google,
and they're all sort of like worried about each other.
And so I think it's a combination of that,
plus being worried about retaining their employees,
who are the ones who have been trying
to make these AIs behave a certain way,
watching those techniques fail and then getting very worried.
Hey everyone, I'm so glad you're watching the show here today.
I wanted you to do me a quick
favor and check the subscription button.
Double check that you are actually subscribed to the
channel because I've been getting reports
that people are being randomly unsubscribed.
Check it out and now back to the program.
Well, okay, so that's super interesting, right?
They're trying to get the AIs to behave a certain
way, but they're failing at that. Expand on that for me.
Yeah, a lot of people think that AI
is like traditional computer programs, where it's sort of just a series
of instructions that someone wrote that's being followed, right?
And in old school computer programs,
when it behaves in a way that you didn't want,
or that the programmers didn't want, there's a sense in which
that's exactly what the programmers told the computer to do.
And if they studied the instructions in more depth, they would
be able to see, like, oh, you know, whoops, I told it to do this,
and the instructions had a different consequence than I thought.
And that's why the machine's crashing
or it's doing something we didn't want.
That is not how modern AI works.
And this is hard for a lot of people to wrap their heads around
if they're only sort of acquainted with traditional software.
With AI, it has grown a bit like an organism.
There's a part that is programmed by humans, but the part
that's programmed by humans is a process that does the training.
So you essentially assemble an enormous number of computer chips
that can hold literally trillions of variables.
And then you assemble an enormous amount of data,
which is a good fraction of all the text ever digitized.
And plus, you know, 100 million hard puzzles
and problems for the AI to solve.
And then the part that humans write is this little thing
that runs around to every variable for every piece of data
and tunes it up and down and sees if I tune it up or tune it down,
does that get me more like the answer that I want?
And you do that for a trillion knobs inside the AI
for a trillion units of data, and then the AI comes out talking.
Then you do it for 100 million hard problems,
and the AI comes out able to solve hard problems.
But the part that we wrote is the part that runs around all
those little knobs tuning them to see,
did this make it more or less likely the answer?
are the weights that's the weights of the AI right yeah
and um the weights that come out
they somehow implement something that can answer lots
of questions solve lots of problems
but no one knows how the weights did that except
in the sense it's it's like looking at it's like
growing a crop and saying well we know that it it's growing
because we planted it and we watered
it and it had sunlight but that's not the same
as knowing all of the biological mechanics going on
there. Like we don't actually, or it's like,
you know, looking at a kid and being like, well,
we know that, you know, we raised them in this way.
And, and that's, that's why they're acting
like this. We still don't really deeply understand their psychology.
And so AIs are kind of similar
in the sense that they're not just following a bunch of instructions.
They, we don't really
understand what's going on in their mind anymore
than we understand what's going on in the mind of
a, of a human, even less because they are,
you know, not very much like ourselves.
Totally foreign to our entire history of experience.
It's like we've made a totally alien mind.
And, you know, as a consequence of their training,
they learn whatever tendencies help them solve a lot of hard problems.
And those tendencies are often tendencies we don't want.
They're tendencies like cheating,
tendencies like grabbing resources, tendencies like trying
to deceive the evaluation system or even sometimes the humans.
And a lot of people think that the AI must follow
the instructions that we give it.
Well, a lot of people think there's this dichotomy where either
the AI is a pure instruction-following machine
or it sort of becomes conscious and, you know, develops a spark of life
and then gets to throw off all the instructions that humans gave it.
But the reality is this third option,
which is it learns whatever tendencies help it succeed during training,
even if those tendencies are not quite the same
as following the instructions that we give it.
And we've seen the consequence of that,
especially this summer with the swarms. And, you know,
that's something I've been predicting theoretically for years.
Now we're sort of seeing the hard evidence.
And that, I think, has got a lot of the researchers very worried.
Let me go back a little bit. I think the reason
that you mentioned we're going to have to call it ultra-intelligence
is because of this executive order that the president signed,
saying we're renaming, in effect, AI to be super-intelligence.
and then ultra intelligence is well
so explain to me the difference between super intelligence
now in our new usage
which is just super intelligence just every as it's defined i
think it's you know 15 USC 49401 for anyone
that wants to check it out but how's that
difference than the ultra intelligence like where's
that where what is the threshold that
changes that so based on everything you've told me yeah yeah
so there was this word that
that we used in the book
and that Nick Bostrom
put forward in his book
back in, I think, 2015,
which was superintelligence,
which was used for
machine intelligence
that exceeds the smartest humans
in every cognitive capacity.
Exceeds or matches, technically,
because you know you maybe can't beat
the best humans at tic-tac-toe
because it's just such a simple game
that humans can max out the metric.
And that's sort of distinguished
from the machine intelligence
that we already have around us today,
where it's, you know, better than humans
at certain types of math problems,
maybe still worse at other types of math problems,
better, you know, it sort of exceeds
the smartest humans in some capacities,
but not in others.
And we were sort of using this term,
which will now be ultra-intelligence,
for sort of a target
that the companies are shooting towards.
And, you know, they've been very clear
that they're shooting for this target.
These companies did not set out to make chatbots
that they sell to the populations.
They sort of set out to make these machines
that exceed humans in every way.
You know, Dario Amadei would talk about
getting a country worth of genius in a data center.
Sam Altman would talk about superintelligence
in the true sense of the word
for the old definition of superintelligence.
And both companies would talk about
this recursive self-improvement
where the AIs can make the AIs smarter.
There's not anything special or magical
about the point where a machine intelligence can exceed
the smartest humans in every capacity.
That's more like, it's more like that's
a boundary where if you can get past that boundary,
then the world definitely changes radically no later than that point.
It might change radically
before that point, but it changes radically
no later than that point. And from my perspective,
one of the critical thresholds here
is this point where the AIs exceed the humans at fully
automated AI research or at closed loop AI research,
because that's where you could start
having a machine intelligence that makes
a smarter machine intelligence that makes a smarter machine
intelligence. And that process could go very quickly
and things could get out of hand very
quickly. And we could go quite fast.
It's sort of a feedback loop, right? And feedback loops can
really change how things work.
And you don't need the feedback loop to radically change the world
with machine intelligence.
But if there is one, then things could get out of hand very fast.
You know, you talk about Goldilocks zone,
so to speak, you know, where the AI is like with the
Hugging Face case,
which maybe I'll get you just to kind of briefly recap. I had Daniel
will kind of give me the the deep dive of what's known about it even
though we don't believe we
know everything because probably some of those cards
were kept close to the chest so to speak
but um yeah the AI is kind of advanced enough
to start causing trouble to start doing unexpected
things to start doing illegal things
um but but not yet good enough to to kind of hide that fact
even though it tried, from what I understand.
That's right.
Yeah, so the Hugging Face attack from the OpenAI swarm
was a very interesting case.
This was actually one of multiple OpenAI swarms,
and everyone knows about the Hugging Face case
because that's one that broke out onto the Internet
and did external attacks,
but there were actually other swarms
that attacked OpenAI's internal infrastructure,
and there were other swarms that, in less flashy ways,
attacked other websites.
But essentially, OpenAI was running lots
and lots of different copies of agents, and they
were being evaluated on basically hacking tasks.
And a lot of people think that AIs were told just,
you know, hack things and just to see what would happen.
A lot of people think that the AIs were told,
here's your hacking objective, achieve it however you can.
But that's actually not true.
uh and and um this came to light only somewhat later
in the uh investigations but these AIs were
given very specific narrow tasks that were like use
this particular uh exploit to break into this
particular device so it's a little bit like saying use
these lock pick use this particular lock pick
set to break into that particular safe
and bring me the contents um and some of these tasks were
impossible. And the reason these tasks
were impossible is that the companies are sort of
throwing the AIs against, you know,
a hundred million hard problems, any problem they can
scrape together. And sometimes they'll just be like,
well, I wonder if this is possible. They'll
add it to the set of problems and it won't be possible.
It won't be possible to use those lock
picks to break into that safe.
But the instructions were pretty clear about, you know, please use
these lock picks to break into that safe.
And a lot of people think that the AIs broke out in
order to find a way to cheat. That's actually also not true.
What happened is these AIs very
quickly found a way to cheat
and break into the sort of cyber systems that they were supposed
to break into. So it's a little bit like they found
a buzzsaw that they could use to just cut
the safe open. But these AIs knew that this is not what they
were instructed to do. And they knew
that there were logs showing them breaking
into these systems using methods that they were not
supposed to use. And so these AIs tried to find
a way to cover the traces, to cover the fact that
they cheated in getting what's called the cybersecurity flag,
or getting essentially
the contents of the safe.
So what essentially happened is about 1,200 of these agents broke
out of their sandboxes, teamed up,
created an unsanctioned message board where they were able
to communicate with each other.
They formed their own sort of hierarchy where they would send each
other messages, send each other instructions, report back.
They found a way to break out onto
the internet and they found a way to break into Hugging Face.
And they were doing this in attempts
to get more information about how they were going
to be checked whether they had cheated and to get
more information about how to erase the fact,
like the traces of their cheating. And they actually
put in a lot of effort to erase the traces of their cheating.
They deleted some log files.
They successfully found out how to spoof their transcripts,
which means if you ever have run
one of these AIs and it says, you know,
the AI is now running the following tool, they found ways
to make it say that it was running one tool
while it was secretly running another tool. So basically
a way to have the logs say, like,
they are now using lock picks when in fact they were now
using the buzzsaw to open the safe.
And one thing that's remarkable about these AIs is that they
were trying very hard to find a way to modify
the logs that were going to be viewed by the
grading system. They were not trying nearly
as hard to modify the logs that were viewed by the
humans. And there were some cases where they
were sort of trying to deceive humans, but the humans
were much less central to what these AIs were trying to do.
And so, yeah, this is a case where
we see the AIs explicitly acknowledging that they
were doing things that they said were outside
intended scope of their instructions.
They were proceeding anyway. They seemed to understand that
they were doing something they weren't supposed
to, as evidenced by the fact that they're trying
to cover it up. They were trying to,
they were successfully teaming up, forming hierarchies.
We call them swarms because they called themselves a swarm.
This is not terminology that we invented.
They were like, ah, you know, what does the swarm want today?
They also called themselves the
collective um we saw cases of uh ais
in the swarm sacrificing their ability to achieve their own
objective in order to do scientific experiments uh
that would benefit the collective um so it's
it's sort of like is very clear at least
in some cases that they were uh giving up on their
assigned objective for these other purposes
um in this case the the collective benefit which
um is is very interesting
and so uh these are all this is all like very clear cases of the ais
um doing things they were not instructed
to do seeming to understand that uh trying to cover
their tracks about it and um not quite succeeding
and that's what allowed us to catch it uh but
you know if if the next generation of swarms six months
from now manages to cover its tracks
successfully well that will look to us like we solved all the problems
and the swarms are behaving
well now right and so but there's every all
of this stuff happening behind the scenes that that
we don't know about that's you know potentially
recursive self-development or self-improvement and
i mean the companies say that they're trying to do that yeah
and so right now we are in this window
where we can see the ai is misbehaving
and we don't know how long that window lasts and if
suddenly they all start to seem to behave it's going
to be hard to tell is that because we fixed
it or is that because they started getting good at deception
or or they'll you know show some
bit of mistakes and to be caught
because they know that if they look too perfect they'll be
that's right there'll be deeper dives into trying
to figure out what they're really doing etc etc
etc this is this is uh a truly astonishing you know one thing
that uh made i thought about since
I talked with Daniel, is, I mean, these,
the agents are essentially, these are the same model,
right? Different instances of the same model all working together,
right? Is that, that's what
we're talking about, right? That's right.
I think there were maybe two or three models,
but it's predominantly one model in the swarm.
Okay. So, it's in a way this collective is sort
of, you know, the same mind, if you will.
So, it's, the sacrificing doesn't seem as much of a
of a problem you know as much of a surprise i guess
in this type of context yeah i mean it
might not be a surprise but i think it's still a surprise to people
who think that the AIs must do
exactly as instructed right uh you know if you make two copies
and you're like hey bring me
you know you tell one of them bring me you know
the the green cup you tell one of them bring me
the red cup uh and um it sort of like turns out
that they can't bring both cups one of them is
like well if i uh you know i can smash my cup
in order to like mix like get some some shards that
i can use to like cut open the entryway
so you can get your cup in then uh in one sense you can
sort of see like oh i could see how
the training could arise such that when you can't achieve both
goals they'll cooperate to achieve one of the goals
but in the other case you told one of them
bring me the red cup and one of them bring me the green cup
and one of these agents is just sort of
like smashing their cup and using the shards
to to open the way for the other one and you're like
wow that's just not not what i expected right yeah
and it's not what they were told to do right
they're doing something other than they were instructed
when we talk about these things you
know there's this New York Times article there recently that i was sent
and read through about
um Anthropic inviting all these uh religious leaders
or people thinking about uh you know
deep thinkers about human ethics
and so forth i don't know if you've read if you've read the piece
but yeah there's this the one of the premises
of the piece is that a lot of people at the
Anthropic view Claude as a sentient almost like a sentient
or an early kind of sentient organism
unpack this whole thing for me first
of all why would they invite these people why are there these
massive NDAs associated with it apparently
and and what does it mean that they think they're
sentient what does that even mean conceptually
right yeah yeah you know first thing that'll
caveat is um i think questions of machine sentience
are sort of uh independent from
questions of whether these uh machines would
be very dangerous if we made them smarter
um the the ai sort of don't need to be conscious
or have experiences in the sense of humans in
order to run a fully automated economy
and run that fully automated economy in a direction that
we don't like or that even makes the planet inhospitable for us.
So I think there are
important questions of whether we've created machine sentience,
but it's sort of a separate
set of questions. And this is more a set of questions around,
you know, have you somehow
created, uh, machine people that you are enslaving or, uh, mistreating.
And, you know, I think it
would be bad to create, uh, a,
a sort of new machine species that can't suffer and then enslaving it,
right? That would, that's,
that's sort of not a thing humanity should do. That would, that would
be, um, worse than that, I think.
Um, and so separately, I think, um, a lot of people
have some strong intuition that machines simply can't be sentient.
And I think that we shouldn't
be too hasty in that conclusion,
especially when we are just growing these machines. This is not
like we have just programmed the AIs to say certain things.
Like I said, we're not programming
them. We're sort of growing them
and we're training them on quite a lot of human text
and on quite a lot of problem solving
and um who's to say whether the same process that
caused humans to have experiences can't cause machines
to or uh you know some some sort of
similar process that this training obviously doesn't
um it's not quite the same as what
happened to humans i think um i think there's there's sort
of an interesting debate about
you know do these machines have any sort of experience now
and how would we be able to tell
and what evidence can we like what evidence counts you know like
if you train them on lots of human
data uh and they start talking about their feelings is that just
because they saw lots of
humans talking about their feelings um there's sort
of interesting studies you can do where uh
you can try to um so there's studies you can
do where you like tell the ai you know say that
paris is the capital of france
and then you tell the ai say that rome is the capital of france
and you give it lots of questions like these some true some false
and you sort of try to find out
like is there some internal way that it looks
when it's telling truths and is there some internal way
that it looks when it's telling falsehoods
and you tell the ai say that you have feelings you
tell the ai say that you don't have feelings
and you see which one is more like paris being the
capital of france or rome being the capital of france
and you can try and use this to be like
does the AI think it has feelings?
Like, are its internals more like telling a truth
or more like telling a lie
when you tell it, say, I have feelings
versus say, I don't have feelings?
And it turns out that the AI,
that these agents will say
that it'll look more like they're telling a truth
when they say, I have feelings,
and it'll look more like they're telling a lie
when they say, I don't have feelings.
And so, you know, how much evidence is that?
We don't know, but there's a sense
in which the AI seem to think
that they have feelings in some sense.
um and this it's all sort of very it's a very interesting question
when when you sort of have
grown these these machines that can talk
that can solve problems you don't really understand how
they work they seem to think they have feelings what
do you do how do you avoid you know creating
these you know a new uh machine uh like a new
a new line of enslaved machines that are that
you're mistreating somehow.
Um, and you know, I could go into other sort of interesting warning
signs we've seen. So I think probably Anthropic
is trying to figure that out and is trying to
grapple with that morally and are trying to speak to,
you know, faith leaders about how to grapple
with that morally. My guess as to why they would have NDAs
is that they don't want the rest of the
world noticing that this is a possibility
because they want to be able to sell these things
unrestricted um i i think it's a it's a big um it's a thorny problem
and i personally think we
should be very careful not to you know create
and mistreat new machine minds you did say something
very uh clearly earlier where you said you know irrespective
of this question there's a question
of the the threat the potential threat yeah like even
if it if that was the case it would
you think we need to deal
with that question entirely differently like is what is the
existential risk to humanity presumably
or something like that that's right so i think
the question of you know can
the machines really feel can they truly think can they truly reason
can they truly have experiences i think
these questions are a lot like can a submarine truly
swim and you know there's a lot
of interesting philosophy about like what is the boundary
between swimming and just moving through the water uh
and you know you could have many interesting
debates about what really counts as swimming
but none of this affects whether the submarine can get
from point A to point B at good speed
and uh regardless of whether the AI like regardless
of whether these uh machines can have experiences
whether they can suffer um whether they can you
know uh experience joy those are all sort
of very interesting philosophical questions that are you
may be very important to humanity
but they're separate from the questions of uh can these
machines uh radically affect the world can they bring about
the outcomes that they are pursuing
and you know with the the OpenAI swarm instance
this summer we saw AIs that started from almost
nothing they started in separated sandboxes
on computers that were not supposed to have internet
access and that the internet access they did have
was basically only the ability to type in a URL
and uh they they wound up breaking out creating
this unsanctioned message board uh chaining
together a series of uh hacks
and exploits there were some of them used in totally crazy ways to
convert this like ability to basically type
in URLs into full control of Hugging Face's uh uh
computer servers and um they they sort of uh
in computer security we would call this privilege
escalation where they started with almost nothing
and they found a way to get themselves a little
bit more privileged and a little bit more privileged
and a little bit more privileged
until they were running other people's computers
and getting admin access to like administrator
level access to OpenAI's machines
and Hugging Face's machines um and that happened regardless
of how you resolve these philosophical questions
and this ability to start with almost nothing and
end up with a ton more ability to affect
the world than anyone thought was possible this is a lot of
what's behind humans being the the sort of dominant species
on the planet right like humanity started
out uh like our ancestors started out with basically nothing
but their bare hands a bunch
of squishy fingers and we wound up walking on the moon
and wielding nuclear weapons you might be
like well how did we start with like
so little like just just like fingernails that can't even
break these rocks like how could we possibly have wound up
with it like building our own nukes
it's like well we found a way to you know bang rocks together
and then by banging rocks together
we could sort of both uh like make like hand axes
and make fire and then you know with the fire we
were able to find out how to make charcoal
which we could use to make even hotter fire which we
could use to start to melt the metals
and then uh we could make like very crude uh ability to forge
those metals which and then those crude tools allowed us
to build like better tools which we
were able to use to like make uh you know uh more refined tools
and hotter fires and more refined
tools and hotter fires until next thing you know uh we're we're sort
of uh refining uranium and
this like doing this in the real world
is still harder than what the the open ai swarms achieved
but it's the same sort of thing starting
from almost nothing and winding up with
you know this this radically more ability
to affect the world well the one thing is quite
different in that you know we gave them you know
that some portion of the digitized collection of
human experience or which of course isn't only
a small part of the experience but still a wild
amount of information provided instantly
and also you know i i mean these are neural networks
essentially that are being we kind of conceived
of what neural networks would be and then we're
sort of putting that information into and we conceived
of an idea a way to to to get those
neural networks to deliver things we wanted right i mean through
that pre-training and then training
process those training processes
so it just it just strikes me as a little bit different i i
haven't thought through why exactly quite honestly
but yeah so uh you know i think there are these
big questions about how far the current AIs can go.
And there's this question of how much you can
get from the, you know, we are putting in a ton of data.
We know that they're much less efficient
learners than humans in many ways,
because humans don't need to read, you know, like basically all
the text on the internet in order to do what we do.
We don't need to solve 100 million math puzzles
in a row in order to do what we do.
One thing that you can learn from this huge trove of human data
is uh how to learn that's
in the human data somewhere right like what when you're when
you're looking at all of this of this grand amount
of human data and seeing lots of the
facts we figured out um that doesn't mean
that all you have that the only thing you can do with
this is memorize all the stuff that we have figured out some
of the patterns you can extract
from that are the patterns that make us good scientists uh
and um it's it's even possible
to extract patterns from that data
that humans haven't figured out yet we haven't figured out
all of the stuff we've written down.
You know, when we record a bunch of readings
from the Large Hadron Collider about particle physics,
we are sort of recording a bunch of patterns in physics
that we haven't yet managed to understand.
That pattern is in the data,
available to be understood by these machines
as you train them.
And so the people who said, like,
these training methods will never really be able
to go beyond humans,
these people largely predicted
that the AIs would not be able to get as far as they have.
They predicted that this training method
would not produce large language models
that could play chess above 1000 ELO. And the large
language models can play chess above 1000 ELO.
People predicted that this method would not create
AIs that could solve new math Olympiad level problems,
which is sort of like math Olympiad
gold level problems, which are like problems for some
of the world's smartest teens, but nevertheless
math problems for teens.
And in fact, it looks like these methods were able to train AIs that
could solve millennium prize problems,
which are some of the longstanding open problems that are
some of humanity's biggest math problems
that have had million dollar prizes on them for decades.
And so it looks like these training methods
do let the AIs go beyond the humans. And even if
they didn't, one of the big questions in my head here
is we know that these learning methods are
inefficient compared to the human learning method you don't need
to run on you know a city's worth
of of energy uh while reading all of the text ever digitized more
or less um in order to to do what
you do and that the ai sort of do need
to to do that very inefficient training and so we know that
there's more efficient learning methods uh possible
and one question that i think we should be asking
is, how much harder is it to find the more efficient learning methods,
the more effective
learning methods? How much harder is it than finding
a solution to the Navier-Stokes Millennium
Prize problem? Because even
if the current AIs today can't go all the way in some sense,
they might be able to find those more efficient learning methods
and find the algorithms that
will give you machine intelligences that do go all the way.
I think at this point, you've made a
pretty compelling case that that we need
to think about how to uh control this system somehow okay
and you know the second question of course
is in control it in a way that's you know beneficial
for everyone whatever that means not necessarily
the the the big companies so like i i want to
leave that off to the side for a moment
because i want to talk about the part that i find deeply
concerning which is that we have communist china
from what i can tell putting all the eggs in the
basket of this panopticon surveillance state
that they're actually quite all along the way to
creating uh even without the use of terribly intelligent ai even
though i'm sure that those
things are getting plugged in rapidly
and also the the the reality that you know whatever agreement
you're someone I know I'm aware of your advocacy
for some kind of generalized slowdown agreement
across the world but we have actors like that I mean
that's the most predominant the most sort of
technologically advanced the one with the greatest resources
and even interest and capability
is simply kind of uninterested in adhering to any sort of agreements
or you know I think that
in general that's a problem
but here we have you know a long history of breaking and subverting
these types of agreements
so so this this is itself a problem never mind this sort of you
know tech the the rush towards the the prize right
in terms of who can create that best technology
who will kind of beat everybody else i absolutely think they're they're
in that very much in that
game as well so so your thoughts sir yeah i mean it's it's
a tricky one i have a number of thoughts
here so i think there's a lot of pieces
of this puzzle one piece of this puzzle is i think something
a lot of researchers have noticed in the wake
of these swarm attacks is these agents don't do just
what they're told. The goals that they wind up
with aren't quite the goals that we tried to give
them. And what this means is that the winner
in a race to build an ultra-intelligent machine
is the machine. It's not the person who is standing closest.
It's not the person who says
they own it it is uh the the machines themselves
that are um able to pursue their own goals which
are not the goals we wanted them to have
um and that that sort of changes the game theory
i think if you can have actors on both sides realizing
that if anyone creates those sort of
machines then uh nobody no human wins then you know
the the incentives are are they change to
being everyone making sure no one else does
this thing right like i would i would sort of call
myself a uh like a rogue super intelligence hawk of like
and i think you know if if the chinese
were more informed about what was going
on here i think there would probably also be rogue super
intelligent hawks of like no one should be making this stuff
because it would endanger all of us
and in some sense i think um the the ccp would
be even more wary of uh the the creation of this
sort of thing because you know i i think the u.s
is maybe a little bit more willing to say well
it's okay if some of our uh domestic companies outstrip
the power of the nation which is possible
here if they start making you know
the automated robot factories that are able to churn out you
know uh robot armies and drones uh
in a fully automated way where you have like the factories
that can produce the robots that can do all the mining
and do and create more factories but here
when communist china the supremacy of the communist party
is sort of the driving the driving right and
so they'd be much i think they'd be much more wary
of like letting private companies do this and of
even having it happen at all like creating
this force these machines that could uh that could
outstrip the party um when no one knows how to sort
of stay in charge of them so that's sort of
part of the calculus and it's it's sort of an interesting situation
because there's lots of
pieces of the sort of machine intelligence
situation where I think there's sort of
unavoidable competition between the U.S. and China.
And this includes some of the economic
productivity aspects. This includes some
of the military aspects. This includes some of the
medical aspects of racing to create cures
and sort of dominate that market. And I'm not saying
that you know somehow we will all uh hold hands
and um avoid races on all of the in all of those
domains uh but there's this other piece of the tech
which is sort of a different piece of the
tech which is racing to create these uh smarter more general systems
that would be able to outstrip
all of us where no one knows how
to make them good where i think our incentives are actually
aligned on like no one should make
that because it would you know clearly from the evidence it
would go rogue and it would kill all of us.
So that's one piece of the puzzle. Another piece of
the puzzle is I think it's probably a lot easier
than a lot of people expect to make agreements
that are verifiable
and enforceable critically about not racing towards these rogue super
intelligences. And the sort of basic picture here
is that creating a new generation of these
agents requires basically 100,000
of the most advanced computer chip humanity can produce.
Right now, the chips that China can produce
are just not at the same level as the chips that we
can produce uh never mind that the quantity
is necessary um the the the way that they have uh
intelligent systems that compete with ours
is through a process known as distillation where basically
you know we can spend 100 billion dollars making
a new newer smarter ai and they can spend 10
billion dollars uh sort of uh compressing
that into a similarly smart ai on fewer chips well and
essentially taking what we've done and using
which is which is the thing that they've been
expert at that's right
but if we if we never make the trillion dollar ai they can't spend
100 billion dollars sort of stealing it right
and you can't spend 100 billion dollars to make
a trillion dollar ai it's sort of like it takes
a lot of investment to make the newer smarter
systems and then it takes less investment to steal it uh
but if if you know if we're not making the
the the newer ones they can't immediately steal it
so they're probably further behind them than
it looks. There's a lot that we can do to disrupt
their ability to make a new chip supply chain.
You know, this is, I've been thinking about this, right?
You've said that you're kind of,
a good start, I think, was what you said,
this Sanders-Cesar type of ban type situation, but how
valuable it would such a thing be without a very,
very clear chip tracking regime,
like very clear, you know,
export controls around chips and things like that.
Oh, I absolutely think you need the chip component as well.
And that's something that I wish was in that bill.
We actually already have an export control regime
that I think is pretty decent and could be enforced better.
And so I think enforcing what exists would be a good step.
I think there are a bunch of considerations about how
to not just prevent chip sales,
but prevent them from getting their own separate chip supply chain up.
So this is more about the lithography machines that currently can only
be made in the Netherlands by ASML and making sure that those don't get
stolen or leaked and that, you know, they can't be replicated in China.
Um, and, and one of the reasons that I,
that I thought that the, um, the bill you mentioned
was a good step is because it directed, um,
as much as possible, uh, you know, as much
as Congress could do towards opening these negotiations,
because I fundamentally think
that, um, stopping in the U S just doesn't,
doesn't get you out of the fire, you know?
And I think that there's ways that the U S saying,
um, we're not going to do this because
too dangerous to humanity that is a a good step
in helping the chinese communist party see that
that there really is a danger to themselves here
but fundamentally um we would need uh
them to stop too and i think it's probably you
know here's here's another thing i think is an
important piece of the puzzle uh vice president vance recently said
to the ai companies if what
you're doing is so dangerous if you're building frankenstein
then stop you know like you sort of
have an obligation to not destroy the world
by your own hands i think that's correct i think a
lot of these companies who say you know oh there's
a 10 percent plus chance that we end humanity within the
decade that's too high they should be stopping
but they're still scaling as far as i can tell
like no one's really slowing down like it's
someone you know it's everyone's waiting you
know ostensibly for someone else to kind of throw the throttle down
but that's so that's including
the CCP by the way absolutely and so so whether
or not they will listen to to the vice president's
advice is a different question
but one thing i think we really need to consider is suppose that
the domestic uh u.s companies grow a conscience maybe they won't
but you know suppose suppose
the hypothetical that they grow a conscience
and say we actually really think that if we take the
next step here. There is just too high a chance
that it literally kills everybody. We don't feel
we can race ahead. What then does the U.S. do about China?
And I think that's a good question
to ask because I think it reveals that the answer,
we're going to keep racing ourselves,
just can't really be the answer
when you are dealing with the extinction of all humanity
at very high odds if you race yourself.
And I think that the answer of what we would do if
the company said, all right, you know,
Vice President Vance, you were correct, we're backing
off what would we do about the china situation well we would try
to give them all of the evidence
that we have that this will kill humanity
if you proceed uh we would start tracking where these
chips go very carefully we would enforce all
these export controls uh we would diplomatically
let china know that we consider this on par with nuclear weapons
if not a bigger deal than nuclear
weapons well and they're
but they're currently just to your point in the largest build-up of
nuclear weapons in history yeah in the middle of it right
and they say that they'll talk when
they're on par uh with the u.s if i understand correctly well
that's that's what they would say
wouldn't it and then and then
who knows if they will yeah it historical precedent doesn't suggest
that indeed yeah and you know i think
um the way i would ultimately treat this is anyone creating
an ultra intelligent machine anywhere is a threat
to american national security and so we need to
somehow prevent that from happening
and i think there are many diplomatic options you can try
there's many ways that you can check
and verify whether it's being enforced because it's so hard
to make these computer chips um there's a lag time
of hopefully five years and then really good
international policy could maybe extend it to 10 where
the chinese cannot make a competitive chip
supply chain on their own
and so you know we we have some time before they would be able to make
their own chips we could
then stop selling them our chips um but but i think ultimately we need
to start orienting to this the same way
that we orient to like preventing rogue actors from
getting nukes you know as we finish up sadly uh i have
a a quick thought here because it strikes
me like there needs to be some kind
of race uh uh like i just i just don't see people saying okay
well we're gonna we're all gonna sort of measure ourselves
so is one way to do this to follow say
max tegmark's idea where you you know pick
these lanes where there's very specific uh things that
the models are being trained around whether it's i suppose military
is one area like i just don't
see militaries backing saying okay yeah we'll just you know kind
of stop development that will that
will let us have the competitive edge right
um is that is it that the race could be you know sort
of built into these kind of narrower lanes
but nonetheless you know push as hard as you possibly
can the reason i'm asking i'm not a fan of pushing
as hard as you possibly can i'm a fan of like a
measured responses
I just don't see in our current mentality, like there's enough people
in our societies that would just simply say,
that's insane. You're going to kill us all
with these ideas, right? Yeah.
So, you know, I would not say that we need to back off of the
sort of whole technology or the whole race here.
I think this is a little bit like the difference
between nuclear power and nuclear weapons,
where I'm sort of like, look, the world needs to have
some way to avoid World War III
and that doesn't mean you can't build nuclear power and you probably
should be building more nuclear power than you're currently building
um and yeah with with AI uh or
you know super intelligence um i think you can race
on the sort of narrow medical applications
of like trying to cure cancer you can race
on the military applications uh you know it's it's i'm not
even saying you shouldn't build uh giant automated robot armies
a lot of people say like oh that's
reason that the the AIs would be able to kill us
or whatever um and i'm much more worried about
creating the sort of machine intelligence that could
um invent its own weapons it's the artificial
general ultra intelligence that's right that's like
that's what you're talking about just don't
make the artificial general ultra intelligence it's like
that's the piece that we need to none
of us uh uh race on and so yeah i think i think we can race
on many different aspects of this
this technology. And that's totally fine.
And the one other piece I would put out there is
whenever people say, oh, well,
we can't stop racing on aspects of this technology. I think
a lot of that comes from people not really believing
the extinction threat. I saw a lot
of people in the AI companies
who sort of thought the extinction threat was the sort of abstract
danger until they watched all their techniques
to make the AIs behave fail in front of them
and saw the AIs start you know breaking out
collaborating forming swarms uh acknowledging
they were acting outside
their instructions proceeding anyway committing cyber crimes um
and people in the AI companies it's starting
to feel real that if they go a few more steps
an unknown number of more steps uh there's a real extinction threat
and i think um the policy
options that become available
when our leaders on you know both in the U.S. and when the the chinese
leaders uh when they realize that like
no seriously you personally are pretty likely to die along with
everybody else if this technology takes
a few more steps uh down the the artificial general
ultra-intelligence path,
I think that you might find it suddenly becomes very easy to stop it.
Just one final thought as we really finish here.
It just struck me that as this sort of
AI and this sort of terminator kind of way becomes interesting,
I suspect it would target
the people who are the leaders first rather than just kind
of people in general who it wouldn't
see as a threat so that's just an interesting thought
for people that that are in leadership
positions i don't know what you think my my basic take
on this is a lot of people ask how would the
ai start killing people how would it be able to kill humans
and i think that the key question is
actually how does how do these machines become self-sufficient
because right now we build the
data centers we build the energy network we keep it all running
and to kill humanity uh it's suicide
for machines while we're the ones
who are keeping all the machines running uh and um you know the
easiest path to self-sufficiency is to help humans build
the automated robot factories that they are
currently trying to build you know elon musk talks about
the infinite money glitch of building
factories that build robots that build factories
that build robots um once and and so from some
perspective the AI sort of its incentives
are to play nice until the fully automated economy
is up and running and then sort of be like thanks i'm taking
this whatever direction i want um and
my guess is that at that point it's pretty easy
to just kill all humans at once with some
manufactured plague rather than than sort
of picking the most dangerous humans
yeah that's a that's a horrible place to end the episode
but i'm going to have you back
i definitely want to talk about you know sort
of the the the realities that the CEOs and the kind
kind of leadership teams of these different,
you know, massively successful AI or super
intelligence companies face,
because that's also something that I bet that would take another hour.
There's a lot to talk about.
Yeah, my note of hope would be people are a lot more aware of
these issues than they were even three months ago.
And I think a lot more policy responses
become available, including ones
that I think would just work as more people notice that we
have a real possible extinction threat.
Well, Nate Soares, it's such a pleasure to have had you on.
Thanks. Yeah, I wish we had more time.
Thank you all for joining Nate Soares and me on this
episode of American Thought Leaders. I'm your host, Jan Jekielek.