"Godfather of AI" Shares Prediction for Future of AI, Issues Warnings

Bring ETHICS Back
145 followers
2 Video Views·Jul 25, 2025
Nobel laureate Geoffrey Hinton, often called a "godfather of artificial intelligence," spoke with Brook Silva-Braga at the Toronto offices of Radical Ventures about the future of AI earlier this month — nearly two years after they first sat down to discuss the evolving technology. He shares some of his early takeaways about AI, which he says has evolved "even faster than [he] thought."
The last time we spoke, two years, one month ago.
I'm curious how your expectations over these two years
have evolved for how you see the future.
So AI has developed even faster than I thought.
In particular, they now have these AI agents
which are more dangerous than AI that just answers questions
because they can do things in the world.
So I think things have got, if anything,
scarier than they were before.
I don't know if we wanna call it AGI, super intelligence,
whatever, very capable AI system.
Do you have a timeline in mind
for when you think that's coming?
So a year ago, I thought it was,
there's a good chance it comes
between five and 20 years from now.
So I guess I should believe there's a good chance
it comes between four and 19 years from now.
I think that's still what I guess.
Which is sooner than when we spoke
because you were still thinking like 20 years.
Yeah, I think it may, there's a good chance
it'll be here in 10 years or less now.
So in four to 19 years, we reach this point.
What does that look like?
So I don't really wanna speculate
on what it would look like if AI decided to take over.
There's so many ways it could do it.
And I'm not even talking about taking over.
We can talk about that.
I'm sure we will talk about that.
But putting aside that kind of takeover,
just like a super intelligent artificial intelligence,
like what kind of things is this capable of
or would be doing?
So the sort of good scenario is we would all be like
the sort of dumb CEO of a big company
who has an extremely intelligent assistant
who actually makes everything work
but does what the CEO wants.
So the CEO thinks they're doing things
but actually it's all done by the assistant.
And the CEO feels just great
because everything they sort of decide to do works out.
That's the good scenario.
And I've heard you point out a few areas
where you think there's reason to be optimistic
about what this future looks like.
Yes.
Yeah, so why don't we take each of them?
So areas like healthcare,
they will be much better reading medical images,
for example, that's a minor thing.
I made a prediction some years ago
they'd be better by now.
And they're about comparable with the experts by now.
They'll soon be considerably better
because they'll have had a lot more experience.
One of these things can look at millions of x-rays
and learn from millions of them and a doctor can't.
They'll be very good family doctors.
So you can imagine a family doctor
who's seen a hundred million people,
including half a dozen people
with your very, very rare condition.
They'd just be a much better family doctor.
A family doctor who can integrate information
about your genome with the results of all the tests on you
and all the tests on your relatives,
the whole history and doesn't forget things.
That would be much, much better.
Already, AI combined with the doctor
is much better at doing diagnoses in difficult cases
than a doctor alone.
So we're gonna get much better healthcare
from these things.
And they'll design better drugs too.
Education is another field.
Yes, in education we know that
if you have a private tutor,
you can learn stuff about twice as fast.
These things eventually will be
extremely good private tutors
who know exactly what it is you misunderstand
and exactly what example to give you
to clarify to you so you understand.
So maybe you'll be able to learn things
three or four times as fast with these things.
That's bad news for universities,
but good news for people.
Do you think the university system will survive this period?
I think many aspects of it will.
I think it's still the case that a graduate student
in a good group in a good university
is the sort of best source of truly original research.
And I think that'll probably survive.
You need a kind of apprenticeship.
Some people hope this will help solve the climate crisis.
I think it will help.
It'll make better materials.
We'll be able to make better batteries, for example.
I'm sure AI will be involved in designing them.
People are using it for carbon capture from the atmosphere.
I'm not convinced that's gonna work
just because of the energy considerations, but it might.
In general, we're gonna get much better materials.
We might even get room temperature superconductivity,
which would mean you can have
lots of solar plants in the desert
and we can be thousands of miles away.
Any other positives we should tick off?
Well, more or less any industry
is gonna make more efficient
because almost every company
wants to predict things from data.
And AI is very good at doing predictions.
It's better than the methods we had previously,
almost always.
So it's gonna cause huge increases in productivity.
It's gonna mean when you call up a call center,
when you call up Microsoft to complain
that something doesn't work and you get a call center,
the person in the call center will be actually an AI
who will be much better informed.
Yeah.
When I asked you a couple of years ago
about job displacements,
you seemed to think that wasn't a big concern.
Is that still your thinking?
No, I'm thinking it will be a big concern.
AI has got so much better in the last few years
that, I mean, if I had a job in a call center,
I'd be very worried.
Yeah, or maybe a job as a lawyer or a job as a journalist
or a job as an accountant.
Yeah, and doing anything routine.
I think investigative journalists,
I think will last quite a long time
because you need a lot of initiative
plus some moral outrage.
And I think journalists will be in business for a bit.
But beyond call centers, what are your concerns about jobs?
Well, any routine job.
So a sort of standard secretarial job.
Something like a paralegal, for example.
Those jobs have had it.
Have you thought about how we move forward
in a world where all these jobs go away?
So it's like this.
It ought to be that if you can increase productivity,
everybody benefits.
The people who are doing those jobs
can work a few hours a week instead of 60 hours a week.
They don't need two jobs anymore.
They can get paid lots of money for doing one job
because they're just as productive using AI assistance.
But we know it's not gonna be like that.
We know what's gonna happen is the extremely rich
are gonna get even more extremely rich.
And the not very well off
are gonna have to work three jobs now.
I think no one likes this question,
but we like to ask it.
This idea of P-Doom, how likely it is.
And I am curious if you see this as a quite possible thing
or it's just so bad that even though
the likelihood isn't very high,
we should just be very concerned about it.
Where are you on that scale of probability?
So I think most of the experts in the field would agree
that if you consider the possibility
that these things will get much smarter than us
and then just take control away from us, just take over.
The probability of that happening
is very likely more than 1%
and very likely less than 99%.
I think pretty much all the experts can agree on that.
But that's not very helpful.
No, but it's a good start.
It might happen and it might not happen.
And then different people disagree on what the numbers are.
I'm in the unfortunate position of happening
to agree with Elon Musk on this,
which is that it's sort of 10 to 20% chance
that these things will take over.
But that's just a wild guess.
I think reasonable people would say
it's quite a lot more than 1%
and quite a lot less than 99%.
But we're dealing with something we've got no experience of.
We have no real good way of estimating
what the probabilities are.
It seems to me at this point,
it's inevitable that we're gonna find out.
We are gonna find out, yes.
Because it seems extremely likely
that these things will get smarter than us.
Already, they're much more knowledgeable than us.
So GPT-4 knows thousands of times
more than a normal person.
It's a not very good expert at everything.
And eventually, its successors
will be a good expert at everything.
They'll be able to see connections
between different fields that nobody's seen before.
Yeah.
Yeah, I'm also interested in understanding,
okay, there's this terrible 10 to 20% chance, but-
Or more.
Or more. Or less.
Or less.
But let's just take as a premise
that there's a 80% chance
that they don't take over and wipe us out.
So that's the most likely scenario.
Do you still think it would be net positive or net negative
if it's not the worst outcome?
Okay, if we can stop them taking over,
that would be good.
The only way that's gonna happen
is if we put serious effort into it.
But I think once people understand
that this is coming,
there will be a lot of pressure
to put serious effort into it.
If we just carry on like now,
just trying to make profits,
it's gonna happen.
They're gonna take over.
We have to have the public put pressure on governments
to do something serious about it.
But even if the AIs don't take over,
there's the issue of bad actors using AI for bad things.
So mass surveillance, for example,
which is already happening in China.
If you look at what's happening
in the West of China to the Uyghurs,
the AI is terrible for them.
To board a plane to come to Toronto,
I had to take a facial recognition photo
from our US government.
Right, when I come into Canada,
you put your passport and it looks at you
and it looks at your passport.
Every time, it fails to recognize me.
Everybody else it recognizes.
People from all different nationalities it recognizes.
Me, it can't recognize.
And I'm particularly indignant
since I assume it's using neural nets.
You didn't carve out an exception, did you?
No, no, it just,
there's something about me that it doesn't like.
I have to find some place to work it in,
so this is as good a place as any.
Let's talk a little bit about the Nobel.
Can you paint the picture of the day you found out?
So I was sort of half asleep.
I had my cell phone upside down on the bedside table
with the sound turned off.
But when a phone call comes, the screen lights up.
And I saw this little line of light,
because I happened to be lying on the pillow
with my head on this side and it was up here.
Facing the phone rather than facing away.
Just happened to be facing the phone.
So this little line of light,
and I was in California,
and it was one o'clock in the morning.
And most people who call me
are on the East Coast or in Europe.
You don't use Do Not Disturb?
No. No, okay.
I just, I turn off the sound.
I turn off the sound. Got it.
And I thought, I was just curious about
who on earth is calling me
at four o'clock in the morning on the East Coast.
This is crazy.
So I picked it up and there was this long phone number
with a country code I didn't recognize.
And then this Swedish voice comes on
and asks if it's me.
And I say, yeah, it's me.
And they say, I won the Nobel Prize in physics.
Well, I don't do physics, right?
So I thought this might be a prank.
In fact, I thought the most likely thing was
that it was a prank.
I was aware that the Nobel Prizes were coming up
because I was very interested in whether
Demis Hassabis would get the Nobel Prize for chemistry.
And I knew that was being announced the next day.
But I sort of, I don't do physics.
I'm a psychologist hiding in computer science
and I get the Nobel Prize in physics.
Was it a mistake?
Well, one thing that occurred to me is
if it's a mistake, can they take it back?
So, but for the next couple of days
I did the following reasoning.
So what's the chance a psychologist
will get the Nobel Prize in physics?
Well, maybe one in two million.
Now, what's the chance if it's my dream,
I'll get the Nobel Prize in physics?
Well, maybe one in two.
So if it's one in two in my dream
and one in two million in reality,
that makes it a million times more likely
that this is a dream than that it's reality.
And for the next couple of days,
I went around thinking,
you know, are you quite sure this isn't a dream?
You've walked me into this very wacky territory,
but it is part of this discussion.
Some people think we're living in a simulation
and that AGI is almost, not evidence,
but hints toward maybe that's the reality in which we live.
Yeah, I don't really believe that.
I think that's kind of wacky.
Okay, so let's put that one.
But I don't think it's totally nonsense.
I've seen the matrix too.
Oh, okay, okay, wacky but not totally.
Okay, here's where I kind of wanted to head with the Nobel.
I think you've said something to the effect
you hope to use your credibility
to convey a message to the world.
Can you kind of explain what that is?
Yes, that AI is potentially very dangerous.
And there's two sets of dangers.
There's bad actors using it for bad things,
and there's AI itself taking over.
And they're quite different kinds of threat.
And we know bad actors are already using it for bad things.
I mean, it was used during Brexit
to make British people vote to leave Europe in a crazy way.
So a company called Cambridge Analytica
was getting information from Facebook and using AI.
And AI has developed a lot since then.
It was probably used to get Trump elected.
I mean, they had information from Facebook,
and it probably helped with that.
We don't know for sure
because it was never really investigated.
But now it's much more competent.
And so people can use it far more effectively
for things like cyber attacks,
designing new viruses,
obviously fake videos for manipulating elections,
targeted fake videos by using information about people
to give them just what will make them indignant.
Autonomous lethal weapons.
They're all the big arms selling countries
are busy trying to make autonomous lethal weapons.
America and Russia and China and Britain and Israel.
I think Canada's probably a bit too wimpy for that.
The question then is what to do about it.
What type of regulation do you think we should pursue?
Okay, so we need to distinguish
these two different kinds of threat.
The bad actors using it for bad things
and the AI itself taking over.
I've talked mainly about that second threat,
not because I think it's more important
than the other threats,
but because people thought it was science fiction.
And I want to use my reputation to say,
no, it's not science fiction.
We really need to worry about that.
And if you ask what should we do about it,
it's not like climate change.
Climate change, just stop burning carbon
and it'll all be okay in the long run.
It'll be terrible for a while,
but in the long run,
it'll all be okay if you don't burn carbon.
For AI taking over, we don't know what to do about it.
We don't know, for example,
the researchers don't know if there's any way
to prevent that, but we should certainly try very hard.
And the big companies aren't going to do that.
If you look what the big companies are doing right now,
they're lobbying to get less AI regulation.
There's hardly any regulation as it is,
but they want less because they want short-term profits.
We need people to put pressure on governments
to insist that the big companies
do serious safety research.
So in California, they had very sensible bill,
1047, where they said that at least what big companies
have to do is test things carefully
and report the results of their tests.
And they didn't even like that.
So does that make you think regulation will not happen
or how does it happen?
It depends very much on what governments we get.
I think under the current US government,
regulation is not gonna happen.
All of the big AI companies have got into bed with Trump.
And yeah, it's just a bad situation.
Elon Musk, who is obviously so enmeshed
in the Trump administration,
has been someone concerned about AI safety
for a very long time.
Yes, he's a funny mixture.
He has some crazy views, like going to Mars,
which I just think is completely crazy.
Because it won't happen
or because it shouldn't be a priority?
Because however bad you make the Earth,
it's always gonna be way more hospitable than Mars.
Even if you had a global nuclear war,
the Earth is gonna be much more hospitable than Mars.
Mars just isn't hospitable.
Obviously, he's done some great things like electric cars
and helping Ukraine with communications with his Starlink.
So he's done some good things.
But right now, he seems to be fueled by powering ketamine
and he's doing a lot of crazy things.
So he's got this funny mixture of views.
So his history of being concerned about AI safety
doesn't make you feel any better
about the current administration?
I don't think it's gonna slow him down
from doing unsafe things with AI.
So already, they're releasing the weights
for their AI large language models,
which is a crazy thing to do.
These companies should not be releasing the weights.
Meta releases the weights.
OpenAI just announced they're about to release weights.
You think that's a bad idea?
I don't think they should be doing that.
Because once you release the weights,
you've got rid of the main barrier to using these things.
So if you look at nuclear weapons,
the reason only a few countries have nuclear weapons
is because it's hard to get the fissile material.
If you were to be able to buy fissile material on Amazon,
many more countries would have nuclear weapons.
The equivalent of fissile material for AI
is the weights of a big model,
because it costs hundreds of millions of dollars
to train a really big model.
Not maybe the final training run,
but all the research that goes into
the things you do before the final training run,
hundreds of millions of dollars,
which a small cult or a bunch of cyber criminals
can't afford.
Once you release the weights,
they can then start from there and fine tune it
for doing all sorts of things for just a few million dollars.
So I think it's just crazy releasing weights.
And people talk about it like open source,
but it's very, very different from open source.
In open source software, you release the code,
and then lots of people look at that code and say,
hey, that might be a bug in that line.
And so they fix it.
When you release the weights,
people don't look at the weights and say,
hey, that weight might be a little bit wrong.
No, they just take this foundation model
with the weights they've got now,
and they train it to do something bad.
The problem with the argument though,
as articulated by your former colleague,
Jan LeCun, among others,
is the alternative is you have this tiny handful
of companies that control
this massively powerful technology.
I think that's better than everybody controlling
a massively powerful technology.
I mean, you could say the same for nuclear weapons.
Would you like to have just a few countries controlling them
or don't you think everybody should have them?
One thing I'm taking from this
is you have real concerns about,
it sounds like all of the major companies right now
doing what's in society's best interest
rather than what's in their profit motive.
Is that the right way to hear you?
I think the way companies work is
they're legally required to try and maximize profits
for their shareholders.
They're not legally required.
Well, maybe public interest companies are,
but most of them aren't legally required
to do things that are good for society.
Which, if any of them,
would you feel good about working for today?
I used to feel good about working for Google
because Google was very responsible.
It didn't release these big,
it was the first to have these big chatbots
and it didn't release them.
I'd feel less happy working for them today.
Yeah, I wouldn't be happy working for any of them today.
If I worked for any of them,
I'd be more happy with Google than most of the others.
Were you disappointed when Google went back on its promise
not to support military uses of AI?
Very disappointed.
I was very disappointed,
particularly since I knew Sergey Brin
didn't like military use of AI.
Why do you think they did it?
I can't really speculate with any inside information.
I don't have any inside information about why they did it.
I could speculate that they were worried about
being ill-treated by the current administration
if they wouldn't use their technology
to make weapons for the US.
Here's the toughest question I'll probably ask you today.
Do you not still hold a lot of Google stock still?
I hold some Google stock.
Most of my savings are not in Google stock anymore.
But yeah, I hold some Google stock.
And when Google goes up, I'm happy,
and when it goes down, I'm unhappy.
So I have a vested interest in Google.
But if they put in strong AI regulations
that made Google less valuable,
but increased the chance of humanity surviving,
I'd be very happy.
One of the most prominent labs has obviously been OpenAI,
and they have lost so many of their top people.
What have you made of that?
That OpenAI was set up explicitly
to develop superintelligence safely.
And as the years went by,
safety went more and more into the background.
They were gonna spend a certain fraction
of their computation on safety.
And then they were engaged on that.
So, and now they're trying to go public.
They're now trying to be a for-profit company.
And they're trying to get rid of all the,
basically all the commitment to safety,
as far as I can see.
And they've lost a lot of really good researchers.
In particular, a former student of mine,
Ilya Sutskever, who's a really good researcher,
and was one of the people largely responsible
for their development of GPT-2,
and then from there on to GPT-4.
Did you talk to him before all that drama
that led to his departure?
No, he's very discreet.
He doesn't talk, he wouldn't talk to me
about anything that was confidential to OpenAI.
I was quite proud of him for firing Sam Alton,
even though it was very naive.
So, the problem was that OpenAI was about
to have a new funding round.
And in that new funding round,
all the employees were gonna be able
to turn their paper money, in OpenAI shares,
into real money.
Paper money meaning really hypothetical money.
Hypothetical money that would disappear
if OpenAI went bust.
Tough time for an insurrection.
So, a week or two before everybody's gonna get
maybe the order of a million dollars each
by cashing in their shares, maybe more.
That's a bad time for an insurrection.
So, the employees massively came out
in favor of Sam Altman.
But it wasn't because they wanted Sam Altman,
it's because they wanted to be able
to turn their paper money into real money.
Yeah.
So, it was naive to do it then.
Did it surprise you that he made that mistake,
or was this kind of the principled,
but maybe not fully calculated decision
that you would expect?
I don't know.
Ilya is brilliant and has a strong moral compass.
So, he's good on morality,
and he's very good technically.
But in terms of manipulating people,
he's maybe not so good.
I mean, this is a little bit of a wild card question,
but I do think it's interesting and relevant to the field
and relevant to people discussing what's going on.
You talked about Ilya being discreet.
There does seem to be this culture of NDAs
throughout the industry.
And so, it's hard to even know what people think
because people are unwilling or unable
to even discuss what's going on.
I'm not sure I can comment on that
because when I left Google,
I think I had to sign a whole bunch of NDAs.
In fact, when I joined Google,
I think I had to sign a whole bunch of NDAs
that would apply when I left.
And I have no idea what they said,
I can't remember them anymore.
Do you feel at all muzzled by them?
No.
Okay.
Do you think it's a factor though
that the public has a harder time
understanding what's going on
because people aren't allowed to tell us what's going on?
I don't really know.
You'd have to know which people weren't telling you.
Okay, so you don't see this as a-
I don't see it as a big deal.
As a big deal.
Got it.
I think it was a big deal that open AI
appeared to have something that said that
if you'd already got shares,
they could take the money away from you.
That, I think, was a big deal.
They rapidly backed down on that when that became public.
That was what their public statement said they did.
They didn't present any contracts for the public
to judge whether they had reversed that,
but they said they had reversed it.
There's a number of just important
kind of hot button-y things.
Hot button is actually not even a great word,
but relevant issues I'd just like to get your feedback on.
One is the US and kind of the West's orientation to China
in their efforts to pursue AI.
Do you agree with this idea
that we should be trying to restrain China?
There's this idea of export controls,
this idea that we should have democracies reach AGI first.
What's your thinking on all that?
First of all, you have to decide
which countries are still democracies.
My thinking on that is, in the long run,
it's not gonna make much difference.
It may slow things down by a few years,
but clearly, if you prevent China
from getting the most advanced technology,
people know how this advanced technology works.
So China's just invested many, many billions,
maybe hundreds of billions,
of the order of 100 billion, I think,
in making lithography machines
or in getting their own home-based technology
that does this stuff.
So it'll slow them down a bit,
but it will actually force them
to develop their own industry.
And in the long run, they're very competent and they will.
And so it'll just slow things down for a few years.
But race is the right framework.
We shouldn't be trying to cooperate with communist China.
I wouldn't describe it as communist anymore.
I use the loaded term specifically
because why wouldn't you cooperate, right?
The only rationale to not cooperate
is if you think they're a malignant force.
Well, there's areas in which we won't cooperate,
where we is, I guess, I'm not sure we is anymore
because I'm in Canada now.
And we used to be sort of Canada and the US,
but it's not anymore.
Obviously, the countries are not gonna cooperate
on developing lethal autonomous weapons
because the lethal autonomous weapons
to be used against other countries.
But we've had treaties and other types of weapons,
as you've pointed out.
We can have treaties not to develop them,
but cooperating in making them better,
they're not gonna do that.
Sure, sure, sure.
Now, there's one area where they will cooperate,
which is on the existential threat.
If they ever get serious
about worrying about the existential threat
and doing stuff about it,
they will collaborate on that,
ways of stopping AI taking over
because we're all in the same boat.
So at the height of the Cold War,
the Soviet Union and the US collaborated
on preventing a global nuclear war.
And even countries that are very hostile to each other
will collaborate when their interests align.
And their interests will align when it's AI versus humanity.
There's this question of fair use,
whether it's okay to have the content
of billions of humans created over many years
kind of scooped up and repurposed into models
that will replace some of those same people
that created the training data.
Where do you fall on that?
I think I sort of fall all over the place on that
in the sense that it's a very complicated issue.
So initially it seems, yeah,
they should have to pay for that.
But suppose I have a musician
who produces a song in a particular genre
and ask, well, how did they produce a song in that genre?
Where did their ability to produce songs
in that genre come from?
Well, it came from listening to songs
by other musicians in that genre.
So they listened to these songs,
they kind of internalized things
about the structure of the songs,
and then they generated stuff in that genre.
And the stuff they generated is different,
so it's not theft, and that's accepted.
Well, that's what the AI is doing.
The AI is absorbing all this information
and then producing new stuff.
It's not just taking it and patching it together.
It's generating new stuff
that has the same underlying themes.
And so it's no more stealing than a person does
when they do the same thing.
But the point is it's doing it at a massive scale.
And no musician has ever put
every other musician out of business.
Exactly.
So in Britain, for example,
the government doesn't seem to have any interest
in protecting the creative artists.
And if you look at the economy,
the creative artists are worth a lot to Britain.
So I have a friend called B. Bankidron
saying we should protect creative artists.
It's very important to the economy.
And just letting AI walk off with it all seems unfair.
UBI, universal basic income.
Is this part of the solution
to the displacements of AI, you think?
I think it may be necessary to stop people starving.
I don't think it totally solves the problem.
Even if you had quite high UBI,
it doesn't solve the problem of human dignity.
For a lot of people, who they are is,
particularly of academics,
who they are is mixed up in their work.
That's who they are.
If they become unemployed,
just getting the same money doesn't totally compensate.
They're not who they are anymore.
I tend to think that's true as well.
I saw you give this quote at one point, though,
where you said you might have been happier
if you were a woodworker.
Well, yes, because I really like being a carpenter.
And isn't there an alternative
where you're born a hundred years later
where you don't have to waste all your time
on these neural nets
and you just get to enjoy woodworking
while taking in a monthly income?
Yeah, but there's a difference
between doing it as a hobby
and doing it to make a living.
Somehow it's more real doing it to make a living.
So you don't think a future
where we get to pursue our hobbies
and don't have to contribute to the economy?
I think that might be fine.
If everybody was doing that.
But if you're in some disadvantaged group
who are getting universal basic income
and you're getting less income than other people,
because employers want you to do that
so they can get other people to work for them,
that's gonna be very different.
I'm interested in this idea of robot rights.
I don't know if there's a better term to describe it,
but at some point you're gonna have
these massively intelligent AIs.
They're gonna be agentic
and doing all kinds of things in the world.
Should they be able to own property?
Should they be able to vote?
Should they be able to marry humans
in a loving relationship?
Like what rights?
Or even if they're just smarter than us
and if it's a better form of intelligence
than what we've got.
Should it be fine for them to just take over
and humans be history?
Yeah, well let's go to that bigger idea second.
I'm curious on the more narrow idea.
Unless you think the narrow questions are irrelevant
because the big question takes precedence.
No, I think the narrow questions are relevant.
So I used to be worried about this question.
I used to think, well, if they're smarter than us,
then why shouldn't they have the same rights as us?
And now I think, well, we're people.
What we care about is people.
I eat cows.
I mean, I know lots of people don't, but I eat cows.
And the reason I'm happy eating cows
is because they're cows and I'm the person.
And the same for these super intelligent AIs.
They may be smarter than us,
but what I care about is people.
And so I'm willing to be mean to them.
I'm willing to deny them their rights
because I want what's best for people.
Now, they won't agree with that.
And they may win.
But that's my current position
on whether AI should have rights,
which is even if they're intelligent,
even if they have sensations and emotions
and feelings and all that stuff,
they're not people and people's what I care about.
But they're gonna seem so much like people.
I feel like it's gonna be hard.
They're gonna be able to fake it, yes.
They're gonna be able to seem very like people.
Yeah, yeah.
Do you suspect we'll end up giving them rights?
I don't know.
Okay.
I tend to avoid this issue
because there's more immediate problems
like bad uses of AI
or the issue of whether they will try and take over
and how to prevent that.
And it sounds kind of flaky
if you start talking about them having rights.
Most people, you've lost most people when you go there.
Even just sticking with people,
there seems to be real soon, if it's not already here,
this ability to use AI to select what babies we have.
Are you concerned at all about that line?
Embryo selection?
You mean selecting for the sex
or selecting for the attributes of the developed adult?
The intelligence and the eye color
and the likelihood to get pancreatic cancer
and the list goes down and down and down
of all the things we might select for.
I think if you could select a baby
that was less likely to get pancreatic cancer,
that would be a great thing.
I'm willing to say that.
Okay, so this is a thing we should pursue.
We should make healthier, stronger, better babies.
It's very difficult territory, right?
It is, that's why I'm asking about it.
But some aspects of it seem to make sense to me.
Like, if you're a normal, healthy couple
and you have a fetus,
and you can predict that it's gonna have
very serious problems and maybe not live very long,
it seems to me it makes sense to abort it
and have a healthy baby.
That just seems sensible to me.
Now, I know a lot of religious people
wouldn't agree with that at all.
But for me, if you could make those predictions reliably,
that just seems to make sense to me.
I've been a little bit holding us back
from kind of the central thing
that I think you want people to take away,
which is this idea of machines taking over
and the impact of that.
So I'd like to just discuss that as fully as you'd like
or that we can.
Like, how do you wanna frame this issue?
How should people think about it?
One thing to bear in mind is,
how many examples do you know of less intelligent things
controlling much more intelligent things?
So we know that things are more or less equal intelligence.
The less intelligent one can control
the more intelligent one.
But with a big gap in intelligence,
there's very, very few examples
where the more intelligent one isn't in control.
So that's something you should bear in mind
that's a big worry.
I think the situation we're in right now,
the best way to understand it emotionally
is we're like somebody who has this really cute tiger cub.
It's just such a cute tiger cub.
Now, unless you can be very sure
that it's not gonna wanna kill you when it's grown up,
you should worry.
And to extend the metaphor,
you put it in a cage, you kill it.
Well, the point about the tiger cub is,
it's just physically stronger than you.
So you can still control it
because you're more intelligent.
Things that are more intelligent than you,
we have no experience of that, right?
People aren't used to thinking about it.
People think somehow you constrain it.
You don't allow it to press buttons or whatever.
Things more intelligent than you,
they're gonna be able to manipulate you.
So another way of thinking about it is,
imagine that there's this kindergarten.
There's these two and three-year-olds.
And the two and three-year-olds are in charge.
And you just work for them in the kindergarten.
And you're not that much more intelligent
than a two or three-year-old,
not compared with super intelligence,
but you are more intelligent.
So how hard would it be for you to get control?
Well, you just tell them all you're gonna get free candy.
And if they just sort of sign this
or just agree verbally to this,
you'll get free candy for as long as you like,
and you'll be in control.
They won't have any idea what's going on.
And with super intelligences,
they're gonna be so much smarter than us,
we'll have no idea what they're up to.
And so what do we do?
We worry about whether there's a way
to build a super intelligence
so that it doesn't want to take control.
I don't think there's a way of stopping it
to take control if it wants to.
So there's one possibility,
never build a super intelligence.
You think that's possible?
I mean, it's conceivable,
but I don't think it's gonna happen
because there's too much competition
between countries and between companies,
and they're all after the next shiny thing,
and it's developing very, very fast.
So I don't think we're gonna be able
to avoid building super intelligence.
It's gonna happen.
The issue is, can we design it in such a way
that it never wants to take control,
that it's always benevolent?
That's a very tricky issue.
Just people say, well, we'll get it
to align with human interests,
but human interests don't align with each other.
And if I say I've got two lines at right angle,
and I want you to show me a line parallel to both of them,
that's kind of tricky, right?
And if you look at the Middle East, for example,
there's people with very strong views that don't align.
So how are you going to get AI
to align with human interests?
Human interests don't align with each other.
So that's one problem.
It's gonna be very hard to figure out
how to get super intelligence that doesn't wanna take over
and doesn't wanna ever hurt us.
But we should certainly try.
And trying is kind of just an iterative process.
Month by month, year by year, we try to...
Yeah, so obviously, if you're gonna develop something
that might want to take over,
when it's just slightly less intelligent than you are,
and we're very close to that now,
you should kind of look at what it'll do
to try and take over.
So if you look at the current AIs,
you can see they're already capable of deliberate deception.
They're capable of pretending to be stupider than they are,
of lying to you so that they can kind of confuse you
into not understanding what they're up to.
We need to be very aware of all that and to study all that
and study about whether there's a way
to stop them doing that.
When we spoke a couple of years ago,
I was surprised at you voicing concerns
because you hadn't really done much of that before.
And now you're voicing them quite clearly and loudly.
Was it mostly that you felt more liberated to say this stuff
or was it really a really big sea change
in how you saw it in these last few years?
When we spoke a couple of years ago,
I was still working at Google then.
It was in March and I didn't resign till the end of April.
But I was thinking about leaving then.
And I had had a kind of epiphany before we spoke
where I realized that these things
might be a better form of intelligence than us.
And that got me very scared.
And you didn't think that before
just because you thought the time horizon was so different?
No, it wasn't just that.
It was because of the research I was doing at Google.
I was trying to figure out whether you could design
analog large language models
that would use much less power
and I began to fully realize the advantage of being digital.
So all the models we've got at present are digital.
And if you're a digital model,
you can have exactly the same neural network
with the same weights in it
running on several different pieces of hardware,
like thousands of different pieces of hardware.
And then you can get one piece of hardware
to look at one bit of the internet
and another piece of hardware
to look at another bit of the internet.
And each piece of hardware can be used
and each piece of hardware can say,
how would I like to change my internal parameters,
my weights, so I can absorb the information I just saw?
And each of these separate pieces of hardware can do that.
And then they can just average
all the changes to the weights
because they're all using the same weights
in exactly the same way.
And so averaging makes sense.
You and I can't do that.
And if they've got a trillion weights,
they're sharing information like a trillions of bits
every time they do this averaging.
Now you and I, when I want to get some knowledge
from my head into your head,
I can't just take the strength of the connections
between neurons and average them
with the strength of the connections between your neurons
because our neurons are different.
We're analog and we're just very different brains.
So the only way I have of getting knowledge to you
is I do some actions.
And if you trust me,
you try and change the connection strengths in your brain
so that you might do the same things.
And if you ask, well, how efficient is that?
Well, if I give you a sentence,
it's only a few hundred bits of information at most.
So it's very slow.
We communicate just a few bits per second.
These large language models running on digital systems
can communicate trillions of bits a second.
So they're billions of times better than us
at sharing information.
That got me scared.
Right.
But what surprised you or what changed your thinking
was you were thinking the analog
was going to be the path previously?
No, I was thinking if we want to use much less power,
we should think about whether it's possible
to do this analog.
And because you can use much less power,
you can also be much sloppier in the design of the system
because what's going to happen is
you don't have to manufacture a system
that does precisely what you tell it to,
which is what a computer is.
You can manufacture a system with a lot of slop in it
and it will learn to use that sloppy system,
which is what our brains are.
Do you think the technology is no longer destined
for that solution,
but is going to stick with the digital solution?
I think it'll probably stick with the digital solution.
Now it's quite possible
that we can get these digital computers
to design better analog hardware, better than us.
I think that may be the long-term future.
You got into this field
because you wanted to know how the brain works.
Yes.
Do you think we're getting closer to that through this?
I think for a while we did.
So I think we've learned a lot at a very general level
about how the brain works.
So 30 years ago or 50 years ago,
if you asked people,
well, could you have a big random neural network
with random connection strengths
and then could you show it data
and have it learn to do difficult things
like recognize what someone's saying or answer questions
just by showing lots of data,
almost everybody would have said, that's crazy.
There's no way you're going to do that.
It has to have lots of pre-wired structure
and it comes from evolution.
Well, it turns out they were wrong.
It turns out you can have a big random neural network
and it can learn just from data.
Now that doesn't mean we don't have
a lot of pre-wired structure,
but basically most of what we know
comes from learning from data,
not from all this pre-wired structure.
So that's a huge advance in understanding the brain.
Now, the issue is how do you get the information
that tells you whether to increase
or decrease the connection strength?
If you can get that information,
we know that we can then train a big system
that starts with random weights to do wonderful things.
The brain needs to get information like that
and it probably gets it in a different way
from the standard algorithm used in these big AI models,
which is called back propagation.
The brain probably doesn't use back propagation.
Nobody can figure out how it could be doing it.
It's probably getting the gradient information,
that is, how changing weight will improve the performance
in a different way.
But we do know now that if it can get
that gradient information,
it can be really effective at learning.
Do you know if any of the labs now are using their models
to try to pursue new ideas in AI development?
Almost certainly.
Okay.
And in particular, DeepMind is very interested
in using AI for doing science.
And one piece of science is AI.
Sure.
I mean, was that something you guys were trying
when you were there, like this bootstrapping idea
of maybe the next innovation could be created
by the AI itself?
So there's elements of that.
So for example, they were using AI to do layout on chips
that were gonna be used for AI.
So Google's AI chips, their tensor processing units,
they used AI to develop those chips.
I'm curious if just in your normal day-to-day life,
you despair, you fear for the future
and assume it won't be so good.
I don't despair, but mainly because even I find it
very hard to take it seriously.
It's very hard to get your head around the fact
that we're at this very, very special point in history
where in a relatively short time,
everything might totally change,
a change of a scale we've never seen before.
It's hard to absorb that emotionally.
It is.
And I do notice, even though people
maybe are concerned, I've never seen a protest.
There's no real political movement around this idea.
The world is changing and no one really seems
to care that much.
Among the AI researchers, people are more aware of it.
So the people I know who are kind of most depressed
about it are serious AI researchers.
I have started doing practical things.
Because AI is gonna be very good at designing
cyber attacks, I don't think the Canadian banks
are safe anymore.
So Canadian banks are about as safe as you can get.
They're very well regulated compared with US banks.
But over the next 10 years, I wouldn't be at all surprised
if there was a cyber attack that took down a Canadian bank.
What does take down mean?
Suppose that the bank holds shares that I own, right?
Suppose the cyber attack sells those shares.
Now my money's gone.
So I actually now spread my money between three banks.
Okay, so not under your mattress.
That's the first practical thing I've done.
Because I think if a cyber attack takes down
one Canadian bank, the others will get a lot more serious.
Okay, anything else like that, what else?
That's the main thing, that's where I noticed
I actually did something practical
that flowed from my belief that very scary times are coming.
When we spoke a couple years ago, you had said,
you know, AI is like an idiot savant,
but humans are still much better at reasoning.
Right, that's changed.
Okay, explain.
Previously, what the large language models would do
is they'd spit out one word at a time, and that would be it.
Now, they spit out words, and they're looking at the words
they spat out, and they will spit out words
that aren't the answer to the question yet.
They'll spit out words,
it's called chain of thought reasoning.
And so now they can reflect on the words
they spat out already, and that gives them room
to do some thinking in, and you can see
what they're thinking, it's wonderful.
Well, it's wonderful if you're a researcher.
And a lot of people from old-fashioned AI said,
well, you know, these things can't reason.
They're not really intelligent because they can't reason.
And you're gonna need to use old-fashioned AI
and turn things into logical forms
in order to do proper reasoning.
Well, they were just utterly wrong.
Neural nets can do the reasoning,
and the way they're gonna do the reasoning
is by this chain of thought, by spitting out stuff
that they don't reflect upon.
You said at the beginning that the last two years,
the development has been faster than you expected.
Are there other examples of that,
things you've seen that if you said, wow, it's fast?
That's the main example.
It's got much better at generating images and things, too.
But the main thing is that it can now
do reasoning quite well,
and that you can see what it's thinking.
Like, why is that important,
or where does that lead that is meaningful?
Well, it's very good that you can see what they're thinking
because there's these examples
where you give it the goal, you give it a goal,
and you can see it doing reasoning
to try and achieve this goal by deceiving people.
And you can see it doing that.
It's like I could hear the voice in your head.
Yeah.
The other thing we've moved through,
but maybe, I don't know if you have anything more
to say about, is just, it's remarkable
that there are so many tech figures
that now have an important role in Washington, D.C.,
at this very moment where what Washington, D.C. does
could be really important to the evolution,
the regulation of this technology.
Does that concern you?
How do you see that?
Those tech figures are primarily concerned
with their companies making profits.
So that concerns me a lot.
Yeah, I don't see how things really change
unless either there's strong regulation
or this moves away from this for-profit model,
and I don't see how those things happen either.
I think if the public realized what was happening,
they would put a lot of pressure on governments
to assist the AI companies develop this more safely.
That's the best I can do.
It's not very satisfactory,
but it's the best I can think of.
And more safely means more resources
from those companies towards safety research.
Yes.
For example, the fraction of their computer time
they spend on safety research
should be a significant fraction, like a third.
Right now, it's much, much less.
There's one company, Anthropic,
that's more concerned with safety than the others.
It was set up to be concerned with safety
by people who left OpenAI
because OpenAI wasn't enough concern for safety.
And Anthropic does spend more time on safety research,
but still probably not enough.
There is this view among many
that OpenAI has talked a good game about these issues,
but is not living out those values.
Is that your perspective?
Yes.
What evidence do you see of that?
That all their best safety researchers left
because they believe that too.
That they were set up as a company
that was gonna develop AI safety,
and their main goal was not to make profits,
but to develop AI safety.
And they're now busy lobbying
the California Attorney General
to allow them to change to a for-profit company.
There's lots of evidence for that.
And I should give you a chance
to hold up anyone as a good actor here
that people should feel better about.
You mentioned Anthropic.
Is that the name?
Do you see?
Of the companies,
Anthropic is the most concerned with safety.
And a lot of the safety researchers
who left OpenAI went to Anthropic.
And so Anthropic has much more of a culture
concerned with safety.
But they have investments from big companies.
You have to get money from somewhere.
And I'm worried that those investments
will force them into releasing things
faster than they should.
And when I asked you
which you'd feel comfortable working for,
you said none of them, I think,
or just maybe Google.
I should have said maybe Google or Anthropic.
Okay, thank you so much for all this time
and the rest of your time today.
I really appreciate it.
Okay, you haven't got the rest yet.
I haven't got, I'm counting on it.
