On the Road With DRIVE AV

Jensen Huang
2 followers
2 Video Views·Mar 14, 2026
NVIDIA founder and CEO Jensen Huang joins NVIDIA Vice President of Automotive Xinzhou Wu for a drive through San Francisco, discussing what it takes to deliver autonomous driving that feels comfortable, confident, and safe.
Well, this is incredible. We're sitting outside Bucks in Whitside.
So we're going to San Francisco today.
That's right.
I was going to say, let me know when you're in autonomous mode, then I can be less concerned about my safety.
We have the only stack in the world that has an end-to-end AV stack.
That's right.
As well as a traceable, human-engineered AV stack.
That's right.
Which is very critical, because for an end-to-end model, it's very difficult to verify.
Basically, it's always safe, right?
That's right.
And the classical stack is developed based on the protocol and the process.
We know how to ensure safety software.
Yeah, the thing that's a miracle about Alpamayo is that it does amazing things.
And it drives like a human.
That's right.
Because it's end-to-end, whenever there are things that need to be improved,
you know that you can back-propagate that through the whole model, and you'll improve it for sure.
I think the challenge, of course, is Alpamayo, as incredibly smart as it is,
and it can reason about the circumstance, we don't know what it can't do.
That's right.
And so that's the challenge, and that's the reason why our classical stack is so incredibly important.
That's right.
The thing that's really great about Alpamayo and our stack is that
I don't ever feel that Alpamayo would ever be allowed to do something that is out of distribution.
And the reason for that is because our classical stack gets involved and prevents it from happening.
Right.
One thing great about Alpamayo is that the development speed is just amazing.
We have iterated, right now the version is, I think, 2300.
We label based on the number of iterations we have, essentially.
And roughly speaking, this is roughly over a year, right?
So we have an average of seven models being iterated every day.
Obviously not every single one of them make it to the car,
but because this process is just imaging and trajectory out,
the manual steps in between in the development process are just minimal.
That's why we can iterate super fast.
Well, the thing that's really great about end-to-end models is that
whenever you're improving the model, because it's one giant model that's end-to-end,
That's right.
you're not going to fix one thing and break something else.
The classical models, because there's so many different modules, you fix this one thing.
The logic sometimes contradicts with each other.
That's right.
And so it's so brittle.
And you fix one thing, you're chasing cats and dogs around the table, and it's really hard.
Most control systems, which are built the classical way, have the same problem.
And it has so many steps.
For example, today we're actually going to see speed bump based on our model.
If you think about how do you detect speed bump and react to it properly,
you have to break into many, many steps.
You start from perception, and it has to be added in the word model, in the context fusion.
And then basically you have to decide how early you have to slow down by how much.
And maybe even based on different speed bumps, we have to do things differently.
But with the end-to-end model, it basically just happens.
Actually, we didn't even plan for it to just happen.
Yeah, right.
So it's pretty amazing.
There are so many people building end-to-end models.
Right.
But I really think that it's the fusion of the two that's revolutionary.
That gives you the superhuman, well, the amazing driving capability.
And the safety.
That's right.
Back to the model, actually, it does that because it's so big, right?
Actually, we do see sometimes when you try to address something,
something else might get regressed.
Right.
So being able to establish a basically full-blown, very comprehensive,
basically closed-loop evaluation, what we call CLE, is super critical.
This is how we gate our model to make sure sort of the bad babies get fielded out.
The thing that I still love, though, and I think this proves to be true,
that even when you have a regression on El Pomoyo,
your classical stack would always guardrail it.
That's right.
Actually, that's why basically our development speed has been so good
because you always have a backup, essentially.
That's right.
Even if the model has minor hiccups, that's okay.
We will keep retraining and just keep getting better.
You never have a loss of safety.
You have a loss of capability but not a loss of safety.
Yeah.
This is going to be so critical for our robo-taxi when that happens.
Is that what I'm saying?
That's right.
That is the advantage we have as well as we move into L3 and L4
because we have redundant stack, even L2++,
to scale it basically for L3 and L4.
It's just so natural.
That's right.
Yeah, that's critical.
And then there will be the tele-operations as a backup system.
Every robo-taxi service should have an air traffic control center.
And the thing that's really great with our technology is that
for whatever reason, if the robo-taxi were to get stuck,
all you have to do is give it a couple of waypoints,
the human-injected waypoint,
and it should be able to navigate itself right out of it.
Right.
And because it has a safety stack, a backup stack,
no safety would ever be jeopardized.
That's right.
And the other thing our model does really, really well is lane change capabilities.
Because the classical, when we do lane change, we have to select a gap.
We have to do a lot of basically rules and heuristics, essentially,
to make sure it's safe.
So sometimes it's a bit timid, essentially.
When you have cars in the front, you have cars in the back,
basically.
There's so many rules.
That's right.
Yeah, yeah.
And our model, basically, in the city,
it's just doing an amazing job in the lane change.
It's also very human-like, essentially.
It never gives you the feeling of rigidness.
Well, it imitates the human from all the drives that it's been on,
and then it generalizes from that.
That's right.
You know, the basic concept of deep learning
to essentially memorize and generalize
works just fine for a self-driving car.
AI is the single most important technology
that makes self-driving possible, essentially, right?
Yeah.
But I have been working on this, obviously, as you know, for a long time.
But the last two, three years, my God, you know.
Yeah.
It's iterating so fast.
Yeah.
Particularly using an end-to-end model.
That's right.
It was a big deal.
Yeah.
Gen-AI, basically.
To some extent, you can think that the whole industry iterates
on the first architecture, which is, you know,
ML-based perception only, right, for 10 years.
Yeah.
And we made lots of progress.
You know, Robotaxi can be done that way.
Yeah.
But, you know, over the last three years,
we're iterating on, you know, we first bring an end-to-end model.
Now we're adding reasoning to it.
It's really two generations in three years.
The two fundamental technologies that we're building on ToddLift,
you know, so, of course, this end-to-end model architecture is revolutionary.
Uh-huh.
But what differentiates us,
and what's going to make it incredibly fast for us to deploy,
to get to level three, four, and level taxis,
is the fact that we have a classical stack as a guardrail,
as a safety system.
Because we have a reasoning system.
Uh-huh.
So the long tail of experiences that are probably not in our dataset
can still be decomposed into routine type of experiences
that are in our dataset.
Yeah.
And the car will be able to reason its way through,
you know, how to navigate, and so how to operate.
And so I really think that between these two ideas,
no matter what the reasoning system decides to do
or achieve,
you always have a safety guardrail behind it.
And so this iteration system that we have
is going to go move incredibly fast.
That's right.
Reasoning and guardrailing is going to be just revolutionary.
That's right.
All of our L2++ work,
that entire flywheel,
you know, collecting data on the road,
and, you know, processing the data.
Uh-huh.
Now we enhance that by basing URAC and the Cosmos.
Yeah, good job.
Good job.
Who knows what a human driver would have done?
That's pretty good.
Yeah, that entire flywheel is essential.
Yeah.
As I said, right, in the model side,
you know, close group evaluation is so important.
We're running essentially, you know, as you know,
2 million tests every day.
And they are all pixel, basically,
reconstruction of the world.
It's just amazing, basically, how the technology evolves.
URAC is so great.
Yes, that's right.
It's so critical.
And the Cosmos can be introduced as well,
which we are working on, basically,
to add Cosmos can cover the real rare corner cases.
And also it can use to amplify the URAC.
URAC is just reconstruction,
but if you put a Cosmos on it,
it can, you know, change the, you know, background,
change the weather.
Yeah, you could up-res it.
That's right, exactly.
Yeah.
The combination of these technologies
has really been revolutionary.
It's probably an accident here somewhere.
Well, they really ought to get one of our cars.
Look how beautiful it is.
Here we are driving autonomously into San Francisco.
You know, this interchange here
always gives me a lot of anxieties
because there's so many different highways.
And, you know, if you just miss one,
you're going to be gone.
You're going to be late for about 45 minutes.
And so it's so terrific to do this autonomously.
So exciting.
You said 10 cameras.
Five radars.
And how many ultrasounds?
Twelve.
Twelve ultrasonics.
This spot actually can be pretty tricky.
Wow, that's nice.
Because, basically, there's a lot of occlusion,
and when you get on the highway,
you know, sometimes the car just stops there,
but you don't see it until, basically,
your turn makes its turn.
So you have to control speed.
Well, I know this exit very well.
I come here all the time.
You know, people underestimate
how gigantic of an effort it is.
You know, you've got to build a chip,
build a computer,
and even building the computer that is functional safe
so that you've got the chip,
you've got the system itself,
the system has functional safety designed into it,
the software stack,
the driverless software.
Then, you know,
every component of even the classical stack
need to get to a certain ASO, basically, level.
We are going to make a lane change here.
Let's see how we do it.
See, this is how.
Yeah, fantastic.
Very human-like.
Yeah, that's so human-like, right?
Yeah.
Hard to describe in code, essentially.
That's right.
Most rule-based would have had the car
completely pass us.
Yeah, yeah.
And then we make one deliberate move.
That's right.
And that's one stage,
what we call speed adaptation.
We try to speed up or slow down,
try to get in the gap,
then start a lateral move, you know?
But that's not how humans do it, obviously.
Our mind is just mimicking
what everybody else is doing,
so it's amazing.
You know, there's, of course, the AI in the car,
but our AI starts with
even AIs to curate the data that we collect.
That's right.
So that...
All right, that's an interesting scenario.
Let's see how we handle it.
Oh, we detect from here.
We decide to do the lane change.
Uh-huh.
That makes a lot of sense, actually,
because once you cross the street...
That's right.
Once you cross the street,
that's not very much distance to stop.
And so we're going straight.
That's going to be interesting.
So we'll go past them
and then do a lane change back, I think.
But it's like the whole street is passed.
This actually makes sense.
Yeah.
Let's see how we do this.
It's blocked on the right.
Yeah.
It's already blocked on the right,
and so it anticipated that,
and so it moved us to the middle lane.
Yeah.
Oh, look at this.
This is great.
So this will be tricky.
Let's see.
We're about to pass a bus.
All right.
Okay.
Wow.
That's so graceful.
Yeah.
It's more peaceful than humans driving.
Actually, this is what the media reporters,
they feel, right, safer, essentially.
It's confident.
It's calm.
It's safe.
So we have all the AI models inside the car.
We have the AI models just to curate the data.
Right.
We don't want a whole bunch of wasted, same thing,
to retest it with the same thing,
and so you're looking for out-of-distribution things.
You're looking for like circumstances.
We have a terminology for this.
It's called a functional scenario tree.
If you think about it, it's really basically a bunch of nodes
which are the cases we need to cover,
and then we have an AI tool to curate data
to fill in these different nodes.
And when we see a new scenario pop up on the road testing,
then we're going to create one more layer, right,
or one more node on this functional scenario tree.
Yeah. Okay.
So this is called the functional scenario tree.
Yeah.
So this is a double-parked vehicle, basically.
Yeah, literally we crossed the lane,
but it was perfectly safe to do so.
That's right. Yeah, yeah, yeah.
You'll see some hesitation sometimes.
We're definitely, I would say,
more conservative than an average driver,
because, for example, we'll get to at least a moment of full stop
in front of every stop sign,
which is the right thing to do, obviously, but yeah.
And so we have the data.
We have AI models for your functional scenario trees.
Mm-hmm.
You have AI models for synthetic data generation.
That's right.
We have AI models for Neurac, for sure.
Right.
We have, yeah, for drive log to reconstruction.
Right, yeah.
When you do engineering,
you can keep progressing and keep adding more capabilities
only if the foundation is strong.
Is strong, yes.
You know, when you know you're taking,
you're investing all of these new capabilities,
but you know you can only do so with confidence
if you know there's no regression.
Right, right.
That's really one of the most powerful things
about our classical stack.
It's so safe.
Every one of these iterations, every single month,
you're quite confident deploying it into the road.
You know, between the evaluation systems
and the validation systems and the methodology
and the guardrails.
That's right.
It's always safe.
From here on, our velocity is going to be so high.
Yeah, yeah.
So this is a residential area.
So in this area, as I said, the bus is a challenge,
and then a lot of random parked cars,
we call them DPVs, double-parked vehicles.
A nudge around them is really, really critical.
You've got a lot of delivery vans and trucks.
A lot.
That's right, exactly.
Weekends are better, but weekdays,
you know, there's just a lot of them.
And DPV nudging is also, you know,
quite difficult to get right.
But the model, again, it's learning, you know,
what a human does, essentially, which is great.
But even sometimes humans make mistakes, you know.
If you make the wrong decision,
you will end up in an oncoming lane, you know,
and sometimes you get stuck, you know.
That's why I think what we are doing
in the current implementation
is still slightly on the conservative side
because the cost of making a wrong decision is too high.
Yeah.
Safety first.
Safety first, exactly.
It's okay to get stuck, essentially,
for an L2++ product.
Yeah, exactly.
I think it's okay.
Yeah, that's right.
If the human's in the car.
That's right, yeah.
That's so great that we got the highest rated NCAP.
I know, yeah.
First time out, it's incredible.
That's right.
This is the right decision as well
because it's oncoming.
We are so patiently waiting and...
Human might be slightly faster,
but I think we...
I don't think so.
I think what he's done is probably the same.
Interesting.
NCAP 2025, so rigorous,
and the Europe NCAP,
the standards are so high.
Yeah.
And we got the highest score.
That's crazy.
That's so exciting.
I knew we were going to pass,
but to get the highest score is really incredible.
Yeah.
That's, again, many years of hard work.
Yeah.
Which is also very important because all this work...
But that's our safety guardrail.
That's right.
We'll be translating to L3 and L4 capabilities.
Not bad, not bad.
This is our image.
Oh, wow.
Excellent.
That was a good speed bump.
Look at how comfortable it is.
Okay.
Oh, good move.
Very nice.
Yeah, makes sense.
Yeah.
So some people just drive right over a little pothole.
I kind of drive around it personally.
Uh-huh, uh-huh, uh-huh.
I think we have to work on it,
but that's a very important feature to have,
especially on a highway, you know?
Yeah.
If you don't avoid a pothole...
But what do our drivers do?
Our drivers?
Just now, it would probably...
Oh, you mean the data collection drivers?
I wonder what they do.
I think they will get around it.
You think so?
I don't know, yeah.
But...
I would say just now...
Hopefully, some of them do,
and we can always select those data, you know,
in our fine-tuning step.
And we can also do reinforced learning.
Yeah.
Reinforced learning is very effective.
Really have a more capable model, right?
This is what we are working on for the next generation, right?
I think reason is one thing,
and having memory is very, very important.
It can get...
It will need less help from the classical stack.
Yeah.
It's so incredible that a billion-parameter model
can encode in, you know, all of this.
But I guess in the end,
the question is, for the car,
looking at all these features,
what are the salient features it needed to encode?
It probably is not that much information.
Right, yeah, yeah.
But the salient features necessary to predict the trajectory...
Uh-huh, uh-huh, uh-huh.
...is probably extremely, extremely small.
That's right.
We see a lot in the world,
so we are impressed by the amount of information,
but how much of it does it really...
does really necessary to contribute to trajectory prediction
is probably very, very small.
Very small, yes.
That's so great.
Yeah.
It started to turn before the car even passed us.
Uh-huh, yeah.
That's what the model does.
That's what people would do.
That's what people would do, yes.
Channelized cone, Jensen.
We set it up all for you.
That's so great.
Channelized cone is also amazing with the model
because it's so hard to get it right with the classical stack.
You have to kind of generate lanes from cones.
Right.
I would have zero concerns doing my email
all the way through San Francisco, you know?
How nice is that, right?
Uh-huh.
Nice.
Very nice.
What a great drive.
The Alpamayo AV stack drives very much like a human.
It imitated a human and generalized incredibly
to all these different scenarios,
and the classical AV stack gives us the safety guardrails
that has been fine-tuned and tested,
and with all of the engineering rules that have been put into it
gives us incredible safety.
And so when we were driving,
the two of us were just talking, enjoying San Francisco,
and it was just incredible.
This is the way transportation should be like.
That's right.
100% comfortable, 100% confident,
and completely revolutionary.
