AI is the third brain of a jet fighter | David Fravor and Lex Fridman

Neural Jetstream
2 followers
1 Video View·Sep 20, 2024
Can we just pause on that just for me from like a human factors perspective and also an AI perspective?
What's?
How difficult so there's like when there's two people there's also a third person. That's the
AI part the some level of automation like autopilot maybe that's correct
Maybe you can kind of talk about the psychology of like you said making decisions really quick 80%
How do you deal with another brain?
Working with you and then also the automation. Is there an interesting interplay that you get to learn?
And also as that changed throughout your career, I imagine it got you gotten better in terms of the automation or perhaps not
Well, I can tell you so that let's say there's a bunch of questions. No, this is this is good
This is good. And this is I'm enjoying this because now we actually get to talk about something other than a tic-tac. So
So let's start with the a6. The a6 was really an analog airplane
That was built in the 60s. All right, and
There's been studies done on the crew coordination, which is the interaction between the pilot and the bombardier navigator
So we would fly low at night in the mountains. So I was stationed up in Whidbey Island, Washington
so you've got the Cascades and
incredible amount of time and we would get in the simulators because unlike
Normally people think terrain following and there's the radars the 111 the b1 has a system like this
But it'll the radar can see and it'll fly it basically flies a straight line
So it goes up and over mountains and back down and up over mountains where the a6 was really manual
So you do this low-level routes where you're gonna you're gonna fly in the mountains at night
You're gonna be at you know, 500 to a thousand feet above the ground
Ripping through like fog layers because you don't need to see outside. You're you're literally a little TV screen
What are you looking at most of the time? So you just as a screen? It's this really primitive
If you look at it now what we did you think wow, that was crazy, but it was really fun
So it's similar to like the flare stuff. Is that is no
Are you this thing is totally radar based now the airplane had a flare balls target recognition and multi-sensor was called a tram
You're looking at like basically like dots of hard objects
He's no actually what it is is the bombardier navigator had a radar and he was getting raw feed off of a pulse radar
In front, okay, so it's just basically mapping the mountains
So if you look at a mountain on a radar and you're coming up on it
The front side is gonna be it's gonna give you a really bright return and then the back side
It's just gonna be a giant shadow because you can't see on the other side
So the bombardier navigators would do that and we they would have charts and they could shade their charts knowing that hey
If we turn a little bit left here, we can get in this valley
We can sneak up this valley and then go around the back side of the mountain
Which is what the airplane would do and it's and sorry to interrupt. I'm gonna just keep asking dumb questions
I apologize, but the pilot
Can you at a high level say what the pilot does versus the bombing bombardier
So you're you're actually just control. I'm flying the jet
I have the throttles the stick and I have a it's about a probably a
four inch or six inch wide by maybe
Four inches five inches high. It looks like it's literally a CRT
that's how old it is a CRT screen and what it would do what the radar would do is the
The bombardier navigator is looking at his radar
He's looking out about twelve and a half miles in front of the airplane
So he has the range really scoped down because the radar can see a lot further
He's looking at about twelve and a half miles when we're in the terrain mode where we're dodging mountains and stuff
And what the pilot has is there's they're called range bins and there's eight of them
so the very far range bin is the twelve and a half mile, you know and
the closest range bin it's a thing and it'll be like between like a half a mile and
Or a quarter mile to three quarters of a mile
The next one might be three quarters of a mile to two miles and then it just keeps going out like that
So if there's a mountain in front
Let's say we're on a flat plane and there's a mountain out in the distance at 15 miles and we're just driving right at it
So when we get to the point where it hits twelve and a half miles where the radar is going to see it on his
Scope my 12 my range bin for that would pop up and it would show like a big bump like a mountain
And then as I got closer to it
The next range bin would pop up and show it and I could see that that bump was moving towards me
And then if I turned a little bit, you know to go over here
I'd see the mountain go over to the right-hand side and I could do that, but it wasn't like a
Video game it's literally like if you think of the original Atari's. Yeah, but you build up
I imagine that you start to get a really deep sense of like the actual three 3d
Environment based on that little Atari's it's you're exactly right and you have to you have to train
So there's been studies the matter of fact a lot of the basis and people probably argue with me, but it's true
There were studies done watching a six crews in our simulators
We call it the wist the weapon systems trainer
And it was not even a motion
It just kind of sat there and you just you could fly these things and train that they would inject into the system
But the crew coordination so you get so my first
My first fleet bombardier navigator who I'll name him his name's Chris Sato
He's a works at Apple
Pretty high up the MIT grad. I think computer engineering
He's scary smart. So Chris could really work a matter of fact all the guys that flew us
So there's another guy Matt who also worked at Apple who's now at SAP. We did our first night traps together the bond between us
I mean, it's one of those things that you just you're never gonna forget
But Chris and I when we started flying together, we were actually the most junior crew in the squadron
We'd spent a lot of time training and and and Chris was amazing at how he could work the system
One because he was extremely brilliant. He was to had that inquisitive mind of oh
We can do all these different things and there's all these degradation modes
but we spent a lot of time to see how good we could actually get because and it's
You almost talk in partial so as the BN is looking at his radar scope Chris would say
I've got rising terrain
That's just what they say
Showing rising terrain at 12 miles and I'd see the little bump and I'd say got it
This is gonna go to your question on the autonomy and how you work with two heads
Yes, so when you first get together the interaction
It's it's it's almost like you have to rehearse it. You have to know and you talk in full sentences
The more and more we fly together
Chris could go
I'm showing and he'd get like rising out and before he finished I'd say I've got it
So you end up starting to talk in partials because I have to trust him
Like I mean, I think there can be no I can have no doubt that he knows how to do his job
Because I'm literally looking at this little scope
That's not giving me this continuous picture of that mountain moving
Remember the mountains here and then it's gonna pop up here and then it's gonna pop up here
Because there's gaps in the coverage on how the system was set up remember
It's an analog system to where he is telling me like I can't see all the way to the left and he he's got a wider
Scope on the radar, but my screen doesn't show that
So he's telling me start a left turn how to avoid a hard turn, you know
And we would do that. So my truck is all happening quick very quick
Well, you're doing we would typically fly between 420 and 480 knots of ground speed miles an hour
Well
420 is 7 miles a minute. Okay between 7 and 8 miles a minute is what you're flying at night
I mean, I broke out of clouds. I mean, I remember him and I flying we were on it's the IR
It's called an IR route an instrument route. That's low. They're all around the country
There's IR 344 that we used to fly which would coast in off of Oregon fly from the land
you go out over the ocean turn around and then you could practice actually coming in on a coastline and
we were flying and
We ended up in the clouds keep in mind
We're between 500 and a thousand feet in the mountains and we're in the clouds
Like you can't see anything and I had to turn off our red lights that flash, you know
They're called anti-collision lights because it was reflecting off the clouds and it starts to bother you just gets annoying
so I turned it off and we were flying we're flying we're flying we break out of that coastal marine layer and
We break out and it's it's a decent night. And this is right by Mount St. Helens
This is kind of we're coming in
So we're coming in from the east and we're just north of Mount St. Helens is where the route goes
And you look up, you know because you can kind of see the silhouette of this mountain that's right next to you
But you're flying along you're just like, you know, you got to trust and you can see houses. You can see the lights
They're above you. We're literally below people's houses flying down these valleys and stuff. So just incredible experience
So when you take that and then you move into an f-18f, so now we're into modern technology. It was actually built in this century
And you're flying so now, you know, the Wizzow is behind us and we're not doing those night low levels
But that same type of crew coordination that has to happen
Because what you're doing is you're sharing the load
So most of the communications that go out of the airplane the Wizzow does all the talk and he's got actually he uses it with
That's the weapon systems operator in the back of an f-18f. So he's gonna run well
Yeah, the radar kind of runs itself now, but we have a situational awareness display and it's it's linked to all the other air
Just like curiosity. What's the situational awareness display because that term comes up a lot
Think of it as a God's eye view
So if you have a the back of the Super Hornet as well
The block twos has about an 8 by 10 display for the Wizzow's
That they can look at the pilots is smaller
It's down between is it's a 6 by 6 between his legs and they're they're getting ready to redesign that
Boeing is but when you looked it'd be like if you put your airplane and you're looking down
So all the stuff like if your radar seeing bad guys out in front of you be like looking down going
Oh, I'm right here. I know there's bad guys out here. My wingman is over here and it shows everything
It's just like it gives you you can look at that display and go. Oh, yeah, I can see where everything's at
I can see if one guy's trying to target another guy you can it shows you all this
It's an incredible amount of knowledge that comes up for the crews to maintain
The overall picture of what's going on big picture sensor was because it's happening so fast
And this is with that autonomy piece. This is the third brain
So we're all looking at it and the third brain is doing fusion. It's pulling stuff together going. Oh, this is all this guy
This is this guy. This is this guy. It's sending it out through the link
So all the airplanes are talking to each other through this digital network
You know that we don't even see it just says that airplane says hey, I'm over here and it tells us and we go
Oh, he's right there. And then we can go he's his airplane says, oh, I'm looking at this airplane this bad guy and it shows us
Oh, he's he's over there and he's looking at this guy. I mean, it's an incredible amount of
Visual intake because your eye you can hear a lot
The third brain is doing the
sensor fusion
The integration of the different sensors and gives you a big picture view
What about the control like is there and I apologize is if this is a dumb question
But you know people use a high-level term of autopilot. How much is there?
Let's use a loose term of AI. How much automation is there? How much AI is there in helping you control there?
The AI piece would be more of a control loop because the digital flight controls so the airplane actually
They had to make the airplane easier to fly and when I say easy
It's relative because people go I could do it because I did it on flight sim
It's real life is a lot different in flight sim. You have no apparent fear of death
you'll do things on a simulator that you would never do in real life, but
The the autonomy in the airplane to allow you to manage. I mean because you think about it
You've got a radar that's feeding you data
you've got a targeting pod that's feeding you data all that stuff is hooked to your head because you've got a joint helmet mounted queuing system on
That it basically maps the magnetic field in the cockpit so it can tell where your heads at looking
So if I turn my head to the right the radar will actually look to the right the targeting flare will look to the right
I know by the way
The back seater has a helmet on too so he can look to the left and he can do things
So depending on what sensor he's controlling
So if he's got control of the targeting pod and he looks left the targeting pod looks left
But if I have something where I want to lock a guy up that I don't see that maybe the radar didn't see but I can
Get over and now point the radar, you know get the because it's a it's a phaser a radar now. It doesn't really scan
There's there's all kinds of cool stuff that that technology
Brings because if you just if you went back 30 years and said hey or 40 years ago and said hey
We're gonna have this helmet and you're gonna be able to slew everything to your head
And I don't mean a mechanical setup, but I mean literally you're just gonna map magnetic resonance and go
oh look and then I can I can literally slew my sensors this fast and then mash a button and
Transfer, you know high quality coordinates from a system into a joint, you know
JDAM which is a joint direct attack munition
That is the GPS bombs that you see all the time and then let that thing fly and I'm solving this problem in seconds
Minutes or hey, I got it
We're gonna have to menstruate coordinates and you know
You bring back the data and then they do all the targeting for it
And then they send another group out to get it instead of all that now. It's that fast
So there's a okay
I mean
We probably don't have enough time to talk about the beautiful fusion of minds that happens when two people are flying
Controlling the plane but at a high level. This is a really interesting question
For people who don't know what they're talking about like me, which is
What is the difference between a human being and an
AI system like what?
can
What is the ceiling of a current AI technology for controlling the plane? Like how much?
Does the human contribute is it possible to have automated flight? For example, like what is the hardest part about?
Flying that a human does
Expertly that an AI system cannot in warfare situations in in flying
fighter jet plane, so I would say
AI systems are usually black and white when you write the algorithm for an AI system
It's it's it's it's really it's basically you're taking thought and turning it into a giant math problem is really what you're doing
Right, so you've got this logical math problem math problems are there's there's there's a line
It says I can or I can't and it's a it's a very finite line, you know
But you can go up to the line where a human we all have gray areas where we go
Maybe yeah, I'll try it
So he's gonna operate within that gray
So if you took if you take an airplane and say and I'll just take a hornet for a while
It was super hornet doesn't matter any airplane and you go here is the flight performance model of the airplane
So if you know in an EM diagram is the energy so it basically says the airplane can fly as slow as this
It can go as fast as this it can pull this many G's force of gravity, you know
So one two, three, four five six seven and then based on the airfoil design and everything else and how it can pull
Here's how it's gonna fly, you know, because it's really physics based
Well, if you depending on how you write the AI, but typically AI you don't want the airplane to leave controlled flight, right?
You want to maintain it so that it is flying in a controlled envelope
Where there are times and you can go back to World War one
Where people intentionally departed the airplane from controlled flight in order to obtain an advantage
Which is that's where the human goes. Can I do this?
I know it's outside of where I would normally go
But I can do that so you can do some crazy things now, especially since the flight control logic in modern airplanes with digital flight controls
They're extremely forgiving so you can literally
I've done things in Super Hornets that literally even as a pilot inside the airplane
You're just like wow
I cannot believe it just did that like it'll flop ends
Which defies most logic and I guess you know in a way you could probably program it
But I still think when you get to the edges that may or may not give you an advantage
There are things that a human can will do
That AI won't and I don't think we've got to the point because how do you how do you map illogical solutions?
You know most AI is logical. It's it's based on some type of premise when you write the algorithm to control it
There's bounds
