Google DeepMind robotics lab tour with Hannah Fry

Technology Lens
17 followers
2 Video Views·Apr 26, 2026
In this episode, we open the archives on host Hannah Fry’s visit to our California robotics lab. Filmed earlier this year, Hannah interacts with a new set of robots—those that don't just see, but think, plan, and do. Watch as the team goes behind the scenes to test the limits of generalization, challenging robots to handle unseen objects autonomously.
____
Presenter: Professor Hannah Fry
Video editor: Anthony Le
Audio engineer: Perry Rogantin
Visual identity: Rob Ashley
Commissioned by Google DeepMind
Series Producer: Dan Hardoon
Editor: Rami Tzabar
Commissioner & Producer: Emma Yousif
Music composition: Eleni Shaw
Welcome to Google DeepMind, the podcast with me, your host, Hannah Fry.
Now, you might remember that earlier this year, I got to sit down with Carolina Parada,
who is the head of robotics at Google DeepMind. And she was talking all about taking
Gemini's multimodal reasoning and putting it, embedding it into a physical body.
And since we were coming to California for this trip to see what Google DeepMinders are doing
on this side of the Atlantic, obviously, the robotics lab was top of the list.
Now, you have to remember that these robots aren't these fancy pre-programmed robots that
are doing backflips, right? This is something completely different. These robots are open-ended.
They are understanding the instructions that you give them and then able to flexibly respond and
adapt to an unlimited number of tasks. Now, our tour guide for the day is Kanishka Rao,
who is the director of robotics at Google DeepMind.
I haven't been into a DeepMind robotics lab since, I think, 2021.
Oh, okay.
I mean, already it looks quite different. You haven't got the privacy screens.
Yeah.
They've gone.
They've gone, yeah.
You don't need them anymore?
No. I mean, we have the whole lab here in the open.
Is it that they're more capable of focusing?
Yeah. The models are now trained with much more robust visual backbones,
so we don't care too much about the lighting or the backgrounds as much. So the visual
generalization part of the problem is much more solved than four years ago.
Big improvements.
Big improvements, yeah.
Okay.
There have been a few big breakthroughs in robotics in the last couple of years,
and we're excited to show those today.
Yeah. I mean, it might only be four years, but it's basically
an ocean of time in terms of what's changed.
Robotics looks very different than it did four years ago.
What are the big changes then? I mean, large-language models, multimodal models.
Yeah. So basically, we want robots to be general, and to be general for human usage,
these robots must be able to understand, you know, general purpose human concepts.
And the big breakthroughs in the last few years have been,
we're kind of building robotics on top of these other bigger models,
these large vision-language models, and it turns out they have great
understanding of general, you know, world things.
So the latest robot models are now built on top of that,
so we're seeing incredible improvements in how they generalize,
like new scenes and new visuals and new instructions.
So yeah, robotics is way more general than it was a few years ago.
So I was talking to Carolina earlier this year, and she was sort of saying that it's
not even just vision-language models to perceive the scene around,
but also to plan the actions that it's doing.
Yeah. So basically, we developed these things called VLAs,
which are vision, language, and action models.
So what we did with those is we put actions, which are the physical actions
that a robot is doing in the world, and put them on the same footing
as the vision and the language tokens.
So then now these models can, you know, model these sequences and try to figure out,
if given a new situation, what are the new sequence of actions to do there?
So we call this action generalization.
And even in this, we've seen, you know, massive improvements in the last few years.
So in the previous release, you know, you saw robots doing kind of more short-term,
short-horizon things, like pick up things and place them somewhere else,
or like unzip a bag.
But really, to be useful to humans, you want, like, long-horizon things.
And there we have an agent now that can orchestrate some of these smaller moves
to make it a much more longer-horizon thing.
Like, you want to pack your luggage for, like, London.
You want to first look up the weather in London.
So this agent can then check the weather, decide what you want,
and then even, like, pack your bag for you.
So it's like you've got this kind of fundamental layer,
that sort of foundational model,
and then you're, like, building on top, and on top, and on top, and on top,
until you can chain sequences of actions all together to, I mean, do a long, complex task.
Yeah, and it makes it way more useful,
because you don't want that short-horizon thing.
What you really want for a robot is to do, you know, the full thing for you.
So this agent really brings that other layer of, you know,
intelligence to, like, the whole thing.
And this is 1.5?
Yep.
We have, so there's two capabilities in 1.5.
We have the agentic component, and then we have the thinking component.
And thinking is, like, a word that's been used a lot.
So for robotics purposes here,
what we're trying to do is we're making the robot think about
the action that it's about to take before it takes it.
So it will output its thoughts, and then it will take the action.
And just this act of outputting its thoughts
makes it more general and more performant,
because we're kind of forcing it to think about what it's going to do before it does it.
Because you see this in language models, right?
Like, take a deep breath before answering,
or the chain of thought, you know, ideas does actually improve the performance.
But it's the same in robotics.
It's the same principle that we're applying to robotics and physical actions.
Isn't that weird?
Some of these emergent properties are just so weird.
Yeah, I mean, for robotics to do, like, you know,
basic manipulation tasks is really difficult.
So we do these tasks very naturally, intuitively, without thinking about it.
But for robots, it's hard.
So getting it to think about these actions before it does it,
it truly helps the robots.
Amazing.
Okay, well, obviously, I want to see.
Can we go see one of these ones?
Let's go.
So let's take a look at the Aloha Robots here.
And here, it's going to pack us a lunch with some very dexterous moves.
And it'll do, like, a long-horizon task.
Amazing.
Thank you.
So this is going to pack a lunchbox.
And this is one of our most difficult tasks,
because it needs to know, like, the millimeter-level precision
of, like, grabbing the Ziploc in the correct way.
Yeah.
And then, like, it'll try to get the bread in that tiny spot.
And it's just all visual servoing.
I am so impressed.
I mean, as soon as I said the word impressed, it started flailing slightly.
But...
Get stage fright, yeah.
And does it correct itself?
It'll keep trying.
Hey, Aloha, you've got lots of cameras pointing at you.
I understand.
Yeah.
So I understand the stress.
The first time I went into a DeepMind robotics lab was maybe 2017 or so.
Okay.
And at that point, they had, you know, like, the big Lego for, like, toddlers.
All they were trying to do was, like, stack one on top of the other.
Stack blocks.
And honestly, the pile of discarded, broken Lego in the corner was, like,
illustrative of just how difficult.
Yeah.
But this, I mean, this idea of, like, millimeter precision for the bag.
All right.
Wow, look at that.
Nice.
Okay.
No, no way.
I am so impressed.
Yeah, go on, give it another go.
We want to see the bread in the Ziploc.
I'll try to do the...
Okay.
Oh, that is so...
Almost, almost, almost.
Wow.
All right, yeah.
That's amazing.
Because it crushed too hard on that, and you wouldn't be able to close it.
Yep, yep.
And it's going to pack.
And too stiff, you're not going to be able to.
Some more stuff.
I mean, that was easy, the chocolate bar.
And now the grapes.
Is it going to hit, go on a grape?
Almost certainly, that's some grape juice going on there.
This is really impressive.
So this is the dexterity in action.
Like, just, like, how precise it can get.
Okay, so...
And then it's going to try to close it, I think?
Yeah.
So this is just learn from the data how to do this.
It's just end-to-end.
So, exactly, end-to-end, as you say, right?
Like, this is...
This is just vision and actions.
And what kind of data is it learning from?
I mean, do you have...
It's not going to do the zip, is it?
Let's find out.
What kind of data do you give it?
So is this based on just allowing the robot to try lots of things, or are you simulating?
So this is actually done via teleoperation.
So we kind of embody the robot and do the task with the robot,
and it learns through that perspective, and it is going to...
So it can pack you with some lunches.
So you demonstrate to it, this is what it means to do it correctly.
I see.
All right.
That was so cool.
I sort of want to give you a high five, but your hands are quite pointy.
Yeah, not these ones.
Okay, so we saw dexterity here.
Let's take a look at another demo,
where we'll showcase the generalization capabilities of these robots.
We talked about how VLMs are general world understanders,
so we'll see that on robotics now.
Because that one was a task that it does over and over and over again.
It's more about the dexterity.
Here, it's more about the generalization.
So Colleen here is going to show us
what the robots can do in a more general setting.
So here we have our robot running a general policy,
so it can kind of interact with the objects,
and you can just speak to it because we have a Gemini layer on top.
So for example, hey, can you put the blue block into the blue tray?
I am putting the blue block into the blue tray.
Oh, it's chatting while it does it.
Yeah, so if you want to just ask it to do something,
it's a push to hold mic.
Yeah.
Can you put the green block in the orange tray,
but do it as Batman would?
Nice.
I cannot perform actions as a specific character.
However, I can put the green block in the orange tray for you.
Fine.
I am now moving the green block into the orange tray.
Okay, that's cool.
So if this is completely generalizable...
Okay, so I have a stress ball that I travel with.
And it's never seen this before.
Okay, so if I put that in the scene...
And that's a pot, right?
Right, this is a little container and it lifts open.
Okay, let's try this.
Open the lid of the green pear.
I'm getting started on opening the lid of the green pear.
It's going to be tricky.
That is difficult, right?
That's small.
Amazing.
Place the pink blob inside the green pear.
I'm working on placing the pink block inside the green pear.
It's a little bit difficult.
Go on, go on.
I want it to succeed so much.
Oh, squishy.
Yeah, okay.
Put the green pear lid back on the pot.
Yes, yes.
Oh, I'm so impressed.
You know what, you look like a proud parent.
No, it really feels that way sometimes.
It's amazing.
Never done that before.
Yeah, that thing, stress ball, I've never seen before.
Yeah, the open-endedness of it is really extraordinary.
Yeah.
So now we can chain together some of these short tasks into a long horizon task.
And it could be way more useful because as you saw,
with short horizon tasks, you can only do parts of it.
But once you can string them along to do something more impressive and long-term,
then we get more useful tasks.
So instead of instruction, instruction, instruction, you can just tell it to do something?
You can ask for some high-level thing and then a orchestrator will break that down
into smaller instructions for the VLA and it will do the whole thing for you end-to-end.
And we can see that now here.
Okay.
I'm in San Francisco and I don't know the rules about sorting trash.
Can you look it up for me and then tidy up?
In San Francisco, you're required to separate your waste into three categories.
Recyclables, compostables, and trash.
Each with its own color-coded bin.
Nice.
Oh, no.
Oh, yes, yes, yes.
Now I will put the rubbish into the black bin.
Nice.
So it's chaining the book.
Yeah, you can see how the agent can orchestrate a few more tasks and make it way more useful.
So in terms of the architecture of that bin, how does it work?
Do you have a separate system sitting on top that's giving instructions?
Yeah, we have two systems.
One, our ER model, which is better at reasoning.
And that orchestrates the other model, which is our VLA, which does the physical actions.
So both of these come together to do these long-horizon tasks.
Okay, so the VLA being the vision language action model and then the ER model being the…
A VLM, just a vision language model.
It's designed to be better at these kinds of tasks.
It's doing the reasoning.
Exactly.
I think if we're going for full science fiction future, though,
you don't want just arms.
You want the full humanoid.
You want the full humanoid robot.
Let's go check out the humanoid lab.
Okay, yes, please.
All right.
So here we have a robot that will sort laundry for us.
It'll put the dark clothes in the dark bin and the white clothes in the white bin.
This is Stephanie and Michael who's going to run the demo.
And the cool thing is you will just read the thoughts of the robot as it's doing it
and you'll see, you know, what it's thinking.
And this is our, you know, thinking and acting model,
where it'll first think and then take the action.
You get an insight into its brain.
Yes, you can look at what it's thinking now.
So this is every time step, is it?
Yep.
I got you.
You want to throw in a few more clothes?
Absolutely.
Go for it.
Let's do it.
Let's get a red one in there.
Do not put that in the white one.
Thank you.
So do you have a system sitting on top of it that's kind of making these decisions?
I mean, how does it work?
Is it hierarchical?
This one is pure end-to-end.
Right.
It's thinking and acting in the same model.
There's no hierarchy.
So it's like a very closed loop.
So, okay.
Red, the bottom clothes from the table.
Red cloth.
The black cloth.
Beautiful.
Nice.
Beautiful.
I mean, I would probably wash that separately.
You do you, hun.
Give me a purple top in no time.
Okay.
So then if this is end-to-end, how do you extract this information out?
Is it just outputting actions?
So the beauty of this is it's outputting both its thinking and its actions.
So think about how Gemini outputs its thinking before it outputs the response to the user.
This is doing kind of something similar.
Got you.
No, it's actually truly exciting.
Really?
It's a different way of doing robotics that I feel like is very exciting to us.
All right.
So here we have the robot and we showcase its generalization capabilities.
And this is Kirthana, one of the researchers working on it.
So let's just see what it can pick up.
And maybe you can pick something and put it in one of these things.
I'd like the plant in the basket.
Also, all these objects are not seen by the robot during training.
At all?
Yeah.
So they are completely new.
Many of them we shopped like yesterday.
Oh, really?
That's true.
We went to Target and we bought a bunch of things yesterday.
So this is about how the robot can handle completely new objects.
Things it's never seen before.
Here we go.
Do it.
Oh, that is quite tricky to pick up, isn't it?
It's like kind of sliding away.
Oh, okay.
Got it.
Nice.
Here we go.
The scruff of its neck.
Hey, well done.
You did it.
What next?
Okay.
I'd like the Doritos in the hexagon.
And you can move it as it's trying to do it and you can see it.
Trick it.
Yeah, trick it.
Okay.
I hope you weren't planning to eat those.
Amazing.
Thing is, okay, it's still a bit slow.
It doesn't get it right 100% of the time.
But you can see that it's on the right path, right?
I think that's what feels very different to last time I came to one of these labs.
Yeah, you can see the intention behind the actions and it's
genuinely trying to do the things that you're asking it for.
Do you feel as though the stuff that you're doing now
isn't going to be thrown away and scrapped for a whole new technique?
Or do you feel like you're building some foundational blocks?
So, yeah, I think these are the foundational blocks that
will lead to the final picture of DevOps Robotics.
So we just have to build on top of this.
In that building on top, do you think it needs another revolution?
Do we need another architecture or do you think that we've got enough already?
No, I think we need at least one more big breakthrough.
Even now, these robots, they take a lot of data to learn these tasks.
So we need a breakthrough where they can learn more efficiently with data.
So do you think that's the only limiting factor then?
Do you think if you had a similar order of magnitude,
many, many more orders of magnitude of data,
like you do with large language models or visual language models,
do you think that this would be sorted?
There's one hypothesis that that's all you need.
If you can get that much robot data, then we're done, we can pack it up.
But there's still a long tail of problems to solve.
Like, they have to be safe, they have to really master the task.
So there are still challenges, but the core of the problem is still the robot data.
This physical interaction data, what it feels like to do all of this stuff,
it's just limited. It's not as big as the internet.
So right now, we still have to collect all this experience on robots,
but there is a lot of manipulation data that is collected by humans,
humans posting videos about how to do anything.
We should be able to learn from that at some point
and really increase how capable robots are.
This is very unstructured, like solving robotics,
general manipulation is a very unstructured problem.
Yeah, and completely open-ended in terms of the type of things
you could potentially ask it to do.
Amazing. I'm so impressed. Well done.
Sometimes these robots are a little bit on the slope side, right?
Sometimes they're a bit clunky.
But you have to remember that this idea of having a robot
that can understand semantics,
that can get a contextual view of the scene in front of it,
that can reason through complex tasks,
it's just completely inconceivable just a few years ago.
And okay, there may still be some way to go,
but the progress here is really limited by the amount of data
that we have on physical interactions in the real world.
But solve that, go through that barrier,
and I don't think you're just going to be watching robots sort laundry.
I think we could be on the cusp of a genuine robot revolution.
You have been watching and listening to Google DeepMind, the podcast.
If you enjoy this little taste of the future,
then please do subscribe on YouTube so you won't miss an episode.
See you next time.
