Physical Intelligence is building a foundation model that can control any robot to do any task — what the team describes as the GPT-1 moment for robotics. The company's cross-embodiment approach trains across many different robot platforms, and recent results show tasks being performed zero-shot that last year required hundreds of hours of data collection.
In this episode of The Lightcone, co-founder Quan Vuong sat down with Garry, Jared, Diana, and Harj to talk about why robotics is finally ready for its scaling moment, how PI runs its models in the cloud rather than on-device, and the playbook for what Quan sees as a Cambrian explosion of vertical robotics companies.
00:00 The new robotics startup equation 00:41 Intro: GPT-1 moment for robotics 03:05 How AI unlocked robotics (RT-2, PaLM-E) 06:17 Breakthrough: multi-robot scaling (Open-X) 09:12 The real bottleneck: data 13:10 Emergence: zero-shot robot skills 16:01 Real-world demos: laundry & warehouses 22:21 Robotics becomes a data + ops problem 23:16 Cloud-controlled robots (big unlock) 29:03 How to start a robotics company today 32:33 The coming explosion of robotics startups 43:53 What’s still missing (and what comes next)
The equation, I think, for starting a robotic business has changed and will continue to change
at an accelerating pace because the upfront cost is not that high anymore.
Everyone's sort of spending a lot of time in the digital world and it feels like,
you know, now is the time to start thinking about the world of atoms.
You literally just gave people the playbook for how to build a vertical robotics company.
This has really been our mission from the start is to create that Cambrian explosion.
It still like blows my mind.
I didn't know if this would exist even in my entire lifetime.
Welcome back to another episode of The Light Cone.
Today, we have a very special guest, Kuan Vuong.
He's one of the co-founders of Physical Intelligence,
which we think might be the robotics AI lab that brings about the GPT-1 moment for all of robotics.
Kuan, thank you for joining us.
Pleasure to be here.
Has been a longtime admirer of YC and our mission is to build a model that can control any robot
to do any task that is physically capable of and to do so as such a high level of performance
that's going to be useful to people in all walks of life.
And so, GPT-1 for robotics, you know, what is it?
You know, is the chat GPT moment for robotics real?
Our perspective here is that we want to build a model that's really intelligent.
We want to build a platform that allows us to externalize that intelligence to the rest of the
world and allow them to use it to build very interesting applications in all sorts of vertical
and robotics.
And we think that it's going to be more like a peeling an onion analogy where you start from
a really strong base model that has all sorts of common sense knowledge and already works
to some extent on your robot.
You have then a mixed autonomy system, very similar, for example, to an autonomous driving
car today.
And then you actually deploy that system to do a real job.
That system might make a mistake.
It's okay.
And then over time, by actually exposing the system to the complexity and the edge case of
the real world, that system gets incrementally, even just slightly better over time every day.
And you know, one day you wake up and you suddenly have a system that is just fully
autonomous and just provide tremendous value.
Might be helpful to give the audience a bit of a mini history lesson on why robotics is
so hard.
And there's been a lot of breakthroughs in the last two years.
And I mean, just to simplify the robotics problem is three pillars, semantics, which
I think we got a lot of locks in with language models that somehow we ported into robotics.
Then you have the planning.
And then the last thing is control, which needs to be done in real time and interact
with an environment that changes.
Walk us through the seminal papers that a lot of the team of Pi Robotics published that
gave you the inkling that the GPT-1 moment is near.
And that started in 2024.
Yeah.
So the dream to build general purpose robots has been a longtime dream, I think, in humanity.
We're not the first to say that our mission is to build a model that can work on any robot.
And we're really fortunate to be in this moment in time, in history, where we feel
that it's possible to kind of walk back a little bit.
A few years before, there was, I think the first is Seikan, which to me was the first
demonstration of language model and how you can bring all of the common sense knowledge
and language model into robotics.
And therefore, that significantly kind of reduces the need to collect robot-specific
data.
So for example, if you have a task of, oh, I want to go to the YC office to record a
podcast, what are the steps I need to take?
You can ask a language model, you know, just show me the step and show me the plan.
And that worked incredibly well.
And then the way kind of language model infiltrate, if you will, in robotics is it starts at the
planning level, at the semantic level.
And then, but there's still the control problem, you know, at the end of the day, you still
need a mechanism to convert the plan into low-level action that can actually actuate
the robot.
And that bring us to POM-E, and that bring us to RTT, which stands for Robotic Transformer
2.
And what this two work really show is that if you start from a vision language model
that is really powerful, and you kind of use robotic data to adapt this model to speak
robot language, if you will, then you see a lot of transfer from the kind of knowledge
that exists in the vision language model down to the low-level action.
Like one of my favorite example when we did the RTT2 project was you can have a picture
of celebrity on the table, a picture of Taylor Swift, you have a picture of the Queen of
England.
And you can ask the robot, you know, pick up the Coke can and move it to Taylor Swift,
even though the concept of Taylor Swift, it just doesn't exist in the robot data at all
in that work.
You can do other examples such as kind of spatial reasoning that doesn't exist in the
robot data at all.
Like, for example, move the dinosaurs next to the red car, and these are all just completely
unseen object in robot data.
And so that was RTT2, and that was POM-E.
Now, RTT2 and POM-E are single embodiment exercise.
Just for the audience, single embodiment, meaning it worked for a very specific robot?
It worked for a very specific robot.
In robotics, you can ask the question, how do you scale?
Especially how do you scale data collections?
And one of the insights that we had back then was, you know, maybe the data from one robot
is not that different from another robot's anyway.
If you have enough robots in your training data, maybe what the model learned isn't to
control one specific robot, what the model learned is something that's more abstract,
which is, how do I kind of learn a general notion of what it means to control any particular
robotic platform?
And therefore, I will be better at controlling any particular platform.
And that brings us to what we call Open Cross Embodiment and Robotic Transformer X.
That was a big paper, because it was the first that showed potential scaling laws that apply
to robotics, because now you could start training all these models across multiple kinds of
hardware, not just one, which has never been done in robotics ever before, because from
all the research labs, they would all train with a very specific set of sensor actuators
or motors, and it was all very finicky with that particular hardware, right?
Yeah.
One of the really interesting results from Open Cross Embodiment, and let me provide
the context here, is that you can take, let's say, 10 different robot platforms, collect
data from them, train a policy, and really optimize the policy to work well on that platform.
So let's say, you know, you have that, you have 10 different platforms, 10 different
policies, and now if you simply take the data and absorb it into a model that is high capacity
enough to really absorb that data, and you can compare, like you have this generalist,
right, that learned to control the 10 different robots, you can compare it to the specialist
that has been optimized to work well on a particular embodiment.
How does it compare?
And the interesting result from OpenX is it was 50% better.
Wow.
And that was really surprising, because in robotics, it's hard enough to get your model
to work on one particular robot platform.
And one of the reasons why I say that we're really fortunate to be in this moment in time
in robotics is because OpenX was really only possible because of the support that we received
from the robot community.
It was a huge collaboration across the robot community.
And the reason why that's really important is there is this joke in robotic grad school
that, you know, if you want to add two years to your PhD, just work on a new robot platform.
You know, by that logic, if you want to have 10 robot platforms, that's 20 years.
Why is that?
It takes like a year or two to just get the platform up and running to even collect the
data.
Yeah.
Is it fair to say that the dataset that was created from EmbodimentX is similar to the
scale of an impact that ImageNet did for vision?
Because it was huge, and it was the first large dataset across multiple hardware, huge
collaboration.
I still think that ImageNet was more impactful in the vision community.
And the reason for that is a few.
The first is that ImageNet also allowed for reproducible evaluation.
OpenX as an effort was more about making data available for people to use.
And evaluation is a really difficult problem in robotics that OpenX did not solve.
And the second is, I think, OpenX is a drop in the bucket at this point in the robotic
community.
If you measure in the scale and the volume and the diversity of data that the community
is collecting, I think OpenX at this point is a drop in the bucket.
I mean, I guess we started talking about sort of GP1, but even GP1, that was sort of this
moment where you can prove...
Alec Radford figured out that there was a neuron based on a very specific input and
output, and then that allowed the scaling laws to sort of take hold.
The biggest problem in robotics I've heard is basically actually exactly what we've
been talking about.
It's the data problem.
Language, you could bootstrap off of the sum total of what you could get off the internet,
which is actually quite a lot.
Can you give us a sense for scale?
Is it petabytes?
What do you think is necessary as an input to the true GPT-1 of robotics?
The data scarcity problem in robotics, there's a few ways to look at it.
The first way is that it's really two problems in this case.
There is the generation, data generation problem, and there's data capture problem.
And the difference is that the data capture is that there might already be lots of robotic
data that is being generated, but there's just never been really an incentive to capture
it, to make it easy for digestions in training.
And that's one of the goals that OpenX was trying to solve, which is if you have robotic
data, it's a really good idea to capture it and make it possible to train on.
The second way to look at it is that robotics is very different from language model.
There is not an internet of robotic data that you can use.
And so you see this kind of very operationally heavy effort to collect data.
And there's the question of, is it going to scale?
Well, the way that I look at it is, let's take the US GDP, 24 trillion US dollars.
Let's say if we actually solve robotics, a model that can control any robot to do any
task, napkin math, maybe contribute 10% to US GDP.
Well, that's already a massive number.
And I think that promise is one of the reasons that warrants the investment into data collections
in robotics.
And the third way to look at it is we're very focused on cost embodiment.
And cost embodiment, there is the data collection aspect as well, which is to really make sure
that your model and your organizations and infrastructure are set up to consume data
from many different sources of robots.
And that actually allows you to scale easier.
For example, if I were to contrast our approach compared to, let's say, a company that has
a particular hardware platform that they optimize for and they scale, it's not an approach that
has really allowed people to scale because it's just much harder to figure out how do
you manufacture 1,000 units of something for now compared to making sure that you yourself
are ready to absorb data from 1,000 different types of robots that are already in the community.
I mean, it's a crazy problem, isn't it?
I mean, the hardware itself, even within the same design of embodiment, if there's a hardware
run that goes awry or one of the servos is slightly different, you see it in the data,
right?
And then how do you control for that?
Yeah, so I think we were doing kind of like an inventory of robots in the company and
we were so shocked by that there are no two robot platforms that are the same.
And if you ask people in the robot community, sometimes there's debate about multi-robot
versus single robot.
And the argument is that single robot is simpler to scale.
And actually, that's not how it plays out in practice.
How it plays out in practice is even if you have a single robot that you're optimizing
for, over time, that platform is going to drift.
Maybe you want to make hardware change or you have software change.
You end up in a situation where it's much harder for you to reuse old data.
Because in machine learning, if you want to generalize from a distribution, you would
like many samples from that distribution.
And if you just have one robot platform that have a major change every three months, maybe
you have a few data points from that distribution.
Whereas if you start from the hypothesis that if you have many robot platform in your fleet,
your model is going to learn something more abstract, which is how do I control a robot,
not any particular robot, then the model will be able to ingest data from a slightly different
robot better.
And actually, we're starting to see emergent property in this kind of robot large flotation
model.
That's good news.
What we're doing, where you start to see interesting transfer between different data
sources.
For example, today, it's possible to perform tasks zero-shot, zero-shot meaning you don't
collect any data.
And these are the tasks that last year might have required hundreds and hundreds of hours.
What are some examples?
Yeah.
Do we have any videos we can see that show it?
So, you know, I get some flack when I come back because this is not a published result.
Hopefully, this will come out soon.
So, you know, I want to reserve the excitement for that.
I'm kind of like building up the excitement a little bit.
So, hopefully, this will come out soon.
All right.
These are not simple tasks.
These are like actually difficult tasks that just last year required like hundreds of hours
of data collection.
You hear it here on Litecone first that there's some emergent properties that are going to
come out of Pi.
Can you give us a sense of like the flavor of the tasks?
It's really easy to fool yourself.
And so we wanted to test across like a few different tasks of different flavor, a task
that requires precision, a task that requires reasoning with multiple objects in the scene.
It all seems to have this property.
That's really nice.
So it does seem like that's something that's kind of a more general property that emerged
rather than we just, you know, got lucky and suddenly the model started working on one
particular task.
Could you help us understand where we are now in terms of like what's working and how
well it's working?
Like, we're not quite at the chatGBT moment yet.
Like, where are we?
I think you brought some videos that you were going to show us to like help everybody visualize
what the current state of the art actually looks like.
I think where we are is I think if you have a task where it's okay for the robot to make
a mistake and it's possible for you to set up a mixed autonomy system where you have
a person that takes over when the robot make a mistake and provide corrections.
It is possible to get to a level of performance where it starts to make sense to think about
scaling robot deployment.
And the example that I specifically want to highlight here is this blog post that we did
with Weave and Ultra.
And it's great that these are both YC company.
I want to provide a little bit of context here first.
The context is that Pi is a primarily research organization.
We want to focus on building the best model.
But we also want to not be tunnel vision.
We want to make sure that the model that we built is actually going to be useful and actually
perform tasks that people in society cares about.
And one of the really good way for us to do so is to partner really closely with company
that want to get robot out there today.
And the way that these relationship work is that we treat each other like we're on the
same team.
Very free flow of information.
And we design a system that try to get the best possible performance for the tasks that
this company care about.
So let me talk about Weave first.
What you're seeing in this video is a system that we built together, folding really diverse
item of laundry in a real laundromat in the mission.
You can see people walking outside.
And why this task is difficult is because there's just infinite possibility of observation
space.
Like, you know, clothings are deformable.
And no two item of clothing here are the same.
And these are also unseen.
And these are not clothing items that are seen in the training data.
Yeah, I love this team.
They are some of the most cracked people out of Apple I've ever met.
Gary was the partner for Weave.
Maybe you want to explain what Weave is and what their company is.
Yeah.
I mean, they're actually shipping their first robots into the home.
We sort of talked about it as being able to do household tasks like this.
And I think they were very inspired by physical intelligence's first demos with laundry folding.
So it's actually a total trip to hear about it.
You know, a year ago, we were talking about them doing it.
And then now to see them do it working hand in hand with you is really awesome.
I think this is a great example of like, you know, you need the model smarts, you need
the data collection, and then the hardware and the sort of system integration all working
together is just hard to nail.
So yeah.
And to get back to your question about why robotic is hard, it is a really hard system
problems.
You need everything to work well and work well together to get this result.
And Weave is such an incredible team for us to work with to get this result.
And it actually didn't even take us that long to get this result.
It was roughly, well, we set a goal and maybe it was like two weeks afterwards where we
got a model and a system that was good enough at performing this task.
It still like blows my mind to see a robot actually folding laundry.
I didn't know if this would exist even in my entire lifetime.
Because like folding laundry, I mean, it's always been like the Turing test for robotics
because there's no way to like deterministically program a system the way that you did like
pre-AI to do this because the space is like so infinite.
And like, we've shown that it's possible for us to do, like basically everyone can
do this, like robots will be able to do everything.
It's only a matter of like improving it from here.
There was a funny story where when we first published Pi Zero, people thought of us as
the laundry company.
Because the demo was just focused on laundry and actually picking home tasks, especially
tasks that has to do with deformable objects.
It's a very intentional choice on our end.
We're not just after the home.
We really want to make it broadly applicable, but picking home tasks for us to start with
has a few benefits.
One is relatable.
You know, you can see the laundry folding demo and you can kind of like grok how this
is going to be useful and you can get a sense of why it's hard.
And the second is that it's really easy to set up to test generalization.
You can talk about Ultra, which is your company, Jared, a demo of it.
Yeah, this is Ultra.
The thing that I love about this video is you see, you know, it's bright outside and
you see this is 4x speed and it's 100 minutes.
If I scroll to the end, the sun has set.
Oh, wow.
That was one of the big problems in robotics where it would be so sensitive to the environment
in lighting and mess up the vision system, the semantics and part of it.
Yeah.
And the interesting thing here is that it is possible to get to the level of autonomy
that the robot is just performing the task.
This is autonomy at scale.
Like this is ready to be scaled.
Quan, because this task is less familiar than laundry folding, do you want to explain what
the robot is doing here and what Ultra is like doing as a company?
Ultra is a company that want to make it really easy to adapt robot to, you know, new tasks.
And right now they're focusing on logistics space, which is really important because there's
lots of labor shortage in logistics.
And the task that we focus on together here is, you know, if you order an item from Amazon,
you sometimes get this soft pouch that item gets shipped from.
And the task here is you have a tray of these items here and the robot is supposed to pick
one of them at a time and place it inside this pouch.
The machine would enclose it and then pick up the pouch and put it on the left here to
be ready for shipping.
Now, this part is hard because there are many different types of objects that can be in
this tray.
And the opening here is actually very narrow.
So you see this interesting example of the robot kind of nudging the item to go into
the pouch.
And that's really hard, like that requires a very good understanding of the scene and
like very precise motion to nudge the object into the pouch.
The other thing that's hard about this task is the level of autonomy that's required.
Like this is running for an entire day.
There is still human intervention, I want to say, in this like full day operation.
But the level of intervention is actually quite minimal.
This is not just like some like demo station, right?
This is actually recorded in an actual e-commerce warehouse where they're actually shipping
real products to real customers.
This isn't just like a like a lab.
This is packaging real customer, real order for customer to be shipped out in a real
warehouse.
So this is real operations.
So I think this is really cool because I think when people think about robots, they tend
to think of the consumer use cases like weave because that's, you know, what we're familiar
with in our daily life.
What I find really interesting is that there's like a million applications like this ultra
thing that you wouldn't think of as obviously like, oh, who packs the like soft pouch of
things that you get from like Amazon?
Well, there's some person like who does that.
And this is like a job that we can now build a robot to do.
The interesting thing about the approach is that you're converting it from a very difficult
engineering problem into an operation problem of how do I identify the use case and how
do I collect the right data, which is in some sense more scalable because you can build
a system that allows you to collect data for many different tasks.
So, you know, it's not a problem of how do I scale data collection rather than, you know,
for every new task, how do I design a really difficult engineering system to solve it?
YC Startup School is back.
We're hand selecting the most promising builders in the world and flying them out to San Francisco.
July 25th and 26th to discuss the cutting edge of tech.
Apply now for a spot.
OK, back to the video.
I think one thing that the audience may not know is that you have a very unique technical
insight that in the past robotics folks would have kind of gasped and be shocked because
robots need to run in real time.
A lot of times all of the compute runs in on device.
But you guys have done something very different.
Can you tell us more about that?
So that this works in real time with large models and really well.
YC So the context here is that, you know, we
talked to many companies that would like to deploy robots.
And one of the first questions we get is what compute units should we get on the robot?
You know, it's expensive.
It's going to increase the bomb cost.
And they're worried that it's going to go out in fashion very quickly because the model
changed, the model gets bigger.
How do I make sure that the hardware that I'm going to commit to today is going to be
viable for a couple of years?
It's a very difficult question.
People are often really surprised when I tell them that almost all of the robot evaluation
that we run at Pi today, including the really complicated demo that we have shown, making
coffee, folding laundry, mobile robots navigating around.
The model actually hosted in the cloud.
And, you know, this is not like a cloud as in a server in the office.
It's a real cloud.
The model is hosted in a data center somewhere.
And within this high frequency control loop that is controlling the robot, the robot is
actually querying an API endpoint that hosts the model, sending it images and language
command and getting back action that then execute directly on the robot.
And this is surprising because of precisely the reason that you mentioned, you know, how
do you actually make it work?
This is why it's really important for Pi to couple system hardware and model development
and research very tightly together because it allows us to solve for this problem.
For example, one of the insights that we have here is that you can actually bury the inference
time within the robot control loop because, you know, if I'm a robot, I have enough action
for me to execute for the next hundred milliseconds.
There's no reason for me to wait until I finish executing that action to ask my model for
a different action.
I can do it as fast as inference, essentially.
And so, you know, maybe when I only have 50 milliseconds of action worth left, I can ask
for the next sets of action.
And when the current 50 milliseconds is over, I have something that's ready for me to continue
with my next 100 milliseconds.
So that's one of the insights.
The other kind of algorithmic improvement, we refer to them as real-time chunking.
Design inference in such a way that you know there's going to be a delay in how long it
takes to query the model on the cloud, basically.
Like the problem here, if I get a little bit more technical, is an action chunk is a sequence
of action that I can execute on the robot.
So, you know, it's not just one action.
And if I have an action chunk that I can execute for 100 milliseconds and 50 milliseconds in,
I want to predict another action chunk and I'm going to transition to that new action
chunk after my current 50 milliseconds is over.
How do I make sure the two are consistent?
How do I make sure that if I'm moving this way, the next action chunk is going to allow
me to continue to be smoothly moving this way?
You can pre-compute.
Yeah, you can pre-compute.
And that's one of the algorithmic improvements that we've made to make inference using model
hosted in the cloud possible.
I studied computer engineering, so I'm not really an algorithms person.
But when it comes to systems like that, like pipelining, like get me all over that.
That sounds great.
That's so interesting.
I mean, this simplifies, it's a brilliant choice because it simplifies so much of the
system for the robots.
You don't need all these clunky, I don't know, people have two operating systems at
some times for robots embedded, RTOS and then the regular one and all these complex giant
compute and power.
And this is what the initial versions of Waymo used to run, basically a server on the trunk.
And you can't afford to do that with general day robotics, which is brilliant that you
figure out how to do it.
Yeah, you don't have to.
I mean, you can do things, some of it, there obviously has to be some compute there, but
a lot of the compute can happen elsewhere.
And then is there, there must be a video, like this, this thing that we're looking at
in the top left, like how much of that is sort of like video feedback, how much of it
is like local processed.
I mean, is there any compute locally on this robot or is it just like a dumb, like video
camera that streams data to the cloud?
For this, I am not a hundred percent sure, but I am inclined to believe that it's just
a dumb computer, like for this specific video.
I don't remember, but I'm just a hundred percent confident that we can make this work
with a dumb computer and a robot.
And the one other interesting thing about our collaboration with Weave and Ultra is
one, I've never seen that robot in person.
Two is I have very little idea about how the robot actually works.
Interesting.
And that's a very intentional choice.
I want to stay away from that as far as possible.
I also don't know how they collect data.
Like I intentionally don't ask them this question to understand whether it's possible for an
organization like Pi to parachute into their existing system and to work really closely
with them on the thing that actually matters to get the system to work and not have to
learn about how they've set up their system, because in a way that's like a more scalable
recipe.
Yeah.
You completely decouple a lot of the hardware control loop choices from the semantics and
planning, which just works, which is brilliant.
Yeah.
I mean, I'm really surprised.
It works.
And when we started the company, we thought that real deployment is only going to be in
a conversation like five years into the life of the company.
Because the problem is it's really hard.
And we're two years in, and this is the result that we have.
And real deployment and scaling the number of robots is a really serious consideration
today.
And so the pace of progress has just been very pleasantly much faster than we expected
originally.
Often on this podcast, we talk about what all this means for startup founders.
I think that might be an interesting question for us to explore here.
So if you imagine someone was listening to this podcast, maybe they're like a college
student that's studying computer science, and they think robots are really cool, and
they want to do something like this.
How should they get started?
And what are the skills that they need?
Do they need to be a mechanical engineer to be able to build a robot like this?
Can they just buy an off-the-shelf robot arm and camera system?
And load Pi, and you're off and running in like a day.
Before I actually answer your question, let me provide a few more context.
The first is that robotic is traditionally really hard because it's an extremely vertically
integrated business.
You need to have your own customer relationship, your own hardware, your autonomy stack, your
own safety certification, your own everything.
And the barrier to entry is just really high because of that.
And one of the things that we're trying to change is that we're trying to provide a
foundation of physical intelligence that the community can build on top of that allow them
to onboard autonomy onto their robot and their task much quicker than before.
So that's the first.
We want to provide that kind of seat of intelligence that allow people to move much faster so that
they can focus on other problems.
The second thing is that I think the recipe for starting a vertical robotic business today
is one, have a really good understanding of the existing workflow because the robotic
system needs to fit into existing workflow.
And the second is to be very meticulous about identifying where the opportunity is.
You know, if there's a workflow that needs X number of work today, where is the robot
when you insert, it's going to make the biggest difference.
And two is to really be scrappy when it comes to hardware and data collections.
You don't need an incredibly expensive robot that is capable of very precise motion.
Today we have to do this task.
And the reason why is this model is really reactive.
And so they can compensate for some of the inaccuracy in the actual robot movement.
And to ensure that you have the ability to collect data and to run evaluation,
especially evaluation in real deployment.
The next step after that is to get a mixed autonomy system that allows you to get to
the point where it's break-even.
Like break-even economically.
Break-even economically.
Because the reason why that's important is because it allows you to then scale the number
of robots.
Because if you lose money on every robot, it's very hard to scale.
That has been historically one of the biggest challenges for robotic companies as they grow
into growth stage.
It's just the payback hack period.
It just doesn't make sense.
Yeah.
So the equation, I think, for starting a robotic business has changed and will continue to
change at an accelerating pace because the upfront cost is not that high anymore.
And now, what is the upfront cost?
The upfront cost is much cheaper hardware, ability to collect data, ability to collect
evaluation, and ability to understand the use case to see where they should insert the
robot.
It's not about having incredibly expensive hardware.
It's not about having your own proprietary, I think, autonomy, classical stack anymore
to be able to do this task.
And so it allows companies to focus on the component that will actually allow them to
differentiate themselves from the rest of the space.
Now that you've sort of unbundled it and you no longer need to build this fully vertically
integrated company in order to build a robotics company, are we on the precipice of a Cambrian
explosion of vertical robotics companies where there's going to be like a thousand companies
like Ultra going after every menial job in the economy and getting a deep understanding
of the customer, building a robot that can solve that problem, doing a mixed human machine
deployment until it can run fully autonomously, and building a company in every sector?
Is that the future that you see people building on top of Pi?
It's funny that you mentioned Cambrian explosion because when we wrote this blog post, there
was that term that was very kind of like hotly debated.
We are, I think, academics at Turk, and we want to be kind of very major when we communicate.
But myself personally, I believe there's going to be a Cambrian explosion of robotics companies
across the entire world and across many, many different verticals, just because it's just
so much cheaper to build.
And it doesn't require someone with 20 years of experience in robotics to start anymore.
It requires someone that is really scrappy, that can move really quickly, can do the system
integration, can understand customer what they want to start the deployment.
I mean, what's coming up for me is, obviously, we work with a lot of robotics companies and
to meet a lot of founders, and it feels like there's this continuum.
One is to use an analogy to personal computing.
You could argue that industrial robotics today is basically like mainframe or mini computer
level, like if you look back in the 70s, huge public companies like digital computer
that just did these sort of very, very expensive deployments, but they were very, very specialized,
and it was all extreme enterprise.
The idea of a personal computer was ridiculous, right?
It took the Altair and then Apple 1 and Apple 2 and then IBM PC XT to create personal computing.
And then the traditional advice for robotics for many years is go after dirty and dangerous.
And then, of course, those are sort of the industrial cases, like you have these giant
Tesla robots in the Gigafactory and things like that.
It feels like what you said around profitability is really, really big.
So does that mean that the people who do the vertical robot Cambrian explosion moment,
or the people who are sort of first in that, it sounds like they would be the first to
be profitable and not dirty and dangerous?
I think this is already happening today.
I think we have the fortune of having lots of visibility into the robotics community
because people would like to talk to us, people would like to learn what it's like to build
a foundation model for robotics, and people would like to know how do I get the same level
of autonomy.
And there are so many companies and businesses that we talk to that would love to put a robot
into their space, that it's okay for the robot to make a mistake, and they just need it so
much.
I really believe that the recipe that I mentioned earlier of identify where the robot can fit
in, focus on cheaper hardware, collect data, run evaluation, mix autonomy, break even,
scale robots, will work across many different vertical.
And I'm seeing it play out today, and it's just incredibly exciting to see.
And this is pretty cool that you literally just gave people the playbook for how to build
a vertical robotics company.
This is a playbook that could possibly be followed successfully hundreds or thousands
of times.
And the reason why I want to mention it is because I do want to see that Cambrian explosions,
and we want to help enable it.
For Pi, if we talk about why Pi is going to fail, it's probably going to be because the
problem is just way too hard.
Maybe it takes 50 more years to solve the robot problem, and not a couple of years,
5, 10.
And so we want to enable the community.
We want to accelerate progress, and that's why we're very open.
We publish our research.
We open source Pi Zero and Pi 05.
And people are sort of shocked when they ask me, is there any difference between Pi Zero
and Pi 05 that you open source versus the model that we use internally, Pi Zero and
Pi 05?
And the answer was, actually, no.
It's the same model.
The pre-trained model weights that you're using that we open source is also the pre-trained
model weights that our researchers internally use for Pi Zero and Pi 05.
And so we really want to help accelerate progress in the community, and to create that Cambrian
explosion.
Yeah, that's very inspiring.
I mean, I feel like everyone's sort of spending a lot of time in the digital world, and it
feels like now is the time to start thinking about the world of atoms.
And this is sort of the perfect mix of actually, like, how do you take electrons and turn it
into abundance in the atoms world?
And I think about Dario Amadei's essay, All Watched Over by Machines of Loving Grace.
And when you really think about the perfect manifestation of that, it's not like perfect
agents that look over you just like in the electronic world.
It's actually something a little bit more akin to what we're seeing here.
Yeah.
And this has really been our mission from the start, is to create that Cambrian explosion.
And this is why we choose to focus on the model, because we believe that is the bottleneck
to just really make robots useful across many different tasks in the world.
And that's why we also focus on cross embodiment.
Success for us is not defined as only our model on a robot performing tasks that is
useful.
The surface area for success is actually much larger, which is our model performing really
useful tasks on somebody else's robot out there.
Maybe we don't even know what that robot is like in a way that's useful to the end consumer.
Could we maybe talk a little bit about the humans behind the robots here?
How did the company get started?
Who are your co-founders?
How do you get together?
And what skills do you each bring to such a complex problem?
Sometimes the joke I make here is that the humans behind the robots are also robots.
Yeah.
So Pi is a very, I would say, untraditional company.
We have larger than average founding teams.
And some of us worked really closely together when we were at the robotics team at Google.
And the robotics team at Google was, I think, a really, really great environment.
For seeing the sign of life and creating the relationships in the community that allow
the robot community and these advances to flourish.
There is Loki, which we met when we were thinking about starting the company and has just been
really instrumental in making sure that we're a good business.
And there is Adnan, our hardware lead that came over from Andro.
And Adnan has a really difficult job because if you want to work on cross embodiment,
remember my joke about how if you want to add two years to your grad school, you bring
on one more robot.
The hardware problem and the operational problem for us is how do we build, improve, and scale
a fleet of heterogeneous robots.
It's just not one robot platform.
And because we built the organization from scratch in the beginning to support that,
I think we're able to do it.
It's just a really hard problem because there's just no two different robots in the fleet.
How do you make sure that everything runs smoothly?
We're really good at divide and conquer, if you ask.
How many co-founders are there in total?
We have Brian, we have Chelsea, Sergey, myself, Loki, and Adnan.
Is it just necessary to have that many co-founders to solve a problem as big as this?
Or was it a case like you were already sort of like a unit together, you'd already worked
together, and whatever you started, you'd already work together?
One common question that we have is, why band together?
And the first is that we really enjoy each other's company.
We spend a lot of time at work, and it's, in some sense, give meaning to life.
And so we really want to enjoy the relationship we have at work.
And the second is that any one of us could have started a company and be successful.
But the problem is just so incredibly hard.
And the chances of success is just so much higher that we band together,
and we can divide and conquer the problems.
And that's, I think, one of the main reasons
why the progress has been much faster than we expected.
What were the differences of you working before in either academia or a big industry,
big company like Google, as opposed to now in a startup?
Because this is the first time for a lot of you doing a startup, right?
Yeah, this is the first time for a lot of us.
One of the really surprising things that we learned when we started the company is that
the infrastructure for supporting large-scale general-purpose robot, which is not there.
And this starts from the software itself.
How do you collect data?
What device do you use to collect data?
How do you manage the data?
How do you annotate the data?
How do you get visibility into the data?
How do you run evaluation?
How do you build operational process?
Like, there wasn't a company that offered this kind of services,
which is very different from software.
And we were really surprised to find out.
And so we ended up writing a lot of the software at Pi ourselves.
But I think this is another area of incredible opportunity,
of kind of building services for a robot company.
Like, if you can offer remote teleop, for example,
if you can offer data collections, if you can offer annotation service.
Because, you know, these are functions that doesn't need to be repeated
from one company to the next.
So I think there's lots of opportunity to build
kind of support for growing robotic business.
So that's one thing, like one surprising thing that I learned.
And the second is, I think, one of the reasons why we have managed to achieve such progress
is that there is a really tight loop of collaboration in the entire life cycle
of model development, going from what task you collect data for.
You collect data for the task.
How do you do it?
What hardware do you use?
Once after you collect the data, how do you get visibility?
How do you ensure data quality?
How do you then make sure that you can easily train on that data?
After you train on that, how do you run evaluation?
Evaluation is a really hard problem in robotics
because it scales super linearly to model capability.
Like, let's say you have a model that can perform a two-minute task.
Running evaluation for that is very different from
running evaluation for a task of 20 minutes.
It's not 10 times harder.
It's more than 10 times harder.
After you run evaluation, how do you can distill the learning from that evaluation
to know how to improve the model further?
One of the really side projects I would love to take on
is to build an automated robotic research scientist,
which is really one of the bottlenecks we have today
because this is a really difficult skill set
that requires intuition about the entire stack.
I would love it if there is a model that can ingest multi-model data such as this
and analyze failure modes,
understanding, oh, is the robot performing this way
because of the data that was collected,
or the way that it was annotated,
or the way that we train the model,
and then suggest ideas and actually try them
to figure out if those hypotheses are correct.
That's something that I would love to have.
It would dramatically unlock us.
Sometimes I make the joke in a company that we should record all of the meetings
and then train a model to basically just make prediction
about what is the next set of experiments.
Oh, you could. You totally could.
What if it's OpenCLAW and Obsidian and Markdown files
and a brain.md with ontology that's custom to your use case?
And what if it's 100 OpenCLAWS in the background that you orchestrate?
I think there's two sides to this.
The first is that we already see a little bit of a side of life
where for simple failure modes during evaluation,
if you can describe the way that the robot fail in text
very precisely and very clearly,
then you can ask the language model to make
very reasonable recommendation about what the next step is.
But the flip side is that this only works for simple cases today.
And the reason why that's the case is because I think
it's pretty fundamental limitation of the model that we have today,
which is that they are not at the core model that take action in the world
and see the consequences of its own action,
especially action that changes the physical world.
And so I think this kind of very fundamental understanding
about how the physical world works is missing
from the really large foundation model.
And I think that that's one of the ingredients that's missing
to be able to build this automated robot research scientist.
What's interesting about OpenCLoud?
I don't know.
I mean, basically it can go and it can just do things,
which is interesting.
And then at that point, it's on the research lab to provide
CLI, MCP endpoints to the things that might control robots
or reconfigure rooms or...
I mean, I think Karpathy feels like he's starting to talk a bunch about this,
where if you mix auto research plus what he's been talking about with markdown files,
it might just happen in the open.
There's a sort of sense that you have to make something much,
much more complicated to make it work.
But what if that's just wrong?
What if we just have markdown files and agents
and you could make it yourself with literally cloud code and MCP today?
What if it's not an algorithm problem?
It's just literally an integration challenge.
We have a version of this internally that I use a lot.
There was a point when I was spending an embarrassingly large
amount of money on API queries.
Yeah, yeah.
And my team was like, Juan, what are you doing?
Oh, I'm that guy at Y Combinator right now.
So to give you an example,
we have a cloud scale that's essentially serving the role of a pre-training on-call today.
So we have these pre-training runs that are really large.
It's very, I think, a difficult exercise to keep them alive,
for them to continue to churn,
just because there's so many things that can go wrong.
And we have a prototype, a pre-training on-call
that kind of babysits the run
and have the permission to take action to remedy error that it sees.
And one of the surprising outcomes of that exercise
is that it leads to about 50% improvement in compute usage,
like just overall compute utilization for that large pre-training run,
which is huge for us.
And this is just a small, simple prototype that I built.
And I think there's a lot more to be done.
Juan, this is incredible.
Thank you so much for everything.
Thank you for making physical intelligence.
Thank you for showing us these incredible demos.
And honestly, the thing that gives me the most hope is this idea that
there's an entity, there's a research lab out there
that is focused on giving this to the world,
about to create this Cambrian explosion of robotic startups.
So someone watching right now will be inspired by this
and start playing with your models.
And they might create a robot
that touches billions of people's lives for the good.
Thank you for having me.
Been a pleasure to the listener.
The one takeaway that I want you to have
is I think robotics has changed a lot.
And the cost of building in robotics has decreased,
and I think will continue to dramatically decrease.
And it also requires a very different kind of scrappy skill set
that a young startup needs.
We hope to enable really an explosion of many, many, many different robotic use cases.
And always reach out to us if you want to collaborate.
Thanks, man.
Thanks so much.
Thank you.