How Millions of Blobs Are Transforming Robotics

Infinity Robotic
4 followers
Are we living in a simulation? If you're a robot, the answer might be yes.
------------
#machine #Robotics #robot
AI Generated·May contain errors
What you're seeing right now is not real. I mean, this is a real place, but we're inside a simulated
version of it made up of millions of spherical blobs. It's called a Gaussian Splat, and because
they're so cool, in this video we're going to share with you how they're made, how they can
render at much faster speeds than traditional 3D methods, and how we're using them to help
train our humanoid robot, Phantom. The short answer is that you just take a bunch of pictures,
you feed them into a computer, and out pops your result. But that describes how photogrammetry
works too, so what's the difference and why are we using Gaussian Splats instead? They both start
out in a similar way, by triangulating the position of an object in 3D space by comparing two different
views of it. If you do this enough times, you can create a point cloud, or a bunch of dots in 3D
space that roughly create the depth of the scene. Where the two techniques really diverge is that
photogrammetry tries to connect all those dots into tons of 2D polygons, and then texture those
shapes with the computer's best guess of what pixels should be printed onto each of them.
Gaussian Splats, on the other hand, use those dots to help recreate the original picture itself,
not the actual objects depicted within them. It does this using machine learning,
where each next iteration of the scene changes the position, number, shape, size, color,
and opacity of each of these blobs to get closer and closer to matching what the camera saw in
the first place. When you repeat this process in parallel with all the other input images you
provided, you get a Gaussian Splat. Which still might sound like jargon to you, so let's quickly
unpack that name, and it'll actually give you a much clearer idea of what's going on under the
hood. Gaussian comes from Carl Friedrich Gauss, who is famous for, among other things, using the
normal distribution, which is also called a bell curve. You've probably heard of it, it looks
something like this. But what if this axis was opacity, and this axis was distance from the
center? And then, what if you depicted that same curve across three dimensions? Congratulations,
you now have a 3D Gaussian distribution, which is what all these things are. But they aren't
splats yet, and I know a lot of them look flattened already, but that's only because one of
their axes was squished during the machine learning gradient descent process, the one that recreated
all those camera views. It turns out the most efficient method to depict most objects is with
flattened disks instead of full spheres, but they all still are 3D shapes. Still with me? The real
splatting is basically just projecting all these shapes onto a single 2D plane, which becomes your
viewport of the scene, like the one you're looking at right now. That's the step when all the 3D
volumes of the scene truly become two-dimensional, as they are projected or splatted onto your view.
It's like throwing colored snowballs at a wall, and that's the name, Gaussian splat.
Both of these techniques have limitations, like if my input images don't include the backside of
this couch, I won't be able to see it in the final result. But there are two main benefits for using
Gaussian splats for robotics training. The first is how well they handle reflections and small
details, and the second is how fast they are. Photogrammetry has a difficult time with shiny
things. And even if it gets the geometry right, if you want to mimic the real world, for example
for the purpose of providing higher quality training data for your robot, you still have
to recreate the reflections by simulating light, which is computationally expensive, a problem that
Unreal Engine has been working on for years now. Meanwhile, you get realistic reflections with
Gaussian splats basically for free, because instead of bouncing millions of light rays off of millions
of surfaces, whenever you move around a scene, none of the splats actually move. And even though
the color of the splat can change depending on your viewing angle of it, from the computer's
perspective, splatting is just a quick one-time calculation per frame that allows the scene to
render at over 100, even 200 frames per second. And that gain in performance makes a meaningful
difference for robotics training. It means we get to simulate realistic views for the robot's cameras,
but at a much cheaper computational cost. However, because there's no geometry inside of Gaussian
splats, we can't actually simulate any physics or interactions with them alone. That's the main
limitation of using them. But luckily, other tools like photogrammetry, like we just discussed,
can help close that gap. If your goal is to train robots for industrial settings, like this example
here, there's a limited scope of where you'll need to go and what you'll need to interact with,
so there's really no need to recreate the physicality of the entire scene. You can just
focus on just the parts that you need. For us, this all works perfectly. Our engineers have
combined Gaussian splats, real physics, and object collisions into a single simulator, where we can
now train robots and digital clones of real-world situations. This makes robotics training cheaper,
faster, and even better than trying to get the same outcomes using traditional 3D visualization
methods alone. With Gaussian splats, the real universe is our render engine. No need to reinvent
it. It's still not perfect, but kind of like how the screen you're watching this video on right now
is tricking your brain into seeing this image by only using red, green, and blue light, our robot
can't seem to tell the difference between the real world and millions of blobs.
That's all for now. Thanks for watching.
