AI’s Biggest Secret: Why It Defies Physics

Structural lectures, you use this famous phrase, which I've always felt, I hate to say it, Nigel, and I know he's a hero of yours, but Phillip Anderson's, you know, more is different. Yeah. I always felt that was kind of simplistic, but maybe I'm wrong. I'm just a, you know, dull-headed experimental cosmologist. So tell me, what is the significance, is there anything really significant about, I mean, of course, like where does a sand grain start to become a sand pile? Where do you, you know, protons, so tell me, what does it mean to you? Why is he a hero? Why do you start with that in that lecture? You ask, well, Anderson's a hero, okay, many things. He won the Nobel Prize for his work on assorted electrons, but there's no field of condensed matter physics which was left untouched by his intellect. So in condensed matter physics, he is a giant and is in the same way that Einstein, Hawking, and others. Bohr. And I'd say more than Bohr, actually. The more is different. The article was immensely influential. First of all, he really was the first person. I mean, there's technical ways in which more is different is important. For example, you can't have phase transitions unless you have, you take the sort of thermodynamic limit. But it's not just that when you have phase transitions, it's just that you can have previously unanticipated complex behaviors that you would never have otherwise expected based on looking at the thing that stuff is made of. So again, let's go back to our fridge magnet. You've got your electrons and the material and they have magnetic dipole moments. You would have, unless you did a particular calculation, you would never know that this thing could be used to stick your kid's drawings on the door of your refrigerator. Okay, it's a cooperative effect. It is a conspiracy of the atoms. And I actually do an experiment, which we can do right here, if you want. I do this, why wouldn't I? So the experiment is this, okay? So there's the ceiling up there, okay? And we're going to move the ceiling, okay? And we're going to do it like this. Take a finger, this finger, okay? And push. Well, my ego made all the do it. Come on, come on, put some effort into it. But this is a gas, not a solid. It's a gas, not a solid, so we didn't move it. The thing is this, the Hamiltonian, the formula for the energy of the gas is exactly the same as the formula for the Hamiltonian of the solid. They're no different. And yet, when I take this and I push my water bottle, you know, my fingers don't go through. All the atoms in this conspire. Is that really true, Nigel? I mean, at some level, just study VanderWaal's Hook's Law. No, no, no, it's really true. And this is why it's important, because when we're talking about emergence, remember I said we're talking about new laws of physics. When you have a solid, there are new laws of physics. The atoms have decided that they're not just going to sit at particular sites in a checkerboard lattice that somebody has conveniently laid out for them. They've actually conspired that they're going to keep their relative separations the same. And because of that, they minimize their free energy by doing that as a statistical mechanical description of what is happening. That means that you now have new excitations, which are, first of all, you have the rigidity, the stiffness, the emergent rigidity of a solid as measured by the Young's modulus and things like this. And you have the ability to transmit sound waves and other waves as well, of course. So at the level of description of the material, you now have new laws of physics. And the only thing that's changed is the temperature. You haven't changed the interactions between the atoms. You have changed the correlations, and that's the important thing. But it's a statistical property, and it's not one that you can see just looking at two atoms. Sorry, it's just looking at two. You have to look at the whole ensemble. So that was a thing that Anderson was very interested in and understood the depth of its significance more than other people. And later, he wrote that essay in 1972. In 1985 or so, when I came to the University of Illinois, the first project I did was with my cousin, Paul Goldbart, who's also a very well-known theoretical physicist and will next year be the president of the American Physical Society. And we worked on this question of why rubber is solid. So everybody knows that rubber is stretchy and can expand and stuff. But that's not the right question. It's stretchy, but, and you know, the piece of, you know, with this cable here can bend and deform and stretch, but it's still solid. The first question you should be asking is not why is rubber elastic, but why is it solid in the first place? Because you have a bunch of polymer molecules. They are stapled together by cross-links. It's like having a bucket of worms. They're all flapping around in a thermal equilibrium. You go in and do an experiment that you'd never get an IRB to do, even though a worm doesn't have a backbone, which is staple the worms together in random. And then you find that the thing is not just connected like a fishing net, which would just be floppy, but it's actually rigid and like a gel. And then, you know, you can tap it and it will wobble. It has sound waves and things like that. And those are all emergent phenomena. It's very complicated to calculate them because the whole thing, the polymer chains are going at random. The cross-links are at random. Everything is random. It's a very hard technical problem, but it's an example of this more is different. And the point about more is different and the point about emergence that everybody misses is that when you talk about emergence, something emerges. What is it? And the answer is it is a particular type of rigidity, which is a generalized rigidity, as Anderson called it, which basically technically comes from a certain type of response function of how does the system respond to perturbations when you poke it. And that was the lesson of that. And it wasn't really understood in those terms. And once you understand in those terms, then you can, that lays the groundwork for applying it to more complicated materials and more complicated systems. If I may, I'd like to give you another example of more. Okay. So the most stunning example of more is different is something that all of you know, all the viewers know and use, and that is AI. Okay. When you ask how, as I do, how is it possible for AI to even work in the first place? Okay. Now, let's think about this. So the first thing you would say is, let's suppose I've got a time series of data points, you know, whatever it might be, stock prices, who knows? And you say, well, I want to make a model of that. So the first thing you do is you take your data and you say, I'm going to make a model that's just a straight line, goes through some of those data points, but it doesn't go through all of them, the data points, you know, wiggle and twist and turn. And so the straight line, if you ask, does it fit all the data? Of course it doesn't. If you ask, does it make good predictions? The answer is, well, you know, not really, because it's too simple a model. So then you might say, look, I've got, you know, 50 data points here that I'm training my AI on. You know, why don't I just use, you know, a hundredth order polynomial? A much more complicated equation. That equation will fit every single data point that you want to train the AI on. Every single point. Every single, a hundredth order, I've got 50 data points, I've got a hundredth order polynomial. No matter how you embed it, right. So, well, I can find a way to, I can find, you know, I have a lot of data, but I can find a way to make it fit through every single data point so there'll be no error in the way that it fits the data. You say, this thing was going to be really great at making predictions in the future. Except it's not. And it's not because I've fitted the data, but I've also fitted the noise. And so if you make predictions, they're basically contaminated by the noise. So then you would say, well, okay, so if I have a large error, but I only have a linear fit, that's not going to do very well fitting a complicated data set. I've got a very complicated formula that's also not going to fit very well because it's fitted the noise. Somewhere in between those extremes, there should be a sweet spot where the two things balance out, and that should be the place where you should try to make your model. That's what AI should do. And that was the conventional thinking, okay? And so you say, that's the answer. Except that's not what happens. In fact, we have, when we fit our data with modern AI, we are fitting it far more than 100 parameters. We're fitting with a trillion parameters, right? Yes, and you're way, way into that regime where you're just fitting noise and the whole thing shouldn't work. That's a mystery. How is it that having such a huge number of parameters can work in principle? Obviously, it obviously shouldn't, and yet it does. And that's a problem that we have a sort of release for a very simple, the simplest sort of non-trivial model of how AI works with a student here, Chan Lee. But the answer is that there's a phase transition in the statistical physics of the learning process. And that phase transition has a rigidity, of a generalized rigidity, just like the rigidity of Moodle missiles, which nobody knew was there because they didn't do the calculation that we did. And so we could understand the transition. We could understand... Does it have critical exponents? Does it have renormalization? It has critical exponents. It has data collapse, all the phenomena that you'll find in my book. And the phase transition turns out to be very similar to the superconducting phase transition. And there's a lot that can be said about that, if you want to talk some more about that. But my point is, this is an example of more is different. Okay, it's not just that you have more things and so you can fit more things. There's actually... It's different is the important thing, not the more. And different means that there is a... It's qualitatively different, not just, well, there's a slightly different number, you know. And it's that phase transition, the qualitative difference, that means that a material or a stochastic computer algorithm, say stochastic gradient descent that is used to train AI, has new behavior when you go beyond a certain point. And that's the thing that why, you know, I think more is different is so important. Because it's not just, yeah, having more money is better than having no money. Yes, I could buy a slightly better car. There's a qualitative difference that comes when you have a... And that's the message of Anderson's article. But here's my pushback, with respect to you and Anderson. Here's some ice. If I told you this is ice that I collected at the South Pole Antarctica, you'd say, no, it's not, it's water, right? And if I had more, and actually this was chock full, I filled it up to the very brim, and then it melted, and now it's this, right? So underwent a phase transition from the South Pole to San Diego, right? Now, if I keep putting more and more ice in there, more should be different, right? And actually, that's what Anderson's telling me. But if I keep doing it, it's just gonna be more of the same. So there seems to be, yes, I agree, more is different. There's a water molecule. It's not like this liquid in here. But if I doubled the amount of, if I added more and more and more, it doesn't behave quantitatively different from this, right? So at what point does the more start to be the same? It does. When it becomes a black hole, at what point? No, no, it is different. So the amount of water and ice that you have in there, if you measure, you've mentioned the critical exponents, like, say, how the heat capacities diverge. It's a first-order phase transition, so you don't have critical exponents. But let's suppose we were talking about, say, the magnetic transition. So yes, you would find that as you, that there's a temperature where the, say, the divergence of the heat capacity, which you will see in an infinite system, it'll literally go to infinity. When it's a finite-size system, it won't diverge. It will start going up, and then eventually it'll smoothly go over. And that's important because you literally see that in, say, granular superconductors. And if you look at a machine learning, at a neural network, in the ideal case where you have an infinite number of neurons and infinite numbers of data sets and some appropriately taken asymptotic limit, you can make a very sharp mathematical theory for that. And only in that case, mathematically, do you literally have the ability to say there's a phase transition and non-analytic behavior and so on. If you, on the other hand, make the system be finite, then the computer scientists would call this ridge regression or regularization. Then, in fact, this infinity goes away, and the behavior is different. There isn't a transition. You won't be able to see that if you put more and more ice in there. If you were doing this with a magnet or you were doing it with a superconductor, you can do the experiment, and you literally can see that only when I go to infinity do I see the sharp phase transition. But, you know, I describe it in my book, if you get to within, you know, 10 to the minus 12 degrees of the critical temperature, then you will start to see the fact that you don't have an infinite number of atoms in your water bottle.