Gradient Descent in TensorFlow — Learn [AI] Deep Learning from Scratch

So, hello there and welcome to another tutorial. My name is Tamei Bakshi and today we're going to be going over, well, the third episode in the Learn Deep Learning from Scratch series. Now, today's episode is actually really exciting because we're finally going to get to take a look at some code. Of course, first episode was an introduction. Last episode was all about derivatives and how function optimization works. And now we finally get to cover actually implementing that function optimization in Python. And I'm going to be showing you some actual use cases for technologies like gradient descent. It's going to be really interesting stuff. Now, before we begin, I do just want to say that if you enjoy content like this and you want to see more of it, please do consider subscribing to the channel as it really does help out a lot. And please make sure to like the video as that does help out too. Of course, any questions, feel free to put them down in the comments section below and I'll get back to you as soon as I can. All right, so now to begin today, as you may recall from the last episode, I had shown you an example of a couple of different functions that we use gradient descent to optimize. So what that meant was we wanted to reduce the value of the output of the function. And therefore, we actually found the derivative of the function with respect to its input. And using that derivative, we were able to reduce the value that came out from the function by changing that input a little bit. Now, we're going to be doing something similar today. And we're going to actually implement a perceptron in TensorFlow. Specifically, we're going to implement what I call the number negator, we're going to implement a super simple perceptron that all it does is takes a number and gives us back the negated input. Let's get started. Now quickly before we begin, just to recap, if you remember, from the last episode, I had shown you that perceptrons are very simple units, they take an input, and they figure out their output by multiplying the input by a certain weight. Now the thing about perceptrons is that we don't really know what the value of the weight should be, right? We know what input we can provide. And we know what output we expect. But we don't know what value of a weight to provide to make it so when we give new input where we don't know the output, the perceptron can predict the output for us, we don't know how to figure that out. But with the power of some mathematics, specifically, the same function optimization that you learned in the last episode, you can actually figure out that weight automatically. Now I'm going to be showing you how to implement this exact same concept in TensorFlow. So let's take a look at this code. As you can see, what I've gone ahead and done is I've pulled up Google Colab. And Google Colab is a great environment where you can actually run Python code in the cloud. You know, you don't have to have a you know, really fancy computer run all this, it comes with a lot of dependencies pre installed. And so just to get started right off the bat and not need to sit down and install your dependencies and set up your Docker containers, do whatever, we're just going to be doing this through Colab. In the future, though, when we get to slightly more complex examples, I will show you how to set up environment locally to make it so you don't actually need to keep pulling up Colab to run this code. Now, as you can see that we are already in Colab. And this entire first example for today's tutorial is just three cells of code. And I think it's pretty incredible just how simple this is. And we start off, of course, with the classic import TensorFlow is TF. So we're basically just importing the TensorFlow library, telling Python that we can refer to it as TF within the rest of our code. And then what I do is a little bit of setup. So I'm first of all setting up some of the inputs and outputs and the weight for the perceptron. Now, of course, the input can be anything. In this case, I just gave it the number three. And the output is what we expect the perceptron to give us if we were to feed in this three. In this case, that is negative three. Remember, we're doing the number negator. Now, technically, if we were to feed in negative three, then the output would be positive three, because negative i and i is negative three. So negative negative three is positive three. So the output would automatically become positive three. But in this case, we're just starting off with three going to negative three, but you know, both of them are both ways, the multiplication of the perceptron would be mathematically equivalent. So then what I got to do, though, is I've got to start off with some sort of guess as to what the weight for the perceptron should be. This doesn't need to be a good guess. Of course, if it is a good guess, it helps, but it doesn't need to be a good guess, it can just be a completely random number. And in this case, I decided to start with just 0.5. Again, it doesn't really matter what it is. But in this case, I did choose 0.5. Now, if you were to do three times 0.5, you would get 1.5. But what we want the perceptron to output is negative three. So as you can tell, the outputs vary off. So we're going to actually fix that in just a moment in the next cell of code. And so what I'm going to do quickly here is run this cell. And just like that, I define my input, output, and the weight for the perceptron. Now, let me tell you what this next cell does. But before that, really quickly, just in the previous cell, this last line of code, what that's doing is it's taking I, O, and W, three values that we did or created, and it's converting them to floating point values, so they're no longer integers. And then after they're all converted to floating point values, they are converted to tensors. Now, a tensor is essentially what TensorFlow uses to wrap individual numbers or even arrays of numbers, such that TensorFlow can actually track the operations that you run on those numbers. Now, why exactly TensorFlow needs to track them, I will show you in just a moment. But for now, think of a TensorFlow tensor as almost like a NumPy array, except it can be a scalar value. So it can just be a single, you know, the number three. Or it could be an entire array or array of arrays or three-dimensional or four-dimensional. It's n-dimensional arrays, you know, as many dimensions as you want it to, or even none, so scalar value. For now, that's what your conception of a tensor should be, not 100% technically accurate, but we'll get to exactly what a tensor is again at a later time. For now, I want you to understand the number negator. So going back to the example, in the next cell, I'm doing something a little interesting. So first of all, I've just got a for loop from 1 to 40. And you'll see why it's a for loop in a moment, but for now, you can just sort of ignore that. Then what I'm doing is I'm creating a new instance of something called a TensorFlow gradient tape. And I'm defining a block of scope in which that tape is valid and its internal memory is still fresh. Now, what I'm doing here is this gradient tape is going to allow us to tell TensorFlow that, hey, look, I've got a tensor, and I'm going to run some operations on this tensor. After I run my operations, I want you to give me the gradient of the operations that I ran. All right? So if you go back to the last tutorial, just, you know, last week, you'll realize that I had shown you a plot of a function, and that function's derivative. Now, I had manually calculated that derivative function. Or rather, I used TensorFlow to help me figure it out. But here's the thing. With computers, when you're writing a complex function, you don't want to have to manually write out the complex derivative. So what you can do is you can simply tell TensorFlow, all right, I'm doing these operations on my tensor. Now, I want you to give me the derivative of these operations as a function with respect to some tensor as part of that operation. So it's really interesting the way that this works. And let me show you what I mean. Now, after I create this gradient tape, what I'm doing is I'm telling the tape to watch the weight. All right? So think about it. The perceptron function has two inputs. It has the input that we want to give the perceptron, and it has the weight, and the output is the input times the weight. Now, if we want the output to match what we want it to, we can't change the input because that doesn't make any sense. Instead, we have to change the weight. And in order to optimize the function, we have to change the inputs that we give it. So we somehow have to modify the weight using function optimization techniques in order to make the output of the perceptron closer to what we want it to be. Okay? That's why we're telling the tape to watch the weight. Because when it's watching the weight, it can find the gradient of our operations with respect to the weight, telling us how to move that weight in order to make the perceptron output what we want it to. Now, then what I'm doing is I'm calculating a value known as loss. Now, what's a loss function, you may ask? Well, a loss function is pretty simple. A loss value is pretty simple. It's basically just telling us how bad is the neural network's output. So just how off is this neural network. Now, here's what's really interesting. The loss function, although you as a human can think of it as how off is my neural network or how bad is my perceptron, really what the loss function is doing mathematically is it's telling us where should I move the weight in order to make the neural network or the perceptron less bad or more good. That's what it's telling us. Because take a look at this mathematical little function that I've put together. You might already start to see some of the, you know, patterns here. For example, I times W. What is I? It's the input. What is W? It's the weight. So this is the perceptron prediction function right over here, right? That part is what the perceptron is predicting according to the current input and the current weight. Then, if you go ahead and take the output and do output minus the prediction, suddenly you've got the distance between the output that you expected and the actual output from the perceptron. Now, this can be a negative or a positive number. So what we do is we square it. And what this does is, first of all, it increases the magnitude of that error, but it also makes sure that it's always positive. All right? Now, what does that mean? That means that the lowest this value can ever be is zero. It can never be below zero. So what we want to do is we want to bring that loss value to zero. Because if the loss value is zero, that means that we did zero squared. And how can we get zero squared? Only if the output is equal to the prediction from the perceptron. Therefore, the same two values minus each other, zero. So if the output is equal to I times W, then the output minus that value is zero. Zero squared is zero. Therefore, the loss is zero. And we have a perfect perceptron. So we want that loss value to be zero. But as you can tell, initially, of course, that's not going to be the case. Now, here's what's really interesting. What we've done is we've defined this as, you know, a bunch of operations that we do on a tensor. But right after that tape goes out of scope and we tell TensorFlow, all right, we're done running operations now. What I can do is I can say, all right, now go back in the tape and find the gradient of the loss with respect to the weight. Now, it's actually giving me that gradient value. And just like how it would do the previous input plus the derivative times the negative learning rate, I'm doing something similar here. So I say that the weight is equal to itself plus the gradient multiplied by negative 0.01. So what I'm doing is I'm reducing the magnitude of that gradient and I'm making it less steep. Remember, the gradient's telling us how steep we're moving. I'm making it less steep. And then I'm flipping it to the opposite direction. And by doing that, I'm able to make it so the gradient tells us where to go to reduce the loss, not increase it, right? We want to do gradient descent, not gradient ascent. So by doing that, I have a better weight value. And then guess what? All I do is do the same thing another 39 times. And each time we update that loss value, each time we update that weight, we're getting better and better of a perceptron. It's getting closer and closer to what we want that value to actually be. Now take a look at this. When I run this code, it's going to continuously print out the current loss and the current weight after the update from the loss. So at the beginning, the loss is a huge value. It's 20.25. And the weight has gone from 0.5 to just 0.23, all right? And of course, that weight continues to get smaller and smaller, and the loss also gets closer and closer to 0. You can see over here that the loss is at 0.007. And over here, the weight is starting to approach negative 1, because that's what we want. It's a number negator. And so we scroll down and scroll down. And now we're at 3.8 e minus 6. And the weight is nearly at negative 1. Now it's not there exactly, sure. But really, in deep learning and with deep neural networks, this kind of precision doesn't really matter. If you're 0.0001% less confident that this image contains a cat or a dog, it's not the end of the world. As a matter of fact, for the sake of performance, we usually do sacrifice a lot of precision with our floating point math. And so you really don't need to worry about that. But if you really, really wanted to push the limits, I guess, sure, you could give this 1000 iterations and rerun it and you would get like super, super, super close. But you're always going to hit a limit. It's floating point precision. It's, you know, it's one of the things that programmers hate most. And so you can get close, but you're not really going to get there exactly. Although that doesn't matter. Just to sort of recap, what we did is we took that perceptron sort of prediction function, and we wrapped that as part of a larger function, which is differentiable, meaning we can find the derivative of this function. And this function was able to tell us how off that prediction function, which was a component of it, was from what we expected. And when we find the derivative of this larger error function with respect to one of the components from the predictor, which in this case was the weight, we can figure out how to optimize that weight to make it so the output is what we expect. So that is the power of gradient descent in action. But now that you've learned about the world of perceptrons, I would like to invite you to join the next episode of the series where I'm actually going to be showing you, just before we get to the more advanced neural networks and multi-layer perceptrons, how you can actually run what's known as dimensionality reduction as a real world example of where you might want to use gradient descent. The number indicator was a pretty contrived example just to help you learn. But this is something that you might actually end up doing in real world scenarios. But it's also not as complex as a real neural network. So it's going to be a great learning experience. Apart from that, though, if you do enjoy this content, you want to see more of it, please do make sure to subscribe to the channel as it really does help out a lot. And also make sure to like the video if you did enjoy it. I would love to answer any questions you may have down in the comments section below. So feel free to leave them there and I will get back to them as soon as I can. Thank you very much for joining. Goodbye.