Model Architecture - The Real Reason That Puts Grok AI Ahead
Penny Li
1 follower
2 Video Views·Nov 19, 2023
Python Codebook: https://colab.research.google.com/drive/1twEPQAk6P0-U5CfJJCYIDIDp3PPDtlS6?usp=sharing
Tutorial in Text: https://open.substack.com/pub/luckypen/p/grok-vs-gpt?r=1v1xeq&utm_campaign=post&utm_medium=web
Neural Network article: https://luckypen.substack.com/p/neural-networks-in-fraud-detection
CodeChat:
Meeting Link: https://meet.google.com/ezj-stmz-cdv
Sign Up Sheet: https://forms.gle/Suhra6vUrbVXCQGQA
Hi everyone and welcome to this YouTube channel.
My name is Penny and this is Penny's Coding Bits.
This channel is dedicated to bring you weekly bite-sized videos with practical solutions
for your everyday data analytical challenges.
And this video is going to bring you a comparison of the background model architecture used
by Grok AI released by Elon Musk early this month and that of ChatGBT.
And before we get started, if you're interested in joining a group of coding enthusiasts live
in a meeting setting, please go to the sign-up page in the description box below to sign
up for a code chat that's coming up on November 24th at 5 p.m. Eastern Time.
Alternatively, you can directly go to the meeting link at the time of the meeting.
Now without further ado, let's get started.
So in order to compare the background model architecture used by this new Grok AI compared
to that used by ChatGBT, I came up with this task.
So the model is supposed to learn from this array as an input and try to predict this
output.
So 0, 1, 1, 0, that's the ideal output that the model should target.
And since the Grok AI model used JAX, which is a deep learning numerical computing model
within Python, I have just imported this in and used this to compile a neural network
model, a simple one though.
So the output that it gave me was this.
So for 0, it gave me this number.
For 1, it gave me this number.
And for the second one, it gave me this number.
And the last 0, it gave me this.
Close enough, but it can be improved further.
But the important thing is, look at the training time.
It only took 0.29 seconds to train this and come up with an output.
And that's an important thing to note in this case.
Now let's look at what the GPT model gave me.
Same problem.
So learn from this array and come up with 0, 1, 1, 0.
And the GPT model came up with this prediction.
It's marginally improved from the JAX model, but pay attention to the training time.
It took 17 seconds.
So this reminds me a lot of when the storage disks first came out.
So first, when we have the portable disks, we carry these huge storage units around.
And they came in these huge sizes.
And now we only have the small USB drives that we carry around.
And they can carry huge amounts of files.
So I think this is where the large language models are going in the future.
They will become more and more efficient.
And training time will be the change maker in the leading models that are on the market.
So let's have a closer look at the JAX model we're using in this case.
The JAX model we're using in this case is simply a simple neural network model with
three layers.
And a neural network model is simply a model that resembles how your brain works.
So think of when you're trying to solve a problem, and you're trying to come up with
different ideas, and you're activating different parts of your brain.
That's how a neural network model works.
And for those of you who are interested in seeing how a map of different activated neurons
look like, you could visit one of my previous Substack pages.
And this is a visual of how different parts of your brain essentially is being activated
would look like.
And for this model, we are having 32 neurons in our first layer.
And for the second layer, we're having a rectified linear unit activation, which outputs the
input as itself, if it's a positive number, otherwise, it will output a zero.
That's all it does.
And the third layer is a dense layer with just one neuron.
That's all it is to this model.
But before we can use it, we need to do some additional work.
First, we need to set a seed so that every time we run the model, we get the same result.
We also need to give the model a name called model.
And we need to set its parameters to equal to the model being initialized based on the
seed as well as our x defined above, which is our training input.
We also need to define a loss function, which is the difference between the predictions
and the target.
Remember, the target is 0, 1, 1, 0.
So the difference between the model's predictions and this array to the power of 2.
We also need a gradient function so that the model would know how to fix its parameters
to improve its performance.
So this gradient function is being defined with the model's values and gradient.
So gradient is the adjustments to its parameters to improve its performance.
And we also need the loss function because the loss function tells the model how much
it's off by.
This JIT right here is also important because this ensures the gradient function is ready
just when it's needed because it saves a version of the model run in a cache, much
like when you visit a website, it saves the essential elements of that website into your
browser's cache.
So that the next time you visit that website, it pulls from memory instead of running from
the website server.
So this is key to the short training time of this model.
Next, we need to define the learning rate, which is the rate at which the model is learning.
And we also need a place to store a history of the loss values of this model.
And we also need to have an idea of when the model started running so that we can have
an idea of how long it ran for.
And with all those defined, we can put them into a training loop.
And for our purpose, we are training this model by running it 100 times.
And its loss value and gradients is equal to the gradient function working on the parameters
that we have defined above, as well as the x and the y.
And this line is essential because this line defines the parameters of each iteration based
on the parameters of the previous run minus the learning rate times the gradient.
So that each time it runs, the parameters are adjusted according to the learning rate
and the parameters of the previous run and how much it's adjusted by.
So that the performance improves with each iteration.
We also need to collect our training values and have a list that's printed like this
so that we can see how the loss value improved over time.
So that's about it for this model, simple as it is, but really efficient.
Next, we can have a look at how the GPT model works to solve the same problem as the JAX
model.
So one thing to note in the GPT model is the embedding feature, which converts the input
the model receives into a numerical representation that the model can understand.
And in our case, we are converting it into a vector of 32 dimensions.
So if you search on Google and just say vector, you'll likely come across something that looks
like this.
So this is how a normal vector would look like.
But imagine this in 32 dimensions.
That will give you an idea of what the model is working with in this case.
And similar to the JAX model, we also have layers in the GPT model, and we have defined
the number of layers to be two.
And if we go to the transformer encoder layer function, we can see some of the features
that's included in its layers.
So first of all, multi-head attention allows the GPT model to pay attention to different
parts of its input that it's receiving, as well as the context of the input that it receives
at the same time.
And in our case, we are defining the number of heads to be two.
And of course, you can increase this as you wish.
Next, we have the dropout rate.
This is similar to the gradient value we have in the JAX model.
And this is telling the model how much of the model should be ignored in a certain iteration.
So in our case, we are saying the dropout rate is 0.1.
So 10% of the model should be ignored.
And layer normalization really just keeps the model on track without going to one extreme
or the other.
And we have the input shape that needs to be defined in the GPT model, which is 2 comma.
That means two columns and one row.
So that's what our input array looks like in this case.
And once we put that into model encoding, we have a model definition that looks like
this.
And if we run this, we came up with the results that we had before.
So that's a quick rundown of a comparison of the Grok AI model and ChatGPT.
And I thank you for joining me on this video.
I'll see you in the next.
