Lecture 36

IIT Kanpur July 201
20 followers
Sep 26, 2024
In this lesson, first we'll introduce the maximum likelihood estimation with its background
and motivation.
Next, we'll fix the ideas with a simple demonstration with univariate unconditional
variable and normal distribution.
Next, we'll obtain MLE estimates of a conditional variable with a two-variable regression model
construct.
Then, we'll formulate a log-likelihood function for a binomial distribution and obtain the
corresponding maximum likelihood estimates.
Lastly, we'll also formulate LLF for a logit function and obtain the corresponding MLE
estimates.
In this video, we'll discuss the background and motivation of maximum likelihood estimation.
We'll fix up the simple ideas related to distribution and setting up the likelihood
function.
Let us start with a simple two-variable model like y.i equal to beta1 plus beta2 x.i plus
mu.i.
We are assuming that the data that is y.i is normally distributed.
So it has a mean mu and variance sigma square.
So data is distributed with a mean mu and sigma square and it has two parameters beta1
beta2, this bivariate function.
So every time these parameters change, the distribution changes.
So this distribution is conditional to these values of beta1 beta2 and the value of x.i.
So changing with beta1 beta2, you'll obtain a different distribution function and we are
assuming that it's a normal distribution function.
That means data is normally distributed.
So with changing parameters, the distribution changes.
Now what we are essentially trying to do is that given these data points y1, y2, y3, and
so on up till yn, these are the given data points, we are trying to infer what set of
parameters, what is the set of parameters beta1 beta2 sigma square that would have generated
this data.
Now, since we have assumed a continuous probability distribution, which is normal distribution,
we'll use the probability density function corresponding to this normal distribution
to define the likelihood functions that we want to maximize.
Please note here, the likelihood means the likelihood of probability of observing the
data and joint probability because there are multiple points.
So we have y1, y2, y3, y4, and so on.
Assuming the normal distribution, each will have its own probability driven by that normal
distribution function.
We are trying to estimate that joint probability, that is the likelihood of observing the data
and later on we'll try to maximize that as well.
So the likelihood is basically the joint probability of observing the data that we have.
Now assumption here is that each data point, whether it is y1, y2, and so on, they are
taken from the same population.
So their distribution is identical and also independent of each other.
So there is no bias.
So when you observe y1 or y2 or y3, it doesn't affect the probability of observing other
data points.
And because we started with the assumption of bivariate setup where yi was a function
of xi, this kind of setup.
So this is a conditional probability distribution, conditional to the values of xi.
So the mean of y would be conditional to the given values of xi and yi can be expressed
as a normal distribution function with the mean mu given that conditional to mean mu
and sigma square, where mean mu and sigma square will change with the values of xi.
So it is a conditional probability distribution function, changing with the values of xi,
the probability will change.
The distribution will change with changing values of xi.
Now because we have assumed normal distribution function, its probability density function
is given like this.
So each yi will be distributed according to this probability density function with a given
mean mu and variables parameter sigma square.
So you have two parameters mu and sigma square and for each parameter you will observe the
probability as this which is conditional to these parameters as these parameters change
with the values of beta1, beta2, x, the probability density will also change.
Now because you are looking at n data points y1, y2 up till yn, corresponding to each x1
you have y1, corresponding to x2 you have y2 and so on, corresponding to xn you have
yn.
So you have n data points and you would like to maximize this probability.
So before maximizing you need to compute that joint distribution function.
So what is that joint distribution function?
So for example if probability of y1 is py1, probability of y2 is py2, you are multiplying
all of them to get this joint likelihood function.
So to summarize, in this video we fixed the ideas related to likelihood function, we started
with the bivariate case, we had set up yi as a conditional distribution and its probability
density function for each of the yis and we noted that from each of the individual probabilities
of yi we can multiply them to get the joint distribution which essentially forms the overall
joint likelihood function, joint probability density function which is the multiplicative
form like this.
And assumption here is that each of the points y1, y2, y3 and so on are independently and
identically observed.
In the next video we will further develop these ideas to construct our log-likelihood
function and how to maximize it.
In this video, with the help of a simple univariate unconditional model, we will try to set up
the log-likelihood function and its maximization.
Recall, in the previous discussion we said that for all the n values of y, let's consider
them as univariate distribution and we said that all these values are independently and
identically distributed, so their joint probability function can be written like this which is
in the multiplicative form, multiplication of each of the individual probabilities p
y1, p of y into p of y2 into p of y3 and so on.
This is typical joint distribution function or likelihood function.
This is with respect to parameter mu, recall that parameter mu and this is our typical
log-likelihood function.
Now if we are trying to maximize this function with respect to the parameter mu, we are essentially
trying to ask what would be that value of mu that would assign the highest probability
of to the data that we observe.
That means what would be the parameter mu that will result in the highest probability
of data that we actually observe, which makes sense.
So put this in a very simple form, you will find this expression joint probability or
likelihood and the value of mu that will maximize this joint probability will be our maximum
likelihood parameter.
So we are selecting the model parameters like mu or sigma square based on their ability
to probabilistically explain the data.
So this is our typical maximum likelihood estimation.
We are trying to find those parameters mu and sigma that will maximize the probability
of data that we observe.
Now in practice, we maximize log-likelihood because our function is a product function.
We convert into sum by taking log, particularly natural log, a very simple probability of
property of log or natural log is this log of a into b is equal to log of a plus log
of b and also log is a univariate function.
So the location of maxima and minima values are preserved.
Taking log does not change these values.
So if you have the likelihood function like this, you apply the log.
So it typically becomes the log of this and because now in the form of log, we can express
it instead of multiplication, we can express it in the form of sums and we can expand the
multiples inside.
So we'll get this minus n of log sigma minus n by 2 log 2 pi and minus 1 by 2 sigma square
summation this.
So this is our typical log-likelihood function.
Now we can differentiate it with respect to the parameter that is mu and sigma and set
it 0.
For example, let's start with parameter mu.
So if I differentiate with respect to mu, this will be eliminated, this will be eliminated
and I'll have this function as 2 times minus 2 times inside this and you'll get something
like this.
1 by sigma square summation y n minus mu equal to 0.
And here we can solve this parameter for parameter mu, we'll get solving this for mu, we'll get
in these steps, we'll get mu equal to summation of y n upon n, which means the mean of y itself.
So the solution is the y n bar or mean y n solving for this one.
Similarly, we can differentiate with respect to sigma del by del sigma.
So this will be 1 by sigma, this will neutralize, this will be 1 by sigma cube with plus sign.
So we'll get something like this.
This is nothing but del by partial differential with respect to sigma and we'll set it up
to 0 to get this sigma square value, sigma, in fact, sigma hat square because it is an
estimate, which is this nothing but standard deviation of y n itself.
So this is standard deviation of y n.
Obviously, here you need the parameter mu, which you'll get from here.
So we have obtained both the parameters mu and sigma here in this fashion.
To summarize this video, we have understood with a simple univariate setup of variable
y, how to create, convert our likelihood function into a simple log likelihood, which
is easy to differentiate and then differentiate with respect to the desired parameter mu and
sigma, set them equal to 0 and find the most optimum values of mu and sigma.
For example, mu is this and sigma is this, which will lead us to maximum likelihood estimates
for this kind of univariate setup.
In the next video, we'll extend this understanding to a typical regression model in a bivariate
or multivariate setting.
Now that we have fixed our ideas related to maximum likelihood estimation for a univariate
unconditional setup, we'll try to extend it for a regression or a simple two variable
regression model by formulating log likelihood function.
Recall, in a typical two variable regression model, where y i is the dependent variable
as a function of x i, where y equal to beta 1 plus beta 2 x i plus mu i, here the estimate
of y i, which is y i hat, which is also the mean of y, estimated mean, is nothing but
beta 1 plus beta 2 x i, where there is no error term, this error term is not there.
And the variance of variable is sigma square, so the assumption is y is normally distributed
or more specifically in this case, conditionally normal distributed, so it's a conditional
distribution, conditional to x i with a variance sigma square.
Now, y i may take on many values like y 1, y 2, y 3 and so on up till y n, so we need
to understand the joint probability distribution.
And this joint probability distribution with a conditional mean, conditional mean in the
sense that mean this is conditional to the mean value and sigma square, so this joint
distribution conditional or given the mean and variance.
Also because each y i is independent of each other, recall we said at the beginning that
these y i's are i.i.d, independently and identically distributed, so the joint probability
density function can be written as this, which can be further segregated as separately for
y 1 in f of y 1 in conditional to mu sigma square, f of y 2 in conditional to mu sigma
square and so on.
So this is our typical joint probability density function, conditional to mean and variance.
Also we said that this y i is normally distributed, so its density function will also follow the
normal distribution function with a given mean and variance, so recall this was the
mean beta 1 plus beta 2 x i and variance sigma square, so the typical density function of
f y i is a normal distribution function written like this, where this is the mean, so its
like similar to that y i minus mu mean.
Now the joint distribution function which is nothing but the multiplication of each
of these y 1, y 2 and so on, like this, like this, the multiplication of individual normal
distribution density functions, we can write in the multiplicative form which is this.
Now because it is exponent, the multiplication will be converted into summation like this
i equal to 1 to n and this is the same term which is here, so it's that it's being multiplied
n times and because it is e to the power x 1 into e to the power x 2 which is equal to
e to the power x 1 plus x 2, so you have this summative term here.
Now this is the typical likelihood function in parameters beta 1, beta 2 and sigma square,
so this likelihood function is in the form of beta 1, beta 2, sigma square, now later
on we'll convert into log likelihood, so it will be called LLF because it will be converted
into log likelihood that we discussed already, so you can see here this is the log likelihood
function, this is nothing but log function applied to this likelihood, so to this likelihood
function if we apply log to individual terms, this one, this one, you will get this minus
n log sigma, why minus n log sigma because this is the power n, so you'll get minus n
log sigma, similarly minus n by 2 log 2 pi by 2 pi because we have this 2 pi term and
then this one, so exponent will cancel and you'll get this overall this term, the square
comes from this, so you'll get this overall term which we are calling as log likelihood
function, so this is the log likelihood function which we are going to maximize and please
note this log likelihood function, log has this probability of monotonicity which means
in the original function, original likelihood function wherever the maxima minima was that
maxima minima remain the same in the log form also, so the max function which has a certain
point maxima as maybe yi, the natural log of yi will be the maxima of log transformation,
so this is called monotonicity which is followed by log function and therefore this function
will attain its maximum value at the same point as likelihood function, so to summarize
in this video we formulated our typical log likelihood function for a regression variable
with conditional distribution and now that we have formulated this log likelihood function
we would like to maximize it, now that we have understood formulating the log likelihood
function we will try to maximize this log likelihood function and find the likelihood
estimates, let us recall our typical likelihood function, this was our likelihood function
and we would like to differentiate it with respect to beta 1 and when we differentiate
with respect to beta 1 these terms will become 0 and this will be differentiable function
similarly with beta 2, so let's see one by one, so first we differentiate with respect
to beta 1 pretty simple we'll get this function we'll put it equal to 0, similarly we differentiate
with respect to beta 2 we'll get another x here because here you have xi and there
was a power 2 so we'll get this one xi equal to 0 and similarly when you'll differentiate
with respect to sigma square you will get this typical function which we'll put equal
to 0 to solve for it, so you have three equations you'll set these equations to 0 which is the
first order condition for optimization and you'll obtain the estimates for beta 1, beta
2 and sigma square that are referred to as ML estimators, so you will set them equal
to 0 first to get the estimate of beta 1 solve them for beta 1, beta 2 and similarly you'll
substitute beta 1, beta 2 to get a value of sigma, so solving for these we get yi as this
expression and this expression solving for beta 1, beta 2, so you'll recall these are
precisely the same expression which you got when you solve the OLS estimators, so these
are precisely the same as OLS estimates and therefore because these are the same normal
equations for OLS solution we get the same solution as OLS and that is why for simple
linear regression the estimates of OLS and MLE are the same.
In this video we'll try to apply our understanding of log-likelihood function for a discrete
probability case that is for a binomial distribution, think of a coin tossing game where if you
toss the coin 100 times you get n heads with a probability of p and 100 minus n tails with
a probability of p minus 1 minus p because the probabilities are not 1 by 2 of course
it is an unfair coin, but given this data that you get n heads with a probability p
and 100 minus n tails with a probability 1 minus p you can formulate the probability
function probability of getting n heads in p tosses as 100 Cn, this is for 100 Cn combinations
p to the power n into 1 minus p to the power 100 minus n, this is a typical expression
for the probability of n heads with a probability p.
Now this likelihood changes so if you notice this is the probability or what we call as
likelihood of getting these 100 out of these 100 tosses n heads but this number changes
as the probability p changes, we are interested in finding a particular value of p that maximizes
the likelihood of getting n heads so that p that will maximize the probability of getting
n heads is the one that we desire. So let us convert this likelihood function into log
likelihood why because it simplifies the procedure and also the nature of the log function is
which is monotonicity the maximum likelihood is same as the maximizing likelihood we will
get the same point. So when we apply the natural log we will get this function the original
function recall what was the original function this one so this is a into b into c form so
it will be like log of a plus b plus c how log of this plus n because it was p to the
power n so n natural log p plus 100 minus n into natural log of 1 minus p.
Now this function will differentiate with respect to p to find the maxima and because
of why we are doing this because of monotonicity property allows us to do this so we will take
the differentiation with respect to log of likelihood so this will be 0 this will be
differentiated easily n by p this is 100 minus n into 1 upon minus p into minus 1 so this
typical expression will be this which will be set to 0 and then we solve it step in a
step by step manner so first step we get this expression and solving for this we get p hat
equal to n by 100. Now this solution though this this is our typical maximum likelihood
solution for probability p hat or estimator of p hat which is n by 100 although it was
expected because out of 100 tosses we were finding the value where n heads are obtained
so it was bound to be n by 100 but as you can see this is our estimates for maximum
likelihood estimate of the probability for a binomial distribution so this is the procedure
through which we obtain the maximum likelihood estimate in the binomial distribution.
In this video we will try to understand the formulation of maximum likelihood estimator
for logit models let us recall that the logit model formulation under the logit function
was p i equal to 1 plus 1 upon plus e to the power minus z i where this was the expression
for z i beta 1 plus beta 2 x 2 i plus beta 3 x 3 and so on this was our z i and this
was our typical probability under logit function here we define p i as the probability that
event of interest y equal to 1 for example if event of interest is default so if default
occurs then y equal to 1 if default doesn't occur then y equal to 0 so if when event of
event occurs interest occurs then y i takes a value of 1 otherwise 0 and the likelihood
or the probability that y i may happen that probability p i was defined like this now
there are two scenarios whether the event of interest will occur in that case y equal
to 1 and 0 otherwise so in each case the probabilities can be simply defined for y equal to 1 as p
i which is your likelihood and y equal to 0 as 1 minus p i which is the probability
of event or likelihood of event not happening but also recall that 1 minus p i is nothing
but 1 minus 1 upon e to the power minus z i which is also equal to 1 plus e to the power
minus z i minus 1 upon 1 plus e to the power minus z i and this if you multiplied by e
to the power z i it will become 1 plus e to the power z i so 1 minus p i will become 1
upon plus e to the power z i now we can define likelihood function as probability that default
happens that means or event of interest happens which is p i to the power 1 into if it doesn't
happen then 1 minus p i to the power 0 the other scenario would be that event doesn't
happen that means p i to the power 0 into 1 minus p i to the power 1 if opposite thing
happens so the way we can define it is 1 this p i we can substitute with this so 1 upon
1 plus e to the power minus z i to the power y i into this 1 minus p i which is this 1
upon 1 plus e to the power z i to the power 1 minus y i when y i doesn't take place so
overall the likelihood can be written as very simply this function into this function so
suppose if y equal to 1 this is equal to 0 so this is p so p to the power 1 into 1 minus p
to the power 0 which is effectively nothing but p itself or the likelihood function so this becomes
our typical likelihood function that we can maximize so let's further look at this joint
likelihood function and now because there are n observations for i equal to 1 2 and so on you
will have a typical likelihood function l theta for variables x 2 i x 3 i x so on for x care what
are these x 2 x 3 and so on these are these values under the z function now for this function for
each values of i equal to 1 2 3 and so on you will get multiple values and parameters beta 1 beta 2
beta 3 and so on because this is in multiplicative form you will get this function multiplication
multiplicative or product of i equal to 1 2 and this 1 p i to the power y i into 1 minus p i to
the power 1 minus y i now because we are taking it converting into log natural logarithm converting
into log likelihood function this will be now summation summation of i equal to 1 to n y i
into natural log of 1 plus e to the power minus z i plus 1 minus y i natural log of 1 plus e to
the power z i so why this summative form this the summative form is achieved because it is a
logarithm now all you need to do is you maximize this this kind of function you maximize this
function and you get your log likelihood functions maxima so maximizing this function will give you
the log likelihood function maxima that you want want so in this video we summarized how to
obtain the like maximize the likelihood function for a logic class of model to summarize this
lesson maximum likelihood offers an extremely powerful approach to nonlinear models where
OLS is not very useful in the linear models also maximum likelihood approach provides the same
estimates as ordinary square estimates MLE approach finds those parameters that maximize
the joint probability of the observed data MLE is asymptotically unbiased and provides minimum
variance that is as the data grows it becomes unbiased and efficient and therefore is a
consistent estimate likelihood values also offer goodness of fit of a model and can be used in
applications such as testing the overall fit of the model similar to F test in OLS pseudo R square
and computing information criterias MLE is more general than OLS and is employed for a wide range
of model applications
