Lecture 28

IIT Kanpur July 201
20 followers
Feb 23, 2024
[Music]
hello friend welcome to the course
multivariate procedure with r so you can
recall that in the last lecture I have
given you an introduction about uh
estimation of
parameters and we had discussed about uh
Point estimation and interval
estimation now there are various
methods for finding out the point
estimate as well as uh interval
estimates for any
parameter and among them uh I would like
to give you here some idea that how
these parameters are estimated and what
type of values do they get and how to
compute them in a real life's data
situation so as I said in the past also
couple of time the objective of this
course is not to teach you the complete
statistical inference all all the topics
in the statistical inference like as
Point estimation confidence interval
estimation and another related topic but
my idea is that I want to prepare here a
background and I want to give you a
sufficient uh background so that uh you
can understand that top ICS in the
forthcoming lectures so keeping this in
mind in this lecture I am going to talk
about maximum likelihood estimation
technique so this maximum likelihood
estimation technique gives us the values
of the unknown parameter on the basis of
given sample of data so as the name of
this topic suggests maximum likelihood
what is
likelihood likelihood is essentially the
probability or the chances of occurrence
of an
event for example if you try to see the
black CL clouds in the sky then you say
that okay there is a very uh good
likelihood that it may
rain and in case if you see a clear sky
then you always say that okay the
chances of rains are very remote they
are very less or in some way I say that
okay the likelihood that it will rain is
very very low so if you try to
understand this sentence what are you
trying to do you are trying to take some
decision about the some unknown process
unknown data based on the value of the
probability and you are trying to take
the decision which is corresponding to
the maximum probability of occurrence so
now if you consider a sample the sample
is coming from a population whose
population parameters are unknown to us
so when the population parameters are
unknown to us I want to know what can be
the value of the
parameter such that the sample is coming
or originating from that population has
the maximum probability so basically we
try to write down the joint probability
of all the observation and we try to
maximize it and based on that the value
of the parameter which is maximizing The
Joint
probability is obtained and that is
called as the maximum likelihood
estimator so now when we are talking
about the maximization there there are
different ways by which we can do the
maximization one of the popular method
which you have studied is is uh the
principle of Maxima and
Minima beside those thing there are
several method there are several
numerical procedure there are several
optimization method which can give you
the value of the parameter for which the
function is
maximum so I will try to give you here
some simple example in
univariate and I will try to take it to
the multivariate setup
also I will try to consider here those
example which have a close form which
are giving you us a very clearcut
solution but in many
situation the value of the estim based
on the maximizing the likelihood
function will be only based on the
numerical data and the optate numerical
procedure that I'm not covering here
because my objective is to make you
familiar with the maximum lity
estimation technique that we are going
to use in some forthcoming lecture so
with this objective let us begin our
lecture now in this uh lecture we are
going to talk about the maximum
likelihood estimation of the parameters
right
so there are different methods in
statistical inference to obtain the
values of the parameter on the basis of
given sample of data method of movements
method of Maximum likelihood method of
minimum K Square Etc and one among them
the method of the method of Maximum
likelihood estimation is a popular
method and as the name implies method of
Maximum likelihood maximum likelihood
means the likelihood is the maximum so
the estimator will be the value of the
parameter that maximizes the likelihood
function so for example let us say that
let X1 X2 X1 be a random sample from a
probability density function or
equivalently a probability Mass function
FX Theta which is here like given like
this because now we are trying to look
at f as a function of X1 X2 xn for a
given value of theta which for a fixed
Theta because the Theta belong to this
parametric space Capital Theta right
then what we try to do here we try to
find out the joint probability density
function or probability Mass function of
X1 X2 XM on the basis of given sample so
we try to find out here this F of X1 X2
xn given Theta now because we are
assuming that they are independent
because they are the random sample so I
can write down this joint PDF as a
product of marginal PDFs or equivalent
to the probability Mass functions and
this joint distribution of X1 X2 X1 is
indicated by here function l l is a
function of theta and X1 X2 xn right so
L is called as the likelihood function
and this is a function of the parameter
Theta given the observed and known
sample values X1 X2 XM similarly if you
for example if you are trying to
consider the discrete random variable
where you are trying to consider the
probability Mass function so then
instead of here joint probability
density function we can also find out
the joint probability Mass function like
here is like here
this probability that X1 is equal to X1
comma X2 = X2 comma up to xn equal to xn
right and then based on this likelihood
function we try to find out the value of
the parameter Theta which is likely to
maximize the probability in the sample
right and so the maximum likelihood
estimator or we shortly call as mle e m
for maximum L for likelihood and E for
estimer the mL of the parameter Theta is
the value of theta that maximizes the
likelihood function L which is the
function of theta and X1 X2 X so now let
me try to take here a simple example to
explain you that uh what do we try to do
for example I will try to take here two
example one for discrete case and one
for say continuous case so in the case
of discret random variable uh we
consider here the barnoli distribution
right that the sample X1 X2 xn is coming
from a barnoli distribution which is
indicated by here B
1 p and it has got got only one
parameter p and the probability Mass
function of the distribution is given by
probability xal to X which takes value
the probability P if x equal to 1 and 1
minus P if x equal to 0 right so all
these X1 X2 X and they are identically
and independently distributed that is
IID so the likelihood function of the
parameter Theta which is here p and X1
X2 xn is found here like this L is equal
to this probility of Jo probability
function of X1 = to X1 X2 = to X2 and xn
= to xn so you can see here this is only
the product of probability of x i equal
to XI because the X1 X2 xn are
independent so this is obtained here
like this power of summation XI into 1us
P powerus sum x i now this is the
likelihood function and we want here the
value of P for which this whole function
is maximum so the maximum L estimator of
p is the value of P that maximizes the
likelihood function l in this case so
now the main issue here is that how to
maximize it there can be different ways
to maximize it right but we are going to
consider here one very commonly used
used method which is the principle of
Maxima and Minima so in order to
maximize this likelihood we consider it
lock or we or more specifically the
natural log which is given by the
function Ln so we try to minimize the
natural Lo function of L using the
principle of Maxima Minima so the log of
L is obtained here by here this thing
and it log comes out to be here like
this that's a very simple calculation
now in the principle of Maximum Minima
we try to find out its firstep
derivative and put it equal to zero so
we try to find out the first derivative
of log l with respect to P which comes
out to be here like this and we equate
it to zero this is here called as the
likelihood equation or the normal
equation right and in case if you try to
substitute it here this will solve to P
equal to xar so I can write down that
the maximum La symmet of P which is
indicated by here p m is equal to xar
and if you want to see whether this
value of p is maximizing or minimizing
the LA function so we can verify it with
the second order condition well I'm not
going to give you here the detail that
how principle of Maxima Minima work but
I'm assuming that you all are aware and
you know it right so thus I can say here
that this xar is the maximum likelihood
estimator of P so now you can see here
that in this function this P was unknown
to
us P was unknown and what we have done
we have got here a sample X1 X2 xn and
based on that we have found that if I
try to take P equal to here the capital
xar that means the estimator of P which
is indicated by P head is equal to
Capital xar so that in case if you get
here a sample X1 X2 here xn then you
simply have to find out here the sample
mean xar like this 1 upon n summation
nxi and it will give you the good value
of the this parameter P right now after
this let me try to take here one more
example from the univariate normal
distribution so what you try to do that
first you try to understand this example
and I will try to extend it to a
multivariate case that is the case of
multivariate normal distribution right
so suppose X1 X2 xn is a random sample
from normal distribution which mu mu and
variant Sigma square whose PDF is here
like this right where X lies between
minus infinity and infinity mu lies
between minus infinity and infinity and
sigma square is greater than zero and
this X1 X2 X1 they are IID so the joint
probability function which is here the
likelihood function can be written as
the product of individual PDFs of XI
which will come out over here like this
it is simply I'm just trying to multiply
this FX now it is easier for us to to
take the log of this function means log
of this L and then we try to maximize it
with respect to mu and sigma squ well we
are assuming here that mu and sigma
Square both are unknown to us right that
that is more interesting is otherwise if
Sigma square is unknown then you need to
differentiate the likelihood function
with respect to only here mu and if mu
is known Sigma square is unknown then
you need to differentiate the likelihood
function with respect to only my square
right and if both mu and sigma Square
are unknown then we need to
differentiate the lackl function with
respect to Mu as well as Sig Square so
we try to differentiate log of L and
equate it equal to zero and once you
just differentiate it put it equal to
zero solve it and then you will get here
the value of mu as your xar and that is
going to be your the maximum likelihood
estimator of mu and similarly you will
get the value of Sigma Square as 1 upon
n summation I goes from 1 to n x i - xar
whole squ which is the the maximum
estimator of Sigma Square now the
question here is that whether these two
values mu equal to Mu head
M and sigma square is equal to Sigma
Square hat M whether are they going to
maximize or minimize the likelihood
function that is what we have to check
so what we try to do here that we try to
consider here the bordered haian Matrix
here like this and we find that it is a
negative definite Matrix at mu = to Mu
hat mle and sigma square is equal to
Sigma Square hat mle right so now these
are the maximum La symmet so now you can
see here once again I would try to
explain you here that we have got here
this probability density function where
mu and sigma squ unnown we try to take
here a sample of size X1 X2 xn and from
there we try to find out here the value
of mu as xar and the value of Sigma
Square as here 1 upon n summation I goes
from 1 to x i - XR s so you can see here
that both this estimator they are
function of
sample values
only so they are actually the
statistic right so now if you try to see
if you want to compute the mle in the
our software so you can compute it by
here mean function and here variance
function V but now you can see here the
divisor here is 1 upon n whereas variant
in R gives you here the value with the
divisor n minus one like this one so we
can make here a small transformation and
by writing n minus one into V divided by
n we will get here the maximum lik
estimator of Sigma square right so this
is how you can do it now our basic
objective in the course is to consider a
multivar setup so now I have given you
the idea that how are we going to
implement the maximum likelihood
estimator in a univariate case in a
univariate normal box ition and just to
make you understand better I will take
here the example of multivariate normal
and I will try to show you that how the
same concept of Maximum lik estimation
is extended to a multivariate case right
so suppose if I say Here let this X1 X2
X1 is a random sample from a normal
population right which has got here mean
Vector here mu and cence Matrix here is
sigma is a positive definite Matrix so
we have discussed earlier that the
probability density function of multiv
normal distribution is given here by
this
thing right which I'm writing here now
if you try to see now we are trying to
say that we need to find out the joint
probability density function of this X1
X2 xn so since all X1 X2 xn Vector they
have been obtained independently so we
try to find out here the the the product
of this X size and if you try to see
here this will this will come here like
this now we try to take here the natural
log of you can here likelihood function
like as here log of N and if you try to
just write down this uh log of L
function and now I can have here three
possible cases that one of the parameter
is known or both parameters are known so
if I assume the that mu is unknown and
sigma is known then in that case
whatever you are trying to do you just
try to differentiate the log of L with
respect to here the mean Vector mu and
put it equal to zero and solve it and
you will get here that mu is equal to
xar Vector so xar Vector means here
because you are saying that you have
here a vector X1 X2 XP so now you are
trying to get here a sample like X1
equal to this and say here xn equal to
here say here like X1 X2 XP so you try
to take here all X1 and then find out
their sample mean this will is indicated
as xar one similarly if you try to take
all the observations on the P variable
and you try to find out here xar which
is indicated by here xar P so this is
indicated here by this mean vector and
uh yeah after that you can also go for
the second order derivative to make sure
that it is the the value of the MU which
is maximizing the log the likelihood
function the second case is that if a
mean vector is known and we don't know
the cience Matrix so in that case you
simply have to somehow find out the
maximum value of the sigma which is
going to maximize here the likelihood
function right here I would try to say
that you have to be little bit careful
watchful that when we are trying to
differentiate the lacel hood function
with respect to a vector or a matrices
then we have to follow some rules and
regulations of the Matrix Theory and
I'm not giving you here the details of
those thing but the the maximum Lac
estimator of Sigma will come out to be
here like this 1 upon n summation I goes
from 1 to n x Alpha minus mu and
transpose X Alpha minus mu right so you
can see here because now here is Mu is
known so that is why this mu is coming
here right so now in this case you let
me assume that both mu and sigma they
are unknown to us so we tried to write
down here the likelihood function and we
try to maximize it with respect to a
mean Vector mu and Coan Matrix Sigma so
obviously the type of algebra the type
of results what are needed to maximize
the uh log of L they are different than
the univariate case and yeah I'm not
giving you here those intermediat step
but after solving them we can obtain
here the maximum like estimator of mu as
a sample mean vector so in such a way
that xar this Vector is has P components
the first component here is X1 bar that
is the mean of the first and observation
on the variable X1 similarly xar 2 is
the mean of the N observation on the
variable X2 and similarly here xar p is
the mean of n observation on the last
variable XP right and uh similarly the
maximum lik estimator of Coan Matrix
Sigma is obtained here like this 1 upon
n summation Alpha goes from 1 to n x
Alpha minus sample mean Vector xar
transpose into xar alpha mean Vector
minus sample mean Vector right and now
in case if you want to compute these
values in the r software for a given set
of data so in order to compute the
sample mean
vector
the r command is c o l m e a NS so if
you try to compile your data in the data
Vector X then you have to write C in
lower case m in upper case and then m e
a ANS in the lower case this is
essentially the column means right and
in order to compute The cence Matrix
then this you have to Simply use here
the command V but as you know that this
V that is the variance command computes
the coari Matrix or the variance with
divisor n minus one so if you want to
have it by this divisor n for example
here like this one upon n then you have
to just modify the VR command by here
this n minus one into VR of X ided by n
right so this is how you can compute the
maximum L estimator of the mean vector
and Co variance Matrix right just to
give you some idea here that how do we
handle the multivariate uh normal
distribution so trying to generate here
first the the observation from a
multivariate normal distribution right
so in order to do it we require here a
library mvt n o RM right this a short
form of multivariate normal mvt n o RM
all in lowercase alphabet and for that
the command to generate the random
numbers but here the numbers are going
to be actually a vector of random
numbers the command here is RMV n o RM
right and then you have to specify here
n is equal to 5 that means you want here
five data vector and you need to specify
the mean vector by here command here
mean is equal to C 10 20 30 for example
it is something like here mu and you
need to specify here Sigma by writing s
i g m a in lower case alphabets and
suppose for the sake of simplity I'm
trying to take it here that's a diagonal
matrix of 2 3 4 like I here 2 3 and four
and all of the elements are in the ofd
they are zero so if you try to generate
here suppose such a x then you can see
here it will come here like this so this
first row is going to indicate the First
Data
Vector First Data
Vector
the second row is going to indicate the
second data
Vector the third row is going to
indicate the third data vector and so on
so now in case if you try to operate
here the command here call means UL L
capital M and then E A ANS in lower case
alphabet X then it will here come out to
be here like this right so what it is
trying to do here if I try to show you
here that it is trying to find out the
mean of the observation in the First
Column and it is repres reproducing here
in at the first value similarly it is
trying to compute the mean of the
observation in the second column and it
is Computing it in the second place and
thirdly if you take the mean of the
observation in the third column that is
reproduced at the third place so this is
how it tries to compute the sample
Vector on the basis of data given in X
right so you can see that it is not
difficult similarly if you try to uh
compute here the co variance Matrix so
you can see here that we have got here
three variables so this is your here the
variance of first variable Sigma 1 1
this uh second diagonal element is the
sigma 22 which is the variance of third
variable similarly the third diagonal
element this is the variance of third
variable X3 right similarly if you try
to come here this variable in writing
this is your here sigma12 which is here
the co-variance between X1 and X2
right which is something like 1/ nus1
summation x i - xar 1 i - X1 bar and X2
i - xr2 like this and similarly if you
try to see this then in the second row
for column this is they are the same
because Co varience between X and Y is
the same as coent between Y and X
similarly this value here this is here
the sigma 13 which is the covariance
between say here X1 and say X3 and this
is here the same value here if you try
to see here as like this and similarly
if you try to see here this value here
is coari between here X2 and X3 right
and this is the same value here they are
going to be the same so this is your
actually here the covariance Matrix
right so similarly if you try to find
out the coal Matrix which has a divisor
n then you have to operate it here by
this here n minus one into variance of x
divided by n and here is the number of
columns C I can find out here n call
command this is equal to here three and
you get here the same outcome but the
divisor n in place of n minus one right
and similarly if you want to find out
the correlation Matrix right so
correlation Matrix can also be found
just by the command here see here C if
you remember seor was also used in the
case of univariate data to find out the
correlation coefficient but now when the
data structure is in the format of a
matrix then it is giving you the
correlation coefficient then similar
interpretation can be given to this for
example this the elements on the first
diagonal element is the correlation
coefficient between X1 and X X1 which is
here one similarly the second diagonal
element is the coalation coefficient
between X2 and X2 and the third element
on the diagonal element is the
correlation between X3 and X3 and that
is why they are one right then this
element the element on the first row and
second column this is the correlation
coefficient between X1 and here X2 and
the element in the second row and First
Column this is also the correlation
coefficient between X1 and X2 which is
the same value like here like this
similarly if you try to take here this
value here 0.330 610 this is the
correlation coefficient between X1 and
X3 and this is here the same as here
like this this is actually here the
coralation coefficient between X3 and X1
and the remaining value here this
one and this one they are the
correlation coefficient between X2 and
X3 and this is here coition coefficient
between X3 and X2 because coition
coefficient between X and Y and Y and X
they are the same so that is why the off
diagonal elements are same actually so
now you can see here it is not difficult
and here if you try to see this is the
screenshot here but let me try to show
you this on the r console so that you
get here more confidence but definitely
my this uh this outcome is not going to
match with you because uh I am I not
used here any set seat so if you try to
see I am uploading first the package mvt
norm and yeah means you have to install
it because I already did it so if on my
computer earlier so now if I try to get
here the value of x you can see here
this is here like this and if you want I
can change it here suppose if I take it
here suppose I want here says
observation so you can see here now I
have here six observation 1 2 3 4 5 here
and in the second case it is one to here
six right and if I try to find out here
the column means here like
this so you can see here this comes out
to be here like this and similarly if I
try to find out here the variance of
here x you can see here it will come
like this and similarly if you want find
the coalation of X you can here find out
here like this right so now you can see
here that
it is not difficult and with this we
come to an end to this lecture so you
can see here that we started with
univariate case uh one discrete one
continuous and then we shifted to the
multivariate
case my objective was to demonstrate you
that how are we going to find such
estimators well uh when we are going to
deal with uh more topics you will see
that in most of the cases in real life
data these parameters are unknown to us
and unless and until we know the value
of the parameters we cannot move ahead
and we cannot draw the statical
inference so at many places I will
simply say okay let us try to estimate
this parameter by the method of Maximum
likelihood or some other
method and whatever is the value or
whatever is the value of those estimated
parameters that will be replace
back so that is why this uh was very
important for me to explain you that in
case if I say somewhere that okay we are
going to replace this mu and sigma by
their respective maximum Lac estimators
then you must understand what I'm trying
to say and now I also have demonstrated
that these methods these estimation
methods uh work for univariate case as
well as for multivariate case the same
thing will happen in the case of
confidence interval estimation so later
on I will try to take uh some techniques
which I explained you in the last
lecture that we tried to create a
statistic which is whose distribution is
not dependent on the parameter and then
using that statistic we try to create
the construct the confidence interval
but that I will try to take in the
further lectures and then uh in the case
of multivariate also the concept of
confidence interval can be extended so
obviously the confidence interval is in
the form of uh some unidimensional space
because we are considering only
univariate case when we have two
variables then this confidence interval
become a confidence region and when we
have more than two variables then it
will become a a sort of uh some
structure some ellipsoid confidence
ellipsoid that is this what we call
it right so all these things will come
in the lectures in
future so but at the moment my basic
objective is that I wanted to tell you
that method of Maximum likelihood in
these cases we could obtain the values
of parameter very easily by using the
principle of Maxima Minima but many
times we have to observe the laute
function and then we have to find out
the value of parameters such that the LA
function is maximized and in many cases
the function is so complicated that we
cannot uh
uh use such clear-cut methods and in
those cases we try to optimize them
using some algorithm or some numerical
technique but in all the cases the
objective is the same that we are trying
to maximize the likelihood function so
please understand that we are simply
trying to maximize the likelihood
function using any of the approach
whatever is convenient and possible to
use and method of Maximum likelihood is
for univ Vari bivariate as well as
multivariate
also as we are going into more Direction
the number of parameters are going to be
increased and so that is why we will be
considering the estimation of the
vectors and matrices also so you please
try to look into book try to understand
more about this uh method of Maximum
likelihood and other possible Methods at
different places I have given you the
direct expression you please try to
solve them algebraically and try to see
uh that you are not facing any problem
this will give you more confidence so
you try to practice it and I will see
you in the next lecture with more topics
till then
[Music]
goodbye
[Music]
