noc19-ma14 Lecture 20-Variation in Data - Mean squared Error,Variance and Standard Deviation

IIT Kanpur July 201
20 followers
Dec 3, 2023
Different measures of variation in data. Mean Squared Error, Variance and Standard Deviation.
Welcome to the next lecture on the course descriptive statistics with R software.
You may recall that in the earlier lecture we had considered the aspect of measuring
the variation by using the absolute deviation and absolute mean deviation.
And if you recall we had developed that tool on the concept of deviations of the observations
around any arbitrary point or around some central tendency value like as mean or say
median like this.
We also discussed that whenever we want to develop these types of tools we had to convert
the deviations only into a positive value.
That means we needed to convert or we needed to consider only the magnitude of the deviations
and for that we have two options.
We try to consider the absolute value of the deviation or second option is that we can
consider the square values of those deviations.
So now in this lecture we are going to discuss the second aspect that how to build up a measure
of variation by considering the magnitude of the deviations by squaring them.
So in this lecture we are going to discuss the concept of mean squared error, variance
and standard deviation and we will try to see that how to implement it on our software.
So before we start let us try to fix our notations once again although I had done it in the earlier
lecture but here I will be just doing it quickly.
So we will be considering here two types of variable.
One discrete variable on which we will have ungrouped data and in this case the variable
is going to be denoted by capital letter X and on this variable we are going to obtain
the n observations and these observations are denoted here like small x1, small x2,
small xn.
Similarly, when we are trying to consider a continuous variable here and we have grouped
data on that variable this means we have a variable continuous variable X on which we
have obtained the observations and those observations have been tabulated in k classes
or say k class intervals and the entire tabulation has been presented in the form of a frequency
table.
For example, here you can see the frequency table in which all the observations have been
converted into the groups and these groups or the class intervals they are denoted by
here e1, e2, e2, e3 and so on.
So we have here k class intervals or say k groups and after this whatever is the midpoint
of this interval that is going to be denoted by here x1.
So x1 is going to denote the midpoint of the first class interval, x2 is going to denote
the midpoint of the second class interval and so on and all these x1, x2, xk here now
in the case of group data they are going to denote the midpoint and not the value of the
observation as it happens in the case of ungrouped data.
And the frequency of these intervals is denoted by here f1, f2, fk so f1 is going to denote
the frequency of the first class interval, f2 is going to denote the frequency of the
second class interval and so on and the sum of all these frequency is denoted by here
n.
So that is going to be our basic notations for grouped and ungrouped data.
So as soon as I say that we are going to define the measures on group data I will be using
these notations and as soon as I say that I am going to develop the tool for the ungrouped
data then I will be using the earlier symbols and notations right.
Now I come on the aspect of developing a tool called as mean squared error and I will
be following the lecture almost on the same line as I did in the earlier lecture.
You may recall that first I define the absolute deviations and these deviations were defined
around any arbitrary value A and then I develop the measure and then I replace A by some measure
of central tendency and we define the absolute mean deviation by replacing A by the median
because median was the value around which the absolute deviations were minimum.
Now similarly on the same lines now instead of absolute deviation I will be considering
the squares of the deviations right.
So if you remember that in the case of absolute deviation I have used the quantity absolute
value of xi-A.
Now I will try to consider the squared values of xi-A and I will write down here the squares
of these deviations as xi-A whole square.
So these deviations are the squared deviation around any arbitrary point A.
Now I will try to obtain this quantity for each and every observation say x1-A whole
square x2-A whole square and up to here xn-A whole square and then I will try to take the
arithmetic mean of all these values and once I try to do it then in the case of ungrouped
data discrete variable this quantity is denoted by here like this.
You can see here this is the same quantity what I have given here.
This quantity is denoted here as say S square A which is called as mean squared error with
respect to A.
And similarly whenever we have continuous variable or grouped data then in that case
the mean squared error with respect to any arbitrary value A is defined here like this.
So you can see here that now this summation is going over the number of classes and here
this is small n is here the sum of all the frequencies and this quantity is a sort of
weighted mean where the weights have been given by the frequency fi.
So this is how we try to define the mean squared error in case of grouped data and ungrouped
data with respect to any arbitrary value A.
Now one can choose any value of A. So this can be shown mathematically that the mean
squared error is going to assume the minimum value when A chooses the value arithmetic
mean or in simple words I can say mean squared error takes the minimum value when the deviations
are measured around the arithmetic mean.
So now what I will do I will try to replace capital A by x bar which is the sample mean
or the mean of the observations and when I try to do it then what happens that I simply
try to replace A by x bar, x bar here is 1 over n summation i goes from 1 to n xi that
is the arithmetic mean and then I try to define here the deviations x1 minus x bar
whole square x2 minus x bar whole square and up to here xn minus x bar whole square and
then I try to find out the average of these things simply arithmetic mean and this quantity
is denoted here in the case of ungrouped data or the discrete variable that these values
are the values which are given here in case of our ungrouped data or our data on the discrete
variable.
So you can see here this is the same thing and this quantity which is essentially here
S square x bar which is in general we denote by here S square this is called the variance
and in case if you try to simplify this expression so you can write down here this same thing
here 1 upon n i goes from 1 to here n xi square plus x bar square minus twice of xi into x
bar. So this comes out to be 1 over n summation n xi square plus this will become here n upon
n x bar square minus twice of x bar summation xi i goes from 1 to n upon n.
Now this can be further simplified to i goes from 1 to n summation xi square divided by
n plus x bar square minus this quantity is simply the x bar so this becomes here twice
of x bar square. So this quantity becomes here 1 upon n summation i goes from 1 to n
xi square minus x bar square. So the alternative expression for the variance is here given
by this thing which is the same thing. So actually you can use any of this expression
to compute the value of variance in case of ungrouped data.
And similarly when we have group data on a continuous variable then in this case the
variance is defined like this so this is 1 upon n summation i goes from 1 to k where
k is the number of classes and then multiplied by fi into xi minus x bar whole square and
where x bar is defined like this and this is small n is defined as a sum of total frequencies.
Similarly if you want to simplify this expression as we did in the earlier case you can see
here this will also come out to be i goes from 1 to k fi xi square plus x bar square
minus twice of xi x bar and if you try to see here this is 1 upon n summation i goes
from 1 to k fi xi square plus x bar square summation fi i goes from 1 to k upon n minus
twice of x bar summation i goes from 1 to k fi xi upon here n.
So this quantity here becomes here same as x bar so I can write down here 1 upon n i
goes from 1 to k fi xi square plus x bar square why because the numerator summation
fi becomes here n upon n minus here twice of here x bar into x bar so this quantity
comes here 1 upon n summation i goes from 1 to k fi xi square minus x bar whole square
this is the same quantity given over here.
So any of this expression can be used to compute the variance in case of grouped data.
Now after giving the definition of variance I would like to address here one more aspect
you have seen that in the definition of variance I am trying to take the average of n value.
So this is defined as 1 upon n summation i goes from 1 to n xi minus x bar whole square.
In statistics there is another expression of variance that is quite popular and this
expression is given by this in case of ungrouped or discrete data this is given by here this
quantity you can see here that this expression is like the earlier one the only difference
here is that instead of here n I have here 1 upon n minus 1 and earlier we had only here
n.
So this is what you have to keep in mind and similarly in case of continuous data or a
group data the divisor that was earlier n now this becomes here 1 over n minus 1.
Now an obvious question comes that why I am using this expression actually the properties
of this expression when we have divisor n or when we have divisor n minus 1 they are
different.
If you have the idea of an unbiased estimator in the context of statistical inference then
when I say or when I use the divisor n minus 1 then this form of the variance is an unbiased
estimator of the population variance.
Whereas when I am using the divisor n then 1 upon n summation xi minus x bar whole square
is not an unbiased estimator of the population variance.
So that is why many times the software uses this definition for example in the R software
this definition is used where the divisor is n minus 1.
So that is the reason I would like to inform you here that whenever you are using any software
please try to look into the manuals of the software and see what they are trying to do.
In case if the data is very large then the value of the variances may not differ much
but in case if the data is small can the values of the variances computed by using the divisor
n or say n minus 1 they may have difference.
So you should know what you are obtaining.
It should not happen that you are assuming that the divisor is n and your divisor is
actually inside the software is n minus 1 okay.
So just be careful.
Now after the variance I come to another concept that is called standard deviation.
You have used possibly 2 types of terminology.
One is standard deviation and second is standard error.
Possibly in general people do not differentiate between these 2 names.
But here I would try to explain you that what is the difference between the 2 but to start
with I will try to use the common terminology and I will use here the standard deviation
right.
So when I say that S square is going to denote my variance then it is actually the sample
variance.
Sample variance means that the variance calculated on the basis of given set of data or given
set of or given sample of data right.
So essentially we are trying to compute the sample variance but we always call it without
loss of generality as variance.
When I try to take the positive square root of S square then this is called as sample
standard deviation.
So once again you can see here I am writing here the sample word inside the bracket just
to indicate you that the common language is simply the standard deviation.
But here actually we are trying to compute the standard deviation on the basis of given
sample of data right.
Once I am saying that the sample variance or the variance or the standard deviation
has been computed on the basis of given sample of data, what does this mean?
So if you try to see in statistics what happens that you are usually trying to collect a sample
of data and on the basis of sample of data your objective is to compute the population
counterpart.
You may recall that in the beginning of the lectures we have discussed the concept of
population and sample.
So suppose I would like to compute the total number of people who are eating say more rice
in the country like India which is a very huge country, a very big country.
Now if I want to find out the arithmetic mean, the average number of people who are eating
more rice than wheat then what I have to do?
I have to compute the number of such persons all over the country which is very difficult
to compute unless and until you execute a census.
So we try to take a sample of data that means I will try to choose a small number of observation
and based on that I will try to find out the mean and that will be called as sample mean.
Similarly if I want to find out the variance of the data that I have collected inside the
sample then that will be called as sample variance.
But there will always be a counterpart like a population mean or population variance.
Population mean means the arithmetic mean of all the population.
Simply the population variance means the variance of all the units inside the population.
So what happened that when we are trying to compute the positive square root of the population
variance then this is called as standard deviation.
But the problem is this we do not have the entire population in our hand.
So we always work on the basis of a sample of data and that is why in a common language
people do not differentiate much between the two definition, standard error and standard
deviation.
But once you are trying to do a course on statistics as a student you must know it.
So and that is my idea to explain this concept here in the next couple of slides okay.
So now I will try to denote sigma square as the population variance and the positive square
root of sigma square that will be called as standard deviation or this is actually the
population standard deviation.
But as I said this using sigma to denote the standard deviation and using sigma square
to denote the population variance is a more popular notation among the practitioners.
Now next question comes what is the advantage of having a standard deviation or say standard
error in place of variance.
Now if you try to see suppose I am trying to collect the data on the height of some
children, say in meters.
So the arithmetic mean will be in meters but the variance will be in meter square.
So if I try to see that say that there are two data set whose variances are say 16 and
36 then it is more convenient to compare the two values if they are in the same units as
of the original data set.
So what we try to do?
We try to find out the positive square root of 16 and 36 which are 4 and 6 respectively.
So now once I say that I have got a data set in which I have measured the heights of the
children in meter and then they have got the arithmetic mean of say 1.2 meter and standard
deviation of say 0.5 meter.
Then it is more easy to understand and similarly if I say that I have got two data sets in
which the standard deviations are 4 and 6 then it is more easy to understand that the
standard deviation of the second data set is higher than the standard deviation of the
first data set.
That is the only reason actually.
So this standard deviation or standard error has an advantage that it has the same unit
as of the original data.
So that is easy to compare.
For example if I have a variable in which I have taken the observations denoted as say
small x then if this small x have been obtained in the unit meter then variance S square will
be in meter square which is not so convenient to interpret also.
On the other hand if I have obtained the observation x in meter then the standard deviation S will
also be in meter and which is more convenient to use more convenient to interpret.
That is the reason that why people prefer to use the tool of standard deviation or standard
error.
Now the question comes what does this variance or standard deviation actually measures?
The variance or say equivalently standard deviations measure how much the observations
vary or how the data is concentrated around the arithmetic mean.
For example if I try to take here a data set and suppose this data set is like this and
suppose there is another data set which is here like this.
So you can see here in both the cases the mean is somewhere here but this deviations
in the case of red dots and in the case of green dots they are different and the deviations
in the case of red dots they are larger.
So in this case if I try to find out the value of the variance say here variance 1 and here
variance 2 then if I try to compute the value of variance 1 and variance 2 on the basis
of given set of data then we will find that variance 1 is smaller than variance 2.
So whenever I have a value of variance say variance equal to 4 and variance equal to
here 10 then this obviously indicates that the data in the variance 1 is more concentrated
around the mean value and the data with variance 10 suppose I am denoting in the red dot that
is more scattered around the mean value which is here like this.
So this is how we try to take the interpretation of the value of the variance.
So obviously when we want to make a decision on the basis of given value of variance then
the lower value of variance or equivalently the standard error standard deviation indicates
that the data is highly concentrated or less scattered around the arithmetic mean whereas
the higher value of variance or the higher value of equivalently standard deviation or
standard error indicates that the data is less concentrated or highly scattered around
the mean.
So this is the interpretation and obviously on the other hand if I have a data set which
has got the higher value of variance or the standard error or standard deviation then
I can say simply that the data set has got more variability.
In statistics usually we always prefer to have a data set which has got the lower value
of variance or lower value of standard deviation or standard error.
So in case if I have got two data sets and suppose we compute the variances and suppose
these variances are coming out to be VAR1 and VAR2 then if VAR1 is greater than VAR2
then we say that the data in the variance 1 is having more variability or less concentration
around the mean than the data in the data set used for computing the variance 2.
Now there is a very simple rule we would always like to have a data which has got lower variance
and if you remember in the initial slides I had discussed that one of the basic objective
in a statistical tool is that we would always like to have a data in which the variability
is less.
Okay now in case if you try to compare the variance and absolute mean deviation then
you know that when there are some outliers or some extreme observation inside the sample
then the median is less affected than the arithmetic mean.
And in this case means if the data has very high variability or the data has extreme observation
or say outliers then in this case using the absolute mean deviation is a better option
and it is preferred over the variance or standard deviation.
Whereas variance has its own advantages for example if you are working with the statistical
tool then the variance has some nice statistical properties.
So from the statistics point of view from the algebra point of view from the statistical
analysis point of view it is more easy to operate on the variance mathematically algebraically
then on the absolute function like as absolute median deviation or we call it say here absolute
mean deviation.
Next we try to understand what is the difference between standard deviation and standard error.
To understand this thing we have a concept which is called as a statistic.
You see the spelling of the subject statistics is STATISTICS but we are not using here the
last S and this is called only as a statistic.
A statistics is a function of random variables.
So if you have random variables x1, x2, xn then any function of x1, x2, xn is called
as a statistic.
For example if I say you have random variables x1, x2, xn and you try to find out the arithmetic
mean like as here capital X bar 1 upon n summation i goes from 1 to n xi then this is this X
bar is itself a random variable.
So this is called as statistic.
Now the concept is very simple whenever you are trying to find out the standard deviation
of a statistic then the outcome is again going to be a function of only the random variables
and this standard deviation is called as standard error.
So whenever we try to find out the standard deviation of a statistic it is called as a
standard error.
What does this mean actually?
Actually ideally what happens that standard deviation is a function of some unknown parameters.
For example if you try to understand suppose mu is representing the population mean right
the mean of all the units inside the population which is very very large and practically it
is very difficult to find out the mean of entire population.
So usually it is unknown then ideally the standard deviation is defined as the positive
square root of the variance of all the values in X which is denoted here as a square root
of sigma square which is equal to here sigma and that is equal to square root of 1 upon
n summation xi minus mu whole square.
So mu here is actually the population mean of all the values xi but since this mu is
not known this is unknown so you cannot compute it this value cannot be computed why because
how you will get the value of mu.
So then in that case the sigma square cannot be found.
So one option is this that we can replace the value of mu by its sample counterpart.
So when I say mu is here what this is the population mean.
So this is equal to 1 upon the total number of units in the population which is population
size i goes from 1 to here population size and say here xi.
So I try to replace it by the sample value.
So this is for population and now for sample I try to replace by 1 upon say here n number
of observation i goes from 1 to n say here xi and I denoted the arithmetic mean to be
here x bar.
So now mu is unknown to us sample mean x bar is known to us.
So what I can do I can replace mu by the sample mean x bar 1 upon n summation i goes from
1 to n xi and then in that case the standard error which is the positive square root of
the variance becomes like this square root of 1 upon n summation i goes from 1 to n xi
minus x bar whole square and this quantity is called as standard error.
So standard error always remember standard error will always be a function of observed
values.
So in simple language I can say that the standard error will always refer to a standard deviation
which can always be computed on the basis of given sample of data.
You have got the data you are asked to compute the standard deviation you can compute it
and in that case this will be called as standard error.
And in that case if you try to see here more specifically the population variance was defined
here like this and now this becomes S square is equal to 1 upon n summation i goes from
1 to n xi minus x bar for the case when we have ungrouped data and 1 upon n summation
fi xi minus x bar whole square in case of we have grouped data.
So this is basically the definition of sample variance and in common language we usually
do not call again and again it to be sample but we simply call it that find out the variance
of the set of given data.
Now after this I will come to the aspect that how are we going to compute the variance or
standard deviation on the R software right.
So I will take the first case here the case of ungrouped data.
So in this case the data vector is going to be denoted by small x and the R command for
computing the variance is here VAR and inside the argument you have to give the data vector.
But remember one thing this command VAR and inside the data vector x gives the variance
with the divisor n minus 1 that is the this value.
So in case if you want to obtain 1 upon n i goes from 1 to n xi minus x bar whole square
quantity then what I can do I can multiply and divide it by the quantity n minus 1.
So now this I can write down here n minus 1 upon here n into 1 over n minus 1 summation
i goes from 1 to n xi minus x bar whole square.
So in case if you are very particular in getting the divisor n in the variance then in that
case I would like to suggest that you try to use this command try to multiply the variance
of x by the factor n minus 1 upon n how for example you can see here now I am writing
in red color this quantity will give you variance of x and this quantity will be the factor
by which if you try to multiply the variance you will get the variance with divisor n and
where n is the length of the x vector that with the number of observation present in
the data set.
Now we are going to consider the case when we have the group data now in the case of
group data you know that there is no built in command in the base package of R. So I
need to compute the mean value of the given data set separately along with the midpoints
from the frequency table and the frequencies from the frequency table.
So in this case we are going to compute the mean as say x mean separately and in this
case if you try to see your expression for variance was i goes from 1 to k fi xi minus
x bar whole square. So now this x bar becomes x mean and then this quantity here I am trying
to write down here x minus x mean whole square and then it has to be multiplied by the frequency
vector here f and then this has to be sum means all the elements in this vector will
be sum divided by the n which is the sum of all the elements in the frequency vector f.
So this is how this expression has been obtained to find out the variance in case of group
data and in case if you have some missing data then in the case of ungrouped data if
the data vector x has some missing values as say n a then we are going to denote this
data vector as say here x n a and in that case the command remain the same but I have
to give here the option n a dot Rm is equal to true right.
And similarly if you want to compute the standard deviation that is very simple simply try to
find out the square root of the variance that you have obtained earlier. So if I try to
find out the square root which is the function sqrt of the variance of x that we had earlier
obtained then this is going to give you the standard deviation or the standard error right.
But in this case you notice that the divisor is going to be n minus 1. In case if you want
the divisor to be n then in that case simply try to take the square root of the earlier
expression that we had obtained right. So finding out the standard deviation is simply
equivalent to finding out the square root of the variance and considering its positive
value.
And similarly in case of group data also whatever the expression you have obtained for computing
the variance just try to take the square root of the variance and this is how you can compute
the standard deviation in the case of group data.
Now I try to take the same example or the same data set that I considered earlier where
we have the data of 20 participants who participated in a race and their time taken to complete
the race was recorded and this data is stored here in this data vector. Now I try to find
out the variance of this data vector and I use the command var and inside the argument
the name of the data vector that is time and I get here this value 283.3684.
And here you can see the divisor here is n minus 1 that you always have to keep in mind
and I will show you that how to find out the variance with divisor n in the next slide.
Similarly if you want to find out the standard deviation then you simply have to take the
square root of the variance. So whatever variance I have obtained here I am trying to operate
the function SQRT. SQRT is a function to find out the square root and I get here the value
16.8335.
Similarly if you want to have the variance or standard deviation with the divisor n then
in this case we have learnt that we need to multiply the variance x with the factor n
minus 1 upon n where n is the length of the data vector. So I simply try to multiply this
n minus 1 upon n in the given variance that we have obtained earlier and I will get here
the value of variance with divisor n.
And similarly if I try to take the square root of this value then I will get the standard
deviation which is based on the divisor n. So you can see that it is not difficult to
operate or get the variance or standard deviation on the given set of data and here is the screenshot
which I will try to show you.
Now next we try to find out the variance in the case of group data. So we consider the
same example and we try to convert it into a group data. We try to find out the frequency
table and from there we will try to find out the frequency vector, vector of midpoints
and the arithmetic mean. We already had discussed this aspect that how to find out the frequency
vector, vector of midpoints and how to create the frequency table.
So I will not discuss here but I will very briefly give you the background so that you
can look into the earlier lectures how to get it done. So you see the given data has
been classified in 6 class intervals and these are the midpoints and these are the absolute
frequencies that we already have obtained and now we need to find out the frequency
vector and mean from the given data. So we had used the command breaks and then cut command
and whatever is the outcome of this cut command we had created the frequency table using the
table command and after that we had operated the as numeric on the data obtained from the
earlier command which has given us the frequency like this one.
And we had created the vector of the midpoint from the given output from here like this
right. So now we have obtained the x and f vector. So now I can say here that in case
if you want to find out here we already have obtained x, we have already obtained f. Now
I need to find out the mean of x. You can see here that in this case mean of x is going
to be defined by summation fi xi i goes from 1 to k right. So you can see here I am trying
to define here at x mean and this is sum of fi xi divided by n which is sum of f.
And if I try to do it here I get here the value of here x mean to be 56. Now I try to
use the command or syntax that we had defined earlier using with the given set of data here
and this gives me the value here 269 and if I try to find out the standard deviation of
this quantity this is here this will give me the value of the standard deviation in
this case.
And this is the screenshot whatever we have obtained here and similarly at the same time
I would try to show you that if you have some missing data. So then how to handle the situation
how to find out the variance and standard deviation. So in the earlier example I am
trying to take the first 2 values to be missing and I am denoting the data inside the new
vector time.na where 2 values are missing and if I want to find out the variance it
is simply the variance of time.na with the command na.rm is equal to true.
And this will give me the value of the variance as 250.2647 this will have the divisor n-1
and in case if you want to convert it into the divisor n then you know how to get it
done. And if I try to take the square root of this value this will give me the standard
deviation when we have the missing values are present in the data. And this is here
the screenshot you can see here.
Now I will try to come on the R console and I will try to show you that how you are going
to obtain these values okay. So I try to first take it here the time. So you can see
here I am copying the data on time. I will put it in the R console and I try to find
out here the variance of here time and you see I get here this value. If I want to find
out the square the standard deviation I simply have to find out its square root and you can
see here this is the value.
And similarly these values have been obtained with the divisor say n-1. If you want to have
the divisor n in the variance then you just use the same command and you get here the
outcome and if you want to find out here the standard deviation where the divisor n then
you simply have to find out this square root of the earlier expression and this gives you
the value 16.40732 and this is here the same screenshot which I have shown you here right.
And similarly if you want to find it out in the case of here this grouped data then first
I need to create here the data vector x and f which I already have done. So you can see
here I can clear the slides x is here my midpoints f is here like this and if I try
to find out the mean in case of group data this is coming out to be 56.
And now if I try to find out the variance in this case this is going to be by the same
command that we discussed here like this is 269 and if you want to find out the standard
deviation you simply have to find out its square root this is coming out to be like
this and similarly if you want to see that how the things are happening in case of missing
values in the data. So I simply need to use this command here and you can see here that
this data time.na I already have copied here and if you try to find out the variance when
this na.rm is equal to true this give me this value and if you want to find out the standard
deviation in this case then this is simply the square root of this value that we have
just obtained and it is coming out to be here like this.
And if you try to see here this is here the same screenshot that I have shown you here.
So now I would like to stop in this lecture and I would request you that we already have
discussed the concept of variance which is one of the very popular tool to find out the
variability in the data please try to understand it please try to grasp it and try to see that
how this measure has been developed and please try to take some data sets and try to compute
the variance with divisor n and with divisor n-1.
Try to compute the mean square error surrounding any arbitrary value and get comfortable in
computing the variance or standard deviation or standard error using the R software. So
you practice and we will see you in the next lecture till then goodbye.
