noc19-ma14 Lecture 29-Association of variables- Correlation Coefficient using R Software

IIT Kanpur July 201
20 followers
4 Video Views·Dec 3, 2023
This lecture discusses how we can compute correlation coefficient in R .
[Music]
[Music]
welcome to the next lecture on the
course descriptive started with our
software
you may recall that in the earlier
lecture we started a discussion on the
concept of association between two
continuous variable and
we learned about the
correlation coefficient and
we also have understood now that how to
interpret the values of correlation
coefficient with respect to the
magnitude and direction of the
Association
now in this lecture I am going to
demonstrate that how you are going to
compute the value of correlation
coefficient using the our software and
how you are going to implement it how
you are going to use it when you get a
data set
so first just a quick review what we had
done earlier so you may recall that we
had discussed the concept of
covariance between the two variables x
and y and for which we had obtained the
end pairs of observations as x1 y1 x2 y2
x and y n based on that we have computed
the covariance between x and y years
like this and this was for the ungroup'
data and similar definitions fall for
the group data also and we also had
discussed that that if I have two data
vectors in our software which I denoted
as x and y then the command CoV that is
covariance between x and y so you write
CoV and inside the argument write the
data vectors this will give us the value
of covariance but remember one thing
this command the covariance between x
and y in our software is going to give
you the value of covariance in which
this divisor is 1 upon n minus 1 and
whereas we had defined the covariance as
1 I goes from 1 to n X I minus X bar Y I
minus y bar which has the divisor n so
after this we had define the correlation
coefficient this was defined as here say
R and finally we had obtained the
expressions here like this a simplified
version of this coalition coefficient
and after that we had understood that
how to interpret the values and sign now
the next question is how to compute this
correlation coefficient inside the our
software so if I say that I have the
same setup that I have two data vectors
x and y then the correlation coefficient
between x and y is computed by the
command cor and inside the argument we
have to write the data vector so the COR
inside argument x and y will compute the
correlation coefficient between x and y
that is the data vectors x and y and
when you try to look into the help of
the COR function then there are several
options and I am going to detail here
some important things which we are going
to use actually
you will see here the function here you
cor and inside the argument this and
this I am writing several options so
Y now we try to understand them one by
one this x and y that we know these are
going to denote the
data vectors here like this
now there is here another command here
use and I have written here inside the
double quotes as everything now what
does this everything means and what is
the use of this syntax you se use
actually this use is an optional
character in this cor function which is
trying to give a method of computing the
covariance in the presence of missing
values if you remember earlier we had
used the command like na dot R M so this
command also have a similar utility so
then in this case we have several option
to give here say everything when you try
to use all the observations or say all
observation complete observation na dot
or dot complete or Sapir wise complete
all the relations and so on
because there are different test
situation in which one would like to
compute this coefficient of collision so
we are simply going to use here the
option here everything where I want to
compute the correlation based on all the
data
right remaining details you can look
into the help menu
after this there is another option here
which now I am trying to denote in blue
color so that you can see clearly this
is here method and inside the cecum and
inside the argument I am writing here
three options Pearson's candle and
Spearman actually there are several
types of correlation coefficient up to
now what we have studied the R and if
you remember I had told you that this R
is also called as say this Carl
Pearson
coefficient of coalition and this is how
this PSN is coming here second or say
another coalition coefficient is rank
correlation coefficient
which we will discuss in the next
lecture and this rank coalition is also
called as
Spearman
Spearman's correlation coefficient or C
Spearman rank correlation coefficient so
this option here which now I am
highlighting in red color Spearman
inside the double quotes this option is
used when we want to compute the rank
correlation coefficient and
similarly there is another form of the
collision coefficient which is defined
by the candle which we we are not using
at the moment so essentially we are
going to use here the option
Pearson the correlation coefficient that
was defined by say here summation X I
minus X bar Y I minus y bar and square
root of summation and X I minus X bar
whole square and summation I goes from 1
to n y I minus y bar whole square this
is actually computed by this option
Pearson right and this is Methodist
mentioned here in the next slide that
method is a gives us a character string
indicating that which correlation
coefficient of the covariance is to be
computed for say Pearson for Kendall or
say Spearman this has to be every rivet
weighted inside the
method okay and default will be Pearson
now I try to illustrate you first some
example if you try to see here I have
taken here two data vectors one two
three four and one two three four same
values and I am trying to compute the
covariance between them
once I try to do it this value will come
out to be one point six six and so on
and here you can see the slide and now
what I do in the next example
which I am denoting now in say here
purple color I try to take here the same
data set see one two three four but in
that next the data set I try to change
the signs
from the second data vector so now if
you try to see what happens here
the value of the covariance comes out to
be minus one point six six and so on so
you can see here that these two values
which I am now highlighting in red color
this and this the magnitude of this
values are the same but only the sign is
opposite and here is the screenshot
what is the meaning of this now if you
try to look into this slide of the
definition of correlation coefficient
we had understood that our can take
positive value and our can take negative
value but if you try to see in the
denominator this value will always be
greater than zero this value will always
be greater than zero so this is the only
value which is the covariance between x
and y this can be greater than zero or
this can be smaller than zero
so the direction of the correlation
coefficient is determined by the
covariance
so I
can say that the sign of
correlation coefficient is
determined
by the
covariance
and this is what I am trying to show you
here that if the data points here in the
first case here and in the second case
here they have got a opposite sign then
this is given by
this negative sign here right and if you
try to plot these two data points how
they will look like
for example now I try to find out the
correlation coefficient and by which I
will also show you the direction
so when I try to find out the
correlation coefficient between the two
data vectors one two three four one two
three four they are the same
then this correlation coefficient comes
out to be here one that you can see and
you see this is also indicated in the
scatter plot which is given here you see
these are the point 1 2 3 4 and they're
lying exactly on the
straight line so this is a case where we
have exact positive linear dependence
and this is the screenshot of the same
operation I will try to show you on the
our software also and similarly if I try
to take the data vectors with exactly
opposite sign say first get a vector
with the all
positive sign data and second data
vector - 1 - 2 - 3 - 4 having the
opposite sign and if I try to find out
the correlation coefficient between the
two this comes out to be minus 1 and you
can see here this is indicated in the
scatter diagram here these are the four
points and this relationship is
decreasing and this is the case of exact
negative linear dependence between the
two variable and here is the the
screenshot of the of both the things now
if we're taking an example let me try to
show you this over the our console also
so you try to see here I try to find out
the covariance between
one two three four and
here
minus 1 minus 2 minus 3 and minus 4
you can see here this comes out to be
minus 1.6667 and if I try to remove the
signs and suppose both the data vectors
have got the same value
then in this case the magnitude remains
the same but the sign become positive so
in this case you can see here the
covariance is positive now in case if I
try to find out the correlation in the
same case where the covariance is
positive you can see here this is
obtained by the function cor and this
comes out to be 1
for the sign of positive is maintained
here
similarly if I try to take the data set
with negative signs and if I try to find
out the correlation coefficient between
1 2 3 4 and - 1 - 2 - 3 - 4 you can see
here this comes out to be here
minus 1
right now if I try to make here the
scatter plots I will try to show you the
both the things here so you can see here
say I can make here the scatter plot of
1 2 3 4 and - 1 2 3 4 you can see here
this comes out to be like this where the
direction of my cursor is indicating the
negative relationship and
similarly if I try to make it here
positive that means that two data
vectors both are here positive 1 2 3 4 1
2 3 4 in this case you can see here that
this
scatter diagram is trying to give a
positive exact relationship now I try to
take an example to show you that how the
interpretation of correlation
coefficients comes into picture suppose
I try to take care data set
we I have obtained the
marks and the number of hours of study
by the twenty students this is the same
example which I have considered in the
earlier lectures while making different
types of graphics or say bivariate plots
so we have recorded their marks
in the first row in both the tables and
the number of hours per week they
studied in ours here
right it's the same example which we
have considered several times and this
data has been stored inside two
variables marks and here hours
right now the first thing what it comes
to our mind that now we have got the
data and before using the concept of
correlation coefficient we would like to
see what is the type of relationship
whether it is linear or not so we simply
use the option plot and we try to plot
here the max versus of us so you can see
here in the data here is lying like this
all this data here here are your marks
and here is your hair actually ours
right and you can see here this there is
a linear trend here so this gives us a
sort of assurance that the relationship
between marks and ours is well close to
linear and now I'm but still I have made
the trendline by hand and now we also
have learned how to make this trendline
so for that I will use the same command
that we discussed earlier scatter dot is
smooth and I will try to make your line
between the two data vectors marks and
hours and you can see here that this
line is now also indicating that this is
nearly linear and this gives us a
confidence that okay in this case I can
use
the correlation coefficient and then I
try to find out the correlation
coefficient between marks and hours you
can see here this comes out to be 0.96
so you can see here what is this point
nine six indicating if you try to look
into this curve you can see here that
these Aviation which I am plotting in
orange color they are very close to
lines and these deviations are very
small and these points are lying very
close to the trendline so this is
possibly indicating that on an average
the degree of linear relationship
between marks and hours is very strong
and this is nearly ninety six point
seven nine percent
similarly if you try to interchange
those variables so earlier I have taken
say marks and of us and now I try to
take our send marks so you can see here
this value of the correlation
coefficient remains the same this is
what we have discussed in the last
lecture also
right so in this case you can see here
the sign of the correlation coefficient
is positive and this is indicating that
the relationship between x and y is
positive which is here you can see here
and we can conclude that as the number
of hours of study per week are
increasing the marks obtained by the
students are also increasing because
there is a positive relationship and the
value of correlation coefficient is
pretty high so that is why I can say
that whatever the data is telling that
is correct and my conclusions based on
the data they are correct and it is also
showing a linear trend and this is here
the screen shot I will try to show you
it on the our console also before I take
one more example of the negative one so
I will try to show you here I already
have stored the data so you can see here
this is the data for marks and this is
the data of our say hours so if I first
try to plot
marks versus hours or say hours versus
mark whatever you want this gives you us
you can see here nice scatterplot
indicating the linear trend if you try
to make it here trendline also so try to
give a scatter dot to smooth and you can
see here that a line is also plotted
here
right and now I try to find out the
between marks and hours that will
indicate the direction so you can see
here that the value of the covariance is
three sixty five point five two six
eight and it is positive so it is
indicating and it is matching with the
work on every whether were a graphical
conclusion that the relationship is
positive now I will try to find out the
correlation coefficient between marks
and hours and you can see here this is
0.96
right so you can see here that this
collision coefficient is trying to take
care of the
variance as well as covariance between
the two variables
now I will try to take one more example
and try to illustrate something more
suppose there are ten patients and those
patients are given some medicine and the
quantity of the medicine is measured in
milligrams mg and the time saying hours
is recorded and to show that when the
medicine is started showing the effect
and
this data is compiled here say quantity
of medicine and the time in say hours so
this is indicated here so this is the
data set of fish or the person number
one or patient number one so this is
indicating that 30 mg of medicine was
given and it took four hours of time
similarly for the second data set this
is the patient
patient number two and a 45 mg of
medicine was given and he took three
point six hours of time and so on so
this data on the quantity is stored here
inside a variable quantity and
time is stored here in say effect dot
time one thing please don't use their
variable time because time is used by
the our software also so be careful so
now I will try to take this data set and
I will try to first make up plot here so
you can see here this is the screenshot
and
you can see here this is on the x-axis
it is quantity and y-axis this is time
that is the effect dot time and you can
now see here these observations are
coming out to be like this so you can
see here that as the value of x are
increasing the values of Y's are
decreasing so this shows here a sort of
negative trend
right and this information we would like
to verify with the covariance function
or the correlation function but now let
me plot here the trend line also so you
can see here that the trend line is also
indicating that the relationship is
almost linear and we can safely use the
concept of correlation coefficient right
so this is the outcome and now I try to
find out the correlation between
quantity and effect dot time so this
comes out to be here minus zero point
nine eight eight five four five four
well you can control the number of
digits also and
now if you try to see this is showing
the value negative so the sine of
negative is indicating that this
relationship is decreasing as the values
of XS are increasing the values of Y's
are decreasing and
the magnitude here is zero point nine
eight eight five four five four close to
0.99 so close to 0.99 means the
relationship is going to be nice and
linear and the degree of linear
relationship is pretty high the maximum
value is 1
in the case of 1 what will happen that
all the observation will be lying
exactly on the line and we are in this
case this is just point 988 so it is
very close to the line so now this give
us a information and confidence that we
can use here the concept of correlation
coefficient and
this degree is coming out to be 988 so
now I can conclude that as the quantity
of medicine is increasing the number of
hours to effect are decreasing and I
would try to show you this thing on the
our console also and this is the
screenshot what I have done okay so let
me first try to copy this data the
earlier data was there because I had
used it earlier so this here is quantity
first let me
clear all the things view the screen by
ctrl L try to get this quantity and then
I try to copy therefore the effect dot
time variable and the data contained in
it so this is here effect that time so
you can see here quantity is obtained
here effect dot time is obtained here
now I would like to know what is the
nature of relationship so I try to make
here a plot between see here quantity
and
effect
dot time and you can see here this is
the plot it's the same plot that we have
just obtained in the slides and now if
you want to make it here I scatter a
smooth plot by adding a trendline so I
have to use the command here
scattered dot smooth and
this gives me here this graph which is
the same points but with a trendline and
now first I try to find out here the
covariance between the two because
covariance will assure us that what is
the sign of the direction whether this
is positive or negative so you see this
covariance comes out to be minus
twenty-two point six two five so this
minus sign is indicating here that the
relationship is negative and this is
verified here in this graph also if you
try to observe the direction of my
cursor on the screen this is decreasing
right now I try to find out the
correlation coefficient between the
quantity and effect dot time and this
comes out to be here minus zero point
nine eighty so once again this sign is
coming from the covariance and it is
indicating that the relationship between
quantity and effect of time they are
negatively correlated as the quantity
increases that time taken to affect is
decreasing
right and this is quite obvious also we
know from our experience that when we
try to increase the dose of the medicine
then it acts faster and
time to react becomes smaller and
smaller well up to certain extent after
certain limit
that may say that is not advisable and
you have to depend on that doctor's
advice
right so now in this lecture I have
shown you that how to
attempt or how to take a decision
whether you want to compute the
correlation coefficient or not so the
steps are first to try to find out the
the scatter plot or the smooth scatter
plot whether trendline try to look at
the trend in case if you are convinced
yes there can be a linear trend or the
relationship between the two variable is
approximately linear
this can be positive or negative then
you decide that well in this case I can
use the concept of correlation
coefficient to measure the degree of
linear relationship and
then you try to use the formula for the
coefficient of correlation
so I stop here and I would request you
that you please try to take some more it
assets and try to plot them and then try
to compute the values of correlation
coefficient and see what do you get so
you practice and I will take the topic
of rank correlation in the next lecture
and then I will see you in the next
lecture so thank you very much and see
you soon goodbye
[Music]
[Music]
[Music]
