noc21- ma35 - lec22

IIT Kanpur July 201
20 followers
Dec 3, 2023
Hello friends, welcome to the course essentials of data science with R software 1 in which
we are trying to understand the topics of probability theory and statistical inference.
So up to now you see we have discussed various definition concepts and topics related to
the probability theory and we were always interested in counting the events and then
trying to find out the probability.
But now you have to think in a very different way.
Suppose we are trying to conduct an experiment and we want to compute the probability of
certain event.
So what is your first job?
The first job is to collect the observations on what is your objective you want to find.
Suppose if I say I want to know the probability of head and tail.
So you will toss the coin and you will record head, tail, head, head, tail, tail and so
on.
And now we have simply said okay after this you can find out the probability but my question
is how?
Can you really add head and tail or if I say if you roll a dice we will have numbers 1,
2, 3, 4, 5, 6 right.
They are the indicator variable.
They are taking values in 6 different categories which are obtained by counting the number
of points on the upper side of the dice.
So now the question is this.
How would you like to integrate all this process in a simpler way so that anybody can understand
it very easily.
So whatever is the process and whatever is our objective based on that we can define
some objective which can convert this qualitative values into a quantitative value.
For example I can say suppose you toss the coin say in a sequence of 5 times and so on.
And then you try to count the number of heads in every trial.
So now in this case every trial is consisting of 5 continuous flips of the coin and then
we are trying to count the number of heads.
So now I can say let there be a random variable which is indicating the number of heads in
the continuous 5 flips of the coin that is all.
So now in this case what can happen that I can define a random variable and that random
variable can be defined like as number of heads in the 5 continuous trials of the coin.
So now what will happen earlier you had something like head, head, tail, tail, tail or head,
tail, head, tail, head but now you can define it in a quantitative way your x is going to
take value 1, 2, 3, 4 or 5 can you take a value 6 no because you are flipping the coin
only for 5 times.
So now you can see I can convert all my qualitative information into a quantitative way.
Now what is my advantage?
Once in statistics you get the information in some quantitative terms you can apply any
of the statistical tool.
So that is what now we are going to start we will try to formally understand different
types of concepts, definition which will formally helps you in formulating a real problem into
a statistical rule as statistical language with which you can apply the statistical rule
and what is your objective means ultimately you want to draw some conclusion that what
is really happening inside the data what type of information the data is containing.
So that can be obtained from such a setup.
So that is what we are going to start today we are going to start a topic on univariate
random variables.
Now as soon as I say univariate that mean there can be bivariate, trivariate or in general
multivariate but to begin with to understand the concept it is very important for us to
understand the concepts in a univariate way what is really happening what is the interpretation
what is the algebra and then we try to convert it into a bivariate, trivariate or multivariate
way.
So we are going to talk about how are we going to define the random variable and what are
the associated properties, definition, concepts etc in this lecture and in the some forthcoming
lecture also.
So let us begin our lecture okay so now we are going to talk in this chapter about random
variables and we are going to talk about 2 different particular type of random variables
which are called as continuous and discrete random variables.
So now the first question comes why do we need a random variable what is the need of
random variable?
So we know that whenever we are trying to work in the data sciences or decision sciences
some concepts are required to draw statistical conclusion from a sample of data from a population
of interest.
Because you see in statistics we are always working only on a sample which is drawn from
a population and as we have discussed we assume that the sample is a representative sample
right for example suppose we know the starting salary of say 100 graduating students from
a college in the subject statistics.
Now we can use this knowledge to draw different types of conclusion about the expected salary
for the population of the all student graduating in statistics.
So what is really happening we have a sample of only 100 students out of say 5000 students
from the college but now in case if you ask that okay what will be the expected salary
if a student graduate from this particular college possibly just using these 100 observations
we can get an answer right.
Similarly if a drug is being developed for say COVID patients then what do we know we
simply try to give the medicine to some patients and then we try to see whether the medicine
is working or not but what is your conclusion what you really want to know out of that think
about it.
So in case if a newly developed drug is given to a sample of selected COVID patient then
some patients may show improvement and some patient may not.
But are we interested only in these values or are we interested only in this small group
of patients or are we interested in the consequences for the entire population of the patients
all over the world right.
So now in order to draw such conclusion we need to have some statistical concept and
that are going to lay the foundation of the statistical inference which is our ultimate
objective.
So random variables are the foundation of theoretical concept required for making such
conclusion and they form the basis for the statistical test and statistical inference
right.
So now I already had given you the case example but now I can explain you formally suppose
we toss a coin and its outcomes are recorded as head and tail.
So now your sample space of this event is going to be some omega small omega where omega
is the head or omega is the tail.
But now if you try to see if I want to know what is the average number of heads coming
in the trial of this 20 flips.
So now you will have 20 values of heads and tail and how will you know that what is the
average value or the average number of heads which are coming in the 20 flips.
You cannot and it is difficult to handle the outcome because these elements they are not
the numbers they are some head, tail, head, tail and so on.
So now in order to solve this issue one simple option is that we can define a function a
mathematical function say capital X and this function is defined which is based on the
outcomes of the experiment which are indicated by a small omega and this function takes the
value 1 in case if omega is head and takes value 0 if omega is tail.
Well if you want you can interchange it also means you can say that it takes value 1 if
omega is tail or it takes value 0 if omega is head that will not make any difference
but that depends on you or the this is the choice of the experimenter that how the function
is defined.
So now you see what is really happening now you can see that here this function X that
is now playing a very important role and it is really helping you this is a real valued
function which is defined on capital omega the sample is based but it is taking us from
omega to a space of real numbers 0 or 1 because omega consists of say head and tail and now
I am saying that this head and tail they are going to be represented by 1 and 0 or 0 and
1 depending on your choice.
This X is actually called a random variable in a very simple language.
So in this case head is going to be denoted by 0 and tail by 1 and your sample space now
omega will be converted from head and tail to 0 and 1.
So this capital omega will be consisting of 2 values 0 and 1 which are going to indicate
the presence of head or tail in the outcome of the experiment.
Now in case if you really want to understand that why do I call it as a random variable
why not a variable.
So you know that when you are trying to toss a coin you do not know the outcome unless
and until the experiment is completed.
You just know either the outcome is going to be head or the tail but you do not know
that after you toss the coin with 100% surety what is going to come that we already have
done and if you remember we had used our software to sample from 0, 1 or say 1, 2, 3, 4, 5,
6 in the case of a dice to understand that the samples that you draw every time they
are changing.
So just for your recollection means I can show you here that we had used the sample
command for drawing a head or a tail and every time I draw it, it will give us a new value.
For example if I try to draw one value out of 0 and 1 so this is going to indicate my
here population and now I am saying a size is equal to 1 that means I need only one observation
out of 0 or 1 and then replace equal to true is trying to say that it is drawn by simple
random sampling with replacement.
So now in case if you try to execute it on the R console for example I am giving you
here some outcome you can see here that when I try to conduct it means every time I am
getting a different outcome.
For example if I repeat it once I get here 1, then I get 0, then I get 0, then I get
here 1 and you can see here this is the screenshot so you can believe on me that whatever outcome
I have shown you here they are really going to work.
But instead of believing on me let me try to show you this thing on the R console also
so that you can see that how this randomness is coming into picture which is very important
for you to understand.
So you can see here when I try to execute this command this time it is giving me 0 now
I try to repeat it is again giving me 0, it is again giving me 0, it is again giving
me 0, now it is giving me 1, now you tell me what it is going to give you 0 or 1 any
answer no you cannot tell me unless and until I press here enter you can see now I am getting
here 0.
So this is what I mean that the values are random.
Now similarly if I try to conduct the experiment with the role of a die and I try to write
down the number as a 1, 2, 3, 4, 5, 6 and suppose we are trying to roll a dice and
then I am trying to see what is coming so now I can simulate it with the same command
by changing the population from 0 and 1 to 1, 2, 3, 4, 5, 6 and now you tell me what
will be the answer it is coming out to be 1.
Now I try to repeat it I get now here 4, I again repeat it I get here 2, now you tell
me I am throwing my dice once again and now what do you expect we do not know unless and
until I press here enter that is equivalent to throwing the dice so I am getting here
1 and now once again you know what are you going to get no so this time you are getting
here a 4.
So you can see here every time you are getting 1, 4, 2, 1, 4 but all of this number they
are coming from say this set 1, 2, 3, 4, 5, 6 and this is what we mean when we try to
say that this variable is a random variable that means X is my number which is coming
on the upper face of the dice right.
So now you can understand that whenever we are trying to conduct a random experiment
we are interested in the outcome of the experiment but we want to convert it in some value we
are more interested in the value of some numerical quantity determined by the result.
We are not interested in all the details of the experiment that at what time you did or
whether it came head or tail we want every information in the terms of numerical value.
So these quantities of interest that are determined by the results of the experiment are called
as random variables right.
Now if you try to see whenever we are trying to conduct an experiment or a random experiment
we do not know what is going to be the outcome.
So now how to take care of this phenomena because you see whatever values you are going
to get they are going to indicate the nature of the phenomena that how the process is working
how the things are happening.
So what we can do that when we are trying to toss a coin the outcomes are head and tail
that we already have converted by indicating by 0 and 1.
But now in the toss of our trial whether you are going to get head or tail how to formulate
this process right because we are not 100% confident.
So one option is this with every outcome we can associate a probability right and then
whatever are the possible values of the variable that it can take for each value of the variable
we can associate 1 probability and that will indicate the process in a much better way
than just by saying head or tail or 0 and 1 right.
So what we can do that we can assign probabilities could the possible values of the experiment
For example if I say probability equal to X equal to 0 is indicating the probability
of a head and I can say that this is equal to 1 by 2 probability X equal to 1 is indicating
the probability of a tail in that toss of a coin so this is equal to 1 by 2.
So now looking at this expression can you conclude that okay we have got an experiment
where we have 2 outcomes head or tail and they have got the equal probability of occurrence.
On the other hand if I suppose write probability that X equal to 0 this is equal to suppose
1 by 3 and probability of X equal to 1 is suppose 2 by 3 what do we infer from here
this is indicating that the event X equal to 1 has higher probability of occurrence.
So there are more chances that if you try to toss a coin then tail will come.
So this is how you can change your statement and you convey what is really happening in
the truth.
So now we may therefore view as a random variable which collects the possible outcomes of the
random experiment and captures the uncertainty associated with them.
So now you can see that the concept of this random variable is very helpful for us to
understand the statistical phenomena and it is associating a numerical value with
the outcome of a random variable as well as it is trying to associate a probability with
that value.
So that you can have a complete picture of the phenomena that what are the possible outcomes
and how they are going to work.
So now suppose it was 2 dice and suppose we want to study about the sum of the points
on the upper face that and we want to know what is the process in which the sum of the
2 numbers on the upper faces of the 2 dice they are coming out to be 7.
So in this case our interest will not be in the individual data point for example out
of those 36 outcome there are such a 6 values 1, 6, 2, 5, 3, 4, 4, 3, 5, 2 and 6, 1 which
you try to sum they will give you the value 7.
Say 6 plus 1 is 7, 5 plus 2 is 7, 4 plus 3 is 7 and so on and we are trying to count
these numbers the numbers which are on the upper surface of the die.
So these quantities of interest that are determined by the results of the experiment are known
as random variables and their values.
For example in this case you can see here if you try to see the probability that you
get the value 2 as a sum of the number of points on the upper face of the die this is
there is only 1 event 1 and 1 that can occur on each of the upper surface of the dice.
So it will give you 1 plus 1 equal to 2 so there is only 1 point so this is going to
be 1 by 36 and similarly if you try to go for the outcome of 3 then 1, 2 and 2, 1 they
are the 2 possible outcomes so the probability is going to be 2 by 36 and similarly if you
try to go for 4 there are 3 possible outcomes similarly for 5 there are 4 possible outcome
and similarly you can see here for each of the number because the minimum number is 2
and the maximum number will be 12 that means 6 and 6 occurs.
So now you can see the probability or the distribution of the probabilities of the values
of random variable in this random experiment.
So you can see here that the maximum probability is here 6 by 36 which mean the maximum chances
are that the sum will come out to be 7 then the next probability is 5 by 36 that is observing
the value 6 or 8 and similarly you can conclude for all other values.
So you can see here this simple experiment has been associated with 2 numerical values
on the outcome and then you have defined the random variable which was the objective of
your study and based on that you have associated the corresponding probabilities and now looking
at the values of the probabilities one can very easily decide that whether the some event
is going to occur or not or what is the distribution of the probabilities among the different values
which have random variable can take right.
Now in this case as I said the random variable is going to take the values whose minimum
value is 2 and the maximum value is 12 and its probability are given.
So now let us try to indicate this phenomena in a statistical way through the help of definition
of random variables and sample space and corresponding probability.
So let capital omega represent the sample space of the random experiment and let R be
the set of real numbers.
Now you can see this random variable is going to take here the values like a 2, 3, 4, 5
so do not you think that I can associate here a set of real numbers the numbers are going
to be between 2, 3, 4, 5 up to 12 right.
So let R be the set of real numbers then the random variable X is assigning one and only
one number to each element of this capital omega which are indicated by small omega and
as you had done earlier that X of head or X of tail you can also define here Xw is equal
to small x and this x is small x that means the x in the lower case alphabet okay what
is the meaning of lower case alphabet and upper case alphabet which I will try to discuss
soon and this small x that is belonging to the set of real numbers and now x is a mapping
from capital omega to R that whatever are the values here they are obtained as 1, 1,
1, 2 etc., etc., etc. they are now translated to a set of real numbers like as 2 and 3 and
so on right.
For example if you are trying to say that x is going to take the value 2 that means
this is the probability of occurrence of 1 and 1 and whose probability is 1 by 36 right.
So now if you try to see this 1, 1 this is your small omega which is belonging to capital
omega and now this is being transported to x equal to 2 which is now a real number.
So you can see here through the help of random variable you can transport the sample space
or the values of the sample space to some real number right.
So now you have seen here that in this example you have obtained the probability of observing
2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 and if you try to sum all these probabilities 1 by
36 this, this, this, this, this, this, this, this, this and this all the probabilities
their sum will come out to be 1 right because this is the probability of omega which was
always equal to 1 that we have discussed in the earlier lectures right.
So now you can see here that the earlier knowledge which you obtain that is now going to help
you in further developments and you want to find out here the probability of say happening
of 2, 3, 4 up to 12.
So this I can express by using the notation probability of union of say x equal to small
i where i is going from 2 to 12 like this that that is equal to x equal to 2 union x
equal to 3 union up to here x equal to 12 and then this is the probability of this thing.
So this is going to be because they are going to be disjoint events right 2 events cannot
happen together.
So I can write down this probability as the sum of individual probabilities and that is
always going to be 1 that you can verify from this example right.
So now I can express all these things in a 1 line definition very simple way and for
that I need here 2 things 1 is capital omega and there is capital R. So let capital omega
represent the sample space of the random experiment and let capital R be the set of real number
then a random variable is a function x which assigns to each element small omega belonging
to capital omega 1 and only 1 number that is x omega is equal to small x. So small x
is going to be a number which belongs to the set of real number. So x is a function from
omega to R.
So now you can see here this is pretty simple if you want to understand it. Now I can give
you this information you can use here that here I am trying to use 2 symbols capital
X and small x what is the meaning of this thing. So now let me try to take 1 very simple
example and then I can explain you very easily.
Suppose you want to measure the heights of 5 students. So let random variable x is representing
the heights of students. Now you call the first student and ask the height. Suppose
this is 168 centimeters. Now you call, so this is the value of here, value of here height,
height of first student and can you try to call the second student and you try to measure
the height of second student. So suppose this comes out to be here suppose 170 centimeter
and similarly you will try to call all the 5 student and finally you will have all the
values of heights of the 5 student.
So now how to indicate this numerical values and how to express this statement. These are
pretty long statement it is not possible to write them again and again height of first
student height of second student and you suppose there are 1000 students you have to write
height of 999th student and height of 1000 students. So 1 simple option is that I am
indicating here with this by capital X the height.
So I can indicate the value of the height by small x. So now this 168 centimeter which
I am taking here this is actually small x. This 170 centimeter I am taking here this
is also small x. Now both of them are small x how to determine what is my first observation
what is my second observation. So I can do here whatever is my first observation I can
write down here x1 and whatever is my second observation I can write down here x2.
So now you can see here that x1 and x2 they are indicating the value of the variable capital
X for the first observation and second observation and similarly means I can have here 5 observations
like x1, x2, x3, x4, x5 which are going to be some numerical values of x. So in statistics
we have a convention that by the capital letters we try to denote the random variables and
their values are indicated by small alphabets small lower case alphabets.
So as I gave you this example here that if x is the height of the student then small
x equal to 168 is the centimeter is the value of the x and similarly say x1 equal to 170
centimeter this indicates the first value of x and x2 equal to 100 centimeter indicates
the second value of x. So whenever I try to write down here let x1, x2, xn be a random
sample you can understand that these are n values which are the values of the random
variable X that is our understanding as simple as that.
So now the way we are going to represent these values is as follows random variable whose
set of possible values can be written either as a finite sequence x1, x2, xn means you
know that only for example in the case of our say here coin you have only two options
so there will be only means suppose you toss it two times so there will be only two outcomes
and if you try to toss three times there are going to be three outcomes.
So we can write down here x1, x2, x3 but similarly this sequence can be an infinite sequence
also and in that case we can write down here x1, x2 dot dot dot. So those random variable
whose set of possible values can be written either as a finite sequence x1, x2, xn or
as an infinite sequence x1, x2 dot dot dot are said to be discrete and the corresponding
random variable is called as discrete random variable.
For example if I say the toss of coin or say let me take better example said that the number
of points on the upper side of the dice when it is rolled it will take value here say here
1, 2, 3, 4, 5 and 6 but if you try to toss here 2 die and if you try to find out the
sum of the numbers on the upper faces the possible values will be 2, 3, 4 up to here
12. So you can see here that these values are finite or countable infinite right.
You cannot say that there are 2.3 number of heads or you cannot say that the value
of the outcome on the upper face of the dice is 3.4 right. So a sample space is discrete
if it consists of finite or countable infinite set of values right. For example a random
variable whose set of values possible values is the set of non-negative integer is a discrete
random variable okay.
For example let me try to take an example to explain you this concept in more details.
Suppose there is a customer care office and that office has 30 external lines for that
phone and at a particular time the system is observed and some of the lines are being
used that how many people are calling and how many lines are actually being used to
answer those calls. So let the random variable capital X indicate the number of lines in
use. Then now if you can see here there are 30 lines. So the first possibility is that
none of the line is going to be used nobody is calling or one possibility will be that
only one customer is calling so only one line is used or second possibility is okay means
two customers are calling to two lines are used and up to now this number can go from
here 0 to 1, 2 up to 30 only because there are only 39.
So if there is a 31st customer the customer will be waiting and then as soon as the telephone
line gets free the customer is going to be entertained right. So in case if we try to
observe this value or the values of capital X which can take then X can assume any of
the integer value between 0 and 30 it cannot be 20.5 no or it cannot be 16.7 no and for
example if I say that at a given time there are 5 lines which are being used this information
can be indicated by writing small x equal to 5.
So you can see here this is our symbolic representation right similarly in another example a batch
of 1000 machine part contain 20 that do not conform to the customer requirements. Now
in this case the random variable is the number of parts in a sample of 5 parts that do not
conform to customer requirements. So somebody is going to take a sample of 5 such parts
and the number of defective parts will be counted defective in the sense that they are
not conforming to the requirement of the customer right.
So now I try to take one more aspect so up to now we have considered the random variable
which is going to take the values like which are say 1, 2, 3, 4, 5, 6 etc and they cannot
take a value like 1.2 or 2.3 like this. So now I try to take the other part where the
random variable can take the value like 2.3 and 3.5 also right. So we try to consider
the concept of continuous random variable.
Suppose our dimensional length is measured such as say vibration, temperature fluctuation,
calibration, cutting tool wear, bearing wear, raw material changes etc. There can be many
such example and now suppose we are trying to see the variation. So in practice what
will happen there can be small variation in the measurement due to many reasons many causes
right. So in such cases in such experiments in case the measurements are represented by
a random variable X capital X then it is reasonable to model the range of possible values of capital
X with an interval of real numbers right. For example the variation in the say this
temperature fluctuation can be between say 1 degree and say 5 degrees and this and any
value between 1 and 5 degrees for example 1.1, 1.11, 1.112, 1.113 or 3.1, 3.2, 3.3 any
value can be taken by such a random variable. So this type of model provides an information
about the precision in the length measurement right.
So now you can imagine that there also exist random variable that take value on a continuum
of possible values right. These are known as continuous random variables right. One
example of this continuous random variable is by indicating the lifetime of a bulb right.
Now if I say the lifetime of a bulb can be between say this 0 hours and say 1000 hours
right that can be anything we do not know and then we have seen that in our homes also
whenever we are trying to put a new tube light or a LED light or a bulb anything we do not
know that how long it is going to last right. So we can say that the bulbs lifetime is assumed
to take on any value in some interval say a to b where a and b are greater than 0 right.
So now once you have understood these two concepts of continuous and discrete random
variables can you recall that when we took the example of a coin toss or the rolling
of dice experiment that we had associated a probability with every value of the random
variable to describe the respective random variable. So up to now we have assumed we
actually assumed that this type of information that is the information about the probability
of such an event that is known to us. So we assume up to now that we have this type of
knowledge however if this knowledge is not there one simple option is to draw a sample
of data consider a sample of data and use it to estimate the unknown probabilities and
other quantities with a given pre-specified uncertainty level right that is our basic
objective this is what we try to do that whenever you are trying to work in a data science the
process is so complicated that you just cannot compute the probability and you do not know
the probability you have to observe the data and then you have to compute such probabilities
on the basis of a sample. Once you obtain it on the basis of sample
there will always be some uncertainty so that uncertainty should be known to you right so
that is what I am trying to say here that we try to estimate such probabilities right.
So now I can conclude this lecture by saying that a sample space is continuous if it contains
an interval which can be either finite or infinite of real numbers and a sample space
is discrete if it consists of finite or countable infinite set of outcomes right this is what
you have to keep in mind and now in case if I try to finally conclude it on this slide
I can say that in general we can say that it is mandatory to know the probabilities
the probabilities for example x belong to a subset A which is actually a subset of R
right that is mandatory for us. So now in case if we try to choose here capital
A to be like this interval open interval minus infinity to close interval say here up to
x where x is belonging to real number then we have probability of x belonging to A is
given by probability that x belong to this interval and probability that x belong to
this interval from minus infinity to x this can be written by here like this statement
that probability that capital X is lying between minus infinity and x and yeah this can be
written as probability that capital X is smaller than x that mean the random variable and less
than equal to its value right. So now if you try to see we can compute or
convert such type of statement into probabilistic statement and this type of considerations
gives rise to the definition of the cumulative distribution function that we are going to
discuss in the next lecture okay. So now we come to an end to this lecture but before
leaving let me tell you very honestly I am dealing here with the data sciences where
you are going to deal with values when we are trying to define the random variables
or discrete random variables or continuous random variables we have got a very strong
mathematical theoretical definitions which are coming from the major theory.
So we have a major theoretic approach for the probability and actually that is the most
general definition and this major theory actually gives us all the fundamental for the development
in the probability theory and whatever we are doing here in the probability theory they
are only a particular cases of those definition which are defined in the major theory. So
those who are from a statistics background those who knows about that definition they
should not get confused my idea here is very simple how are we going to view these concepts
in real data set well in case if I have to teach a class of probability theory or if
I have to teach a class on major theory I will possibly express this thing in a very
different way so that is what you have to keep in mind and do not get confused.
So you try to think about this phenomena try to settle down inside your mind that is
most important because every time now whenever you have to do something first you have to
define a random variable and then you have to take a call whether it is continuous or
discrete and now I will try be showing you in the next coming lectures that the tools
are different for continuous and discrete random variables. So you try to think about
it and I will see you in the next lecture till then good bye.
