noc19-cs33 Lec 28 Big Data Predictive Analytics (Part-I)

IIT Kanpur July 201
20 followers
3 Video Views·Dec 3, 2023
[Music]
[Music]
big data predictive analytics pre-phase
content of this lecture this lecture we
will discuss the fundamental techniques
of predictive analytics basically we
will cover mainly the techniques such as
random forest gradient boosted decision
trees and a case studies of various
implementations using spark ml on
decision trees and and symbol trees
learning for predictive analytics
decision trees let us summarize the
decision tree which we have covered in
the previous session so in that decision
tree what we have seen is that it
automatically handled the interaction of
features that is it can combine several
different features in a single decision
tree that we have already seen so the
decision tree which is covered or which
is drawn as the tree induction which is
called is basically the tree which is
splitted on different features hence
there is a interaction or combine
different features in a single tree now
it can build a complex functions involve
multiple splitting criteria so that
means if the tree is very big obviously
it is built based on very complex
functions so all these things are
automatically handled as far as the
interaction of the features is concerned
that is the important aspect of the
decision tree which we have covered
second part is called computationally
scalability that is there exists an
effect of algorithm for building the
decision tree where every large data set
and with many features but unfortunately
single decision tree actually is not a
good predictor why because of the
overfitting errors and so on and due to
the noise
so computation and scalability is there
that means this particular decision tree
can be implemented can be implemented on
the the clusters and using a spark
system
of hundreds and thousands of nodes which
can be further is killed
therefore it has the computational
scalability of implementing the decision
tree
we have also seen the predictive power
so the predictive power of a single
decision tree typically is not good so
due to the overfitting because of the
noise in the data set therefore the
predictive power is of a single tree is
not typically good due to the
overfitting so a lot of techniques we
have seen about power coming from the
overfitting and thereby improving the
predictive power in the single decision
tree and fourth important part of this
the decision tree which we have seen is
about interpretability so we can
visualize the decision tree and analyze
this splitting criteria of the nodes
hence it has a good interpretability
that means once a decision tree is built
we can understand about the intricacies
of the data with the help of the
decision tree rules and we can also do
the analysis of the splitting criteria
on the nodes and also the values on the
leaves they basically are giving the the
predictions on the new data set by the
tree traversing
in decision trees so all these are
basically the summary of a decision tree
and we have already discussed so
important part here is that it has
though it has a good interpret ability
but it has not that good not so good
predictive powers so here we are now
going to see if let us say we are going
to solve the predictive analytics
problems and this single decision tree
having not that good predictive power
needs to be augmented so we are going to
see in this part of the discussion what
is the the techniques which can be used
for predictive analytics of the Big Data
so before going ahead into the basic
techniques some of the fundamentals
which are required to build the new
algorithms are which is used for the
predictive analytics we will see some of
the features of it as the bootstrap and
the backing so bootstrapping is an
algorithm which produces the replicas of
the data set by doing the random
sampling with the replacement so that
means we using the technique which is
called the random sampling with the
replacement we can build the replica of
the data set that means if the data set
is given we can produce another data set
using bootstrap techniques by applying
the random sampling with replacement so
this idea is one of the important ideas
or essential for the building the random
forest algorithm so let us consider the
data set Z which consists of x1 and y1
so here x1 is the set of features and y1
is nothing but the labels and this
particular data set is called Z
so this particular data set is called
the bootstrap data set if out of this
data set we can generate another one let
us see the bootstrap data set which is
built out of this particular data set Z
it is the modification of the original
data set
produced by applying the random sampling
with replacement that is called
bootstrap data set let us understand how
this is generated so let us see the
random sampling with replacement
technique
so let us understand this random
sampling with replacement using an
example so let us say that this is the
data set which is given let us call it
as Z then we can build another data set
which is called Z star by the
application of the same data set with
with the replacement so let us see that
we will randomly pick an object of a
data set let us say that it is an object
at number three and this is replicated
here in the Z star which is the
replicated data set which we are
building
now next we will pick another object in
Wrentham let us say that this time we
are picking the object number one and we
will now replicate it at the position
number two in the replicated data set
similarly we will now pick the object
number five and we will replicate at the
position number three and we can also
because this is the sampling random
sampling with the replacement we can
pick the the object number five again
second time so it is being duplicated so
that is allowed in this random sampling
with replacement so then possibly you
can pick again the object number five
because at each iteration you pick an
object at random and there is no
correlation with the previous step hence
it is called random sampling with
replacement so therefore five we the
object number five can be picked again
and this is shown in this example
finally we will pick another object let
us say object number two in random and
we will replace it we will replicate it
on the fifth position so this particular
data set which is now generated out of
the original data set which is called Z
and the bootstrap data set which we have
generated called Z star it's called a
bootstrap data set so after
bootstrapping we have a new data set the
size of the data set is the same number
of elements as the original data set but
its contents as you see is slightly
different
now some object may be missing and in
some cases and other objects may be
present several times that is more than
once that we see here that this object
is repeated or is being present more
than once that is allowed in random
sampling with replacement to generate
the bootstrap data set this particular
data set is called bootstrap data set
which is generated using bootstrapping
so after understanding the bootstrapping
let us see what is another technique
which is called McGee so it was the
second idea essential for an
understanding of random forest algorithm
so bagging is in short hand it is called
bootstrap aggregation so it is a general
method for averaging predictions of
other algorithms and not only for
decision tree but for any other models
it can be applied so the bagging works
because it reduces the variance in the
prediction so bootstrap aggregation
bagging is called bootstrap aggregation
or it is the method for averaging the
predictions based on the the different
models which are based on different
learning algorithms so the different
learning algorithms not necessarily the
decision trees but any any algorithm it
can be a learning algorithm
so different learning algorithm will now
give out the models
so this bootstrap aggregation that is
called banking is nothing but everything
of n symbol of our models so it will
give an average of different and symbol
of models
so these models will be generated by
different these models will be generated
by different algorithms learning
algorithms may be that these models are
nothing but an ensample of different
decision trees also so hence the begging
works because it reduces the variance of
the predictions so this particular
method reduces the variance of the
predictions and by doing this averaging
it automatically overcomes over comes
from the overfitting of problems of
which we have seen in the single
decision tree we will see more details
in the further slides so backing is an
important idea which is essential to the
understanding of the random forests so
backing is nothing but a bootstrap
aggregation it is a general method for
averaging the predictions of different
learning algorithms and models and
thereby so it is not only confined to
the decision tree but any other
algorithm many any other learning
algorithm is used here can be used for
bootstrap aggregation so by backing
works because it reduces the variance of
the predictions let us understand the
bagging algorithm in more details so
input to the bagging algorithm is
nothing but a training set that is Z
which comprises of x1 y1 and so on so
where X is the features or in a data set
and Y is basically the the labels which
are given as part of the target
so the input as the training set is
given and let us assume that another
input which is also required in the
backing algorithm is the number of
iteration let us say it is B and also we
require one machine learning algorithm
method let us say it is M so now the
bagging says that for for the iterations
from 1 to B and in each in the step
number two it says that it draw a
bootstrap sample Z star Z star B of size
n from the training data so this
particular Z star is a bootstrap sample
that we have seen how to generate in the
previous slide so once we draw a
bootstrap sample that is called Z star B
the B is the Butte iteration we have
generated this particular training data
to be used in this iteration via the
iteration that is Z star B now we will
apply the machine learning method M on
this data set that is Z star being and
obtain a model let us call it as FX star
B that is at the BH iteration the model
will be FX star B and finally after
carrying out the iterations B we will
generate the N symbol that is the
collection of different models how many
models B models capital B one for each
so the first iteration will have F star
one F star 2 and so on up to F star B so
it will be an instant of be different
models we have generated in this
iteration then this particular model
will be used in the prediction with n
sample so let us see that if the problem
is the regression problem then the
effects that is the prediction that is
the predicted model the model for
the biking model called FX means nothing
but it's an average of all the
prediction menus with the example and if
it is a classification then we will go
for the majority of all the predictions
and which is shown as every model will
give out one prediction and the majority
of that will be the other prediction
with the example to understand this let
us consider an example so let us say
that this is the decision tree and every
DCL let us say decision tree number 1 2
3 and they are represented as f star 1
LF we star 2 and F star 3 now these
different trees let us say will give out
result in the form of let us say fruits
let us say this is orange and this
particular tree will give a fruit which
is called mango and third tree also will
give out the result the prediction
result let us say that it is again
orange now we have to do a voting
majority voting so in that majority
voting we will see that this orange is
appearing twice hence the the prediction
output will be in the form of orange now
the question is why the bagging works so
the model FX has the higher predictive
power than any single decision tree or
any single model and so most of the
situations with any machine learning
method is the core the quality of such
aggregated prediction will be much
better than any single prediction and
why does the bagging works this
phenomena is based on the very general
principle what is called as a
bias-variance tradeoff so you can
consider the training data set to be the
random by it
self so in the real situation the
training data set maybe the user
behavior on the internet for example
while web browsing and which web pages
the user clicks using the search engine
while doing the while clicking on the
advertisement and so on similarly other
examples of the training set data are
the physical measurements for example
temperature location date time and so on
all these measurements are also
essentially stochastic which are not
same so if you can repeat the same
experiment in the same condition the
measurement actually will be different
because of the noise in the measurement
and since the user behavior essentially
are stochastic so not exactly
predictable and so you can understand
this particular training data itself is
random so because of this particular
nature of the randomness in the data or
a stochastic nature so the begging is
also works so the backing is an
averaging over the set of possible data
sets and removing noisy and non stable
part of the models will be achieved
using bagging so after everything the
noisy part of the machine learning model
will vanish out whereby the stable and
reliable part will remain in an effect
so the quality of the average model will
be much better than a single model so by
this way we have understood that you
know what is the bagging and by
averaging the bagging due to the
stochastic nature of the the data set
generation will allow the bagging method
to work and but the important part of
the bagging is that so it will be
averaging of different example
predictions so this averaging is more
accurate because the noisy part of that
machine learning model will vanish out
in this averaging process whereby the
stable and reliable part will remain so
the quality of average model will be
much better than any single model here
in this case of the banking scenario now
in short
let us summarize these two basic
concepts which are going to be used in
the random forest algorithm which we are
going to discuss next
so the bootstrap is a method for
generating different replicas in the
data set and bagging is an bootstrap
averaging aggregation so this method is
based on averaging predictions and
reducing the predictions variance so
backing improves the quality of almost
any machine learning method quality of
prediction based on any machine learning
method so bad thing is very
time-consuming for the large data set
now let us see the random forest
algorithm which uses the bagging and
boosting methods so random forest
algorithm is bagging of the decorrelated
decision trees so again I am repeating
random forest algorithm is a is a
backing of D correlated decision trees
what you mean by this let us understand
the random forest using the algorithm of
random forests so the algorithm of a
random forest requires the input in the
form of the training data set again X I
represents the features of a data set
and why I represents the label which is
given in the training data set so Z is
represented Z is a training data set as
an input to the random forest algorithm
and total number of iterations let us
call it as capital B now we will see how
what steps are followed in each
iterations for small B is from 1 to
capital B total number of iteration let
us see in each iteration that is let us
say step number 2 it will draw a
bootstrap sample which is called Z star
of the size n from the training data set
so training data set
Z will produce Z star let us say that Z
star B is the bootstrap sample for the
iteration number B now then we will
apply then we will grow the random
forest that is nothing but D correlated
decision tree and let us call it as TB
on this particular data set Z star B and
finally we will return this particular
iteration will be repeated at 1 to be
different iterations so that means we
will generate a be different D
correlated decision trees and call it as
T 1 T 2 and so on up to capital B
different trees and this is an example
of the different D correlated decision
tree
now using this particular training phase
a model which is called an n symbol of
different D correlated decision trees
that will be the N symbol of models
which will be generated by the training
phase now this model will be used for
the prediction with these n symbol tree
decision trees and symbol of decision
trees so if it is regression then we
will now as far as the prediction is
concerned which is nothing but an
average of the prediction of all the
models which will be given as the
outputs and if it is a classification
than the majority vote of all the
decision trees will be prediction let us
see again by the an example to
understand what do you mean by the so
classification will require the
categorical output from every and symbol
tree let us say that this particular
example or crease is represented by let
us say a tree number one a tree number
two and let us say it is having a tree
number three these trees will generate
an output let us say this tree will
generate and mango as an output this
tree will also generate let us say apple
there is an output and tree number two
will again generate an apple as an
output so now if we take at the majority
then majority comes as an apple so this
will be the prediction so prediction
will suppress the noise part therefore
these random forest trees are much more
accurate as far as the prediction
compared to the single decision trees
how to grow a random forest decision
trees so the trees is built greedily
from top to the bottom
as we have seen in the in the decision
trees but the difference here is that it
will select em out of P different
features so select em out of P input
variables at random as the candidates
for the splitting so in decision tree
all variables are used as the candidate
for splitting but here this is called D
correlated decision trees will use the
subset of the input variables in random
order
so that is why it is called D correlated
decision tree because not all input
variables are used but subset of the
input variables are used and they are
used for the tree induction that is the
they will be used to decide the
candidates for the splitting now each
split is selected to maximize the
information gained so information gain
as we know that is dependent upon the
impurity of Z and that is the errors
before the split and then errors after
the split and which is shown by this
particular equation now how to grow the
random forest decision trees so select
em out of P input variables at random as
the candidate for the splitting so here
the recommendations from inventors of
random forests are summarized here so
for classification the value of M has to
be chosen as root of P for the
classification problem and the minimum
instance will be per node is 1 and M
will be P times P divided by P by 3 for
regression problems so for
classification the rule of thumb B's
thumb rule says that the value of M
should be root of P and for regression
the value of M is equal to P by 3
so random forest here the results of the
training of two random forest is shown
the first variant is marked with the
with a green one and and either the
variant where each step M equals this
particular case and it means that at
each step we grow the regular decision
trees and you find the best split among
all the variables and the blue line that
is it is the correlated here it is the
decision tree and the blue one is d
correlated decision tree so this is a D
correlated decision tree and this is the
decision tree which is shown as the
green one so this means that at each
step we grow the regular decision tree
and you find the best split among all
the variables and the blue line it is d
correlated decision tree in this
situation we randomly pick M equals
square root of B that is shown over here
square root and all the trees can be
built using different subset variables
as we can see in this particular diagram
that initial stage the variant is is M
equal square root of B is worse before
20 iterations so before 20 iterations if
we do this kind of analysis up to this
point here this D correlated is worse is
going worse d correlated decision tree
is performing worse compared to the
single decision tree
but for the value which is more than 20
before the 20 iterations that means if
the number of trees are less than 20
then the the D coordinated decision tree
you will perform not that good compared
to the single decision tree but as the
number of trees grows that is nothing
but the number of iterations so if the
number of trees are more than 20 then we
see that this your D correlated decision
tree is performing much better start
performing better compared to the single
decision tree but equivalent ly this
variant of random forest algorithm
converges to a better solution that
means after beyond 20 iterations or
beyond 20 number of decision trees this
particular random forest algorithm
converges to a better solution so random
forest is a good method for general
purpose classification regression
problem typically slightly worse than
gradient boosted decision tree we will
see that both the cases so that means we
have seen that random forest is
performing better is better than the
single decision tree but
the random forest is slightly worse than
the gradient boosted decision tree so
that means gradient boosted decision
tree is going to be the best approach
for a general purpose regression
problems and for prediction problems
that we will see in the next slide so
summary that is the automatically
handles the interaction of the features
in the decision tree and of course this
algorithm can automatically handle the
interaction of the features because this
could be done by the single decision
tree so also that particular part or is
also covered in the random forest now
computational scalability is also there
in the random forest of course this
algorithm can automatically handle the
fraction of features because this could
be done by the single decision tree so
therefore in random forests also it
ensures computationally scalability in
the random forest algorithm each tree
can be split independently on the other
trees it is an important feature and I
would like to emphasize it that is why a
random forest algorithm is easiest and
essentially it is parallel so random
forests could be trained in a
distributed environment with high degree
of parallelization so so another aspect
of random forest is the predictive power
as the predictive power of random forest
is on one hand better than a single
decision trees but it is slightly worse
than gradient boosted decision trees
another part is called interpretability
here the interpretability is not that
good
that means you lose the interpretability
because of their composition of hundreds
and thousands of and simple decision
trees involved in the random forest and
therefore it cannot be analyzed
in the interpretation compared to the
single decision tree problem which are
more interpreted interpretable but less
predictive power and so therefore
interpretability what says predictive
power is having a trade out so as far as
the single decision tree will have the
good interpretability
but not that good predictive power
whereas the random forest has were
interpretability that means due to the
complex nature of the tree and sample
and so it is not possible to be
understood by the or analyzed by the
human expert whereas the prediction is
very good good predictive power is there
in the random forest
therefore for good prediction random
forest is preferred compared to the
single decision tree but there is
another better scheme as far as the
predictive power is concerned because
random forest is already lost is good
interpretability aspect so let us see
how better how the best we can do the
prediction so another improvement of
random forest is
called a gradient boosted decision tree
which is having much better predictive
power compared to the decision trees
[Music]
[Music]
[Music]
