noc19-cs33 Lec 23 Big Data Machine Learning (Part-I)

IIT Kanpur July 201
20 followers
1 Video View·Dec 3, 2023
Big data machine learning, preface, content of this lecture.
In this lecture, we will provide an overview of machine learning techniques to explore,
analyze and leverage big data processing.
We will also discuss the tools and algorithm that you can use to create machine learning
models and learn from big data and to scale these models up to the big data platforms.
Machine learning overview.
What is a machine learning?
Machine learning is the learning from the data.
So a machine learning is the field of a study that focuses on computer system that can learn
from the data.
That is the systems often called the model can learn to perform a specific task by analyzing
the lot of examples for a particular problem.
For example, machine learning model can learn to recognize an image of a cat by being shown
a lots and lots of example and images of the cat.
For example, a cat can be characterized by from the examples that it has the triangle
ears and nose, it has a fur, it has a tail, paws and whiskers.
So with these features, you can identify, characterize the cat.
So another feature of machine learning is, so another aspect which identifies or which
distinguish machine learning is that it does not have any programming.
So it has no explicit programming involved in problem solving.
So this notion of learning from the data means that machine learning models can learn a specific
task without being explicitly programmed.
In other words, the machine learning models is not given any step by step instruction
on how to recognize the image of a cat.
Instead, the model learns with the features that are important in determining whether
the picture contains the cat from the data that has been analyzed.
Because the model learns to perform this task from the data, it is good to know that the
amount and quality of data available for building the model are important factors in how well
the model learns the task.
Therefore, it does not have any explicit programming and we have also seen that it machine learning
is the learning from the data.
So there are two different aspects of a machine learning we have covered that is it learns
from the data and second thing is if there is no explicit programming involved in solving
the problem using machine learning.
And third important characteristics of the machine learning is about discovering the
hidden patterns.
Now because machine learning models can learn from the data that can be used to discover
hidden patterns and trends in the data which is hidden.
So the discovering hidden patterns is also an important aspects of the big data of the
machine learning.
Why?
Because it learns from the data so it can also discover the hidden patterns which are
embedded into the data and also which and also captures the trend which is there inside
the or within the data.
Second characteristics which identifies or which characterizes the machine learning
is about the data driven decisions.
So these patterns and the trends led to the valuable insight into the data thus the use
of machine learning allows the data driven applications or decisions to be made for a
particular problem.
Thus to summarize the field of machine learning focuses on the study and the construction
of computer system that can learn from the data without being explicitly programmed.
Machine learning algorithms and techniques are used to build models to discover hidden
pattern and to identify the trends in the data allowing for data driven decisions to
be made.
So machine learning is an interdisciplinary field.
So in applying machine learning to a problem the domain knowledge is essential to success
of the end results.
By domain knowledge we mean an understanding of the application or the business domain
in which we are solving the problem.
So the knowledge about the application, the data related application and how the outcomes
will be used are crucial in driving the process of building machine learning model.
So domain knowledge is also an integral part of a machine learning solution.
Also we can see that the machine learning is an interdisciplinary field that is it requires
the understanding and the background of the roots of machine learning draws from applying
the statistics and also the computer science, AI, these are computer science, AI, statistics
and also the domain knowledge, the intersection of this or it is an interdisciplinary field
of all three together defines the machine learning.
So the example application of machine learning that means where the machine learning can
be applied is in the credit card fraud detection.
That is one of the machine learning application is that you are likely to use in the weekend
or even today is the credit card fraud detection.
Every time you use the credit card, the current purchase is analyzed against your history
of the card transaction to determine whether if the current purchase is legitimate of the
previous history transactions or potentially a fraudulent one and in this process the credit
card fraud can be detected by applying the machine learning on this particular problem.
So if it is, if the purchase is very different from your past purchases that is learned from
the machine learning algorithm using the analysis of the data such as for the big
ticket item in the category that you had never showed an interest in or when the point of
sales is from another country, then it will be flagged as the suspicious activity.
Therefore, such applications like credit card fraud detection can be solved using an
application of machine learning on the big data.
So the big data is the historical data of the transaction records.
So that means from this particular historical data of transaction records, the machine learning
learns the model and whenever a new transaction comes, then it decides whether it is a legitimate
use by the user or it is a suspicious that means which differs from the usage pattern
of that user.
Now another example is about handwriting recognition.
So handwritten digit recognition can be easily done with the help of machine learning.
So when you deposit a handwritten check in an ATM, the machine learning process is used
to read the numbers written on the check to determine the amount of the deposit.
Handwritten digits are trickier to decipher than typed digit due to many variation in
the people's handwriting.
So this handwritten characters can be identified between 0 to 9 the digits using machine learning
techniques.
So this handwritten digit recognition is another such application of a machine learning.
Now further examples of machine learning is about recommendations on the website.
So after you buy an item on a website, you will often get a list of related items.
Often this will be displayed as the customers who bought this item also bought these items
or you may also like that these related items have been associated with them with the item
you purchased by the machine learning model and now being shown to you since you are also
interested to buy in them.
For example, when you purchase an item, let us say a shoe then you are also displayed
the recommended items may be based on your past purchases or the other customer who purchased
this item along with this item what are the other items they have purchased based on that
these recommendations are being made.
So a particular item and its associated items are being identified by the machine learning
and is to be recommended on the website for online purchases of the related items and
these recommendations are given to the customer.
So such kind of recommendations on the website is also an application of the machine learning.
Other applications of machine learning includes the targeted advertisements on the mobile
apps, sentiment analysis, climate monitoring, crime pattern detection, drug effectiveness
analysis.
Now let us decipher what is there in the name behind the machine learning and other such
terms which are related terms which are available.
So machine learning has its roots in the statistics, artificial intelligence and computer science
among other fields together they have contributed and the machine learning field.
So machine learning encompasses the algorithms and techniques that is used to learn from
the data.
Now a related field which is called data mining, the data mining became popular around the
time when the databases were in common use.
Databases and data warehouses were at the main streams in the business operations.
So at that time the data mining are of great importance and there are other practical data
management aspect to the data mining related to accessing the data from the databases,
but the process of finding pattern in the data is similar and similar to the machine
learning and can be used and can hence therefore the same algorithms which are there in data
mining can also be used as some of the algorithmic techniques in the machine learning that we
will see.
So therefore data mining is the field which was around the databases and data warehouses
and therefore there in that term, there in that technique, the algorithms and techniques
are common.
Therefore, some of these techniques from data mining and algorithms are also used in the
machine learning.
So data mining differs from the machine learning in the sense that it was focusing around the
databases and data warehouses, data retrieval and finding the patterns within that particular
databases and data management.
Now another term which is called a predictive analytics.
So predictive analytics is the term which is used in the business houses.
So predictive analytics refers to analyzing the data in order to predict the future outcomes.
This term is usually used in the business context to determine, to describe the activities
such as sales forecasting or predicting the purchase behavior of the customer.
This particular predictive analytics of analyzing the data also uses the machine learning, but
it is the term which is used to predict the future outcomes and data mining is to find
the patterns in the databases and data warehouses.
And machine learning is about the algorithms and techniques used to learn from the data.
Furthermore, the data science is also a related term.
So data science is a new term that is used to describe the processing and analyzing the
data to extract the meaning out of it.
Again the machine learning techniques can also be used here in the data science because
the term data science became popular at the same time the big data began appearing.
So data science usually refers to extracting the meaning from the big data and so includes
the approaches for collection, storing and managing the big data.
So therefore data science and also uses the techniques of machine learning to extract
the meaning out of the data.
So therefore, all these terms related to the which are coined around the machine learning,
we have just summarized.
Again we can say that the machine learning is applied in data science and in predictive
analytics whereas the algorithms and techniques which are used in machine learning can be
drawn from the data mining.
So they are all related terms.
So we can say that machine learning can be applied in data science, predictive analytics
whereas the algorithms and techniques of data mining is in machine learning.
So let us see the machine learning, what do you mean by machine learning models.
So when it learns from the data and discovers the patterns and trends, it allows also it
allows the data driven decisions to be used in many applications.
So how to learn from the data after learning from the data, it forms the algorithm which
is called a model.
So we will see what do you mean by the model.
So it learns from the data which we can also say that the model fits on that data.
So once the model is identified, then when a new data is given, it will discover the
patterns and the trends and this way it allows the data driven decisions to be made and this
particular way that means learning from the data or this particular technique is very
useful in many different applications that we have seen in the previous slides.
Now categories of machine learning techniques.
So machine learning techniques can be categorized into four different types.
So first type is called the classification.
So second type is called regression, cluster analysis and association analysis.
Let us understand these categories of machine learning techniques and then we will go in
more detail of the different algorithms which falls into these different categories.
So categories of machine learning is the first one is called classification technique of
machine learning.
So here in this classification, the goal is to predict the category.
So in classification, the goal is to predict the category of the input data.
For example, an example of this classification is to predict the weather as it is sunny,
rainy, windy or cloudy.
So the weather forecast can be, so weather condition can be a sunny or it can be a foggy,
it can be a windy or it can be stormy.
So this kind of categorization that is the weather which can be categorized into sunny,
windy, foggy and stormy is to that means given this weather input, you can classify into
different categories, then it is called a classification problem.
Some other examples of classification are classifying a tumor from the medical image
as being benign or malignant.
So predicting whether it will rain the next day is also a classification problem.
So that means it is predicting whether it will rain or it will rain or no rain.
So this becomes a binary classification problem.
So determining if the loan application is a high risk, medium risk and the low risk.
So if the loan application can be classified into three different categories, one of these
three different categories, then it is also a classification problem.
Now identifying the sentiments of a tweet or the review as being positive, negative
and neutral is also a classification problem.
So given a tweet and if we can identify the sentiments whether it is the positive, negative
or neutral, then this again can be classified or this can also be a classification problem
of machine learning.
Now let us see the regression problem of machine learning.
So when your model has to predict the numerical value like in the previous slide, we have
seen that it is predicting the category.
But now if it is predicting a numerical value instead of category, then the task becomes
the regression problem.
So for example, an example of regression is to predict the price of a stock.
So the stock price is a numerical value not a category.
So this is a regression task instead of the classification task.
So if you were to predict whether the stock price will rise or fall, that means it will
classify as rise or fall, then it would be a classification problem.
But if you are finding out the price of a or to predict the price of a stock, then it
becomes a regression problem.
But if you are predicting the actual price of it, then it is regression.
So that we have already understood.
So that is the main difference between classification and regression.
In classification, you are predicting at a category and in regression, you are predicting
a value, a numeric value.
So in regression, we are predicting a numeric value whereas in classification problem, we
are predicting a category.
Now further examples of regression problems are estimating the demand of a product based
on the time or the season of a year and predicting a score of a test, determining the likelihood
of how effective a drug will be for a particular patient and predicting the amount of rain
for a particular region.
Now the third category of machine learning is called cluster analysis.
So let us see about in cluster analysis, the goal is to organize similar items in your
data set into the groups.
A very common application of cluster analysis is referred to as the customer segmentation.
This means that you are separating your customer base into different groups and segments based
on customer types.
For example, it would be very beneficial to segment your customers into seniors, adults
and teenagers.
These groups have different likes and dislikes and have different purchasing behaviors.
So by segmenting your customers to different groups, you can target more advertisements
and to this particular group having common interest.
Note that the cluster analysis is also referred as the clustering.
So in this example, we have shown that if it is a group of people, we can segment them
or we can cluster them into different groups of having the common interest and these groups
can further be used for various business applications to target them for the sales advertisements
having the common interest and so on.
Other examples of the cluster analysis includes somehow are identifying the areas of similar
topography such as desert region and grassy areas and mountain regions and categorizing
different type of tissues from the medical images, determining different groups of weather
pattern such as snowy, dry and monsoon and also discovering the hot spots for various
type of crime from the police records.
Therefore, this classification, this grouping together the common or similar data items
is called clustering and this cluster analysis is also called as a clustering.
Now another category of machine learning is called the association analysis of machine
learning.
So in association analysis, the goal is to come up with a set of rules to capture association
between the items or the events.
These rules are used to determine whether the items or events occur together.
So a common application of association analysis is known as market basket analysis which is
used to understand the customer purchasing behavior.
So for example, association analysis can reveal the banking customers who have CDs, certificates,
and also trending to be interested in investment in other instruments such as money market
accounts and so on.
So therefore, the information, this association analysis inside information can be used for
cross selling.
So if you advertise money market account to your customers and likely open, they will
likely open such an instrument that is already seen in these applications.
So association analysis, other examples are such as recommending items which are similar
items based on the purchasing behavior or the browsing histories of customers, finding
items that are often purchased together and identifying the web pages that are often accessed
together so that you can more efficiently offer these web pages at the same time.
So the classification of machine learning technique, we have different categories of
machine learning techniques seen.
Let us say that they are classification, regression, cluster analysis and association analysis
and therefore, depending upon different problems, we have to apply one of these different categories
to solve the problems in the machine learning landscape.
For example, for predictive methods, we can apply the classification or regression depending
upon whether the prediction of a category it is there or a prediction of a value.
Similarly, when the descriptive methods are to be applied, then we have to choose either
between the clustering or between the association rules.
Now furthermore, the machine learning is also categorized as supervised and unsupervised.
So in supervised approaches, the target which is what the model is predicting is being provided.
So this is referred to as having the data with a label and because the target is labeled
for every sample that you have given in the data set.
So referring back to our example of predicting the weather category of sunny, windy, rainy
and cloudy, every sample of the weather data is labeled as being one of these four categories.
So the data is labeled by the domain expert and predicting the weather categories is also
called the supervised task.
So in general, the classification regression both are the supervised activity and approaches.
In unsupervised approach, on the other hand, the target that the model is predicting is
unknown or unavailable.
This means that you have unlabeled data.
Now remember the cluster analysis example of segmenting customers into the different
groups of common of having similarities.
So the sample in your data are not labeled with the correct group.
Instead, the segmentation is performed using the clustering techniques to group items based
on the similarities or it is also called as a characteristics that they have in common.
Thus, the data is unlabeled and the task of grouping customers into different segment
is called unsupervised.
In general, the cluster analysis and association analysis are unsupervised approaches.
So in short, we can classify all four techniques of a machine learning that is classification
regression as the supervised machine learning technique, whereas cluster analysis and association
analysis is unsupervised machine learning technique depending upon whether the target
is available with the data set or not.
If the target is available, then it is called a supervised learning.
If the target or label is not available, then it is called unsupervised learning.
So cluster analysis and association analysis, there the target is not available with the
data set.
Hence, it is called unsupervised, whereas in classification and regression, the target
or the label is there with the data set.
Hence, this activity of applying the data set is supervised machine learning technique.
