#ml#machinelearning#ai#artificialintelligence#datascience#regression#classification 🔥 In this video, we explain every major Machine Learning algorithm. Regression models: Linear Regression, Polynomial Regression. Classification models: Logistic Regression, Naive Bayes. Models used for both: Decision Tree, Random Forest, Support Vector Machines, K-Nearest Neighbors. Ensembles: Bagging, Boosting, Voting and Stacking. Deep Learning: Fully Connected (Dense) Neural Networks. Unsupervised learning: K-Means clustering and Principal Component Analysis (PCA) dimensionality reduction technique.
Heads up! You can't learn Machine Learning in just 22 minutes, a day, a week or even in a month! It needs a continuous dedication, patience, and consistent effort. I’m here to guide you every step of the way with clear explanations, tips, and resources to make your learning experience easier! Don't worry if there were concepts that were hard to understand!
Keep at it, and you’ll get there. Subscribe and like the video if you found it helpful!
Starting with this video, we’ll be posting a quick quiz on our Instagram page to help you review the material and test your understanding! It’s a great way to reinforce what you’ve learned and see how well you’re understanding the concepts. Be sure to follow us on Instagram, keep track of your progress and challenge yourself!
🔔 Don't forget to like, subscribe, and hit the bell icon to stay updated with our latest videos!
🤖 Note that we use synthetic generations, such as AI-generated images and voices, to enhance the appeal and engagement of our content.
🌐 If you have any questions or topics you want us to cover, leave a comment below. Additionally, share with your thoughts about the content, how do you think we can make them better? Thanks for watching!
Hi and welcome back.
In this video, we will explain all machine learning models in a simple and intuitive
way.
But before doing that, I recommend you subscribe to our channel to be sure you don't miss
new videos.
I will go through the models in the following order.
Firstly, we will see the regression models, then classification models, models used for
both, and lastly, two unsupervised models.
Let's start.
Linear regression is the simplest regression model that tries to find the linear relationship
between X and Y, where Y is a continuous variable.
The result is an equation that describes how each unit change in X affects Y.
Thus having only the value of X, you will be able to predict Y by plugging it into the
equation.
To train a linear regression model, you generate an initial, random coefficient or weight for
the variable X and a bias.
Using the initial guess, we predict the labels in our data and compare them with the actual
ones.
The error is then iteratively optimized with respect to the weight and the bias using gradient
descent to find the most optimal values that minimize the error in our data.
Linear regression can be modified by adding polynomial terms for nonlinear data to become
polynomial regression.
By raising X to various powers, we enable it to capture nonlinear relationships.
Note that you manually provide the maximum degree of the polynomial to use.
Regression models also have modifications called ridge, lasso, and elastic net.
All are regularization techniques that are used to combat overfitting but used for different
purposes.
Ridge is helpful for reducing multicollinearity.
It shrinks some coefficients towards zero without making them zero.
Lasso can perform feature selection by shrinking some coefficients and even making them zero,
effectively removing their influence from the model.
You can use elastic net to combine both ridge and lasso.
Now about the models that are only for classification.
Logistic regression, despite its name, is a classification model.
By default, it is designed for binary classification tasks.
The classes are defined as positive and negative class.
It predicts the probability of a positive class given the inputs.
It consists of a linear regression part that predicts a value from negative infinity to
positive infinity and a sigmoid function that transforms it into a range 0 to 1.
This value is often interpreted as the probability of predicting the positive class.
The threshold, which can be adjusted, is used to decide the predicted class.
The error function used is cross-entropy loss because the labels you want to compare are
not continuous but categorical.
Logistic regression can also be modified to handle multiclass classification.
The most popular way is to use the softmax function instead of sigmoid.
You make the model learn the probability of predicting each class separately.
Take note that it changes the number of coefficients.
I like to use the notation of neural networks to explain it, which makes it easier to understand.
Naive Bayes is a probabilistic machine learning algorithm based on Bayes' theorem.
It's naive because it assumes that the features are conditionally independent given the class
label.
This means that if you have a text and want to know the probability of it belonging to
a positive class, where each word is a separate feature, then it assumes that each word is
independent of other words.
This is generally not true, but it makes the computation straightforward and efficient.
As a result, the probability of the text belonging to a positive class can be calculated by this
formula.
There are three types of Naive Bayes models.
Gaussian, assuming that the continuous features follow a Gaussian distribution, multinomial
Naive Bayes, often used for discrete data like word counts in text classification, and
Bernoulli Naive Bayes, suitable for binary Boolean features.
Note that this algorithm was at some point one of the most popular models used for spam
detection.
Now, let's move to the algorithms that can be used for both classification and regression.
Decision tree is one of the most interpretable algorithms in machine learning.
It has a tree-like structure and can be easily visualized, making it ideal for a wider audience.
At each node, the tree chooses a feature to split on, trying to maximize the split between
classes.
The split is achieved by using conditions in the form of if-else statements.
The metric used to evaluate the quality of each split is impurity.
This is pure.
This is not that much.
Decision trees achieve a non-linear decision boundary by using rectangular regions in the
feature space.
Additionally, no data scaling is needed.
However, they can easily overfit the training data if they grow too deep.
There are two common ways to avoid the model becoming too complex.
Firstly, you can pre-prune it, also known as early stopping.
You can achieve it in different ways.
For example, you can provide a max depth that will limit the maximum depth of the tree.
The second method is post-pruning, which involves fully growing the tree and then removing branches
that add little value to model accuracy.
Another main disadvantage of decision trees is that small changes in the data can lead
to completely different tree structures.
Additionally, you should be careful with unbalanced classes.
Some weighting methods are used to grow the tree correctly.
For regression, decision trees split the data based on values of features that minimize
the error in the target variable, like mean squared error.
Each node contains the average or the median of the target variable values within that node.
This average serves as the prediction for any new observation that falls into that particular node.
It is not the best idea to use decision trees for regression, as it is hard to find the
balance between overfitting and underfitting.
Additionally, it fails to provide smooth predictions.
Decision trees segment data into constant regions, which means they produce a step function
rather than a smooth trend.
This can make predictions look unnatural for data that changes gradually over time, like
temperature, stock prices, or other time series variables.
Random forest is an ensemble method that combines several decision trees for prediction and
merges their outputs.
Decision trees are trained using random subsets of training data created by bootstrapping.
Bootstrapping creates multiple subsets the same size as the original data by randomly
sampling the training dataset with replacement.
Training a decision tree is the same with little modifications.
At each step of splitting a node in a tree, random forest randomly selects a subset of
features to choose the best split.
The random choice of features helps in reducing the correlation between trees.
For classification, all trees predict a class label, then majority voting is used to get
the final prediction.
For regression, all trees predict a continuous value, which is then averaged to get the final
output.
Some of the main advantages of random forests include most of the advantages of decision
trees.
It is less prone to overfitting and generalizes better on unseen data.
You can get feature importance scores for random forests, for decision trees as well.
Lastly, they can handle large datasets with high dimensionality.
Some of the disadvantages are it is less interpretable than decision trees.
It is more computationally complex as you train several models instead of one.
Finally, it contains more hyperparameters than the algorithms we discussed before.
Next, support vector machines.
The idea of SVM is to find the optimal hyperplane that separates data points of different classes
in a high-dimensional space.
The separation maximizes the margin, which is the distance between the hyperplane and
the nearest data points from each class.
Those points are the key data points, which are called support vectors.
Support vectors play an active role in defining the decision boundary and are chosen using
a concept called Lagrange multipliers.
In real-world scenarios, perfect separation may not be possible.
Also, you will rarely see perfect linearly separable data.
SVM is designed to handle both scenarios.
It introduces a soft margin hyperparameter called C that allows some misclassification.
Increasing C, you prioritize minimizing classification errors, which can lead to overfitting.
Too small C will allow misclassifications and can lead to underfitting.
SVM uses kernel trick to handle non-linearly separable data.
In higher dimensions, it may be easier to find a separating line or hyperplane.
It is a method that allows the algorithm to work in higher-dimensional spaces without
explicitly calculating the coordinates of the data points in those spaces.
Some examples of kernels include linear kernel, polynomial kernel, RBF kernel, and sigmoid kernel.
RBF kernel is the most widely used.
The concept of the kernel trick is more complex and will be covered separately in a future video.
SVM is a powerful model that often delivers high performance,
especially for high-dimensional datasets.
However, it can become computationally slow when applied to datasets with a large number of observations.
Additionally, it requires careful tuning of multiple hyperparameters to achieve optimal results.
For regression, support vector regressor can be used.
The concept is similar, but now it defines a margin of tolerance
where predictions within that range are considered accurate and are not penalized.
As for SVM, it has many hyperparameters to tune and works well for smaller datasets.
K-Nearest Neighbors, or KNN,
is another interesting algorithm that belongs to a class of lazy learning algorithms.
These are algorithms where you don't train a model or learn parameters from data.
Instead of using the learned parameters for later predictions,
you use the training data directly.
But how?
Let's say this is our training data.
In eager learning algorithms, such as the ones we saw earlier,
you would fit a decision line and use the learned boundary for predictions.
If the new observation is here, then it's a class 1.
If it is here, then class 2.
In KNN, you don't fit a decision line to distinguish between the two classes.
To make predictions using KNN, you need to think about one main hyperparameter k.
If we got a new observation we want to predict the class for,
we first calculate the distances from all other points in the dataset.
The distance metric, by the way, is another hyperparameter.
Then we choose the k closest points, let's say it's 5, and do a majority vote.
So you just use the training data instead of learning something from it.
When you increase k, the predictions are based on a larger group of data points,
which makes the prediction smoother.
This reduces overfitting by making the model less complex.
If you increase it too much, the model may start to underfit
and eventually predict only the majority class in the dataset.
k hyperparameter, and sometimes the distance metric,
should be carefully tuned for better performance.
One major disadvantage of KNN is that each prediction becomes computationally expensive,
especially when you have a large dataset.
Imagine having millions of observations, and for each prediction,
you need to calculate the distance from it to the other 1 million data points.
It becomes very inefficient for large datasets.
Additionally, since it relies on distance calculations,
you need to scale the data to treat all features equally important.
For regression, things are simple.
Everything is similar, but instead of a majority vote,
you take the average of the values or, in some special cases,
the median, minimum, or maximum of the group.
In default KNN, all data points in the group are treated as equally important.
However, there are also modifications of KNN,
where you can give weights to closer observations.
Next, ensemble methods.
The core idea is that groups often make better decisions than individuals separately.
To achieve that, diversity and or collaboration are introduced.
In practice, ensemble methods often achieve the highest score in machine learning competitions.
There are four main types of ensembles.
Bagging, boosting, voting, and stacking.
We have already seen an example of bagging.
Random forests.
The goal of the model is to train several independent and diverse tree-based models.
Then, the predictions are averaged or majority voting is applied to reduce errors
and create more robust predictions.
In random forests, individual trees are often allowed to overfit
their respective bootstrap samples to capture the most information,
but this overfitting is mitigated when the predictions from many diverse trees are averaged,
thus reducing the overall model variance.
Boosting is another popular technique.
The idea is to combine several weak models to create a strong model without randomization.
In this approach, models are combined in a sequential manner.
The first model is trained on the full training data,
and each subsequent model tries to correct the mistakes of the previous one.
Higher weights are assigned to misclassified observations
and smaller weights to correctly classified ones.
During the prediction, models with higher performance are often given higher weights.
Majority voting is often used for classification and averaging for regression.
Voting classifiers are a type of ensembles for classification.
Here, you take different trained models, often of different types,
and their predictions are combined through voting.
There are two main types of voting, hard and soft.
Hard voting is when the models predict the class,
and the class with the majority of votes is selected as the final output.
Soft voting is when the models output probabilities,
and the probabilities are summed up.
The class with the highest total probability is chosen.
The choice between hard and soft voting depends on
whether the classifiers provide well-defined probabilities.
Averaging is used for regression.
The idea is to average the outputs of several models.
For both averaging and voting, you can assign weights to the models
to determine which model has more influence during prediction.
Stacking consists of two main blocks, set of base models and a meta model.
Each base model can be any ML model.
The data is first separated into training and validation sets,
or you can use k-fold cross-validation instead.
You use the training data to train the base models.
Each base model generates predictions on a validation set
or sometimes the original data set, depending on the implementation.
These predictions are then used as input features for the meta model.
The target variable for the meta model remains
the original target variable from the validation data.
The meta model learns how to combine the predictions
from the base models to make the final prediction.
The meta model can be a simple logistic regression model,
which provides coefficients acting as weights to determine
which base model's output is most important in the final prediction.
To sum up the topic, ensemble methods often provide high performance but are slower.
Also, you often lose interpretability as a result of combining different models.
Now let's move to one of the most important topics in machine learning, neural networks.
You try to approximate a function that maps an input to an output.
You are the architect of the function and you can think of a function of any complexity.
The function can be as simple as logistic regression.
Remember this diagram.
This is mathematically equivalent to logistic regression.
You have the inputs, which are called the input layer.
You have the weights.
And you have the output layer, which is the linear combination of the input and the weights.
Sigmoid is applied at the end,
which maps the continuous domain to probabilities ranging from 0 to 1.
Remove the sigmoid and you will get linear regression.
If you have 5 weights, you have 5 parameters to learn.
In contrast to logistic regression, neural networks can also have hidden layers.
These are intermediate layers between the input and output layers
that control the model's complexity.
Such architectures are called fully connected neural networks,
as each node is connected to all nodes in the next layer.
These are the simplest neural network architectures.
However, if we don't include activation functions in a neural network,
each layer would just perform a simple linear transformation, weighted sum of inputs.
This means no matter how many layers we add,
the network would still only be able to model linear relationships,
which limits its ability to solve complex, real-world problems.
Activation functions introduce non-linearity,
allowing the network to learn more complex patterns.
Neural networks learn through a process called backpropagation,
where the model adjusts its parameters, weights and biases, by minimizing a loss function.
If you have a background in calculus and optimization,
backpropagation is essentially an application of gradient descent
that makes extensive use of the chain rule to calculate gradients
through the layers of the network, especially in deeper neural networks.
In theory, you can have as many hidden layers and nodes in each layer as you want.
The problem with adding many hidden layers is that it often quickly fills the memory of the computer.
Additionally, complex models are more prone to overfitting.
For example, different sources mention that GPT-3 has 175 billion parameters.
To use the model, you need 350 gigabytes of GPU memory,
and for training, you will need much more.
This means after the input is fed into the model,
170 billion numbers interact with it to produce an output.
However, don't think of it as just simple, fully connected layers,
but the use of lots of different architectures,
building blocks and training strategies that were developed in recent years.
This includes different activation functions, optimizers,
architectures like transformers, attention mechanisms, and more.
The research in neural networks is very rapid,
and they are the base of almost every complex AI model we interact nowadays.
There is still a lot to talk about, follow us to learn more.
For unsupervised learning, we will cover two topics,
clustering and principal component analysis.
For all the algorithms we discussed before,
we predict a target variable using some input variables.
In unsupervised learning, we don't have a target variable.
The goal is to uncover hidden patterns or groupings in the data.
For example, you have customer data,
and you want to group them into segments to create targeted marketing strategies,
but you don't know in advance how to divide them.
This is where clustering algorithms come in.
K-Means is the most popular clustering algorithm.
Firstly, you need to think about how many clusters or groups you want to find in your data.
Let's assign the letter K to it.
Let's say the number is 3.
Randomly choose K data points to be the centroids of the clusters.
Now you run a loop.
For each data point, calculate its distance to each centroid.
Assign the point to the cluster with the closest centroid.
Recalculate the centroid of each cluster by averaging the data points in the cluster.
Reassign points to the nearest centroid based on the updated positions.
Recalculate the centroids again.
Repeat the procedure until all data points are assigned.
The algorithm stops when the centroids no longer move significantly,
or the cluster assignments no longer change.
While the algorithm is quite intuitive, it has several drawbacks.
You need to predefine the number of clusters, K, which might not always be clear.
Choosing the wrong K can lead to poor clustering.
The result depends on the initial placement of centroids,
which can lead to poor clustering and different results if centroids are poorly initialized.
K-means assumes that clusters are circular and evenly sized.
It relies on distance calculations, so for larger datasets, it may be too slow.
Despite its drawbacks, K-means is foundational
and has inspired many improvements to address its limitations.
This is still the fundamental algorithm that everyone should know.
The last algorithm that we will talk about is Principal Component Analysis.
It is a dimensionality reduction technique that simplifies complex datasets
by reducing the number of features while preserving as much important information as possible.
PCA transforms the data into a new set of variables called principal components,
which are uncorrelated, so there is no overlap in information,
are ranked by importance where the first component captures the most information,
the second the next most, and so on.
PCA creates new features, principal components,
by forming linear combinations of the original features.
To know exactly how, you need a solid understanding of linear algebra,
specifically eigenvalues and eigenvectors.
The mathematics behind PCA can be overwhelming and long, so we won't delve into it here.
Just remember that PCA can be used to reduce the dimensions of your data,
but since it creates new features by performing linear transformations,
it may be more productive when the main patterns in your data are captured by linear relationships,
which you do not always know.
Thanks for watching until the end!
Before you go, just a quick note.
Machine learning isn't always easy, and it takes time to understand it.
If it felt overwhelming, don't worry.
Stay with us, as we'll be releasing in-depth explanations of each algorithm soon.