AI & ML in Finance - Lecture - 11 - Multiple Linear Regression - Further topics

Smart Investing
3 followers
5 Video Views·Mar 5, 2025
11th lecture in the online class "Artificial Intelligence & Machine Learning in Finance" held at Leipzig University, Faculty of Economics.
Hello, everyone, and welcome back to our class in Artificial Intelligence and Machine Learning
in Finance. Now, in this video, I want to talk a little bit about some additional details of the
multiple linear regression model. In the last video, we've already talked a little bit about
how to assess the model fit. We've seen that in contrast to the regular R-squared, we should use
the adjusted R-squared, because actually, when including more and more predictors,
when including more variables in our regression model, the R-squared can actually only go up.
It can only increase. It will never decrease. So that at some point, we'll include unnecessary
predictors, predictors that are irrelevant. And actually, this will reduce the model fit
in the sense that we get overfitting. We have a better fit on paper, but actually,
we're including unnecessary complexity in our model. So that's why in the adjusted R-squared,
we take the R-squared, and we correct the R-squared by the number of variables we've included.
So actually, adding noise variables to an already well-fitted model will decrease the RSS only by
a little bit. So the adjusted R-squared punishes the model for every included unnecessary predictor,
every noise variable that is included in our model. So to see this, okay, so in this slide,
we can see that for the credit data, actually, the adjusted R-squared increases to some point,
and including more and more predictors doesn't really change the adjusted R-squared. So actually
here, for actually, I think this is seven. Yes, this is seven. For seven predictors, we get
an optimal point where the adjusted R-squared doesn't change too much. Actually, we could
have already stopped here, maybe with four or even five predictors. And including additional predictors
doesn't help us in explaining the variation in our response variable. So this is
important now in multiple linear regressions, use the adjusted R-squared rather than the
R-squared. Okay. Now, after having fitted our model, and after having estimated the
coefficients beta 1, beta 0, beta 2, etc., we want to predict values. We want to forecast.
Now, how to do this? Very simple. We take equation 15, which said that we can take the
coefficient estimates, we can enter the values for x1, x2, x3, and so on. And then we get a
prediction, a forecasted value for y hat. And this is what we will do. You take the estimated
coefficients, beta hat 0, beta hat 1, and so on. And you include the values for your predictors.
Can be actual data, but you can also make sort of a simulation. And then if you compute equation 15,
then you get your estimate, your predicted value, your forecast for the response variable. Now,
remember that, obviously, the forecast will include a bias. And this bias will stem from
three sources. First of all, with imprecise estimates of beta 0, beta 1, beta 2, etc., so
even with using OLS, even with using ordinary least squares, we will have some imprecision
in the estimation of our coefficients because we don't have infinite data. We only have a sample
size of, let's say, 1,000, 5,000, even 100,000 observations. This means that there is still some
estimation error that will include or induce a bias in our forecast y hat. Second of all,
the assumption underlying all of this is that we are using a linear regression function instead of
the true function f. Now, in the population, there seems to be, or no, there doesn't seem to be,
but there is a relation between x and y, between our predictors or the predictor variable and our
response variable y. And this relation is the function f. In our modeling approach, we have
assumed that f is a linear function, that we can use a linear regression model. Now, this induces
model bias. Might be that f is actually nonlinear. If that is the case, obviously, we'll have
a bias that stems from the fact that we've chosen the wrong model. And, last but not least,
our third assumption here is that we don't have a clear relation in the sense that
y is with 100% a clear function, linear or nonlinear of f, but our assumption is that
y, the response variable, is a function in our predictors, and we have still some random noise.
That's the error term. We have made some assumptions on the error term, for example, that
it has an expectation of zero, it has constant variance, but we still have the error term. So,
even if our model is correct, if it's a linear function, even if we've estimated the coefficients
perfectly, there is still some irreducible error that stems from the fact that some noise will
always be in the data, and that's the error term. So, never expect your predicted values
to be 100% perfect, because we have three sources of errors, three sources of biases
in our predicted values. Okay. Now, how can we extend the multiple linear regression model?
The first extension is qualitative predictors, our qualitative predictors. Now, in the examples
before, we've already seen some of them, but most of the time we have metric variables,
quantitative variables. For example, it could be income, it could be a price for a stock, etc.
But some variables are actually qualitative. For example, age, gender in our credit card data. So,
how should we deal with qualitative predictors? If the predictor is a qualitative variable,
it's pretty simple. We can use a dummy variable. For example, if we want to code gender
in a very simple way, we can also, actually we can also include more gender types, but if we
only use two, female, male, it's xi equal to one if the i-th person is female, and zero other
y's. And then we get a dummy variable for a predictor with only two levels, and we can include
it in our regression just as any quantitative model, variable, sorry. If we have k levels,
we can actually code this in the following way. We only use k minus one dummy variables. For example,
if we want to code, let's say, age in a very simple fashion, we can say, okay,
okay, we can say dummy variable x one is one for age smaller than let's say 20 years.
And zero otherwise, x one, x two is one for age larger than 20 years, and smaller than 40 years.
And then zero otherwise. And if for example, we only have three levels, then we need two dummy
variables. So if both of these are zero, it's clear that actually, in this case, the person is
older than 40 years. So we have x one for the young people, x two for the people between 20 and
40. And if both are zero, it's clear that the person is older than 60 years. And we can do this
on and on, for example, x three, for example, one for age, let's say 40.
Age and between 60. And then it's clear if all three dummy variables are zero,
it's a person that is older than 60 years. So this is the way how to code qualitative
predictors. But you need to be careful when interpreting the estimated coefficients. Now,
if you have an estimated beta coefficient for let's say, a metric variable, it's clear that
you can interpret these coefficients as a, for example, a 1% change in the predictor leads to
a beta percent change in the response variable. Now here, it's clear that you have to interpret
the coefficient as a switch from zero to one. So what happens if you no longer have a male
observation, but a female observation, then this is the interpretation of the coefficient. Okay.
Now this is a qualitative predictor. How can we extend the multiple linear regression model
even further by including so called interaction terms, for example, imagine predictor x i
increases or decreases the effect of a different predictor x two on our response. To measure this
synergy effect as it's called in marketing, we can use a so called interaction term. So we
set up our regression model response equals intercept plus slope one times x one plus slope
two times x two plus a coefficient that's the interaction term beta three times the product
of x one and x two. This is how you can do it with an interaction term. Again, be careful when
interpreting the coefficients beta one, beta two and beta three. Beta three is the interaction term,
it means that it measures the effect of x one on y if x two changes. The coefficients beta one and
beta two now also measures something different. Actually, it's now even more important when you're
using qualitative predictors, beta one measures the effect of x one, when beta two is actually x two is zero.
So this is the isolated effect of x one, if x two is zero, and beta two measures the effect of x two
on y if x one is zero. So this is a little bit tricky. And also to correctly measure the interaction
term must include x one and x two in the regression as well. So this equation here is correct in the
sense that we've included x one, x two, and their product. This is particularly interesting when you
have an interaction between a qualitative and a quantitative predictor. For example, imagine
x one is income, and x two is gender, male or female. In this case, the interaction term beta
three would measure the effect of gender on income on say health if the response is health.
So how much does gender affect the effect of income on health?
Quite interesting. And beta one and beta two would be the isolated effects, for example, for a male
and female. That's quite interesting when you combine a qualitative and a quantitative predictor.
What else to do? Now we've seen we are talking about a linear model. What to do if, for example,
we see that the effect of x on y is probably a nonlinear one? Well, you can use a polynomial
regression. Let's say predictor x one influences y nonlinearly, we can use a polynomial regression.
That is, we include x one and the squared values of x one, so x one squared. Now see that including
nonlinear terms such as x one squared does not change the fact that this is still a linear model.
It's still linear in the sense that if we have the intercept plus x one plus x two, we could have
also written x one as let's say x two or x three. We could have redefined our variable. We don't need
to know that this is actually a nonlinear term of x one. So it's still a linear model. We can use
OLS. It doesn't change anything, but it can lead to a better fit. And actually using higher polynomials
can lead to such a good fit that again, as was the case with unnecessary noise variables, we get the
problem of overfitting. So this is a problem, but otherwise if we suspect a nonlinear influence,
we can try polynomial regression. Let's look at this in load. First of all, load the library ISLR.
And the linear model is quite simple. We use the car, the auto data. So we are trying to explain
mileage per gallon. So how many miles you can get out of one gallon of gasoline. And we explain
this with horsepower plus the squared value of horsepower. And see that, let me highlight this
for you. Where's my mark? This is strange. Yeah, here's the one. You need to use the function I.
You cannot simply include horsepower hat two, because actually the hat is a different function
in R. So you need to use this function I here. Not very nice highlighting, by the way.
And you can see that this is a model Y equals X plus X squared. And this is the call. These are
the results. You can see the residuals here. And then these are the coefficient estimates.
The coefficient estimates here clearly show that the squared variable, horsepower,
the polynomial term here, is also highly statistically significant in our regression.
So actually, if you were only to include horsepower and then estimate a second model
with horsepower and its squared values as a second predictor, you would also see that the
multiple and adjusted R squared increase considerably. So yes, it seems as if,
sorry, it seems as if horsepower and a polynomial term are needed to better explain
the mileage per gallon response variable. Okay. Now, in this plot, you can see three
different models. You can see the linear OLS fit and the linear model. You can see a polynomial
regression using a degree, second degree polynomial, and also a polynomial term of degree
five. Now, as I said, it is often the case that there might be a nonlinear effect between predictor
and response variable. However, including more and more polynomial terms in this regression
will ultimately lead to overfitting. And this can be seen with the green line. Actually,
you can see that such a line, like the blue line, is fully sufficient to explain the data.
You don't need something that is wiggly like the green one. So actually, including horsepower
squared is sufficient to explain the data. You don't need a polynomial term of degree five.
So overfitting is the case here with the green line, but also the linear model underfits the
data, which is quite clear. So it's okay, but a polynomial regression is much, much better
suited and gets a better fit. Now, what else can happen in multiple linear regressions? Actually,
there are a couple of problems that I only want to shortly comment on. The first one is
nonlinearity of the response predictor relationship. So we have a nonlinear relation
between x and y. We've seen this. One can use residual plots to identify this problem,
and then, for example, use a polynomial regression or more sophisticated models.
The second problem is a likely correlation of error terms. Now, the idea and the assumption
in the OLS model is that the error terms are uncorrelated. If they are correlated,
we have a problem. Our model assumptions are no longer true, and we can only use good causal
inference and good experimental design, if we have experimental data, to counter this,
because usually a correlation of error terms stems from the fact that we have omitted variable bias.
We've missed out one important predictor. We have reverse causality. It's not x that is driving y,
but actually y is also driving x. So we have reverse causality, as we call it.
And all of these problems, omitted variable bias, reverse causality, they lead to a correlation of
error terms because the error terms are picking up something that should have been included,
and then we have a problem in our estimation. And we use causal inference to tackle this problem.
We are likely not talking about this in this lecture, but this is a huge problem
in empirical analysis. The third problem is a non-constant variance of error terms.
One assumption is homoscedasticity, meaning that the error terms have a constant variance.
If it is not constant, for example, we have an increasing variance with the more and more
observations. One central assumption of the OLS model is violated, and we can use, for example,
weighted least squares. Then we have outliers and high leverage points, outliers meaning
extremal values of our response variable, and high leverage points being extremal values of
one predictor variable. What we can do is we can try to detect these outliers and high leverage
points, and then, for example, use Windsorization or simply exclude these data points from our
analysis with all problems that are attached to these methods. And last but not least, we have
collinearity, multi-collinearity, meaning that x1 and x2, for example, share a high correlation,
and this means that one of the variables is obsolete. You don't need one x1 and x2.
The information that is needed to explain our response variable is already included in x1,
for example. Then we can exclude x2, we can drop it, or we can orthogonalize this collinear predictor.
And how can we decide whether we have collinearity? We can use the so-called variance inflation factor,
and I'll get to these in just a bit. So let's start with the non-linearity of the response
predictor relationship. We can look at the residual plots. For example, here, if we use
a polynomial fitting, left it's a linear fit, right it's the quadratic fit with a polynomial
regression. And as you can see, the fitted values in the residuals should actually look
rather like this. In this case, we would say, okay, there's no clear trend, it's around zero,
and we don't see that for very small or very large fitted values, we have a certain trend
in the residuals. Now, with a linear fit, we can see the red line, that's the mean fit. You can see,
actually, yes, there is a trend. It starts out very high, goes back to zero, and then starts again.
So this is an indication of a non-linear relation. And if we do the same for the residual plot,
we can see it's almost constant, and this is how it should be. So from the residual plot of the
polynomial regression, we can see that, yes, it seems to be as if the relation is non-linear,
and a quadratic model is much better suited than a linear one. Now, let's go to the second problem
we have, which is the correlation of the error terms. Now, I told you that this usually stems
from biases like omitted variable bias, reverse causality, and we cannot do too much about this
without using more sophisticated models for causal inference. However, how can we decide whether we
have such a problem here? And to see this, this is a plot, these are three plots, where we have
simulated data with different correlations between adjacent points and adjacent residuals. Now,
if we simulate data, and our assumption of uncorrelated error terms is fulfilled,
we have a correlation of zero, then you can see that we have a residual here, the next one can be
here, it can also be down. So there is no clear trend that, for example, this observation leads
to another observation, another residual being here, or here. Now, if we increase the correlation
between the error terms to 50%, or even 90%, you can see that we have trends, and it looks like a
time series, and this is a problem. Now, here you can see, it's highly likely that you have, when
you have a negative residual, the next one will also be negative. And this is due to the fact that
the error terms, by construction in this case, are highly negatively correlated, no, actually
positively correlated. So this is how you can try to decide whether this is a violation of our
assumption, again, use a residual plot. Now, the third one is non-constant variance of error terms.
You can see one example where you have the fitted values and the residuals, and actually, if you only
plot the mean of the residuals, you would say, well, there's no trend. However, the dispersion,
the variance of the error terms increases dramatically with the data. And this is what you
can see from this funnel shape of the residuals, in contrast to the one you see on the right. Now,
in the left picture, in the left plot, we clearly have the problem of heteroscedasticity. The
variance of our residuals, of the error term in the data, increases from left to right.
But one simple remedy for this is to use the log of our response. And actually,
if we do this simple transformation, you can see it almost looks like the variance stays
the same over our four samples. So very simple remedy, might not be as simple as this. But this
is, again, how we can decide whether heteroscedasticity seems to be a problem.
Next, we have outliers. Now, outliers are extreme values of y. For example, this observation here,
number 20, as you can see, this is an outlier. Now, with the outliers, the problem is that
actually, if we have a look at the next plot, actually here, with the red line and the blue
line, those are the linear fits for the data with and without this outlier. And as you can see,
the blue line in which we fitted a linear model to the data, and we've excluded the outlier,
doesn't change too much. So actually, the outlier for this response value doesn't really
change our linear model. It doesn't change the linear regression line that we are fitting.
However, the problem is that including the outlier can lead to other problems. For example,
it will lead to an increase in the original standard error. So the RSE will increase.
And actually, the adjusted and R squared will decrease when we include the outlier. So the model
fit is getting worse. So we could think about excluding this outlier. You can also identify
this outlier by looking at the residuals in the middle plot here. Again, 20 is far away from the
remaining residuals. And if we studentize the residual, that is, we take it and divide it by
its standard deviation, again, still, it's an outlier. So yes, one could think about excluding
this outlier. Same thing for the high leverage point, which is number 41 here, observation 41.
And this is an extremal value for x for a predictor. Again, you can see that, actually,
if you plot x1 and x2 against each other, this is a high leverage point. Now with high leverage
points, it might be actually, you can see this here, that the regression line, it changes.
Actually, in this case, not too dramatically. But yes, it does change. And this is a problem
for high leverage points. That's why it's also called high leverage points.
One single observation can affect the estimation of a regression line, the regression model,
and the coefficients dramatically. So actually, the coefficient estimates are heavily affected
by high leverage points. And again, one could think about excluding this observation. And for
this, we can also estimate the so-called leverage. And you can see observation 41 has a high leverage
in contrast to the outlier observation number 20. Last but not least, collinearity. Collinearity,
meaning that x1 and x2, for example, are strongly correlated with each other. And here,
we are using the credit card data. And this is limit and age. And as you can see, those two
variables are not really strongly correlated with each other. It's different with limit and rating.
As you can see, there's a strong linear relation between the two. And including limit and rating
in a regression model will lead to problems. Why? Because if you remember the estimation of
the coefficients in the multiple linear regression case, you remember that the coefficient estimates
are given by actually the matrix x transpose times x. And then you have to invert x transpose of x
times x. And you will get numerical problems. Now, if two variables are strongly correlated
with each other, you can imagine that in linear algebra, these two variables form two columns
of our matrix x. And if these two are almost linearly dependent,
the information is obsolete of one column, you can leave out one column. If you're not doing
this, in this case, you shouldn't. The problem is that inverting this matrix becomes numerically
unstable. And this is what will happen. So the coefficient estimates are actually strongly
affected by this, because you now get two numerical problems. And this is shown here.
If, for example, you have beta limit and beta age, those two from this plot here, we saw that
those two variables, those two predictors are not collinear. And actually, it's quite easy to find
the OLS optimum, which is here. So these contour plots, the ones for the RSS, so we are actually,
again, if you remember this, we are minimizing the RSS. So actually, you can see that it's
quite simple if you, for example, as in the optimization algorithm, if you start out here,
it's quite simple to find this point, because this is actually the point where the RSS becomes
minimal. Now, this is the same plot for limit and rating. And as you can see, the contour plots
become quite nasty. And it's now much more difficult to find the OLS estimate of these
two coefficients. Why? Because the function, the RSS function that needs to be minimized,
looks rather, should I put it, this is not as nice to optimize as, for example, this left plot.
So it's a numerical problem, collinearity or multicollinearity. What should you do? Well,
you can simply leave out limit or rating. The information is already included in the other
variable. You can decide on this using the so-called variance inflation factor. And then
you can also orthogonalize it. So you regress x1 on x2, and you use the residual. So you're only
using the information in x2, for example, that is not already in x1. That's one possibility as well.
So this is collinearity. And this is all I wanted to talk about in the section on multiple linear
regression. And in the next chapter, we'll look at penalized linear regression models,
especially LASSO and rich regression.
