noc18-mg22 Lec 58-Linear Regression (Part-1)

IIT Kanpur July 201
20 followers
1 Video View·Dec 3, 2023
[Music]
[Music]
hi welcome back to the lecture series on
microcon omics today we are going to
start our discussion on an empirical
tool called linear regression analysis
linear regression analysis is one of the
tools in the vast field of econometric
methods now what does econometrics do so
micro economic theory or for that matter
any economic theory actually states or
make some hypotheses to explain the
real-life economic phenomenon but these
are all qualitative in nature so for
that matter take the example of the law
of demand that we have seen in the
theory of consumer behavior what does it
say it says that if price of a commodity
increases then quantity demanded of that
particular commodity will decrease given
other things remaining the same so this
is a theoretical result but if I now ask
you that suppose we have a commodity
whose price has gone down by 1 rupees or
one dollar can you tell me by how much
the quantity demand is going to change
the answer is No
from this theoretical result you cannot
quantify the magnitude of the change in
the quantity demand due to a price in
econometrician we'll collect data and
try to fit some kind of statistical
relationship between price and quantity
so that you will be able to give this
answer you will be able to deliver the
answer to this very question now one may
ask that we have also gone through the
concepts like price elasticity right so
he price changes by one percent by what
percent this quantity demand will change
that elasticity can also be computed
only when a functional form a specific
functional form for demand function is
known to you so
to compute the elasticity also we need
to quantify the demand function and
simple linear regression analysis will
help us to establish some kind of
statistical Association ship between
statistical association between the
price and quantity in this case and in
general between any two or more economic
variables so actually the economists
will start with some kind of economic
theory which we will talk about some
true relationship between some variables
for that matter we can assume that we
are interested only on two variables so
there will be some kind of true
relationship which we prevail at the
population level between these two
variables
now there may be a true relationship
between two variables but in the real
life when we observe real-life data then
basically there could be some error or
noise because of many many issues one
factor could be omitted variables so as
we know that economy is a very complex
thing and one variable gets influenced
by not only one particular variable but
many other variables so in a simple
theoretical model when we incorporate
one or two or three variables in a
functional form in a mathematical
functional form we tend to rule out many
other factors which are probably
unforeseen to an economist so that is
why when we observe real-life data on
two or more economic variables we
actually do not observe a curve or a
straight line which we are used to draw
in microeconomics rather we actually
observe a scatter so a true relationship
which can be also called
a hypothesis can be represented through
the notion of population regression
function so let us have a definition for
this concept
okay so let us start with the definition
of population regression function so
here we are talking about a theoretical
relationship between two or more
variables the theoretical relationship
can come from micro economic theory in
our case and you know they had basically
describes some kind of relationship
between a dependent variable and one or
more explanatory variables so just if
you remember our previous discussion
that we just had on the law of demand in
that case the dependent variable would
be the quantity deep and deep a a
quantity demanded of a particular
commodity and the explanatory variable
could be the price of that particular
commodity and income level of the
consumer etcetera etcetera so this is
basically conceived at the population
level so this is some kind of
relationship which is kind of perception
about reality okay
now whether this perception about
reality is actually the case or not that
needs to be tested and econometrics does
so so here we are going to study linear
regression analysis and the purpose of
linear regression analysis is to take a
purely theoretical equation which is
basically a population regression
function something like this why I equal
to beta naught plus beta 1 X 1 and this
is basically my population regression
function 1 I okay so here I define y as
my dependent variable
so dependent variable is basically the
variable whose behavior is explained
through a model and this x1 in this case
is basically my independent variable
or explanatory variable and needless to
say that you can add more explanatory
variables if you assume that there are
more number variables which are
affecting the variation in this
dependent variable Y so you can have as
many as you want in terms of the
variables so this is basically the
notion of the population regression
function so after laying out the
fundamental population regression
function let us now go and study the
regression analysis in deeper details so
in our regression analysis we take a
pure theoretical relation and let me now
simplify the analysis by assuming only
one variable one explanatory variable
but needless to say that this can be
generalized to more than one variables
anytime right okay so only one
explanatory variables X and what is I I
is basically 1 to n these are basically
the number of observations
right okay so now let us go back to the
statistical relationship we have already
told you that in theory we can have a
straight line or a simple curve but in
reality there can be many random noises
in the data and for that reason we
introduce an extra term in this
regression equation and this is called
disturbance
term or some people call it noise
sometimes it is also called random error
okay now quickly let us note down the
interpretations for these parameters
beta naught and beta 1 so beta naught is
basically called an intercept parameter
okay and this beta one is called slope
parameter
so these are the relationships defined
at the population level so this is the
true relationship now what are the
interpretations for this let us first
talk about the intercept parameter so
intercept parameter actually gives a
constant variation in the so beta naught
the intercept parameter actually is a
knife regressor or explanatory variable
which tries to explain the variation in
Y but you know this is a constant term
basically so later we will see that this
is nothing but the sample mean of the
dependent variable okay now let us move
on to beta 1 so beta 1 is basically the
slope parameter and that basically comes
with the explanatory variable so if we
throw one or more explanatory variables
in the regression equation these
variables are expected to explain some
variation in the main variable or
dependent variable Y right so what would
be the interpretation for beta 1 it is
simply say is that it simply says that
if there is one unit change in the
explanatory variable or the independent
variable then by how much my dependent
variable Y is going to change
so now let me try to address what does
regression analysis do so as I told you
that the population regression function
actually talks about the true population
relationship or it's coming from some
theory or hypothesis but the problem is
this we really do not know the exact
values of beta naught and beta 1 and
beta K if there are K number of
explanatory variables we need to know
some numbers for these parameters in the
regression equation so how to find some
proxy values for these true population
parameters our regression analysis
actually tries to find out good proxies
for these population parameters beta
naught beta 1 to beta K and there are
many methods to obtain these proxies we
are going to study only one such proxy
and the name of the method is ordinary
least-squares but before we get into the
details of the ordinary least-squares
method let us have a look at two more
definitions so any regression method
actually leads to an estimator and here
we are writing the definition of an
estimator so an estimator is a
statistical
formula
derived
from
okay so now let us going to have another
definition and this time it is for okay
fine no definition so now let me have a
simple diagram to explain the philosophy
behind regression analysis
I hope this simple diagram will help you
to understand this tool better so let me
have this explanatory variable plotted
along the horizontal axis and the
dependent variable or the variable of
focus Y along the horizontal axis so the
population regression function says that
there is some kind of a linear
relationship between these two variables
and if we plot that then we can say that
say for some value of x say x 1 we
observe some value y 1 and so on so
forth so let me just know have two more
data points there can be many so let me
assume that there is some positive
relationship between y and x and this
comes from some kind of theory or
hypothesis so basically the population
regression function is basically
straight line which joins these points
and this is the true relationship
between y and x so basically one can
write y equal to beta naught
the beta 1 X ok but due to the presence
of noise in the economy and of course as
a result in the data we actually observe
lots of points in this why explain if an
economy trishy on collect some data from
reality then basically this diagram is
called a scatter diagram and now the
task at hand is to find out why there is
a deviation from this pink line or the
population regression line so suppose
let me take a particular value of XA x2
then I expect that my population
regression function as it is giving me
value of y in reality I should see that
Y value only for x2 but I am observing
some data in reality here this big large
blue dot so what is causing this
difference so can we explain this
variation in Y so that is the question
with which regression analysis so this
is the problem with which regression
analysis is concerned now as we really
don't know the values for beta naught
and beta 1 then we can take help of
statistical methods to come up with some
kind of sample regression function so we
know we can devise some estimator which
will now be a fitted line amongst these
data points in the scatter plot so we
are basically saying that
we don't know the true relation between
we don't know the true parameter values
but note we can adopt some statistical
method to come up with you know some
estimator so that we can feed some kind
of a straight line through the data now
note that there could be many estimators
and hence there is no unique straight
line so if you draw this brown straight
line then it will represent one
particular combination of beta naught
and beta 1 and if I now draw this
sky-blue straight line this also passes
through the same scatter plot so if I
now plot the sky-blue straight line
which also passes through the data that
I have or the scatter diagram that I
have but it represents a different
combination of beta naught and beta 1 so
these two lines that I have drawn these
are called the sample regression
functions right the task at hand is to
find the best possible sample regression
function which fits the data okay there
are many methods to find proxy values
for beta parameters we are going to now
study the simplest possible method which
is called ordinary least-squares let us
talk about some estimated equation from
the real life data so suppose we are
given some data points like there are n
number of data points in the scatter
plot note that I am going to restrict my
discussion only on one explanatory
variables in the next class we are going
to expand the number of explanatory
variables so whatever I am going to
discuss now is quite general so it will
hold for key number of explanatory
variables as well so let us start with
in number of data
so we observe some values of the
dependent variable like Y 1 y 2 up to Y
n corresponding to the explanatory
variable or the regressor variable
values X 1 X 2 to X n right so basically
we are observing the data as a pair so
basically this is the way you can think
of the data set so observation number 1
number 2 dot dot number n and here is
basically my data and data is basically
X 1 Y 1 a pair of observed pair of
variables values X 2 y 2 and finally xn
and y n so these are basically the
points in the scatter diagram okay so
now an estimated equation using this
data
would look like
right
so here note that I have introduced this
new notation hat so head basically
represents this term estimated okay so
this is basically represented by this
carrot or hat sign okay so this Y I Y
hat I is called the fitted value or the
predicted value
okay now then this beta naught hat and
beta 1 hat are basically called the
estimates of the population parameter
from the data now the question is how to
find these estimates so the first thing
that we would like to do is to write
down another expression for prediction
error
or Reseda well of course we can
understand that that as we do not know
the true values for beta naught and beta
1 given a dataset we can only produce
some proxy numbers and they may be close
to the original or true population
parameter value or they may not so
whatever be the case you know in most
cases you know they are not equal to the
population parameter values the these
estimates I am talking about
hence basically the Y hat I the
predicted or fitted value that we obtain
from our estimated equation or
regression equation will not match the
actual data which is why I so there will
be some difference and then this
difference is called prediction error or
residual of the regression model this is
given by e I and that is basically
defined as Y I minus y hat I so the
actual value minus the fitted value now
this method
wellis minimizes
the sums sum of squared residuals okay
now why you know we need to minimize
this sum squared of total first of all
our target is two why we have to
minimize this sum squared of residuals
first of all as an econometrician or a
statistician I would like to minimize
the error so that is why I have to
minimize but why square term because
note from that diagram that we had which
depicted the population regression
function and the sample regression
equations that you know error could be
of either positive or negative type so
if you have only sum of errors then you
know positive and negative numbers will
cancel out altogether that is why we you
are squaring the error term so that we
deal with only the positive numbers and
we minimize the sum of squared residuals
so basically you see this sum of squared
residuals now let me expand this so we
can now write
okay so basically what we see the sum
squared of residuals is a function of
the unknown entities like beta naught
hat and beta 1 hat so the target is to
find beta naught hat and beta 1 hat in
such a way that the sum of squared
residuals is minimized so we know how to
minimize any function so we have to
basically take partial derivatives and
said the m2 equal to 0 so there will be
two such derivatives
okay so note that this expression leads
to an equation like this here the sum
ranges from I equal to 1 to n just to
just to have a less cluttered expression
I am avoiding the sum range
okay so this is one equation and this
first-order condition leads to another
equation
so ii fast order condition also leads to
an equation which looks like
oops I mistakenly written one here there
is only one explanatory willable so let
me just remove that one suffix super
subscript
okay so these are called the normal
equations and if you solve these two
normal equations then you will get beta
naught hat and beta 1 hat such that as a
pair they minimize the sum of squared
residuals let us now look at a graph
which talks about the decomposition of
the variation in Y once we have
estimated the sample linear regression
function that means that we know our
estimates for the population parameters
beta naught and beta 1 plot Y bar x bar
here in the graph let me assume any
particular value of x say X tilde okay
and for this value this point is
basically why had I not I Y hat equals
beta naught hat plus beta 1 X tilde
right suppose the original data says
that the corresponding Y value the Y
value corresponding to the X tilde value
is this green dot here right so in that
case basically this gap is my residual
right GI which is basically the
difference between the actual value
minus the fitted value Y I right okay
and then for the diagrams company nests
sake let me extend this Y bar so now we
are going to see some important concepts
which emerges from the valus method the
first concept is of total sum of squares
or abbreviated as TSS
so now this could be broken down into
two pieces
okay so the first component is known as
the explained sum of squares
why this is called explained sum of
squares because of this fact that here
the Y hat I this component is actually
given by the sample regression function
so here we see the role of the
explanatory variables because you know
this Y hat I is the fitted value right
so given the values of XS and
relationship between these variables
that we have assumed this is the fitted
value and this is basically some kind of
value which comes from within the model
so the other component is known as the
residual sum of squares
okay fine okay so we will continue with
this discussion on linear regression
analysis in the next lecture
[Music]
[Music]
