Lecture 36: Need for Evaluation

IIT Roorkee July 201
61 followers
3 Video Views·Dec 3, 2023
In this lecture video, the evaluation method and its type are discussed.
[Music]
welcome friends once again to the
nptel mooc module on exploring health
survey data
we are at the verge of last week of our
module ah here we have completely
targeted to focus on the direct
application of the recent
you know research in health care
by which you know
there are
you know several awards several you know
policy making
ah
taking place
so the recent example that i can cite is
the
nobel prize you know given
ah to
the on the very particular year also in
two years back to officer ajit banerjee
and and their team
ok
ah so policy evaluation has taken the
center place center stage in the present
day research
and
nonetheless health care has taken
the most important you know dimension
in social science
where you know
there are large number of
you know
research researchers
trying to give certain ah know
trying to take certain attempts
to evaluate different policies in
different levels
that is why we have to title the lecture
on the very first you know lecture of
the last week is on need for evaluation
we know the need for evaluation
is understood just by its name but there
are certain techniques we are going to
introduce you
and how the need is actually defined
on the first introduction
we have
kept the title of this particular page
is evaluation
we are going to cite one ah you know
important definition ah given by u u n
evaluation group united nations there
are number of projects going on and they
are sponsoring you know various program
evaluation
you know
projects and and large number of are are
at present
being evaluated in india in in
developing countries in african
countries as well
though as for the definition this is
defined as an assessment
a
systematic and impartial as possible of
an activity
project program strategy policy theme
sector operational area or institutional
performance
ah in continuation to this definition we
are also adding other you know important
features that is this definition also
focuses on expected and achieved
accomplishment
examining the results chain pro
processes
you know contextual factors and you know
causality in order to understand
achievements or the lack thereof it aims
at determining the relevance impact
effectiveness efficiency and
sustainability of the interventions
so basically there are so many things
first of all it is emphasized about the
activities ok
and systematic and impartial you know
approach
strategy policy
sectors operational you know area
institutional role
it has given importance to examining you
know the results with a chain of you
know
you know
strategies
then their processes contextual factor
causality also to
link with the treatment and the control
group
and accordingly we find the impact
effectiveness and their efficiency or
the sustainability of a particular
project
so ah the evaluations are periodic
objective assessments
of a planned ongoing or completed
project programs or policy
evaluations are used to answer specific
questions you know often related to
design implementation or results
ah further evaluation should influence
policy and operational decisions so you
know ultimate
ah target of this evaluation is to go
for policy and to find out the loopholes
of the policies to get a better
implemented one so that you know the
project is going to be more sustainable
ah here is the presentation of summative
evaluation ah
that can be you know subdivided into
outcome evaluations impact evaluations
cost effectiveness or cost you know
benefit analysis and sect some some
possible of meta analysis then secondary
analysis etcetera to examine the
evaluation
first one is called outcome evaluation
that investigates whether the program or
technical technology caused
demonstration effects
on ah
demonstratable you know effects ah on
specifically you know defined target
outcomes
so the target outcomes are are evaluated
largely ah through the outcome
evaluation approach second one is called
impact evaluation is this is ah is
broader and assesses the overall or net
effects
intended or unintended of the program or
technology as a whole so this also ah
deals with you know the direct
you know outcomes but
ah the net effects are mostly emphasized
ah the third one is is more important
that is called cost effectiveness or the
cost benefit analysis addresses the
questions of efficiency by on ah
standardizing outcomes in terms of their
cost ah or the values incurred for
running the project and it is for till
the final outcome
so it is the poly the evaluation methods
are also simultaneously discussed with
their budget ok the budget is very less
usually those evaluation methods are not
going to be very effective
ok it is too less then it is not
effective if it is within a range
generally you know it is considered to
be good
ah second analysis examines you know
existing data to address new questions
or use methods not previously employed
so the secondary analysis actually gives
the gaps in the existing study and
accordingly new know variables or
methods can be adopted
meta analysis as we all know that the
meta analysis is the analysis of
analysis ah studies of studies ok this
gives multiple studies to come to a
conclusion and to give a judgment about
ah certain you know value in question ah
um
you know if we give get you know large
number of studies you know if we review
all of them we can get certain direction
ah usually these days we we apply some
you know meta-analysis approaches like
prisma that is through literature review
but some conclusion can be also derived
for certain policies
so formative evaluation includes several
evaluation types
ah that needs you know that that is one
needs assessment then evaluate ah you
know
evaluability assessment implementation
evaluation then process evaluation
ah needs assessment determines who needs
the program
how great the need is and what might you
know work to meet the need
ah evalu you know if evaluability
assessment determines whether an
evaluation is feasible and how
stakeholders can ah know can help save
its you know usefulness
structure conceptualizations help
stakeholders define the program or
technology
or the target population and the
possible outcomes
implementation evaluation you usually
deals with the monitoring aspect of it
monitors the you know fidelity of the
program or technology delivery
ah that that is precisely called
implementation evaluation
then
process evaluation basically
investigates the process of delivering
the programs to the large masses
especially the technology how it is
reaching to the larger population all
its you know steps processes are
actually evaluated
so these are all important needs
basically assessment ah
you know
then evaluation or evaluability
assessment
ok whether that is you know feasible or
not then you know structure
conceptualization based approach is
followed then implementation how it is
implemented that evolution can be also
done then how the process are rightly
followed or not can also be checked
here is the schematic you know details
ah that is going to guide you
how ah and and which way it follows
evaluation types then when to use then
what it shows why it is useful these all
details are explained in this particular
chart
ah like you know formative evaluation
suppose we just take
as an example ah within that we
we we can you know think of about
evaluability assessment or needs
assessment
ah
that that has you know ah when to use
that we use during the development of a
new program
ah then whether the proposed program
elements are likely to be needed
understood or accepted by the population
you wanted to reach
and why it is useful because it allows
for modifications to be made to the plan
before full implementation begins
similarly the availability assessment
and needs assessment has its
the same kind of you know ah information
like when to use what it shows and why
it is useful
ok
ah like if you go for an example about
process evaluation or program monitoring
these features these points are
important
ah like when to use in these two cases
for process and program monitoring as
soon as program implementation begins ah
the process evaluation is is going to be
you know active ah our program
monitoring is also active ah so this
also gives information about during
operation of an exiting program existing
program
what it shows it shows that how well the
program is working the extent to which
the program is being implemented is
designed whether the program is
accessible and
an acceptable to his target population
or not
why it is so useful because you know it
it checks on the process ok it is
guarantees the you know right
delivery of the the the program this
provides an early warning ah for any pro
problems that may occur
this also allows programs to monitor how
well their program plans and activities
are going to be
applied effectively
outcome evaluation or or objective based
evaluation ah where ah when when this is
useful this is useful after the program
has made con contact with at least one
person or group in the
target population so basically once
persons have been contacted for ah for
they are basically for the treatment
group
how these treatment groups are actually
you know
now evaluated ok ah so they they might
have you know understood about the
objectives whether the objectives have
been raised to the right person or not
why this is useful this tales whether
the program is
being effective in meeting
its right objective that is convinced to
the persons who are the stakeholders ah
can also be evaluated
there are various forms of economic
evaluation that is very important
economic evaluation you might have heard
that is called cost binary fat analysis
or cost analysis cost effectiveness
evaluation
cost utility evaluation etcetera
last one is is about impact evaluation
impact evaluation is is is ah is is
applied when
the operation of an existing program at
at appropriate intervals are taken if
intervals are taken and then and two are
compared then in that case it is called
impact evaluation
or generally at the end of the program
ok
and why this is effective because this
is useful because it provides evidence
for use in policy and funding region
once this is done successfully and
empire evaluation guarantees the right
uh
ah
model was followed then that can
directly be you know applied for no
policy or or funding decisions
the degree to which the program meets
its ultimate goal uh on an overall rate
of you know maybe you know
std transaction transmission how much
has the program x decrease the morbidity
of an std beyond the study
ah sexually transmitted disease etcetera
you know there are may be you know many
other such examples where ah
this really ah going to be effective or
not so that can be evaluated at the end
of the program
ah so evaluation can answer three types
of questions one ah
one is called descriptive questions then
normative questions then cause and
effect ah related questions ah
emphasized by imas henrys two thousand
nine paper ah descriptive questions
asked about what is taking place all
about normative is which what you know
ah compares what is taking
place to what ah should be taking place
ok that is basically comparing what
should be what or to be done ok as
compared to what is being what is you
know taking place
cause and effect questions progression
or attribution they ask about what
difference the intervention makes to the
particular outcome
now we are explaining impact evaluation
which you already ah explained ah in the
previous slides
evolution seek to assess the changes
brought by a project or program rather
than focusing on activities and and
outputs only
the aim to assess predicted you know
change ah although many cover unexpected
or negative change as well
they attempt to assess the contribution
of a you know development intervention
to any identified change they usually
follow a rigorous and accepted research
methodology to you know identify their
change and contribution
impact evaluations are very particular
ah kind of evaluation which is explain
ah a specific you know cause and effect
ah questions
what is the impact
like
or or causal effect of an impact of a
program of an outcome of interest this
basic questions incorporates an
important causal dimension so every time
we are mentioning about causal dimension
this your nobel prize is is on causal
inferences you can just have a check
ah how
the persons have received their
explanation are given just put this
search in google causal influences nobel
prize you will get the information about
it
the focus is on the impact evaluation
one is on is only on the impact that is
how the outcome has been changed
ah due to the treatment
the changes directly attributable to a
program program modalities design
innovation etcetera
impact evaluations can be carried out on
any kind of work including service
delivery capacity development
mobilization and policy influencing and
any sector of work from health
ah through
to governance
and impact evaluation can be applied at
any level of work from small projects
through to complex programs covering
multiple organizations sectors and
geographic locations so in in many
direction that can be
made
impact evaluation you know questions ah
some of the questions are important like
what is the causal effect of
scholarships on school attendance ah and
academic achievement so directly we can
ask about the causal effects of school
you know scholarship or scholarship on
their achieve performance
ah does health insurance reduce the out
of pocket expenditure of the household
to what extent where the nutritional
outcomes of children affected by portion
of vision which has been you know
been regulated at the present government
ah
initiated by the present government
what is the impact of
contracting ah out primary care to pro
you know private providers and access to
health care
here is the detail ah when information
about how world bank has published ah
some guidelines related to impact
evaluation
how it is appropriate they talked about
in uh meaning in different indicators
like how innovativity is how
replicability is how strategically
relevant the you know program it is you
know
whether it is tested or untested
how influential this this could be made
ok ah is the intervention inpu
influential will results ah be used to
inform key policy decisions or not ok
etcetera are are mentioned by ah world
bank you know publish paper
ok
ah coming to the pro
another difference called prospective
versus retrospective you know impact
evaluation
prospective evaluations are developed at
the same time as the program is being
designed and are built into program
implementation that is what is called
prospective
ah the example is that the baseline data
are collected before the program is
implemented for both the treatment as
well as compare comparison group that is
what those are called baseline survey
baseline surveys are made
ah to have certain
you know programs to be implemented
later
retrospective evaluations rcss
ah program impact after the program has
been implemented
ok we can retrospectively examine
whether it had actually impacted the
group or not
this is looking for the treatment and
the comparison groups exposed ok
ah
others are like you know effect efficacy
studies versus you know effectiveness
studies efficacy studies assess ah
asses whether a program can work under
ideal conditions
efficacy study explore proof of concept
often to test the validity of a new
program or a specific theory of change
if the program does not gen generate
anticipated impacts under these
carefully managed conditions it is
unlikely to work
if rolled out under normal circumstances
whereas in case of you know
effectiveness studies assays ah whether
a program does not work under normal
conditions
whether effectiveness evaluations are
properly you know designed and
implemented and the results may be
generalizable to intended you know
beneficiaries beyond the evaluation ah
sample
evaluation design there are key elements
in the evaluation design
ah those are like deciding whether to
proceed with the evaluation or not
identifying you know key evaluation
questions ah the evaluation design
should be embedded in the program theory
ah the comparison group must serve as
the basis for a credible counter factual
we are going to explain just now
ah addressing issues of selection bias
and contagion effects ah findings should
be you know ah trying angulated the
evaluation must be well contextualized
to have a better you know implications
ah theory of change is is well referred
in the context of evaluation ah so let
us understand that a theory of change
explains how activities are understood
to produce a series of results
so since the change is is to be
explained so chain of change has to be
explained ah the this explains ah this
contributes to achieving ah the intended
or object impacts
it is a description of how an
intervention is supposed to deliver the
desired result it is a key underpinning
of a knee impact evaluation given the
cause and effect focus of the research
they explore the conditions and
assumptions needed for the chains to
take place
they also make explicit
make explicit the causal
logic
behind the program and map the program
interventions along logical causal
pathways
when preparing for an impact evaluation
the theory of change should be reviewed
and revised as per the necessity
ah for example the existing theory of
chains may have several gaps or or on
realistic assumptions that should have
been revised on time
the impact evaluation can be improved by
you know using the theory of change to
identify the relevant variable the
particular variable that should be
included or not included
ah identifying the immediate outcome
what could be the immediate you know
success ah from the makers ah in
situation where the impact of interest
will occur after the evaluation time
frame ok i some you know basically some
testing is required immediate outcome
testing is required
ah this identifies aspects of
implementation that should be examined
to see if the failure of to achieve
internet impacts is due to a failure
to implement the intervention
successfully
this identifies potentially relevant
contextual factors that should be
included in data collection
they ah then
then the theory of change also includes
you know ah
some guides about data analysis ah this
provides framework for reporting
findings as well
using the theory based you know um
approach usually you know avoids black
works impact evaluation this means that
you know
those which give a finding on impact
but no indication is to why the
intervention is or is not doing so it is
avoiding the black box impacts as well
so developing a result chain is is
required in case of an evaluation method
a results chain is one way of defecting
the theory of change as we already said
other approaches include theoretical
model logic models theoretical framework
and outcome variables
this establishes the causal you know
logic
ah
from the initiation of the program
beginning with resourceable ah
availability or resources availability
to the end looking at long term goals
etcetera the basic results chain will
actually map the following elements
those are inputs activities and outputs
like inputs where resources at the
disposal of the project including stop
and budget etcetera
should be well you know examine
activities such as actions taken or work
performed to convert inputs into outputs
outputs are the tangible goods and
services that the project activities
produce
these are directly under the control of
the implementing agency
similarly besides you know three ah
important direction inputs activities
outputs
the
basic you know result chain also map
other tool those are outcomes ah not
just output outcomes is the particular
you know target beneficiaries ok the
beneficiaries which have been you know
targeted should have been you know well
dealt with the outputs of the of the
project
ah usually this is not ah you know
directly under the control of the
implementing agency
final outcomes ah the final results
achieved indicating whether project
goals are met or not
the implement of a result chain this
follows inputs to then inputs to you
know activities then to
then to then to outputs ah ok activities
to output then outputs to you know
outputs to your outcomes then output to
final outcomes ok
ah ok then ah inputs we have already
discussed it it includes budgets also
stopping etcetera activity series of
activities taken in in in between
ah outputs basically whether the
products to be you know delivered
ah ok
on time under the control of the
implementing agency this here the
implementing agency control is there
in case of outcome implementing agencies
is not having any control
ok
so these first three sites that is
inputs activities and ah and and you
know outputs
ah is is usually referred as supply side
ah implementation other two ah since you
know we are saying the beneficiaries and
the final you know reachability of the
project we are linking with the demand
side as well as supply side so therefore
these are called the real results like
you know in technical institutions they
do invent many things but many of the
inventions are not catering to the
society
ok so in the program implementation we
do appreciate the stand of the technical
institutions ah but but the end of the
result that those may not be sustainable
because it is not going to deal with the
final beneficiaries
ah so example like health insurance
subsidy program the ultimate objective
of
health insurance subsidy program is
improving the health of the
country's population the government is
concerned that poor rural households are
unable to afford the cost of basic you
know ah health care with detrimental
consequences
for their health
to address this issue hisp ah subsidizes
health insurance for rural ah households
ah that has covered their cost ah
relative they are relative to their
other you know health care and primary
ah health care and and medicinal cost
the central objective of hisp is to
reduce the cost of health care for the
poor families and to reach to the masses
and improving their health outcomes the
hypothesis related to the hisp reform
assume the following that is the
household will enroll in the program
once it is offered ah that enrollment in
the program will lower households out of
pocket health care expenses
ah the key evaluation ah question is
this that you know what is the impact of
hisp on poor households out of pocket
health expenditures
the hisp results chain also follows you
know inputs to inputs to these
activities then you know outputs then
outcomes then final outcomes ok so
accordingly you can think of their you
know supply side indicator and the
demand and demand side indicator as well
along with the supply side indicators
the causal inferences are very important
in case of evaluation method many policy
questions
involve cause and effect relationship
for example does teacher training
improve students test course
the do conditional cash transfer
programs cause
better health outcomes in in children
do vocational training programs increase
trainees out you know in in incomes
etcetera so these are some of the
questions you know usually asked you
know we have referred here for your easy
understanding
all the cause and effect questions are
in common answering them accurately can
be actually you know very challenging
and should
making it scientific is even more
challenging
in the context of cash transfer program
or direct cash benefit ok ah
or direct transfers benefit ah you know
programs ah for example simply observing
their children better health outcomes
after you know getting cash transfer is
not sufficient to establish you know
causality
ah children health outcomes might have
improved even if they did not get the
cash because of other factors such as
you know several vaccination program
usually even if gas is not there by the
other
programs might have benefited so policy
evaluation should be very particular in
establishing what exactly they are going
to observe and how they can utilize
other effects
to establish causality between a program
and an outcome we use impact evaluation
methods to rule out the possibility that
any factors other than the particular
program we are interested in evaluating
really explaining the you know observed
impact
the answer to the basic impact
evaluation question is that what is the
impact or causal you know effect of a
program that is p value or sorry or the
p program success of or or program you
know effect of ah
or the causal effect of a program
ah that is on an
outcome of interest ah y
now p here we are taking a program and
the outcome we are taking with the you
know notation is y is given as
delta that is change in the outcome to
that of the the the change in the p
ok ah why what is it it
outcome with the changed
with the one that is the treatment group
as compared to outcome at zero in the
control group when there is no program p
is equal to zero no program ok ah and
here we are actually subtracting the
impact of change in the program and its
outcome value because of the change as
compared to no change in the
program so that means the control value
as compared to the you know treatment
group if there are any changes that is
captured and that is all about called
you know evaluation
it states that the causal impact of a
program of on an outcome y is the
difference
between the outcome y with the program
in order in other words p when p equal
to one
the outcome ah without the program
outcome that is equal to y without the
program ah when
when
when p is equal to zero
now most important aspect is called
counter factuals counterfactuals are are
are ah you know we discussed in in
program evaluation largely the
counterfactual is is what would have
happened
ah what the outcome would have been for
a program participant in the absence of
the program ok in the absence of the
program if program is not there then
what would have happened to these in a
particular ah group ah is basically
called counter factual you know if we
just go by
no availability of the program then who
are those people ok who are those non
beneficiaries and what has happened with
them without the program they are
actually actually called counterfactuals
in the basic impact evaluation formula
the term y that is p is equal to zero
represents the counterfactual so this is
what is called not contr though we are
saying control but actually called
counterfactuals
ok
ah so ah
since we cannot directly observe the
counterfactual we must estimate the
counterfactual
solving the counter factual
problem would be possible if the
evaluator could find a perfect clone for
a ah you know program participant if
program parchment and a replicated one
without the program parchment is
represented then that two groups can be
compared so comparing with the treatment
group with the counterfactuals are
actually going to give us better results
in terms of evaluation
the key to estimating the
counterfactuals for program participants
is to move from the individuals or unit
level to the group level
the group that participate in the
program is known as known as treatment
group and its outcome is y is eq y given
p is equal to one after it has
participated in the program
just statistically i know identical
comparison group is the group that
remains unaffected by the program and
allows us to estimate the counterfactual
outcome ah with you know p is equal to
zero
the challenge of an impact evaluation is
to identify a treatment group and a
comparison group that are statistically
identical
on average so this is what on average
that should be you know statistically
you know ah
identified and statically identical
so these two groups ah
in the absence of a particular program a
valid comparison group has the same
characteristics on average as the
treatment group in the absence of the
particular program
ah this remains unaffected by the
program and would react to the program
in the same way as the treatment group
if given the program
when the comparison group does not
actually estimate the true contrafactual
then the estimated impact of the program
would be actually you know invalid
in statistical term ah it will be called
biased or giving bias results
two counterfeit you know estimates of
the counterfactual that is called before
and after comparison enrolled and non
enrolled ah comparisons
ah before and after comparisons are also
known as pre post
or you know reflexive comparisons
ah this compares the outcomes of the
same group before and after
participating in the program
enrolled and non enrolled comparison are
like you know ah like you know ah
these are also known known as ah self
selected you know comparisons
this compares the outcomes of a group
that chooses to participate in a program
with those of a group that chooses not
to participate so these these are the
two comparison ah there are some
methodologies in impact evaluation
mostly you know applied those are called
experimental design and called non
experimental design ah experimental
design we we
we use you know randomized control trial
and in non experimental design there are
largely three called propensity score
matching regression discontinuity
approach and difference in difference
approach
ok so these ah these four aspects we are
going to
discuss in our respective in lectures in
this week ok
and we will also try to cite examples
for your reference and i am i am quite
sure that you know this is going to be
useful
and will be stimulating your research
further with this
basic guidance about evaluation i i am
quite hopeful that you have been
recharged
to carry forward you know evaluation
based methods
any sort of doubts you have ah do not
hesitate and put these questions on the
screen of nptel you know query box will
be happy to address it thank you
[Music]
you
