Using Evaluation to Develop Evidence: How and Why | July 21, 2022 | Webinar

USDOHAHS
156 followers
May 22, 2024
This is the first of a three-part series focused on the key principles of evaluation in the federal context. In this session, Elizabeth Yano, Ph.D., M.S.P.H. covers: 1) How and when evaluation is most useful; 2) The varying types of evaluation and when each is most appropriate; and 3) Why evaluation is a vital tool for program planning, management, and priority setting at both the program level and agency/office/component level.
--
U.S. Department of Health and Human Services (HHS) | http://www.hhs.gov
http://www.Twitter.com/HHSGov | http://www.Facebook.com/HHS http://www.Instagram.com/HHSGov
http://www.LinkedIn.com/company/us-department-of-health-and-human-services
HHS Privacy Policy: http://www.hhs.gov/Privacy.html
okay the logins seem to be slowing down
a bit and it's a minute or so after 1 so
I want to welcome everyone to our
training session today my name is Susan
Jenkins and I'm the director of the
division of evidence evaluation and data
policy and as's office of Science and
data policy here at HHS I apologize for
not showing my camera but you know 15
minutes ago it decided it wasn't going
to work anymore but I am happy to be
here with you and I want to give a
little background as to why we are doing
this training and what this training is
so this is part of hhs's response to the
foundations for evidence-based
policymaking act and as part of that we
assess departmental capacity in a range
of evidence building areas one of the
areas that Rose to the top was a need
for better understanding of program and
evaluation amongst people most likely to
benefit from program evaluation so this
is not evalu ERS themselves although
you're welcome to join the session of
course but this is for other folks who
may wonder how does evaluation work why
should I care about evaluation so this
first session is part of a three-part
series there's a session today a week
from today and two weeks from today and
during this session we will cover how
and when evaluation is most useful the
varing types of evaluation and when each
is most appropriate and why evaluation
is a vital tool for program planning
management priority setting at both the
program level and at the agency office
component or division level in the next
session we'll focus on ways to engage
with the evaluation process so that you
can get the most out of it and in the
third session we will address
appropriate use of evaluation data to
inform programmatic and policy
decisions so a few housekeeping items we
are recording this session and so it
will be made available able likely
through the asby
website um in a few weeks or so captions
are available for this session but you
need to turn them on if you want to use
them but they are available to you and
if you have questions for the speakers
please submit them in the Q&A tool
throughout the presentation and we have
planned about 10 minutes at the end of
the presentation for answering those
questions and we'll collect those
questions and if they're more than we
answer we'll do a frequently asked
questions and post that at the same time
that we post the slides the recording
and any other materials related to this
session and the series and so our
speaker today we're very excited the
speaker today and for the next two
sessions is Dr Elizabeth Yano
Becky and Dr Yano is an Adjunct
professor at health and of health and
policy management at the UCLA Fielding
School of Public Health
Dr Yano is trained in healthcare
epidemiology biostatistics and health
policy at UCLA and Rand Dr Yano has over
30 years of experience in Health
Services Research and program evaluation
at UCLA and teaches evaluation of health
services in the executive mph program
and I have spoken to Dr Yano and so
while may sound very nerdy very um
technical which she is she's also very
personable and I think you will very
much enjoy this session so with that I
will turn it over to Dr
Yano thank you so much let us go ahead
and dive
in so it's a pleasure to speak with you
today uh about oh need to share my
slides so
sorry there we
go uh it's a pleasure to speak with you
today about using evaluation to develop
evidence the how and the why as Dr Jen
Jenkins
mentioned so this is as already
mentioned first of a three-part series
focused on the key principles of
evaluation in the federal context and as
mentioned the sessions will be recorded
and made available later in this part we
all are going to focus on how and when
evaluations useful the varying types and
why evaluations a vital tool as already
mentioned and we will have these 10
minutes uh at the end uh and it's my
understanding that we will want you to
actively engage in the Q&A section along
the way and we'll aim to respond in real
time to the extent that we can so what
is evaluation the Office of Management
and budget has laid out a definition of
evaluation as an assessment using
systematic data collection and Analysis
of one or more programs policies and
organizations intended to assess their
Effectiveness and efficiency another
definition is judgments of worth reached
by weighing information against some
explicit or implicit yard stick I prefer
explicit ones because they're measurable
but that yard stick may be statistical
significance clinical relevance
comparisons against a regional or even
arbitrary standard either way it's
important to have those Notions upfront
another definition is a management or
decision-making tool for program
administrators planners policy makers
and other health officials so ultimately
evaluation is about learning if the
program or policy Works does it have the
intended outcomes and this is just an
example to help you think about programs
that vary in terms of new organizational
structures different management
strategies the addition of services not
previously available tailoring to
identified needs of different
populations combining two or more
programs that have worked separately but
perhaps together now may have some
bigger bang for the buck thinking about
changes in financing or new or changes
in existing policy to one or more ranges
of outcomes on access use satisfaction
function many of the others costs uh
cost Effectiveness disparities um all
the way down to mortality rates
depending upon the focus of your program
or your
policy now evaluation can be focused on
process or outcome measures or both from
the process measure side we think about
whether or not the program delivered the
services as intended did they serve in
fact the target population they intended
at the outset did they finish their
activities by a certain Target date and
was it implemented as intended that's
distinguished from outcome measures that
can be quite extensive but just some
examples here are things like improving
guideline knowledge among Physicians
improving Health outcomes and specific
types of outcomes in federally qualified
Health Centers to improve quality of
life perhaps among cancer survivors in
the community to reduce readmission rat
among Medicare patients to reduce Health
disparities for example reducing
maternal mortality among
African-Americans or reducing food
insecurity can be this or again any
number of other outcome
measures now evaluation is grounded in
the social sciences in both theory and
practice so think psychology sociology
anthropology economics I'm an
epidemiologist by training but we come
back to these social sciences to think
about program development and evalu
valuation evaluation is also similar to
other scientific inquiry the evaluation
in questions and standards are thought
of instead of research questions and
hypotheses we have common study designs
there are randomized program evaluations
or time series we often care about
comparison groups we also use common
statistics thinking about how we compare
two or more groups or adjusting for
differences between people who do and
don't participate in the program
evaluations can also be different in
terms of our focus and scope this is not
about pills procedures or devices this
is about Health and Human Service
programs delivery models community-based
programs policy changes or new policies
at the local state Regional Andor
National levels this is about measuring
Effectiveness efficiency equity and
experience as well as impacts and costs
it's real world it's not efficacy under
idealized or homogeneous conditions
the flip side of this is what is not
evaluation it is not opinions from
program developers or other proponents
of a particular program or policy it is
very much human nature to believe in
what we developed or work on or funded
and it's especially true if what you are
doing feels good and or if you like with
whom you work right that's a natural
Prospect but you have to be aware of the
hype that sometimes comes from
opportunities to develop a new program
there's marketing to the community to
potential funders to peers I can tell
you in my own early stages of my work in
evaluation I was working to evaluate a
new Regional pilot to improve primary
care for high-risk vulnerable veterans
and within days of starting my new job I
heard it's incredible I was getting
comments and emails and calls from
Washington DC saying how great it was
that they had heard that this program
was so effective but the program hadn't
even been launched yet and we had not
collected a single piece of data so
important to be careful evaluation is
also not a description of just the
attributes of the problem or its
determinants it's very useful for
thinking about how well the program maps
to the drivers of the problem but it's
not a valuation itself it's also not a
description of the program who leads it
how it's funded what it aims to do but
without data on what it actually does or
has accomplished and importantly if you
Google evaluation you will come up with
many many sites for human resources
evaluation of Staff performance and this
kind of program evaluation has nothing
to do with that kind of HR
approach so what can evaluation do for
us well it answers questions about
contributions of a program or
intervention care model or policy I'll
use those terms somewhat interchangeably
to one or more outcomes so it could
include for example things like
evaluation of patient- centered medical
homes did they fact improve access
continuity and coordination or Child
Abuse Prevention programs was was it in
fact successful in improving parent
education and support the provision and
receipt of Supportive Services did it in
fact have an impact on family well-being
school performance for the children and
reduced incidents of child abuse
overall evaluation also enables us to
compare and contrast the contributions
of multiple programs to identify the
best ones to use now best has quotes
because it can be itself defined in many
ways most effective most efficient or a
combination like cost most coste
effective which doesn't mean the least
costly or best cost benefit or best
return on investment even within those
each term may be defined differently
most effective may be one that achieves
the best outcome on average for everyone
who participated in the program or it
could be focused on the greatest impact
among the most
vulnerable evaluation also allows us to
compare and contrast programs but that
means we also need programs to use
comparable measures so that we're
measuring Apples to Apples not apples to
oranges evaluation can also help us
determine what parts of a particular
program or policy work best to improve
its own efficiency and Effectiveness
over time by way of example you could
think of this about uh the process for
rolling out a multi component program in
stages doing one component at a time to
evaluate outcomes over time to see what
addition of each component yields
discreetly or you could Implement
different levels of program in different
regions to understand which level yields
enough of an impact so I think about
this in terms of taking a particular
area and and perhaps implementing
program light something that has some
basic infrastructure to it versus
another area where you may deploy more
program components at a time and allow
you to explicitly and empirically
compare and contrast how these different
components of the program operate
together I think of it as finding the
key ingredients for what you would want
to take your show on the road for in
terms of spread evaluation can also help
you justify program and policy
decision-making and budget allocations
to optimize population Health to
minimize disparities and other Federal
targets the evidence Act is designed to
ensure evidence informs strategic
planning program and policy design and
budget allocation so evaluation can
support each of these objectives helping
federal agencies demonstrate systematic
rigorous approaches to decisionmaking so
for example does the Strategic process
planning process itself integrate
evidence and or use an evidence-based
planning
process I wanted to also share some
information about some of the most
common types of evaluation and the
descriptions of these varies based on
the disciplines that that generate them
but the key is to think about formative
evaluation as one of the most common
types which ensures that a program or
program activity is feasible appropriate
and acceptable before it is fully
implemented this is an opportunity to
make midcourse Corrections in the
process of a program's development and
evaluation they're usually conducted
when a new program activity is underway
or when an existing one is being adapted
or modified these are extremely useful
um types of
evaluation another common type are
process and implementation evaluations
and those determine whether a programs
activities have been implemented as
intended process evaluations answer the
what when where and how kinds of
questions outcomes and Effectiveness
evaluations measure program effects in
the Target population by assessing the
progress in the outcomes or outcome
objectives that the program is seeking
to achieve for example the changes in
knowledge Health outcomes quality of
life readmission rates and other
outcomes of Interest I mentioned
previously impact assessment itself
assesses program Effectiveness in
achieving its ultimate goals for example
if your program provided local medical
providers with with intensive STD
training to improve the effectiveness of
their screening counseling and treatment
of their patients and the program worked
to improve knowledge attitudes and
skills among members of your target
population and community- based
organizations on the impact side was the
program ultimately able to reduce the
overall STD tra transmission rates in
the community Beyond those in the
evaluation
itself now evaluation is a vital tool
for program planning management and
priority setting so I wanted to talk
some about when it's most useful well I
would say an evaluation is most useful
when the evaluation results the measures
process or outcome are mapped to your
agency measures so even if you're not an
evaluation expert you are in the
driver's seat in defining the measures
themselves balanced by what can be
measured reliably and
validly evaluations are also most useful
when those results are well timed to
make decisions that may sound like an
obvious uh statement but I can't tell
you how many times I've seen evalu aers
come in and say I can answer these
questions in three to four years but the
agency and the policy makers need an
answer in about six months so there
needs to be a connection between the
evaluation design and the timing in
order to make evidence-based decisions
so you want to make sure the evaluation
can deliver in sufficient time for your
reporting and decision-making needs uh
colleagues suggests thinking about cost
quality and speed and picking two of
those where if you need something
quickly but you want it to be high
quality you and lik likely are going to
need to spend more resources on that
particular
evaluation evaluations are also useful
when results are indeed rigorous enough
to be useful in decisionmaking in other
words avoid garbage in and garbage out
if an evaluation is worth doing it's
worth doing right uh and of course
anything that sounds too good to be true
probably is we'll discuss this more in
Parts two and three on evaluation Basics
and evaluation team qualification
and lastly an evaluation tends to be
most useful when the results are in the
context of what else is known about what
works allowing for comparisons across
programs as well hi Becky this is Susan
sorry to interrupt we got a comment um
could you slow down just a little bit I
can indeed thank you thank
you so how to choose among programs
policies and policies to evaluate well
in all honesty it's unlikely that you're
agency or office has enough resources to
evaluate everything it's important first
off that programs be stable and in place
long enough to have some level of output
or outcomes in my agency in the VA they
have prioritized Evidence Act topics
where there's a critical mass of
research or evaluation evidence already
as well as Partnerships with VA program
offices where evidence is being used so
in our case the example include access
to care homelessness and Women's Health
the VA then funded query the quality
enhancement research initiative to help
oversee Evidence Act progress in that
case they then in turn funded a
partnered evidence-based policy resource
center and a field-based series of
evidence-based policy evaluation centers
with a coordinating Hub in planning the
resource center provides technical
assistance on randomized evaluation
designs metrics for use analytic plans
learning agenda development per The
Evidence Act and a leaon they serve a
liazon role between evaluators and
operations and system partners and in
addition to providing programming
support and evaluation briefs to ensure
that any evidence that is generated is
communicated to all the key stakeholders
and participants up the chain and down
the chain the evaluation centers
themselves conduct spefic specific
evaluations linked to agency evidence
priorities these are what I would call
content or topic agnostic centers they
do program evaluation for any kind of
program or policy that is put
forward there are other strategies for
choosing among programs and policies to
evaluate that you should consider one
would be population impact what programs
or policies are likely to have the
biggest bang for the buck for the the
individuals and the communities that you
serve perhaps the focus would be on
disparities reduction again what
programs and policies exist in the
Health Equity or Human Service Equity
space for priority with underrepresented
and underserved
groups may want to think about Big
Ticket programs that represent major
parts of the budgets that you oversee
have they ever even been evaluated I
have seen these kinds of programs in
diverse settings and sometimes the
question is is are they too big to fail
meaning can we in fact have the room and
the will to make evidence-based changes
in some of these foundational programs
and is there room to make Transitions
and adjustments to them over time
another option is to Simply start small
and build evaluation capacity talent and
experience in applying evaluation
results if you've not done evaluation
before this can be a little intimidating
and a little complex and so sometimes
it's better to start with a small team
and a small component of your program
and begin to learn how to develop the
evaluation relationships learn how to
design these learn how to prioritize the
programs and move forward but always
with a plan for how you would support
and accelerate their growth over
time I also wanted to mention
participatory strategies these may
include expert panels stakeholder
councils strategic advice from Key
leaders and agency-wide decision-making
approaches to help you set expectations
around programs and policies and their
evaluation as well it's often very
helpful to get input from a wide array
of people on this path to choosing which
programs and policies to
evaluate I was asked to speak to some of
the stages of program and policy
development when you should even be
thinking about an evaluation and granted
I've been doing evaluations for over 30
years and so perhaps not surprising ly
it's an easy answer to me and that is
early and often I think evaluation from
our experience across a wide array of
local Regional and national projects is
that it evaluation should be integrated
in the early stages of program
development part of that is because the
structure of evaluation design itself
can help structure program development
too for example with its strong emphasis
on clear and measurable goal statements
and smart objectives there's also an
opportunity to consider how something
would eventually be evaluated as you're
thinking about program development it's
what is in fact what in fact do I think
this program or policy can accomplish
and how would I go about measuring that
at some juncture it also enables enables
pre-planning on collecting outcome and
other measures at Baseline before the
program is either completely implemented
or even before it's been started at all
I can't tell you how many times I have
had individuals come and tell me I have
the best program ever and it's really
works and could you now evaluate it and
the reality is is my opportunities and
options for providing any kind of
evaluation have now been limited by the
fact that there was no Baseline
information now that said it's not
always possible to integrate evaluation
in early stages so then what well any
program or policy can be evaluated after
it has been launched even years in some
of that depends on the programs life
cycle some programs are time limited
they are given five years of funding to
accomplish some goals or objectives so
in those cases that can be a little bit
more challenging but others are longer
term or have no specific end dat and
that creates more opportunities to
evaluate both the short and the
long-term benefits to consider what it
takes to maintain those programs and to
be thinking about sustainability as well
so how do you approach evaluations of
existing versus new programs well new
programs sometimes come with new
resources and that can enable
development and evaluation time and F
and funding and that can create a great
opportunity for the combination of
evaluation with program development
proactive evaluation planning is
planning is key if those Innovation
resources are going to go away often
evaluation is used to provide the
rationale the business this case the
return on investment the impacts on why
this program in fact should be sustained
if new programs are to be implemented
however without new resources they may
have to compete with existing programs
for funds in that case evaluation may
actually be even more important in that
scenario that evaluation funding May
squeeze budgets too so you'll have some
Push Pull in that Arena existing
programs typically have the advantage in
contrast of having been fully
implemented so they can be a little less
risky as the new programs don't always
come to fruition or they may take longer
than anticipated to be fully implemented
which then stalls the evaluation I have
been in situations where we have hired
an entire evaluation team but the
program components simply aren't ready
and you either have to put people on
hold or you begin to evaluate early
stages as the case may be now existing
programs typically um also have other
challenges there may be vested interests
with some of them with or without
evidence of Effectiveness and they may
or may not be open to evaluation I have
definitely come to sites that simply
were not interested in anyone coming in
through the doors and thinking about how
to work that through has another set of
issues with
it I also want you to be thinking about
how none of this really starts from a
denovo position I want you to think
about how to integrate prior evaluations
into your evaluation planning in reality
there's very little that is completely
and utterly novel with a few exceptions
so it's really important to take stock
of what's been tried before and learn
from it consider programs and policies
for example that have been tried for a
related problem or for a different
population and think about whether or
not they could be adapted to your
program or population it's important
similarly to compare your program
planning and evaluation to standards and
best practices where possible just a
couple of examples maybe you're you are
supporting a community based program
that's focused on diabetes prevention
but maybe many of the attributes of that
program could in fact be helpful for
hypertension control or hypertension
prevention as well or maybe you have a
program that reduced disparities in
African-American youth but that could in
be tailored to work among latinx youth
in another region so realize there's a
tremendous wealth of information out
there on what people have tried to
accomplish so the summaries of what
works may be available through existing
systematic reviews of the published
literature or other
means so I want to introduce you um to
evidence reviews which may provide
timely and accurate synthesis of
targeted topics of importance to
programs agency staff and policy makers
evidence reviews may be used to develop
policies informed by existing evidence
they may be used to implement effective
services to improve outcomes and support
evidence-based guidance and performance
metrics evidence reviews may be used to
appraise current programs in the context
of new evidence and they may be used to
set directions for future research and
evaluation to address gaps in knowledge
now where evidence isn't sufficient
expert panels may also help you
understand what has worked previously by
combining systematic review evidence
that does exist with expert opinion to
fill key gaps and this is a widely used
process to address those kinds of
problems so I wanted to talk to you
briefly about how to use existing
systematic reviews that might be
relevant to your evaluation there are
several types of systematic literature
review that are available for one is
just what we would call the plain old
literature review process where you get
on some online search engine for the
published literature and it's a fated
information gathering often to help you
define and refine the understanding of
the problem itself the why should we
care component systematic reviews gather
information typically though within a
specific scope they actually include
steps for appraising the quality of the
information often whether or not it's
randomized trial evidence or some such
and they provide you some kind of sense
of the confidence in the findings across
the studies that they look at within
systematic reviews there's also a
process called a review of reviews um
that summarize findings across multiple
systematic reviews for example we
identified over 16 systematic reviews
focused on provider burnout so we
synthesized all 16 to come up with a an
aggregated set of
information metaanalyses are another
form of review that use highly
sophisticated statistical combinations
of evaluations that have similar goals
to come up with some statistically
aggregated answer about what works and
I'll show you an example in a moment and
then on the flip side of those are
evidence Maps which are broad scans of a
larger field of evaluations which give
you a notion of where research and
evaluations been done and where a lot of
work has or has not been
done on the systematic review resources
and these are examples uh this is not an
endorsement of any particular resource
we often use bibliographic search
engines such as PubMed we also sometimes
can work with the Cochran
collaboration this is a Global Group
that emphasizes supporting
evidence-based medicine and
decision-making they've established
standards for literature review such as
search strategies inclusion and
exclusion criteria abstraction standards
peer review quality review criteria as
well as publication standards and there
are Cochran centers around the world
including many in the United States
another example is the arc
evidence-based practice centers the AG
gency for healthcare research and
quality which uses Cochran standards to
generate evidence reports for a wide
variety of health problems so the
message here is really you don't
necessarily have to start from square
one there are lots of resources online
across Health and Human Services I also
wanted to mention the VA has an
evidence-based synthis synthesis program
that's actually linked to the
evidence-based practice centers as
well now I wanted by way of example um
to show you I've been helping our agency
better understand what works to reduce
HealthCare Employee burnout it took less
than a minute to find
118 systematic reviews on health and
safety and work less than a minute it
would obviously take longer to find the
ones that are closer fit but I'll get a
lot of ideas as I review these
two now a review of reviews by way of
example may help answer questions about
comparative effectiveness so agency
leaders wanted to know whether they
should continue to invest in continuous
quality improvement
to improve clinical care delivery and
this review of reviews basically said
yes and no cqi Works in specific
contexts there's lots of evidence
regrettably but it's of low methodologic
quality and no one has cornered the
market on the best framework to use and
context matters as to whether cqi works
or not so it is not a one-size fits-all
solution Now reminder that a
metaanalyses are Soph highly
sophisticated statistic iCal
combinations of published evaluations
and research with similar goals and
outcome measures that comes up with some
aggregated answer of what works I wanted
to show you what one of these looks like
in this example you get a nice visual
representation of how spinal
manipulation Works in a bunch of
randomized Trials compared to usual
practice whatever that was for each one
of these now high scores if you see the
red arrow here high scores are actually
worse pain scores so anything that's
negative is actually good anything on
the left of that line so without
advanced statistics you can look at this
and see that SP spinal manipulative
therapy is an effective treatment in
reducing short-term pain you don't even
have to read the paper if you don't want
to so these visuals can in fact be very
helpful in this example I wanted to show
you how many different types of
interventions can be compared and
contrasted in this case for improving
adult immunization and cancer screening
services you can quickly see in this
example that organizational change is
the most effective intervention for
immunizations cervical cytology and
colon cancer screening but for
mamography it's patient Financial
incentives just as important are the
strategies that are the least or not at
all effective including those simpler
things we often try to do first like
feedback tell people how they're doing
or patient education which may be
necessary but never sufficient on its
own
now here's an Evidence map example on
the other end of the spectrum uh
remember these are broad scans of the
literature to learn what kind of works
been done and where the gaps are here if
you look at the tall bars you'll see
that there are hundred or more studies
in these taller bars in each section
which demonstrate that there are over a
hundred or more studies on health
disparities among veterans who are
racial and ethnic minorities but in
contrast we know very little about the
health disparities of transgender
disabled or homeless those small bars
towards the right of each section in
this way an Evidence map can help you
think about where new programs may need
to be developed and are
evaluated now lastly this evidence map
shows you the types of interventions
that have been tried for different
disparities so on the left side you'll
see system level technology provider
focused patient focused and
multicomponent so you can see at the top
here with those big circles that these
are mostly observational big data
studies but given the different colors
of the circles you can also see that the
evidence in these different types of
populations is also
mixed the purple bubbles the yeses also
show which interventions worked and you
can see here that most of those are in
the technology category or or are
patient focused that means there's a lot
of room for us to think about provider
focused and multi-component
interventions as well as other work that
needs to be done for other population
ations so how do you integrate prior
evaluations into your evaluation
planning the real challenge is what if
you even can't find any despite all the
tools and resources I described at your
disposal it is possible that many
programs have been tried but not
evaluated and for that I say buyer
beware because in the absence of
evidence even if someone has been
running a program for some time you
really can't speak to whether or not
it's been effective there are also other
programs that have been evaluated and
may have some data but are not published
this is often represented in what we
call the gray literature reports working
papers newsletters policy briefs mounds
of government documents even speeches
that sometimes provide a very compelling
argument for why something may work the
challenge with that gray literature is
that it has typically if ever been
peer-reviewed which means they are
representing their work with unknown
rigor they may be opinions and
statements without evidence and the work
has been conducted with uncertain
qualifications expert opinion as I
mentioned before can help fill gaps and
help inform evaluation planning in these
circumstances I thought of two examples
in this area one we're working on right
now on a congressional mandate to
evaluate the prevalence of intimate
partner violence among veterans the
legislation required a literature review
and a a study without specifying much
more to bring these two pieces of EV
sets of evidence together we recently
convened an expert panel of researchers
policy makers providers and community
members to try and sense make across
this information to inform a task force
that will make decisions around an
evidence-based comprehensive ipv
program another example is that perhaps
you have a literature review of what
Strat iies reduce gun violence but the
evidence is limited so in that case you
may again want to convene an expert
panel of policy makers Community leaders
researchers and others to take the what
is known about the literature review
combine it with expert opinion to make
some evidence-based
recommendations either way the
recommendation is to also consider
contacting your evaluation office or
liaison in your agency if you cannot
find any prior evaluations
so as we are working through these
examples and I want you to be thinking
about how you can use evaluation results
for program planning management and
priority setting so let's say you've
gone through all of this you've you're
overseeing or even developed and
implemented a program and you have
evaluation results ready so now what
well as I often say for my uh uh
students and colleagues it depends it
depends on whether the program was found
to be effective or not if it's not
effective that process evaluation I
described earlier could be instrumental
in helping you understand why the
program didn't work was it a pro problem
in how the program was implemented or in
its theory of C of cause and effect was
the program ever in a strong position to
have the outcomes of interest if it was
effective you need to think about
whether or not the program you know and
the program did work it improved
outcomes but did it do it a lot or a
little now that becomes be important
because if it made some statistically
significant but very minor Improvement
you're going to need to make decisions
about whether the program's efficiency
and Effectiveness could be bolstered can
you improve its impact and whether or
not the program costs are worth worth
worth those incremental gains even
though they're statistically significant
let's say now if the program worked a
lot however you define that you still
need to consider the significance of the
gains whether or not those gains were on
average improvements that were
statistically significant let's say or
whether or not there were gains for
people that were already likely to do
well versus gains for people that that
um needed to do much better in other
words do you need to tailor the program
further or push it to make sure that
it's serving people with greatest need
and then you have to consider costs as
well that return on investment the cost
benefit the cost Effectiveness needs to
be part of that consideration when a
program
works the other issue is that it depends
on whether the program was significantly
better than other programs perhaps
aiming to achieve the same goals and
that may indeed have implications for
which is sustained so that evidence
indeed needs to be rigorous if you were
talking about concluding a program on
the basis of evidence it needs to be
quite strong irrefutable perhaps and
strong enough to have the decision make
good sense on the other end if you do
not have that kind of evidence we often
recommend people continue to evaluate
the set of programs aiming to meet the
same goals or consider whether or not
some programs work better for some
communities than others and have that
part of your explicit uh uh priority
setting and management
process so with that I'm going to turn
us over for uh questions and answers um
and I'll stop sharing at this
point thank you very much Becky again I
apologize for not showing my camera but
it decided to stop working around noon
today but um thank you so so much Becky
we really appreciate all the information
that you shared with us and we do have a
number of questions if people have
additional questions please put them in
the Q&A function and we will try to
address them now or have a document that
comes out later quite
frequently asked questions type of
document so we will get to your question
and so the first one that we received is
comparable measures you mentioned that
in one of the beginning slides and you
talked about comparing Apples to Apples
and not apples to oranges could you
offer a little more context regarding
that or an example of how you would sort
of determine what's Apples to Apples
Apple happy to do so um I I think about
some of the work that's been done in
Suicide Prevention and there are
measures of
suicidality um so that you have a notion
of whether or not people are uh less uh
suicidal that they're less likely to to
uh uh attempt suicide and there may be
three or four different validated
measures that you might find on a
literature search the question is is if
you compare multiple programs and
they've all measured suicidality
differently then it's going to be
somewhat difficult to compare those
programs or intimate partner violence in
our expert panels last week we spent a a
surprising actually to me amount of time
on the different definitional aspects
between the Centers for Disease Control
the Department of Veterans Affairs
community-based definitions and realized
that if all these definitions differed
what actually goes into saying someone
has experienced intimate partner or
domestic violence is going to vary and
that means that again we may be faced
with apples and oranges across different
programs that have sought to reduce
intimate partner violence or address
needs after such
exposures does that help thank you yes
um another question is there a
recommended time frame for a program to
have run in order to have a substantive
level of outputs or
outcomes great question um it I would
say again it depends um in part in one
of the main evaluations I did in the VA
years ago um the program itself was a
profound
transformation of uh care teams into
what they called academic Global care
teams that had a novel electronic health
record component substantial training
and even physical plant renovations to
create seamless services around uh
Primary Care uh assigned patient
populations when you're talking about
something that includes space renovation
uh uh training and potential lure issues
um and any other complex components th
any one of those steps is likely to
delay having the entire program in place
right so if I evaluate that program when
half the building is under construction
and not everyone's been hired and some
of the folks have not been trained I'm
giving the program short shrift because
it's not fully uh built right literally
and figuratively uh in other situations
it may be simpler to say okay this
requires training and we're going to
install you know a specific reminder or
we're going to um uh head to 10
different clinics to do some kind of
activity with them and it may be easier
for you to know that the program is in
place so so it depends upon the program
complexity and how melty facet it is uh
how and when you really feel like it's
stable enough for you to go even in that
first example we found we really
couldn't do a full Baseline um because
of the way even the evaluation was
funded and we ended up calling it kind
of early versus late and that's okay too
um so long as you're clear uh that the
program was in its very early phases and
that's when you decided to collect data
in some cases as well it's good to have
many data points if you can of course
every time you collect data it costs
more resources for not just data
collection but analysis and Reporting
but in some of those situations if the
program is going to take quite some time
to roll out those multiple data points
start giving you some insights into how
fully fleshed out the program in fact is
that can also be a good time for what we
described as formative evaluation where
you in fact can feed back to the program
developers and Lead leaders by the way
you you think the program is fully
developed but here's a bunch of the the
areas that haven't been as implemented
as you think they have been you may want
to shore those up before we start the
outcomes
evaluation thank you and I would also
emphasize for everyone particularly
folks that work with the programs know
the programs are topical subject matter
experts that's where as evaluators we
really need your input to really know
are things implemented as much as we
think just as Becky was saying so we
encourage you to partner with the
evaluation
teams another question what have been
some unexpected challenges you've
experienced when trying to plan an
evaluation and I say that with a smile
in my voice thank you and I'm smiling as
well um gosh more than I can probably do
justice to in the time um I have had
uh I think one example is actually a
Child Abuse Prevention project we worked
on years and years ago where suddenly
the volume of um uh referrals from the
uh Family Justice Court uh that stayed
high but the programs uh throughput of
referrals to comprehensive services to
the community uh suddenly took a sharp
fall and we rapidly went in to do a
process evaluation in those sites and
began interviewing program participants
Pro not participants excuse me program
staff and what was fascinating is that
they all uniformly said well actually we
have all of those referrals um and
they're in my bottom drawer right here
and we stopped and said well why aren't
you processing those and they said well
the program leader said if you don't get
everything done in two weeks from when
we get it from the court I don't want to
see it so they operationalize that
explicitly and just put them all in the
bottom drawer so going on site talking
to people understanding what the uh uh
unintended consequences may be of shifts
in directives shifts in planning or
policy can be incredibly important um I
have seen also a substantial turnover in
programs over time where you need to
make sure that both the program and
agency leaders and the evaluators
communicate what's going on and what the
expectations are for evaluation in the
field um I've also during evaluations
found that people think that if
something is being evaluated um that
you're doing some special program that
will probably be taken away so I have
had I've only learned through evaluation
that in some cases when we've
implemented let's say in this one
example a depression collaborative care
model um in a a few doz dozen uh primary
care clinics that one site saw that one
provider in particular was consuming all
of the referral options and none of the
other providers were really using the
new program and we found out that the
person thought this was research and
thought this would really helped him
with his patients to have his patients
depression managed and they got better
so they were able to help manage their
diabetes and their other issues better
um and thought saw it as a constrained
resource that actually didn't want to
share so he missed the part of the
program that was about being a clinical
champion and sharing this program with
others um and in other places you learn
challenges in everybody capitalizing on
the program but the program itself uh
not doing well in evaluation because
they did not anticipate demand and so as
providers began to use a program more
and more and more and more the program
wasn't capable of keeping a pace and
very quickly providers stopped using the
program because they're like well nobody
can answer my calls nobody's responding
to my referrals and so again both the
evaluation and the program have this uh
commensal relationship with each other
uh in terms of how they uh inform uh
future changes to the program and the
policies themselves so I hope those are
some examples um I have had I think one
other was where we did deliver results
that demonstrated that a program did not
have have the intended effects so we had
someone say well we're not going to pay
the contract um that's why you it's good
to have contracts for both the program
and the agency and for if you contract
out with evaluators so that if you're
delivering bad news there is a plan for
how you actually account for that and
how you think about it and how you have
considered that even at the earlier
stages of program development and
evaluation that ultimately worked out
just fine and the program in instead of
being defensive shifted to what else can
we do in the program to in fact
accomplish the outcomes we meant to
there um but it becomes a conversation
and that's where the value and
participatory evaluation lies as well so
just some examples thank you Becky and
on that last point I will also chime in
that the HHS evaluation policy does
require that evaluations be made public
in a timely manner and so having some of
those discussions up front in an
intermediate frame and then at the end
to understand what are we going to do if
the results aren't what we want them to
be how are we going to use that how are
we going to take an approach that is a
helpful approach and we we weave that in
and we do that from the beginning and
again as Becky said make that a
conversation so we are almost out of
time for questions so I do want to
acknowledge that we have some actually
quite sub substantive questions here and
I think that it's best to give quite
substantive answers so we will respond
to those in writing and post that
material when we post the recording so
for example one of the substantive
questions is there seems to be much talk
about evidence-based practice are there
any evaluation methods for
practice-based evidence especially for
Native and Indigenous peoples that may
not have the capacity to integrate or
prove that their healing practices are
just as a effective or even better than
western-based medicine so I don't want
to give that type of a question that
really substantive question short shrift
so we have received it and several
others that are equally weighty we will
respond in writing to those we will
provide some
information on those in terms of one
last
question um let's see what are your
thoughts on evaluating culture changes
like increases in De
eiia so Equity area success measures
such as reduce microaggressions towards
those identified as transgender or
Native
Alaskan excellent question um I I'm
actually uh there's substantial work
going on in that Arena and the
development of reliable and valid
measures of things like
microaggressions um and culturally
appropriate measures so that when you
assess the prevalence of these concerns
in different communities you're doing it
in a way that captures the experience um
in a meaningful way work is underway on
doing better jobs in all of those areas
um and so again I'd like to respond to
that one in writing afterwards um but I
just know that there's a tremendous
amount of work uh to get that right
so that we can we that's a global we we
as program evaluators Health Services
researchers Human Service researchers
and the like can ensure the access to
good measures to use in program
evaluation across the country I don't
think that those um tools are
necessarily available as yet um but I
know that there's a lot of initiatives
out there for improving measurement and
improving
their inclusion in a wide array of
program evaluation activities I know
that in in my own work as well um so we
work with um different program offices
in our area to make sure we're using
measures that are standard within the
field basically so I I hope that's a
helpful initial answer but I agree Dr
Jenkins that it it warrants a fuller
answer after this session today as
well thank you um so with that I will
turn it over to Academy health and we
have a survey we are very interested as
this is the beginning of our Evidence
Act capacity building training series of
this series of three trainings but also
we are developing trainings over the
next several years to address the
capacity assessment that HHS did Under
The Evidence Act so I will turn it over
to Academy Health we have a few
questions that we really encourage you
to respond to to help us to improve
these trainings
