Using Evaluation to Develop Evidence: Getting the Information You Need | July 28, 2022 | Webinar

USDOHAHS
156 followers
May 22, 2024
This is the second of a three-part series focused on the key principles of evaluation in the federal context. During this session, Elizabeth Yano, Ph.D., M.S.P.H. covers: 1) How and when to work with an evaluation team; and 2) How to turn your need for information into solid evaluation questions and maximize the benefit of the evaluation.
--
U.S. Department of Health and Human Services (HHS) | http://www.hhs.gov
http://www.Twitter.com/HHSGov | http://www.Facebook.com/HHS http://www.Instagram.com/HHSGov
http://www.LinkedIn.com/company/us-department-of-health-and-human-services
HHS Privacy Policy: http://www.hhs.gov/Privacy.html
so good afternoon welcome to the second
in the series of how and why evaluation
is important and can be important to you
this series is a result of part of hhs's
response to the evidence-based
foundations for evidence-based
policymaking act in response to that act
as a requirement we did a capacity
assessment to assess the HHS capacity
for evidence building that includes data
building data capacity evidence building
in terms of evaluation performance
measurement foundational fact finding
policy analysis a range of things in
that capacity assessment one of the
things that Rose to the top was a better
need to understand evaluation amongst
people who are not evaluation staff
themselves or not the primary people
responsible for evaluation sort of a why
should you be interested in evaluation
and how can this helpful to you so this
series and I said this is a second in
the three-part series is focusing on
that topic and so we will cover how and
when to work with an evaluation Team how
to turn your need for information into
solid evaluation questions and maximize
the benefit of evaluation and in the
next session which is next Thursday at
the same time we will talk about
appropriate use of evaluation data to
inform programmatic and policy decision
so it won't be a statistical discussion
per se but how to know that the data are
good quality how to use those data how
to communicate those data a couple of
housekeeping things the session that we
had last week was recorded and we do
have a um PowerPoint we will be
providing the PowerPoint later in this
session for those that want it from the
last session we will also be attaching
in the chat a copy of the PowerPoint for
this session so you will be able to look
at that while you're looking at the
presentation we will be recording this
pres this webinar as well and all of the
materials and recordings will be posted
on one of the asby websites and there
will be a link to them when they are
available on the evidence building
Evidence Act part of the ASP website and
we'll send out materials to let you all
know when they are posted and please
submit questions through the question
and answer tool and we will get to those
questions some we will answer on the fly
in writing as we go through but
otherwise we will have time for
questions and answers at the end of this
session so it is my distinct pleasure
and honor as last week to introduce our
speaker Dr Elizabeth Yano Dr Yano is a
an Adjunct professor of health and
policy management at the UCLA Fielding
School of Public Health Dr Dr Yano
trained in healthcare epidemiology
biostatistics and health policy at UCLA
and Rand has over 30 years of experience
in Health Services Research and program
evaluation and at UCLA teaches
evaluation Health Services in the
executive mph program and so while that
sounds really nerdy and she can be quite
nerdy if she wants to be she's also very
engaging and we're thrilled to have her
here so I will turn this over to you Dr
Jana thank you so much let go ahead and
share my screen can you see that all
right great so yes as Dr Jenkins
mentioned this is our part two on
getting the information you
need and second of a a three-part series
for thinking about key principles of
evaluation in the federal context uh and
we will have time at the end for
questions and answers so let me just
dive in on when to work with an
valuation team that depends on the type
of evaluation needed and the
availability of internal resources and
expertise now you may have within your
program your component your
agency strong internal expertise but it
may be that you have difficulty
redirecting staff to evaluation perhaps
everyone's booked solid or they may not
have interest in eval your evaluation
Focus for me whether it's at UCLA or
within my agency finding existing
faculty and staff in the middle of a
fiscal year in particular can be really
tough as people often plan out their
coverage and their Pro their projects
well in advance it can also be difficult
to engage existing folks who have strong
skills but also strong passions for
specific topics and if yours is not
among them you're going to have to keep
looking now it may also be you have
internal expertise in analyzing survey
data but that may not equate to
strengths in conducting qualitative
interviews for example so if an
evaluation incl includes survey data you
may want to consider an evaluation team
that combines in-house agency expertise
with external evaluator qualitative
expertise I'm afraid I speak from
experience in this one I have real
strengths in big data and surveys and I
figured I could easily handle
qualitative interviews no problem I
learned I'm a better talker than a
listener uh perhaps no surprise there
but the challenge was that I tended to
get really excited about people's
responses to the interview questions and
I started putting two and two together
as I would speak with them and I would
start to feed back the things that I was
learning along the way uh when we listen
to the audio tapes of my first couple of
interviews my very kind and gentle
qualitative expert said wow that is just
a really great great quote there and I
beamed and her response was the only
problem is we can't use it because the
interview we had to have said that not
you so onward and upward now you may
also have internal expertise that's
ready to go but there may be value in
having an external objective evaluation
team that can be challenging if in-house
expertise is strong and willing but the
political context is such that Reliance
on an external evaluation team would be
better my story here is really about a
VA colleague who published an evaluation
in a top tier Journal which showed that
the VA outperformed private sector
Health Care in something like 295 of 300
Healthcare quality measures not
surprisingly the agency quoted this
paper all over the place while the media
touted this external evaluation findings
ignoring that my colleague was actually
also a VA researcher in many ways this
was the best of both worlds as the as
the methods were indeed rigorous and
conducted at a think tank while
benefiting from his expert knowledge if
you excuse me if you don't have if you
don't have internal expertise then it's
good to partner with other departments
with such expertise while also seeking
external evaluators it's really good to
get
help so I want you to realize basically
that you have options you can have an
internal evaluation team that works
within the agency and outside of the
program that can evaluate the program
itself you can also build an evaluation
team within the program so long as you
think you can do this with enough of an
Arms Reach to remain objective namely
you have to be able to report good news
and bad news or you can work work with
an external evaluation team that works
with the agency itself or you can work
with an external evaluation team that
works with the program or you could have
a
combination so I wanted to give you an
example from some of my own work for one
of my own evaluations where we had all
the right expertise inhouse for the
practice surveys and the key stakeholder
interviews up top and the administrative
data which required use of our
electronic health record on the bottom
right but we opted to contract out the
provider and staff surveys with a
leading expert at a nonprofit we did our
veteran surveys and the DAT and with a
survey firm that checked all the boxes
for veteran experience and also the data
security that we needed and we
contracted out with a school public
health on the right side to conduct
additional qualitative interviews with
primary care teamlets so that in this
case we were able to leverage our
in-house expertise with some external
qualitative expertise so we could simply
just get all the work done so bottom
line you should do what works for your
agency and your
program now there are varying
perspectives on the value of an external
evaluation team some perceive working
with an evaluation team that's external
to be faster and less costly they may be
able to staff up faster than the agency
I know from my own experience Federal
hires can take a lot of time I had a
three-year project and it took my HR
Group at that time three years to post
the position that was a problem it may
require changes to an organizational
chart to actually be able to staff up or
it may be especially challenging to
accomplish midyear and an external
evaluator may not have those delays it
can also an external evaluator can also
dedicate staff to the evaluation and
perhaps they'll be less likely to be
pulled away to attend to competing
demands or less likely to allow
evaluation to become a collateral
Duty an external evaluator may also be
able to design and launch evaluation
faster if they've got prior experience
so for example interview guides or
surveys they've done for related
programs that they can just tweak and
adapt rather than develop um denovo they
may also flip side of the F of the
hiring issues may be able to dial down
the evaluation faster and easier versus
in our case sometimes having to hire
permanent staff the flip side is that
external evaluators may not be faster or
they may even be quite costly contracts
themselves take time and experience to
get right and this is not a simple
handoff my job is done someone else is
doing the
evaluation these kinds of external
relationships require non-trivial
oversight engagement and management and
I've had also university-based
evaluators say that they have to go
through regulatory review like
institutional review boards for human
subjects protection and that can add a
substantial amount of
time now the plan plans for engaging
evaluation teams require some
pre-thought and I would say the first
step really in this arena is to engage
your agency staff and that it includes
topical and uh other relevant expertise
within your agency and across programs
perhaps as well as those who are in
charge of the outcomes and acting on
them so you need to be able to think
about what the agency's goals for the
program and its evaluation are and
that's relevant regardless of how old
the program is long it's been around and
it's always good I think to revisit
program goals and the assumptions that
there there are therein you also need to
be thinking about the kind of evaluation
information that would be useful um is
it process measures or outcome measures
like we talked about last time is it
comparative effectiveness is are the
agency leaders and program leaders
needing to line up a variety of programs
to really see and compare and contrast
which one's most effective for what kind
of groups Etc do people really need a
change over time or do they need stories
and testimonials and why by stories I
mean combining your statistics which for
some people can be either dry or given
challenges in numeracy difficult to
always understand but instead create a
more using qualitative interviews create
a more understandable meaningful and
often more powerful portrait of what a
program is accomplishing what it means
to those those who participate in their
own
words all of this also requires thinking
through what kind of resources are going
to be available for evaluation and in my
experience it's regrettably not at all
uncommon for me to have to spend several
meetings hearing about someone's new
program and their need for evaluation
helping them think it through only to
find out that they in fact have no
resources for evaluation or that there's
a bad mismatch between what they want
and how far their resources are likely
to go so it's very important to be
explicit
now in that context you have to think
about the scope of work and what the
deliverables will be this is again the
value of setting an appropriate level
and scope of the work and this is true
as true for internal and external
evaluation teams so that everyone's on
the same
page I also want to emphasize that you
have to think through who needs to be at
the table initially for example who's
going to help you select outcome
measures and also think about how you're
going to design the eval valuation and
then think about who needs to be at the
table later on those people who can act
on evaluation outcomes I can't emphasize
enough how useful that latter part is
because even though you may be
collecting the outcomes they care about
you need to also know the kind of
evidence they would need in order to act
I recall one of my early career folks
completing a topf flight evaluation of
women veterans access to and use of VA
Healthcare and presenting our findings
to the agency's head of women's health
in an expert panel meeting to help come
to consensus on what to do in reaction
and response to her evidence the good
news at well the bad news actually is
that to start with that was that the
agency found her work in a single VA
Network to be really high quality but
not useful for making a national policy
change so the good news is that they
decided to actually fund her to take her
work to scale nationally so that they
could then use it to make evidence-based
practice and policy changes that kind of
win-win is what we often
seek in this context it's important to
develop questions for the potential
evaluators and team members you may want
to bring in and decide how you'll assess
their qualifications speak with
references or clients ask for work
samples don't be shy or even consider
whether or not you're going to do a
request for applications and fully
review proposals that come in uh along
the
way now in my you and again in
experience working with an evaluation
team whether it's internal or external
needs to be best done as a partnership
it's a really important to be clear on
the responsibilities of the agency and
the evaluation and team as well as the
program it's really important to
establish useful communication
approaches the what when high and wow
who's how and who's on first for what
the frequency and the mode of
communication should be designed to fit
the agency needs for example I've had
situations where the agency needs to
report progress on a program on a
quarterly basis but didn't build in the
time to make sure they could get the
quarterly reports soon enough to review
them to edit them to ask questions on
the path to them taking that information
up to the next level so it's important
to set those expectations up explicitly
on how soon you need information for it
to be useful for you to communicate up
the chain or across to other agencies or
organizations
it's important to be clear on what
communication is needed do you want to
know when there are problems with the
evaluation plans or there implementation
do you want to know the moment there are
problems with a program or are you the
type where no news is good news um
personally I prefer to hear problems
early so that we can act on them but
every every level of the agency in the
program itself may need different um uh
trigger points for engagement I also
like to identify a communication pathway
so it's very clear who is on point
especially in larger organizations so is
it clear to the evaluator to whom they
ought to communicate for different
issues so I would just say it's
important to make sure expectations
around communication are included in a
contract if you're doing something
external or in a within agency
memorandum of understanding to minimize
problems down the line so in summary in
this area the work plan and the
deliverables need to be clearly
communicated and and you need to know
when there's a flexibility and where
there may need to be consequences now
this is an important point no one likes
to assume there's going to be problems
down the line in an evaluation and the
downside is that hope tends to lead to
the avoidance of discussion of the wh
ifs something goes wrong especially
while you're developing relationships
across teams across programs between an
evaluation team and the like um and
those relationships May last for years
so people tend to again not want to get
into some of these potential
consequences but life happens and in my
experience it is always better to walk
through them ahead of the
game so I wanted to give you an example
uh of how I've explicitly gotten agency
program and Frontline providers and
staff to come to consensus on the kind
of senior leader involvement that uh
that we think is important and feasible
in the context of an evaluation uh this
may be overkill for you but we literally
surveyed those involved at different
levels of the organization ahead of a
group meeting to guide discussion
similar questions you might pose are
what kind of oversight is needed by the
agency who will be involved at the
program level and how how often will
reports on progress be needed and will
you need consumer engagement so these
are just things to consider along the
way there are also many stakeholders
Beyond an evaluation team so programs
and policies not surprisingly have many
stakeholders there are the agency
leaders themselves who may or may not be
able to participate in the evaluation or
any of these meetings I'm talking about
there are the program developers and the
program leaders sometimes they're one
and the same sometimes they're different
as well as program staff they are often
very strong advocates for the program
they understand the program in details
that are very important there are also
those folks who are the program benefit
iici Aries and users it may be patients
and families or providers and staff
there are community members and
community- based organizations as well
as other agencies uh that may be focused
on similar problems that you're working
to solve and the evaluations that they
may be conducting as well that you'd
benefit from there may also be topical
experts from other programs researchers
evaluators who want to learn about what
you're learning in as much real time as
possible and then there are groups that
you need to engage that oversee needed
data you're not going to get access to
electronic health records easily without
going through many layers of Gatekeepers
or often getting access to
administrative data can be quite
challenging if you don't Engage The
Right stakeholders along the way so you
can be the Hub and spoke in something
like this where you're working
individually with each one of these
stakeholder groups and there are times
that that would be appropriate uh but
depending upon the size and diversity of
your potential stakeholders you may also
want to consider a stakeholder Advisory
Group to engage regularly this provides
a structure for regular engagement
collaboration and mutual accountability
and there's plenty of research that's
demonstrated that Community engaged
evaluation has many benefits it can
increase the way findings are
implemented to bring about change and it
can increase inclusion of ideas shared
voice and even more importantly trust in
the results
how long should an evaluation be well I
know I say it depends frequently but it
depends on the type the size the
complexity and the outcomes that you're
seeking in an evaluation as well as how
long the program actually takes to
implement or the outcomes take to be
realized so some evaluations can be
quite brief a matter of months but
that's typically only if the evaluation
focus is very narrow and you've already
been able to readily identify
appropriate staff so I want have a
couple of examples one would be an
evaluation of an educational program
which may only require surveys before
and after completion of the program to
demonstrate improved awareness and
knowledge um or a process evaluation of
a program may only require tracking
specific progress Milestones towards
putting program components in place
perhaps they're easily uh accessible um
and that may require selected number of
interviews and maybe some inter site
visits and you're done there's also
rigorous randomized evaluations
including pragmatic trials and other
large scalea evaluations that can easily
take three to five years there are a lot
of steps from planning and implementing
to collecting and analyzing data before
you even get to reporting there are
plenty of steps in there that also can
take longer than expected I have had a
lot of experience where program
implementation during leadership changes
gets styed completely and you have to be
on hold while the program shifts its
governance structure or has to rehire a
new leader or things like that I've also
experienced delays and challenges in
survey data collection when the response
rates are quite low and require me to
double the data collection period to
make sure we get enough people
responding to a survey for it to not be
biased and there are many more examples
so you need to be prepared for this kind
of evaluation
timeline now some evaluations can also
be quite extensive 10 or more years so
if a program takes time to be
implemented and this I think about when
people are building buildings engaging
lots of stakeholders having to change
policy in order to implement the program
standing up large numbers of staff and
training them and putting them in place
then the evaluation itself adds time to
that entire window beforehand to collect
Baseline information during the
implementation and afterwards I need to
be able to collect and then analyze and
Report data well after the program
itself is supposed to be done per se now
another example is if you're focused on
something like mortality for a chronic
condition it can take many years to see
any movement on the needle on changing
patterns and and rates or if you're
relying on secondary data like Vital
Statistics it can take years before
those data for a particular year or
window are actually collected cleaned
and made a available for others for
analysis so you need to have uh some
thought to those kinds of evaluation
challenges and opportunities the flip
side is no program or policy can wait
five or 10 years for results so it's
critical to have interim objectives and
outcomes throughout the evaluation
because it's going to be essential for
program leaders policy makers and others
to get those interim reports to make
decisions along the way it just simply
can't be a black box and expect others
to have full faith without early markers
of progress and inroads on key measures
so basically you need to build
accountability
in now there are also um types of
information as a result that I wanted to
mention that are needed at program and
agency levels the evaluation questions
need to represent those program and
agency needs the common information
needed by both are things like program
progress on implementation and
deliverables including any early warning
signs that things may be running a foul
and some of this could be integrated
into contract language and scope of work
if you're doing this with an external
group there are also typically demands
for early results especially if they're
needed to continue or secure funding
that's important to be mindful that the
kinds of information funders need have
to be built into the evaluation it may
just be a matter of demonstrating how
many individuals were helped the kinds
of services that were delivered and
their volume and early Trends in
outcomes even if there's not a big
increase or anything early on it's just
demonstrating that things are happening
early results often mean formative
evaluation which we talked about last
time which also requires data cleaning
maybe merging analysis and Reporting so
every wave of results that you need for
interim reporting does mean more time
and resources so that needs to be built
in as well and then of course are the
end results either at the end of a
program's life cycle or the end of the
planned evaluation and funding I always
suggest inclusion of both quantitative
and qualitative data to enable that
storytelling as I mentioned
before so I want to shift us to thinking
about then defining program goals and
activities because that becomes the
foundation for moving on the kind of
evaluation you would want first off
seems simple but you do need to
articulate what you want to accomplish
you need a clear goal statement the
difference you're trying to make the
goal is the outcome you're trying to
change and you have to think then about
what kinds of activities and resources
are needed to accomplish those goals and
think about the logical path from your
activities that should lead to each
outcome and then think about are the
right contributing factors or drivers
being targeted in program
activities in that process of trying to
develop program goals and activities you
then also want to think about what else
is out there in the literature um or
cross agencies that may actually have
been shown to be effective at achieving
your goal so that you have that mindset
that there is in fact uh opportunities
to learn from other programs and you may
be able to compare your program goals
with others and consider adapting
existing ones to your needs but only if
you think they are a good fit for your
agency your populations and your
context so I wanted in case it would be
helpful to share a sort of worksheet I
often have people used to think about
the drivers of the problem they're
trying to solve and the think about how
their program is linked to the one or
more drivers uh and think about then the
outcomes they can expect it's often in
this process of working this through
that people realize their eyes have
gotten too big or that they can only
really manage one or two drivers of any
particular ular problem and that they
have to think differently about their
program and its evaluation this exercise
also helps me better understand their
intentions and their needs which always
helps me design a better
evaluation similarly I want you to think
about the wide range of possible out
program outcomes Beyond those that you
may have planned so here's an example to
help that may be especially gerine to
our lives during the pandemic one
problem on the left side is that adults
do not exercise enough that comes with a
sense that there are some people who
indeed exercise regularly um but too
many who are not you want the regular
exercisers up top to keep it up and you
want to get those people who are sitting
on the couch or sitting in front of
their computers too much to regularly
start exercising um at some point and
then you want the non-exercisers you the
reality is is that the N some
non-exercisers are simply not going to
change no matter what you do and some
regular exercisers May in fact slow down
and stop and you know perhaps illness or
injury but the bottom line is you want
to make sure you're thinking through all
of these possible outcomes when you're
designing the program and conducting its
evaluation so that if you don't
necessarily get all the outcomes you
expecting on the other side you
understand all the changes that occurred
and
why so in defining program goals and
activities they need to to be explicit
and measurable so I would say to say
improving birth outcomes is too broad I
need to know whether or not you're
talking about reducing low birth weight
rates are you talking about reducing
pre-term delivery rates are you talking
about reducing maternal complication
rates or are you going so far as to
focus on reducing maternal
mortality then I'm going to push you on
among whom are we talking about all
deliveries are we talking about
high-risk ones only and if you say that
I'm going to say well how are we going
to Define how high risk and then I'm
going to ask you among whom is it an age
group is it among teenagers it is among
older moms and then I'm going to press
you again on where nationally in
specific uh High need regions in rural
hospitals and how are you going to
Define that these first areas where
you're pushing on this is just to make
sure that you can design an evaluation
again that itself is explicit and
measurable do we have defined outcomes
uh measures for outcomes of interest in
low birth weight fortunately there is a
cut point at less than or equal to 2 200
grams but if there is not some kind of
recognized validated cut Point like that
you may need to do additional work with
your evaluation team to figure out what
kind of outcomes and what kind of cut
points are going to matter for you
perhaps not
surprisingly program goals should be
feasible within some reasonable time
frame within available resources and
that too needs to be discussed upfront
all of these steps affect the
evaluation ultimately the test of a good
program goal is the ease with which
evaluation questions flow from it so if
my goal is program a seeks to improve
colorectal cancer screening rates in a
particular State the evaluation question
is simply the flip of that was
participation in program a associated
with improved coloral cancer screening
rates in that state so this is why this
program goal development is so
important now program goals may need to
be multi-level and there's a lot of
details here you can view at another
time but the key here is to be thinking
about whether or not you can achieve
your outcomes of Interest by working at
only one level versus working at
multiple levels so it may not be that
you can solve the problems of Interest
working just with individuals but
perhaps you also have to work with
families and social supports and maybe
not just providers and team members but
also their organizational and practice
milus um and maybe how think about how
they're functioning in their local
community and State Health policy
environments and so on if the program
goals are multi-level the evaluation
needs to be as
well now every program has some Theory
whether it's explicit or implied of how
and why it should work thinking that
through helps you get the most out of
your valuation for example in theory
investment in preventive Services should
reduce service use and costs but how the
thought was that a preventive Services
package in Medicare would actually
improve individual beneficiary
self-efficacy in how do they change
their behavior and changes in their
autonomy so they don't have to wait for
a physician or someone else to tell them
what to do and that will actually change
their risk behaviors which and if they
change their risk behaviors it'll
improve their function improve their
health status they'll even be more
satisfied and their TR pain will reduce
and all of those things taken in a
series of steps should reduce service
use and costs now in this case they
relied on a bandura social learning
theory now I was not trained in theory
there's some wonderful uh opportunities
to look at Theory at a glance and some
other tools that we can make available
but in this case they they landed on
this one that posits that people learn
from one another via observation
imitation and modeling or what we would
just call attention memory and
motivation so they then um worked
through a program that focused on
recruiting providers and educating them
on their preventive Services recruiting
enroles in Medicare and doing a baseline
risk assessment going through and
providing health promotion and disease
prevention visits and health promotion
classes working through with benefit
series an actual B uh plan for changing
their behavior for one or more health
risks and then repeating some of the
preventive services classes on top of a
life satisfaction class all on this
logical Pathway to reducing functional
Decline and utilization and then the
issue here for evaluation is I already
gets me thinking about possible
evaluation questions like was the
provider education in the farle column
effective at improving their knowledge
were the one year one and year two
preventive services with that behavior
change plan actually able to improve
self-efficacy autonomy and reduce risk
behavior as planned and was the program
ultimately able to reduce functional
Decline and lower use in
costs so let me walk you through another
example and take another step further
now in this case this is a program
process as of establishing a goal to
improve the use of Highly active anti
retroviral therapy among patients with
HIV or human imuno deficiency virus
disease the second step after
establishing that goal was to select
drivers of the problem Target in this
case based on the literature Physicians
generally lacked at this time uniform
knowledge of the anti-retroviral therapy
guidelines they lacked clinical supports
to adhere to those guidelines in real
time such as reminders and there was
some evidence that it was can be hard to
change change position Behavior as new
knowledge emerges so the third step was
to design program objectives that match
those selected drivers and causes so the
team in this case developed a physician
Education and Training Program on that
therapy they designed an audit and
feedback program to share guideline
adherence back to each practicing
physician comparing Theirs to other
Physicians and other practices and they
gave them incentive bonuses to
Physicians who actually achiev Benchmark
rat
of therapy com
adherence now in this case based on that
process I can begin to build a model of
the program and its relationship to
short and longer term goals which again
then helps me drive evaluation questions
so on the left you have the HIV quality
improvement program with physician
training the audit and feedback of their
actual practice on HIV care the
comparison to guidelines and the bonuses
and then the early outcomes were the
Improvement in physician guideline
knowledge guideline adherence and in
this case a focus on improving
satisfaction as well on the long-term
path for the overarching goal or our
primary outcome of improving the
proportion of patients on
anti-retroviral
treatment so to show you how that
process gets you to evaluation questions
we take each one of those to then say
was the Pei program associated with
higher physician guideline knowledge was
it associated with higher physician
guideline adherence was it associated
with improved physician satisfaction and
was it Associated ultimately with
increasing highly active an
anti-retroviral
treatment among participating Physicians
HIV positive patients so I wanted to
show you the onetoone correspondence in
this kind of
effort now as you translate program
goals into evaluation questions there
are a few other lessons to be learned
as I said just like with goals the
evaluation questions must be explicit
measurable and feasible so the first
example is I always have people come in
and say did my program work that's what
I need to know and of course I have to
push back and say that's too generic I
don't know what work means and I don't
know how to measure it so I they may go
back and do their homework and come back
to me and say was the healthy patient
program associated with Better Health
outcomes at lower
costs this is something we call bundled
it's got two outcomes at the same time
you really need to separate those
because it's very possible that the
program could achieve Better Health
outcomes but it could cost more so you
need to be able to answer both questions
separately or was a healthy patient
program associated with Better Health
outcomes well it's better than does it
work but it's still too generic because
I need to know which health outcomes are
going to be of the focus so of those was
the healthy patient program associated
with improved knowledge of Healthy
nutrition habits that's the best because
now I know we're after knowledge I know
what kind of knowledge we're after the
fact that they want to see an
improvement tells me that this is
something I'm going to have to evaluate
before and after so that's a study
design issue and there may also be room
for us to compare this to a group not
exposed at the healthy patient program
which of course is more work as
well now I want to make sure that you
remember that just like there are
outcome measures there are process
measures we very often want process
evaluation questions as well so these
address again the program operations of
who what when and how so some examples
were the program activities accomplished
overall or with the intended frequency
and intensity or dosage if you will of
the program were they accomplished on
time I don't need statistical
significance for a question like that
was what was the quality of the program
components and how well were they
implemented what was the intend was the
intended target population reached I
have had programs where they sought to
get high-risk teens but they ended
getting
low-risk later age individuals which
meant that they missed the Mark um how
did external factors influence program
delivery did our assumptions hold up
about how and why the program would work
I showed you that model for the HIV
program is the logic actually work we
thought perhaps the nature of the
problem was X but based on that
assumption we further assumed that doing
a b and c would address that problem
were we in fact right or were there
other factors of which we were unaware
or where our logic based on outdated
thinking when I think about the HIV
example there is one very big thing
missing in that model and that is that
it's all focused on changing Physicians
but there's nothing in there about the
rest of the teams that in that ensure
that good quality of care is delivered
there's also nothing about the patient
and F or focus on compliance or their
understanding and awareness so process
evaluation questions can help us in
those particular
areas next step on this then is thinking
about standards of Effectiveness so
we've got a series of evaluation
questions now we need to know how to
answer yes or no and it's to do that
that it's best if we have preset
specific criteria on how to answer each
evaluation question now when we think
about program Merit or Effectiveness
it's often set by one of several
standards it may be statistical
significance for example program a
improved continuity in one year and we
get the P less than 0.05 we go oh great
it's statistically significant and in
terms of whatever its change was or we
may have program a improved access
statistically significantly better than
a comparison program uh that only works
if those two programs the program of
interest and In the comparison group are
reasonably comparable meaning they serve
about the same risk patients or in an
area that is somewhat similar to the
area in which the program is to ensure
you're comparing Apples to Apples you
may work on normative comparisons for
example you may have no way of randomly
assigning people or identifying a a
region that's like the one where you
need to develop and Implement and
evaluate this program so in one example
we had a program that was focused on
improving the low birth weight rate
among participating teens and again
there was no comparator so we compared
the program's impact among these
high-risk teens compared to the entire
County's low-risk low birth weight rate
if they could improve it better than
that that was quite
something and sometimes we don't have
comparisons or statistical data
available and so we use arbitrary
standards for example perhaps you want
to set up the standard that at least
half of program seat participants will
engage in all program Services if they
don't that means the program was not
effective in that area Target setting
for progress and end goals is key and
that's the one of the more important
takeaways I want you to have here it
communicates the expectations and
priorities for progress on key outcomes
and I want to say the flip side to all
of this is that it's really important to
avoid perfectionism or exessive of
advocacy no program is all things to all
people if you think about it you know
you're never going to get 100% of teens
to change Behavior it makes me think of
a school-based program that I learned
about in my son's school um years ago in
high school where outside of the magnet
program in which he was uh being
educated the average reading level was
at the second grade level in high school
second grade and one of the teachers
developed the reading program to
intensively help these students and
within a single semester he managed to
bring their reading levels up from a
second grade level to an eighth grade
level but the district or and other
leaders had set up the standard that
those students had to make it to their
grade level in a single semester so the
program was decided to be a failure and
they concluded it so you don't want to
have those unintended consequences along
the
way now to go back to the HIV Qi example
I wanted to provide you with some
example standards as well so was the HIV
Qi program associated with higher MD
guideline knowledge my standard there is
I want to see a statistically
significant increase in knowledge among
those who participated for the second
evaluation question was the HIV Qi
program associated with higher MD
guideline adherence in this case I want
to see a statistically significantly
higher level of guideline adherence a
after participation and in this case I
want a higher level of guideline
adherence among those who participated
in my program compared to people in
another comparison Group which also
means I have to collect data from both
groups which we'll talk about more next
time the third evaluation question was
the HIV Qi program associated with
improved physician satisfaction in this
case you don't have to stick to a single
standard you can have more than one um
so in this case I want want a majority
of the Physicians to report the program
as mostly to highly acceptable that's
part of my thinking about implementation
evaluation as well and I want those
Physicians that were part of the program
to again have a statistically
significantly more satisfied level of
reporting than comparison Physicians and
then lastly my gold standard my my
overarching impact goal is that in fact
this program was associated with a
change in the volume of patients with h
disease getting highly active
anti-retroviral therapy and there I want
to see a statistically significant
increase in the percent of patients
receiving that therapy within the first
couple of years of the program
so I wanted to make sure you have those
kinds of examples in hand now let's say
we go through to the other end and you
have to decide now whether or not the
evaluation questions have been answered
you've gotten your statistical stst
excuse me statis St istically
significant results I would and you're
going to go celebrate but I'd say it's
still critical to lift the lid on those
findings and this is also where there's
value of expert and stakeholder input in
assessing evaluation evidence and that
we'll talk about in part three in the
meanwhile I want you to consider all the
results that you get has a single
outcome of say four or five outcomes
demonstrated statistical significance or
is it really a mixed bag of results
where something worked and some things
didn't or it worked for some areas and
not some others I would say then ask why
it did not work as intended for all the
outcomes to guide your next steps I
would also think about why did some
outcomes improve much more or much less
than others and what does that mean for
the program or the policy does it make
sense for the program to have had the
outcomes noted sometimes we see programs
have incredibly strong outcomes in areas
we weren't necessarily expecting that's
worth exploring further are there
subgroups this is really important are
there subgroups for whom the program
failed to work if yes what can you in
fact do about it can you talk to those
for whom it did not work and can you use
that evaluation information to help the
program further tailor itself or are
there statistically significant
improvements that are simply not so
clinically or programmatically
meaningful that's most important in in
really long Pro large programs Medicare
snap these things that have tens of
thousands or hundreds or thousands or
millions of beneficiaries and program
participants where um the sample size if
you will is so large that a tiny smidgen
of a change drives statistical
significance but it's not enough for
anyone to get hugely excited about
unless it's on the path to bigger
changes over time perhaps and lastly
were the analyses done in a way that
minimized bias and were key other
factors that could have driven outcomes
adjusted for appropriately now in this
particular series we won't be getting
into the statistics but again here's an
opportunity to bring in others to help
you understand whether or not these are
issues in your
evaluation and lastly I want you to
think about the information that's
needed when and in what forms I for
shadowed some of this earlier in today's
talk but again it's best to predefine
what you need when based on agency and
stakeholder needs the common information
that's needed here is the program
description its purpose its goals its
setting and context this is information
we talked about last time is not being
evaluation but it's still important
information you often need to start
talking about program Pro progress with
especially if it's a new program or
changing
and that's where we think about process
evaluation the organizational chart the
documentation of procedures and policies
the Staffing the services delivered uh
we also often have to report on
information about the characteristics of
those who participated in the program
which we'll talk about a bit more in
part three and of course everyone wants
to know early outcomes if they were
collected and of course post program or
later program outcomes and any changes
that were had and here's two that often
aren't mentioned um off the top and that
is it's important to know something
about people who were lost from the
program meaning they started and didn't
continue because that's useful
information on either something else
that needs to be done to retain them or
something they didn't like about the
program that you may need to think about
changing to ensure Equitable access to
the services that were going to be
provided and maybe a summary of the
challenges they faced could be delivered
as well and then you need to be thinking
about unintended consequences good and
bad along the way this is a lot of
pre-thinking to do about what you would
need from an evaluation and the kind of
evidence but it is always necessary
because I have also been in many
situations where after programs
completed we simply don't have adequate
information based on what people learned
along the way uh hence why this is group
and team processes the format for these
may vary as well think thinking about
briefings briefings slide decks
infographs visual abstracts each group
of stakeholders may need something
different so we often need to be
thinking about just as I talked about
how often we communicate it's what kind
of information our different
stakeholders and agency groups need
along the way so with that I'm going to
um conclude that portion and make sure
we have time for Q&A thank you so much
Dr Yano for such a good presentation we
do have a few questions we'll get to as
many as we can those that we don't get
to we do plan to have all of the
questions and a sort of questions and
answer document that we will post along
with the recordings and the um slides
and things like that our first question
is are there stakeholders that you would
recommend not including on the
evaluation team outstanding question um
I would I would pose it a slightly
different way and see if I get to your
interest
um I would ha say that the evaluation
team needs to participate and engage
many stakeholders but they may not be
fully on the evaluation team at the end
of the day the evaluation team has to be
objective and be able to deliver as I
said earlier good and bad news so you I
have noticed in some participatory
evaluations where you get to know
everyone in the program so much that you
have to be careful to not let it bias
the answers you're about to deliver um
and so it is one of those circumstances
where you want to have people
participate and engage in steps along
the way but not always be an actual
member of the evaluation team um there's
also the perspective of um making sure
that there are some I'm thinking about
the if there are any other situations
where stakeholders there would be people
you would not want on the team I I think
if anyone's unwilling to hear bad news
you may need to hold them at arms
length thank you the another question is
are there ways to ensure the program
goals are culturally competent and I
think we will put a more um thoughtful
and complete answer in writing but if
you could just touch on this Dr Yano
we'd appreciate it yeah I another really
exceptional question I think all of this
pre-planning talking about throughout
this um is is in some ways Insurance
about making sure that there are
explicit opportunities to make sure the
program goals and the evaluation goals
are culturally competent I often press
on people to think about every single
evaluation question and think about its
implications for stakeholders its
implications for groups that are
marginalized or underrepresented
uh and to think about how we're even
posing questions and surveys all of that
work requires
intentionality and time and making sure
that the right people are at the table
that to be quite honest poke me and poke
others to make sure that we're we're
framing these things appropriately
measuring things appropriately including
additional measures we might not have
even thought about before um and so it
it needs to be an explicit part of this
entire plan
thank you and we do have other questions
but we're running short on time so we do
want to put a poll here to get your
feedback because we are evaluators so we
want to evaluate our performance here
