Zen and the Art of Evidence Building: An NIH Story | November 12, 2024

USDOHAHS
156 followers
1 Video View·Nov 23, 2024
This training will focus on improving evidence processes through tracking annual evidence activities by highlighting NIH’s work creating the Strategic Tracking and Reporting Tool (START) platform. Ajay Vatave, MD, MPH, NIH Assistant Director for Tools and Analytics, will walk you through core components to consider as you think about building your own processes and tools to track evidence building activities on a regular basis.
--
U.S. Department of Health and Human Services (HHS) | http://www.hhs.gov
http://www.Twitter.com/HHSGov | http://www.Facebook.com/HHS http://www.Instagram.com/HHSGov
http://www.LinkedIn.com/company/us-department-of-health-and-human-services
HHS Privacy Policy: http://www.hhs.gov/Privacy.html
so thank you everyone for making it to
our training our capacity building
training on Zen and the Art of evidence
building an NIH
story um with AJ vat and comfort Kai
from NIH let's move to our next slide
please all right so our presenters
wanted to start to today with we're
using slido um to do some polls
throughout today's presentation and
while you all are feeling that out I
will go ahead and do some brief
introductions so today uh as noted we
have AJ vat who is the assistant
director for tools and analytics within
the National Institute of Health office
of evaluation performance and Reporting
and in his role at with opr Dr vat is
facilitating the develop Vel M of the
evaluation and learning agenda for NIH
and supporting nih's performance
reporting he's also leading the effort
to create the Strategic tracking and
reporting tool or start a digital
platform designed to enhance nih's
ability to collect and report on
administrative activities start
streamlines The Collection tracking and
organization of information to inform
progress on key goals and objectives
across the agency start supports nih's
goal goals to exemplify and promote the
highest standards of scientific
Integrity public accountability and
social responsibility in the conduct of
science Dr Rat is also a Public Health
Physician with nearly 25 years of
clinical Technical and managerial
experience we are also lucky enough to
have Miss Comfort Kai who serves in
nih's office of
OE um I
guess office of UPR uh where she is a
member of both the Strategic tracking
and rep supporting tool team as well as
the performance and evaluation
team uh in her role Comfort oversees
Communications operations for the start
team and provide support for client
engagements and data infrastructure
designs on the performance and
evaluation team she AIDS in coordinating
NIH level performance monitoring and
Reporting activities for nih's programs
policies and operations so without
further Ado let me let the team take
over and give their presentation to us
today thank you so much great thank you
for that and thank you for all those who
were able to jump in and respond on
slido and with that Comfort Could you
actually take us to the next slide I'll
just comment on the fact that we're
bringing slido because we want this to
be interactive um to a degree and as it
relates to being interactive feel free
to drop things in the chat questions as
we go along uh folks from asby will flag
them for me and I'll try to respond to
them uh the best I can so with that
before I dive in just want to kind of
get a sense not only where you were from
which was what the first question was
about and now also just kind of get a
sense from what perspective you're
coming from do you have experience in
evaluation are you doing things like
performance monitoring is it ledge is it
policy you having those types of
informations when it's available be
useful for me I'm kind of gearing the
discussion for
today and as the results come in I'll
just mention a lot of what talk about
today you know is really about bringing
things back to the basics um when it
comes to evidence building and I think
what we found in our journey is that
really supported us both in the short
term and really the long term in terms
of reaching some what I consider to be
fairly high aspirational goals um of
what we're trying to do with our
platform known as start so it looks like
we have a good amount of folks with
evaluation and different levels of
program experience and evidence building
so that is great comfort can you jump
into the next
slide so what are we going to try to
accomplish today to try to understand
what our approach was um in terms of
tracking evidence and building
activities I'm really going to focus a
lot initially on the tracking sheet that
is added as an attachment I believe to
the invite and then you know kind of
share what we thought about you know in
terms of our processes what we've
learned and kind of what we're experienc
as we continue to grow in this space
next slide
okay good question for us to start with
here again one more slide on this prob
be the last one for a bit but this one's
important um this one's important
because I'd like to understand if you
are all starting at a similar place that
I was and my office and my team that is
doing all of this work which was to say
when I was thinking about where we were
in the evidence building space it felt
very
disjointed disorganized you know
underresourced um and I didn't know
that's maybe just unique to our
organization um I suspect that it's not
and if it's not I want to make sure I I
kind of capture a message for all
audiences that are maybe better resource
than we were in particular spaces and
those who maybe were at an equal place
than we were or even in some cases maybe
starting somewhere a bit more
challenging than that yeah so I'm seeing
a lot of terms here that I connect with
for sure scattered insular is another
one working on it this idea
of what it looks like you know growing
nent tricky tricky is a good one I will
be getting into
that
cautious give it a couple more moments
here developing yep we're right there
with
you cool all right these are very
similar themes um that I think that we
connected with as they continue to
populate and among others um and I'm
sure if we sat down looking at this
thinking about all the barriers that
we've all experienced in trying to
advance you know premises like
evaluation performance monitoring
evidence-based decision making there's
any number of issues and hurdles that we
need to overcome in that
space frustrating is a good one let's
end on frustrating next slide and I
hopefully what I showed you in the rest
of this talk will
help at least Identify some Pathways to
kind of work through that frustration in
fact that there are cases where it's
possible to do things that I think at
the outset a lot of people would think
is
impossible next slide
Comfort all
right bridging of data silos okay one
last thing aspects of evidence building
do you find most challenging um again
this will help me a little bit in terms
of what I lead into in particular when I
go into actually the platform so as
you're populating
this uh two aspects of our presentation
today we'll be working through this SL
and series of slides we have connected
to it and then we'll pivot away for a
little bit I'm going to drop you into a
testing version of our platform so you
can all see what we've been doing and
then we'll come back to kind of bring it
home together and this will give me a
good sense of you know the issues and
these are some of the ones that we faced
quite frankly and that's why I'm kind of
getting a calibration from you all that
we were thinking about when we were
undergoing you know our approach to
evidence building and what we were going
to try to kind of need to work through
at at NIH and it's no surprise to me
that I see data silos and and kind of
leadership Buy in close to the top or at
the top here again those are areas that
we are are very well familiar with and I
I will also be talking about
that the database of evidence is
interesting I'm going to talk about that
that's a lot of what we are doing um
with start and I'll have some thoughts
on on how we're able to get there and
recognizing some folks might not be able
to and that goes kind of hand inand with
monetizing data collection one thing
I'll flag for you in these results which
was interesting for us is that reporting
burden is last and I'm going to get into
a lot of discussion about you know the
ways that we sort of championed our
caused and evidence building and how we
sort of Switched our approach on our
messaging angle um to advance you know
evidence building and I will tell you
reporting burden you know or well I will
just say the concept of Staff burden you
know was something that we used a lot in
our discussions in terms of making the
case for doing all the work that we did
which as you'll see is a pretty
impressive body of work all right next
slide so why I start next slide so we
said frustration right it was out there
right on the slido that was one of the
things so just some background our
office is a fairly new office at NIH
it's about four four to five years old
it's kind of a combination of a couple
offices that worked either in the
performance or kind of policy based
spaced and when we came together we
talked just about that we were dealing
with an approach of evidence-based and
collection Gathering that involved
sending out Excel spreadsheets all over
the place and just having hundreds and
hundreds of excels that you know some
fellow was typically you know creating
pivot tables for and then getting staved
on some share Drive where it sat like
Gathering dust by itself right like we
were in a place where we knew we were
doing and we were being responsive but
it just felt like we weren't really
doing things in the best way that we
could and we were fortunate in the sense
that we were able to bring together I
think a really good group of people to
think through like how we might go about
this and I think we had a group that
consisted of folks who had been at ni
for a while folks who had been their
kind of mid-career and some folks who
were in news so we had a combination of
fresh ideas of kind of insightful
perspective of being at an agency over
time to really think about you know who
are the power Brokers for instance what
are the messages that you know resonate
the most with folks so we really wanted
to Galvanize all of that and I happened
to be you know at that time probably the
person tapped with kind of leading a lot
of this effort but truthfully everything
that I'm going to talk about even though
I'm talking about it from my perspective
it is a combination of the work of
people on my direct team like Eileen Oni
and comfort who's on there and Trisha
chaker forthy and folks within my office
as well who were more numerous to count
and folks within NIH so my perspective
here is coming from my contributions in
that lane and one thing to know about me
that was mentioned is I started my whole
career or training in medicine and so
for me it makes a lot of sense of why
ultimately fell into kind of
organizational learning as I think about
it and metrics and evaluation
we've all gone to a clinician right and
there's any number of things that a
clinician could measure when you know
you interact with them but we all
basically do the same thing initially
right we're going to talk to you we're
going to kind of get an understanding
from your perspective of what's going on
we're typically going to check your
pulse we're going to check your heart
rate we're going to check you know your
breathing frequency and that is to say
that in medicine it's similar to
evaluation and performance monitoring
and risk management these other Concepts
you can measure a lot of things but you
have to choose to meure me those things
that are the most accessible and that
can lead you to more informed detailed
measurement down the line you know I
might see that your pulse is abnormal so
maybe I'm going to send you for an Echo
for instance echo cardiogram I just
don't do it right out of the gate and I
think one of the things that we thought
about at NIH that the same is true for
here you know we could really get lost
in a maze of measurement and have so
many measurements down the line it could
get really overwhelming and we really
wanted to try to focus on measurement
that had meaningful action behind it
something that would initiate something
or that was responsive to something so
really getting in front of it next
Slide the other aspect to know about me
is I kind of dipped in and out of
government I was at CDC originally I
went into the private sector and then I
came back into the federal sector at NIH
and one of the things that I realized in
that private sector was one that
measurement is really powerful in
decision- making in particular in the
private sector you know when you're
looking at things like revenues and
growth expansion you know it's really
useful to figure figure out where you
need to Pivot however at that same time
I realized why a lot of these models
were failing in government because in
the private sector a lot is measured
around people time you know how much
they're billing how much there is in
revenue and as we know in government
it's not typically like that right we're
working on these far Rising goals we're
trying to cure cancer we're trying to
decrease the frequency of a particular
disease and we don't really measure in
people time and so when I was coming
back into service I felt like I had a
better perspective to kind of make the
change that I wanted to be a part of
which is combining the best of civil
service with what I saw in the private
sector next
slide so we're going to start here this
is actually the basic basic and it is
part of your takeaway this is actually
coming from our colleague on our team
Eileen oi who brought it over from her
former Institute we call it the
accountability matrix and you're going
to see that I'm going to focus on this a
lot at the beginning because later on
I'm going to get into the tech the tech
part of what we're doing may not be
accessible to you all yet just because
of where your agency is at from a
funding level from a buyin level this
this should be approachable to everybody
it's with Excel and I'm going to
hopefully in this discussion convince
you that this simple Excel can be
extremely aggravating sometimes but
really worthwhile and it can actually
lead to some pretty amazing things when
you're able to be in a position to
translate into a database next
slide all right so this table that we're
leaving with you I think I want to just
highlight some of the principles these
aren't necessarily new principles but I
want to talk about them in terms of how
they've helped us us get to where we
went and that is ultimately what we're
doing with start you know one of the
first things that we do with evidence
building is we help people think through
this table and one of the first things
we want to establish is how do you want
to organize right and you can organize
in a lot of different ways typically for
us in evidence building we have found
that strategic plans make a good anchor
because they tend to capture the vision
um it can be a really challenging
process though once you've laid out your
organization I have applied this Matrix
to my own team and had the fortune of
working with a great evaluator Janelle
banks in her office and she asked me all
the great Annoying evaluation questions
that someone's supposed to ask what do
you mean by that statement AJ how are
you going to measure that statement AJ
like why are you doing that instead of
this AJ like it sounds like it's easy
and a lot of the things on this Excel
look on the service to be easy but I
promise you once you get into it
sometimes it's not and that's okay
because a part of evidence building is
really about kind of rewiring some of
the ways that we think about things and
in particular when we're Consulting with
folks on how to do an evaluation who
might be coming from a program space for
instance it can be a challenge you know
kind of really thinking through what
they
mean forward when
Comfort after we
organize it's also important to identify
who is responsible we have found in our
approach when you kind of generalize too
much and you don't give ownership to one
person it's really easy for the wheels
to fall off and I think when it comes to
evidence building it's important to
communicate you know your approach to
evidence building it's probably equally
as important if not more important to
identify who is responsible at any
aspect of it and without it it's really
hard to ascertain you know where the
information is coming from where you can
make change happen who you reach out to
for more context and you know the idea
of anchoring responsibility in
everything we do around evidence
building I think is pretty Central to
what we've been able to accomplish board
one when should this be done so beyond
knowing who beond knowing the vision how
we want to
organize what parameters around time
right are there Dimensions can we
capture it and this is really important
because if we're thinking about evidence
building along the lines of progress
monitoring risk management you know
what's our Horizon what are we looking
at if we're thinking about evaluation
you know perspective what time
dimensions are we working with so to the
extent that you can capture time and
having to accept that sometimes time
could be infinite some things are
ongoing to the degree that you can
capture or think about the parameters of
time of the types of things that you're
working on incredibly important for one
please what do you know seen this far
too many times you know we get
galvanized it's one of the things I love
about Civil Service we're quick to react
we're often very responsive I think
that's part of the job in terms of a
cycle of of different administrations
coming in and knowing that there's lots
of people with new questions that need
to be answered and we often rush into
being responsive and I think sometimes
at least we have found at NIH that can
bleed over into other ways that we go
about our work and we don't often spend
enough time thinking about what we know
and how do we know it and where do we
know it from all that type of thinking
about kind of thinking through what your
Baseline is is really important it's a
step that folks often skip over not
because they mean to it's just again
because we've kind of gotten a habit of
being in this react nature I think
within the government to a degree
forland what do you want to know I think
all of us are on this call for a reason
in terms of belief
in evaluation um as a discipline in
terms of what it can bring to the table
along with other Concepts like project
management and risk management and
performance
monitoring but I think what we saw more
and a change that I think we're starting
to see happen because of our work is
thinking about what you want to know
upfront you know when we first started
and came together as an office I think
it's fair to say that most often we were
asked to evaluate a program that had
already been in place for 5 years or an
evaluation plan I do think and I do feel
that with what we've been doing as a
group and what we've been doing with the
platform we've been building that
Dynamic has changed we're actually
getting a lot more people talking to us
saying hey we're designing this program
we're planning to implement this what
are some ways that we can think about
this and again this Excel spreadsheet is
a really good way of helping people
connect with the fact that we're doing
things for purpose and it's really
important to be able to identify you
know what questions you want to answer
and by extension how you're going to
answer it you know in that exercise I
just mentioned that when we applied this
to my own team Jamal rightfully pointed
out like how can you know this and the
answer was I couldn't like I felt the
work we were doing was going to
contribute but there was no way that we
could measure it and so we took it out
and so that's another thing when you're
going through this process there may be
some things you choose to take out from
an Evidence building capacity it doesn't
mean that it's not important it doesn't
mean that it might not be you know an
attribute that improves because of your
work it just means that it's really hard
for you to discern in any meaningful way
that your resulted in that work and when
you're prioritizing big lifts such as
building evidence building capacity
within an agency you're going to have to
make
choices forward one
please so that created this whole Vision
around Star right like those kinds of
problems that we were thinking about
that kind of organizational structure
and I'm going to grab the screen here
real quick and then we're going to do a
little quick
demo all right I believe everybody can
see let me move you all here
here great so one of the things that we
wanted to do we really wanted to
centralize this process we felt there
were a lot of pillars which I'll get
into when I pivot back to the PowerPoint
that we wanted to accomplish but I want
to show you like how that simple table
or that way of thinking that that table
captures you know the ways of thinking
about bringing evidence together can
actually translate into a database so
this is start and our test sit so don't
get too long up in the data some of it's
actual some of it's test but that stands
for tracking and reporting tool
something's about start what did we talk
about organization you know we provide
Pathways for people to align information
that exists so you're seeing that right
here who who's responsible who's someone
that is going to be the person I can go
to that when start goes out and has a
data call request for instance that I
know they're going to have that
information when we have time can we get
time let's get time if we have it even
if it's ongoing or if it's unknown let's
say that let's understand that
the basics of what do I know right what
are we trying to do here what's the
purpose where are we going with this
trying to think about those things
upfront getting folks to really
articulate that around their work is
something that we really wanted to bring
to the table in sort of this database
design leveraging those exact
tables and when we do that we get to do
a lot of fun cool techy stuff right then
when we do that so you can imagine we're
just living on a sandwich of all these
Excel spreadsheets essentially we're
going to be able to at our activities
and how they align with our vision and
our goals what's that distribution look
like when it comes to our products we're
going to be able to do the same thing
you know how do that distribution look
what are we doing in these places really
quickly and when I want to get details
it's not a problem it's sitting here in
a database that really allows me to kind
of understand the intention of this
information and in this case what we did
is we bought an information from another
source so again thinking about ways to
leverage evidence we made an alignment
to that source and we help people
provide some additional context around
that Source something that was in Lay
language something that we could
translate that science to say hey we
know you funded this this is why it's
important and this is what we did and
you can imagine that as you start
stacking up this kind of information
across a larger database how powerful
that can
be here's another way maybe because
we've got good thoughtful design up
front we're really building out our
excels we can set a threshold for what
we consider to be low resource along our
activities and then we can identify what
these green low resource areas are and
can start drilling in and we can start
exploring ing like what is going on here
what's actually happening here let's
give our leaders let's give our decision
makers let's give our program folks more
information to work with to make better
decisions and so here I can see sort of
what this funding table looks like I can
see these projects that we have and I
can see what our funding art looks like
do I have to worry I'm not sure oh if I
come down to the narrative I can see
here look they got another 3 375,000 so
looks like there's still a shortfall
here but maybe the problem's not as bad
as what I thought but there's a
shortfall right a way to make
information actionable is incredibly
important and it's really important that
people as part of an evidence-based
building culture act on that evidence
and that is something we are actually
seeing at NIH with what we're
producing other things once we start
bringing in this data we have
opportunity to create conversation so
this is working again off a fairly
similar Excel spreadsheet this is an
amalgamation of our investments and
software applications that are looking
at extra mural or extra mural portfolio
17 seven what are the opportunities for
co-investment here what are the
opportunities for someone to say hey we
could sunset this system and actually
use this system what's the opportunity
for learning here so there's more to
evidence building than just proving a
point evidence building is also about
creating conversation about exploring
and understanding how we can learn
better as an organization and again by
layering our approach and by being
methodical in our approach you have the
ability to translate all of those types
of excels into a database like
this and an offshoot great you have all
this data working for you not only is it
helping you kind of bring that data
together it's helping you to synthesize
that data in ways that are summary that
are bringing all kinds of information on
where things are being highlighted in
the media how it's aligned to the
Strategic plan acoss important topic
areas and if you have to go to another
topic because you're going to the hill
real quick great okay we'll flip to that
we'll start using that data again to
really help us out and really stay on
top of things as they're progressing and
as our priorities change we can continue
to build these out and scale these out
with the different types of data that
we're bringing in
but let's not stop there that's that's
not fun so here's some sneak peek stuff
so we've got all this administrative
data coming in it's answering a lot of
questions for its intended design but we
think we're also bringing a lot of
information that's worth exploring let's
not keep that separate from our
ecosystem let's bring it into our
ecosystem let's create a whole
evaluation resource within this
ecosystem not going to go into too many
details here because I think in the New
Year you'll get a full detailed
discussion on this from the folks who
actually created it which was Jamal
anden in our community of practice um
but lots of great stuff here right so we
have all of that kind of happening in
the background and now we're saying hey
this is evidence building let's give a
place for resources for training let's
actually do more than have static PDFs
about the importance of evaluation let's
actually start building things in and
tools that people can use at different
points in time to execute their
evaluations and let's do more than that
let's actually meet people where they're
at let's understand that evidence
building is a series of events happening
over time and over that time they need
different things let's give it to them
in all inone place maybe you're building
a team great let's give you some
materials that you can Source out let's
give you some tools let's give you some
support on training and let's carry that
all the way through to the extent that
we can where we can and when we can and
let's also recognize the fact that some
of our best work isn't even occurring by
feds it's occurring by the folks who
work with us so let's figure out how we
can access those resources as well and
make it available for folks to Think
Through what these challenges are and
how to overcome them and how about we
not only replicate this for evaluation
maybe we can do something similar for
strategic planning for risk management
for
performance and let's go one step
further now that we have all these great
ideas coming together built off of these
simple excels if you will now we can
improve our transparency of one data
needs to be updated who needs to do it
whether you need to do anything at all
whether you're Institute office or
Bureau what's your data what does it
look like here you can access it maybe
you want to know what other folks are
doing you figure out that there's these
Open Access models that might have a lot
of interesting things like AI for
instance and you want to be able to know
what's going on with new things so let's
have a place for people to explore right
Comfort go ahead and grab it back I will
stop
sharing all right so let's talk a little
bit about in my mind again I'm seeing a
lot of people what it took to to make
what I just showed you happen and again
some of you know me some of you have
seen that if you want a deeper dive into
to start the tool feel free to reach out
to us we are happy to talk in great
detail about what we're doing
there but when it comes to evidence
building and what start represents there
are a lot of challenges you're going to
face whether you're going for something
as complicated as what we're doing with
the database or whether it's simply
leveraging the Excel that we're
providing or some form in between I
think there's a lot of challenges
that you should be prepared to face and
it's okay because you can overcome them
and I think we're a good example of that
one you have to be prepared for the fact
that this may be a bit of a solo Journey
where you have to wear a lot of
different hats IR irrespective of the
number of people you have working on
this when it started out I was you know
someone who knew about evaluation
performance monitoring I had to start
getting into database design now we're
getting into security you know when
you're starting out in these spaces it
is often the case that you're going to
have have to juggle a number of
different roles and that's okay that
just represents the fact that this is
complex and there's a lot of different
partners who are involved in this space
that you need to be thinking about and
you need to go in with a little bit of
knowledge you don't have to have all the
knowledge you just need to know when
your knowledge stops and you need to be
able to bridge with your community to
broker that knowledge with folks who
have the
expertise the other thing is really
understanding your pitch angle and what
I mean by this is I've been at this
evaluation evidence game for a while now
and you know most people at a high level
if you talk to them about you know the
reasons to do evaluation they will
generally connect with it but different
audiences really prioritize different
things for leadership it might be the
ability to have information quickly and
you know be responsive for staff who are
reporting in it may be that efficiency
and reducing Staffing burden is actually
the primary thing that they're most
interested in and so on and when you're
doing these evidence building activities
for an organization so essentially
organizational learning you have to
understand that you may need to focus on
a particular pitch angle for a different
audience because that's what they're
really going to connect with they are
going to get the whole picture generally
but they really want to see you know how
what you're talking about really fixes
something that's a priority to them the
other thing I think that most folks will
will typically face is a lack of
leadership buying and we saw this in the
slide end and I think this is
understandable right you know one if you
look at what we're doing oh my gosh it's
another new database design it's the
latest greatest dashboard why am I doing
this the other part of it is a practical
reality for evidence-based
decision-making that leaders are coming
in you know throughout the government at
different Cycles they are often put in a
reactive situation of needing to know
certain things and so they don't have
time to really think about some of these
more institutional in kind of
infrastructure types of improvements
that need to happen around evidence and
that's okay it's just important to know
about that going up front going into
this and there's ways to address it
which I'll touch on later
then lack of funding and resources never
be enough in this space I I don't have
enough now I didn't have enough then
you're going to start with not enough
money and not enough people and even
when you do feel like you're making
progress you're still going to feel like
I don't have enough money and you don't
have enough people that's just sort of
the way that it's been for us and I
think one of the things about that is
just being aware that that's probably
going to be the case and so it may need
to modulate your ambition right you may
need to prioritize a little bit
differently in terms of how you tackle
things um my team knows me to be someone
who looks to you know kind of get
through any wall that is in my way and
one of the ways that we dealt with both
lack of funding and resources we
branched out we decided that this was
going to be a community-based platformed
and our platform is now sourced across
the entirety of NIH through an idiq
which we manage and people sort of
invest in what we do we actually bring
in expertise and form relationships and
Partnerships across the NIH to really
realize the intellectual property and
capacity that NIH staff have and by
doing that you know we've been really
successful next
slide so how do we tackle the chaos
pitch angle I've already mentioned you
know this is really talking about
thinking about your messaging and
understanding you know what you have to
do I was really lucky in the fact that
our first division director really
pushed us to create a business plan he
really wanted to know that we had
thought it through and I literally wrote
I can't remember how the first draft was
probably like a 25 page document of what
this was all going to be about but going
through an exercise like that and really
understanding the different audiences
and where you're trying to you know make
progress in change is really important
championship of the people I can't tell
you how important the people are you all
know this many of you work in health
agencies Public Health agencies care
delivery
agencies the community is just as
important in internal change as well you
know a lot of times when we didn't
necessarily have the leadership Buy in
you know on a onetoone level they are
office to the OD Office of the director
one of the things that really helped us
continue to make progress is we always
had support of the community when they
weren't hearing about it from us they
were hearing about from someone else I I
kid you not I went into one meeting and
my director said AJ I'm giving you this
money I don't know why I'm giving you
this money but everybody is telling me I
need to give you this money and so there
really is the power of the community you
know and part of that is making them
feel part of the process and being
responsive to their needs and I'll talk
more about that in a moment last Point
confidence
you know it's really important that
whether by individual or as a group of
individuals I'm lucky that I have a
group of individuals both within the
team in the office who I think are
incredible Champions and that are
confident you know there's going to be a
lot of push back on on a lot of
skepticism on some of these things or at
least that's what we found and it's okay
you know people need to to see it to
believe it and we can't be shaky in our
belief in my view that bringing evidence
building to the modern era um and really
getting it going is worthwhile and yes
there's going to be it challenges and
yes there's going to be user challenges
and yes there's going to be funding
challenges but we always I will say in
the four-year history of us as an office
as a team have projected confidence that
this is the right thing for our agency
to do right and we have unwavering
belief in that and sometimes truthfully
that unwavering belief has helped people
kind of come with that say okay these
people are going to keep pushing forward
I can be a part of that next
slide and kind of what it took to build
us so for us where did we start so we
always imagine start kind of resting on
these four primary pillars it has
expanded beyond that um but we started
with strategic planning evaluation
performance monitoring and risk
management now this graphic is
interesting for a number of reasons if
you look at it from top to bottom this
is actually how we're progressing and
there's a reason for that strategic
planning we felt provided us really good
organizational Frameworks for evidence
building it's something that's within
our office's domain as well it's also
something that a lot of folks have it's
something that is approachable so it was
a great starting point for us evaluation
is where we're at now you just saw some
of the evaluation module which is going
to be coming out in the new year and
from there we're going to build on to
Performance monitoring which we have a
sister platform but we'll plan on kind
of supercharging what we have there
based off of what we've done and then
finally you see risk management which
doesn't actually sit with our office
there's actually another functional
office NH that does risk management but
for us it makes sense that information
works well with what we're trying to do
with start and so it made sense for us
to include that you know in our decision
of how we're going to go about building
start but you can see here we're making
choices we're not doing all of this at
once we spent the first few years really
focused on strategic planning we're just
now starting to Pivot into evaluation
and the next year or so we'll start
pivoting to Performance monitoring so
these pillars are important to identify
because it gives you a big vision of
what you're shooting for but but you're
also like I said going to have to make
choices because it's very likely unless
you're a well-resourced office and I I
think typically most folks in our space
aren't um you are going to have to
choose what you focus on next
slide impact and Innovation lots of
things that have come from this right we
talk a lot about what we're doing from a
modernizing perspective we know that
there was like modernized HHS at some
point this is actually truly modernizing
what we're doing but not only in the
platform but also in our way of thinking
and I think that's another thing that
you know our office has done a really
good job of and our team has done a
really good job of that modernizing
comes in lots of forms it's modernizing
of our culture it's modernizing of our
data it's modernizing of our
infrastructure and it capacity lots of
aspects and there's a lot of great
stories to tell when you're building
evidence capacity around
modernization
standardization tricky issue tricky
issue especially for a group of
evaluators probably who are wedded to
standards and would like standards I
think if you're in an agency that can
pull off something Sim ilar to what we
did and out of the gate maybe whether
it's due to regulatory Authority you
have some level of standardization to
bring to the table we would encourage
you to do it if we could build start
over again and we had a blueprint of
standards that we could work with uh we
would do it but the truth is we didn't
but that doesn't need to stop you when
we started working in building start we
accepted that we didn't actually know
how everybody wanted to measure things
and we intentionally allowed a good
variation in how people approached
measurement in the first three years now
this year moving forward even though it
is more work for us we are in a much
stronger position to Pivot towards more
standardization because now we can show
people this is what you've been
collecting it's very similar to this
let's agree that this is what we'll
collect on say collaborations for
instance we are able to have those
conversations because we've worked with
people to collect data in the ways that
they thought they needed it which
sometimes showed them it wasn't the way
that they needed it or maybe the way
that they thought would they were really
unique actually wasn't all that unique
compared to how other people were
thinking about things so standardization
can occur upfront it can also occur like
Midway point but it has to occur
probably at some point to really go to
scale you can operate at a smaller scale
with a little bit of you know lack of
standardization but to go for the scale
that we're work looking to work at you
know we are starting to move in that
direction efficiency I mentioned this
about sta burden all great data
collection systems will fall apart if
they are a pain to work with so we have
lots of fun dashboards to look at lots
of great reports that can save someone
six weeks of time to develop a first STP
but this is a constant hunt for us to
improve ways for people to interact with
our system whether it's dropping in Word
documents or Excel spreadsheets whether
it's how we create our interfaces so
people can update data we have to
remember that part of evidence building
is a human experience a human experience
to generate that evidence and to the
extent that we can make that part of the
process approachable meaningful it's
really important and more importantly
probably than anything else that I would
say in our evidence building Capac
it's really important to get a feedback
loop to talk with these folks what's
working for them what's not working for
them and then to take that feedback and
actually do something about it so you
saw that Ico landing page that I just
showed you in the evaluation model that
comes straight from our community saying
hey we like what starts going there's
some improvements but we would really
like a better centralized landing page
where we just had everything laid out
for us so we could access things and our
ability to continue to bring people on
this mission that we've instilled about
building our evidencebased capacity
really become informed decision makers
really increasing transpar transparency
it's all about creating these feedback
loops to continue to improve things as
we move forward next
slide so summary Lessons Learned
Community matters right I talked about
it at multiple levels whether it was
from our funding whether it was from
being Champions on our part it's never
too early to start engaging your
community and it's never you
know an end to your engagement our
Community is expanding when we first
started this work our community was a
planning and evaluation group of folks I
would now include our community like
nihit folks working in artificial
intelligence our community has expanded
and we have expanded with them and that
is just part of this understanding that
to do these big types of organizational
changes on learning and evidence
building it really does require a
multi-disciplinary community to bring
their different angles to these
challenges because it does take a lot
resilience there's going to be lots of
times where you want to bend there's
going to be lots of times where you may
just want to give up because there's so
much bureaucracy to kind of work through
but it's important to be resilient so
this work is hard and like I said even
my team we do this 247 365 days a year
and when we were applying it to
ourselves it was incredibly hard um we
got through it but it wasn't like the
easiest thing and we do this every day
so you need a certain level of
resilience to kind of carry through it's
easy to stop halfway way push through
it's kind of like going to the gym you
kind of don't want to go but once you go
you're like oh I like going to the gym
expect
obstacles I already mentioned this there
is stuff out there you know your agency
better than I do some of you may have a
leader who is you know really empowering
data driven decision-making that is
amazing take every opportunity to
leverage that others you may not it's
okay to have obstacles obstacles allows
you to plan and I think once you
articulate and take the time to think
through your obstacles a good chunk of
them you can probably think through
strategies of how to navigate those and
more importantly you'll probably think
through like how to prioritize those
strategies and one to implement
them other thing ambitious work takes
time like we're four years in five years
in roughly I will say I'm more than
pleased at the the level of change that
I think I feel like I'm seeing within
our agency I think when I think about
how the M Community is really connected
uh with what we're doing even if they're
not using start even if they're just
using that Excel spreadsheet to kind of
really lay out what they're trying to do
with their strategic plan or their
program it's been incredible um to
really see the amount of cultural change
that we've seen in the last few years
but even still we have a long ways to go
and we have always had a group that's
committed to understanding that this is
something we're just going to have to
chip away at and we're always going to
have to prove oursel each and every
stage of this as we continue to scale
and that's why conviction matters it's
never over in terms of proving yourself
in this evidence base once we started
having this kind of centralized you know
data collection strategy the next
question is why why don't you have ai
you know even when we have ai the next
question is well why don't you have that
there's always going to be these
evolving demands in terms of of what you
need and that takes a certain level of
conviction because at some point it's
easy to say come on haven't I done
enough but the thing is you're getting
people excited even if they're not
saying they're getting interested
they're getting engaged and that's why
they're asking you these questions
that's why they're asking for more
that's why they're pushing and so having
that conviction to kind of push through
all of that and understand that yeah you
know every year you may have to prove
yourself of why evidence building at
your agency is important okay it's a
challenge that I think we all have to
accept because it's important that we do
it last piece educator
mindset so we're always educating you
whether it's ourselves right I just
mentioned right now we're getting
involved in AI I'm a Public Health
Physician I don't know anything about
artificial intelligence but we know that
there's really smart people at NIU do in
other places and so we're reaching out
and we're starting to get educated on
that we also do a lot of educating with
people we walk them through that Matrix
that I showed you and that you'll have
is available to you and help them think
through that to lay out their ideas to
lay out what they're thinking about we
also educate people and you know
database and how we use start and where
we're going with start there's also
educating the educator so the people
that we first were educating two or
three years you know I was probably out
there talking about start all the time I
don't do that as often because the
people who are now using start are out
there educating people how to use our
tool effectively and sometimes quite
honestly that may mean a lot of upfront
work for folks it may require a
Workforce to really think about what do
we mean by this how do we know that some
cases it's easy but it's really
important because that it goes back to
the central point it's all part of the
community and this is the way it cycle
backs to the community and so everything
you know for us at the end of the day
even though we have Excel templates even
though we have a fancy database that
we're continuing to evolve it's all
humancentric people are at the middle of
all of this because we are reflecting
people in our organizational learning at
all levels that it's occurring we're
just providing a better pathway you know
galvanizing the intellectual property we
are lucky we work with tons of amazing
people as I'm sure you all do and we're
just providing that conduit to help them
realize a lot of the things that they
would like to see happen with our agency
which are all things I think that we
generally care about we want to know
we're doing good work we want to know
that we're making progress we want to
know that we're doing making a
difference when something's not right
we're going in there and we're making a
change when we find things that are
working we're replicating those things
to the extent when we can when we find
that they're not working we're
broadcasting it to help people not go
down the same rabbit hole and we're
finding new ways to explore and to
innovate and so with that last
slide I am happy to take any questions
and I'll just end by saying this you
know humble beginnings Excel spreadsheet
not super complicated can lead to some
pretty amazing things um if you have the
opportunity to to take in that direction
but even if you don't those Basics
really matter because when it comes to
evidence building it needs to happen
vertically and horizontally across an
organization and so much of that is
really about helping people think about
things differently than they maybe
traditionally have so with that I am
happy to take any questions
thank you so much
AJ uh while we wait for people to Pew up
their questions and for those who have
questions please feel free to go ahead
and unmute yourselves or put your
question in the chat um could you remind
us how long it's sort of took you to
move from your initial sort of
conceptualization
start to where you are now for instance
yep so for us the timeline was roughly
four years the first year was not much
development it was a lot of you know
finding one partner to work with us the
second year was mostly about two or
three partners with working with us and
then years three and four have just kind
of been the barn doors have blown out
and everybody's coming to work with us
and it continues to scale at a fairly
incredible pace
and when when you started this work was
it did it um and this is probably a an
ignorant question but for those of us
who are like at the at the way back at
the starting um did it essentially start
with like an Excel spreadsheet of sort
of like this is what we want to do or
were you already starting to like you
know kind of program
things yeah no it actually did
start with honestly just a general
concept um we sketched out a pretty good
business plan we had a layout of what we
thought made sense to us from a basic
structural component but we knew we
needed to talk with folks and our first
iterations were actually inspired by the
community not by us I mean that was
intentional we didn't want to come off
on a position to sort of dictate top
down we kind of wanted to work bottom up
because we knew going into this we
weren't quite certain I will say this we
had no dedicated team for this we had no
mandate to do this we had no leadership
mandate to do this we frankly didn't
really have a budget to do this I did
Cobble together about $10,000 or so on
an existing contract we had to kind of
create a prototype and that's you know
what we had but for the beginning of the
year that year one I mean I I always
laugh I I kind of went around with a two
or three deck PowerPoint that that's all
that I had as my resource to work
with
nice um see a question what do you see
as your next step for a start so yeah
it's a big it's a big year for us that
Ico landing page and evaluation module
that I showed everybody briefly it's
transitioning us from being more of just
a data collection vehicle which is what
we were first known for like people are
oh my gosh this is so much better than
just excels that we never attach like
you know there's ways to do assignments
but now we've gathered so much data and
now that we're centralizing you know a
way to really get at this data and
explore this data we're going to be
becoming much more of a resource where
people are going to want to be looking
at that data in other ways than maybe we
originally intended right it's data
that's related to someone strategic plan
or some program and kind of thinking how
we evolve to as for start to be a
resource um is a big piece of it another
thing I would say
is now we kind of know the database
design we actually need to have versus
the one we have so at some point we'll
probably be doing a bit of a a re
architecture based off of everything
that we've
learned and then
third you know
what we'll be incorporating soon and we
have some promising initial stuff that
came today how do we incorporate
artificial intelligence and we think
bringing those three things together
kind of becoming a better resource for
folks by incorporating artificial
intelligence to kind of explore our data
in other ways than initially structured
you know those are going to make some
huge improvements and advancements in
kind of what we bring to the
table um so you know from from the
position of from ASP position in terms
of doing sort of the annual kind of data
calls related to let's say the
evaluation plan for instance are you
able to use start currently for those
sorts of things uh like when we ask for
you know what are your most significant
uh evaluations for instance for you know
upcoming in
2027 yeah we could yeah we if if there's
a consistent data structure that you all
are using to to call it out and I'll
defer to Jamal and the evaluation
Community NIH you know they will look to
leverage that evaluation module when we
have ongoing evaluation needs from HHS
we'll standardize those and put them in
to start and that'll just become part of
our evidence building
capacity
thanks I want to hug all of the
questions okay any other questions that
folks can go ahead and drop in the chat
or verbalize again I know it's a lot to
take in it's a big system and it was a
pretty ambitious project and it still is
a pretty ambitious project um I will say
and this is a plug um and I'm happy to
reach out with you afterwards at any
point I don't think what we have done
needs to
be confined to
ni I truly believe that although there
may be Evolutions or adaptation of what
we're doing or maybe kind of
incorporation you know into what people
already have this is something that
could scale um for other groups to try
out it does what I will say even with
the areas that we need to focus on and
improve and strengthen it does work um
I've been both at CDC and NIH and I've
had some details in some other places
and some of the challenges that I know
are here I know exist to other
agencies um so again to the extent that
that's of interest to folks and they
would ever like to follow up feel free
to reach out to me and I'm I'm happy to
chat for sure
all right I'll put in a last call for
questions oh could you please put your
email in the chat
a I will you have an interested
party yeah and that I am always creating
interested parties in this space because
one of the things that you know I'll
just maybe close with this and left more
questions come
in you know
really to tell our best
story from my perspective in terms of
the HHS
story we need to better tell how our
story relates to one another like our
basic science that we produce how does
that gets translated in sort of the
implementing Wings how does that
translate into the public health Wings
you know each of our work we can point
to um in terms of it's important but I
think every agency gets strengthened by
the fact that we're all interconnected
and I think the more the American public
understands the importance of our
interconnections and how we do our work
and how it benefits not only the people
of this country but quite frankly
globally that can only be a good thing
um and so I have a personal interest in
thinking of ways to do that and I'm not
saying you know our approach is the only
way to do that but we are trying to put
our foot on the path of at least
starting the conversation you know that
hey we could do something like that
let's try it
out I agree thank you for that um that
is always I think I think that the
promise of um you know the Evidence Act
is really to pull that story together
and to help us better um plan and
execute the work that we're doing and to
um really be able to as as you put I
think so well tell the story of what it
is that we're actually doing and that
it's not just sort of like throwing
spaghetti at the wall but that we're
we're actually following your recipe and
uh you know putting together a a meal if
you will um so thank you for that a
really appreciate your time and comfort
your time as well today um yes we uh we
have another comment in the chat yes we
will be sharing the slides as well as a
um a form that AJ and comfort have put
together for us uh that shows us how we
can start thinking through how to put
together our own sort of uh data
collection tracking form and uh a has
shared his email and then if you would
be so kind we also have a very short
survey for you to take um about our
training today so without further Ado
thank you so much everyone for your time
today and we look forward to seeing you
in about three months for our next
training all right thank you so much all
and have a great rest of your day thank
you take care everyone all right bye
bye-bye
