Project Description: Digital data are now widely recognized as valuable assets―research resources that have tremendous potential for reuse in new and innovative ways. As recognized by Jim Gray over a decade ago, and the natural history community decades earlier, realizing this potential will require ready access to extensive bodies of curated research data. Advances in the management and archiving of digital data are proceeding apace, but curation services need to extend to identification of high-value data and enhancement of data to fit new purposes. Through our studies of data practices across more than a dozen sub-disciplines in the earth and life sciences, and associated data representation and identity problems, we are uncovering indicators of reuse value and conditions of fit. This presentation shows key research outcomes are directly informing data curation education and practice and discuss our next phase of work leveraging strategic collaborations with scientists, data centers, and repository developers: the Site-Based Data Curation project, with geobiologists and resource managers at Yellowstone National Park, and the Data Curation Education in Research Centers partnership with the National Center for Atmospheric Research.
Speaker Bio: Dr. Palmer conducts research on fundamental problems in the use of scientific and scholarly information and teach courses on information behavior, scientific information practices and problems, and user study design. My program of research is about mobilizing information for researchers, and it focuses on two interrelated areas: information work in the research process and context-rich digital research collections.
I'm also from the other side of Green Street, so very nice to be invited and have an opportunity
to talk a little bit about our perspectives on data.
So what you're going to hear is a little different because we, our approach is we actually,
our object of study are data and the people who create and use them.
And so just to put this a little bit in the current national context,
many of you who generate your own research proposals know we now have requirements
for data management plans that have to go with proposals.
There are many, that's with NSF, NEH, NIH, all the various agencies are falling into place here.
But in addition to that, there's a lot of effort in this area.
So there's a recent Office of Science and Technology policy memo that came out of the White House
in February where 65,000 people, scientists, publishers, members of Congress,
and the public signed on to say, you know, we want access to not only the publications
for research but the actual digital data.
And so there's a lot of movement in this area.
Federal agencies are now being, going to be required to have clear,
coordinated policies for increasing access.
So our job is really to understand how we get there.
How do we get communities ready to share their data because they're going to be expected to.
So I know this concept isn't familiar to everyone,
so I just wanted to give you a quick background.
Our definition of data curation is the active and ongoing management of data
through its life cycle of interest and usefulness to scholarship, science, and education.
There are many definitions out there.
Wikipedia has happened to pick up ours, so it's getting a lot of play.
But I really like one from the Digital Curation Center in the UK.
It's really focused on the notion of fit for purpose,
curation being about making sure data over time are fit for purposes.
A lot of people associate the term with Jim Gray from Microsoft Research
who was working with the Sloan Digital Sky Survey group back in 2002.
They did a nice paper on the issues of preservation of that huge digital object and resource
and talked a lot about the necessary role of really capturing all the possible metadata,
including project designs, documents, discussions, procedures, software, everything else.
But in fact, the notion really dates back to science and natural history museums
and the curation of objects for study in science museums.
You can go back to a paper in Nature where the curator of the British Museum of Natural History
is starting to allude to this need for really big data,
what they considered big data then,
and trying to manage it and make it useful, justifying the investment in it.
At DOE in 1994 in one of their first bioinformatics summits,
when they're talking about bioinformatics,
really invoked that metaphor of curation from the museum community
and data curators to be the new profession for playing this role.
And then there's some other very nice papers that have come out of the museum community.
So while we've done a lot, in fact we've been involved with curating
the Slowest Digital Sky Survey recently,
most of our research right now is on what's called the long tail of data.
So there's been some predictions that the long tail is going to be two to three,
or is two to three times larger than big data,
the stuff that comes off of instruments and is generated pretty readily.
The claim is it's a lot easier to handle, understand, and archive big data
because it's more homogeneous.
And in fact, small data or the long tail actually has been a huge investment.
So finally, we've been saying this for a long, long time,
one of us finally, when he was a program officer,
was able to do an analysis of all of the proposals over a given year
to show that 80% of the proposals were really single PI, small group science,
not necessarily generating any big data.
And it's sort of under this $300,000 range.
Not as big in total dollars, but a lot of projects, a lot of people, a lot of data.
So when we think about curation, or at least this is sort of the groupings
that I've come to associate very macro level,
you hear a lot of talk now about curation in order to retain the scholarly record,
in part for reproducibility.
A lot of this gets talked about in terms of climate modeling and climate change.
And it is often thought about as data that's directly associated
with the production of papers, models, other products.
But in fact, there's a lot going on and a lot of interest in data
in terms of it as research capital.
I think about this as producer value,
and institutions are getting more invested in being centers of excellence
for certain kinds of data, for certain kinds of research.
And then also, even more so, interested in just the raw materials of research.
This is the consumer value of data,
and we're working very closely on some of this
to develop repository services for data sharing and data reuse.
So the two projects I'm going to talk about and just introduce you to
are the Data Conservancy, and that is sort of closing up,
although there's some new work going on there,
on site-based data curation by Yellowstone National Park.
So the Data Conservancy is a big collaboration.
It was one of the five awards that was through the NSF DataNet program.
It's part of the cyber infrastructure for 21st century.
So about $9 million for a national scale infrastructure for this project.
It's led by Sheridan Libraries, at Johns Hopkins,
and the PI is Sai Chowdhury.
I was a co-PI, and there were also co-PIs in Cornell,
NCAR, and Marine Biological Center.
So the Data Conservancy is basically a community
that develops solutions for data preservation and sharing
to promote cross-disciplinary reuse.
And so for us, when we say preservation,
we mean not just saving the bits and the bytes,
but really requires providing enough representation information,
context, metadata fixity, et cetera,
that someone other than the original data producer
can use and interpret the data.
So we have two teams at Illinois working on this project.
I'm going to talk mostly about the data practices team.
We do qualitative studies. We're social scientists.
We're studying the long-tail complex heterogeneous data,
reuse value, and things like that.
Second group, data concepts group, is led by Alan Renier,
who I didn't know was here. He's back there.
They're studying formal terminology,
identity conditions for data sets,
versions, et cetera, representation levels, and such.
So over the course of this project,
we've been investigating across domains,
and we've been covering earth and life science.
So you see the ones up there on your left
are what we covered at Data Conservancy.
But in fact, a previous project allowed us to bring
other domains into the picture to get a better sense
of the comparative analysis that we were doing.
And again, you might not be too familiar with
these kinds of more qualitative social science approaches,
but we had to...
You know, it's really hard to get scientists
to talk shop about their data.
They take it for granted. It's very mundane.
You know, they want to talk about science research questions,
which we enjoy, but, you know, we want to get down to the data.
So we had to come up with some techniques
that really get us there quickly.
So because we don't want to waste their time,
we get a couple hours,
sometimes follow them around if we're lucky.
So we start now with lead scientists, PIs.
We get in with them.
We start with a pre-interview worksheet
to get them focused on a project
and certain types of data that we're interested in.
We move then to semi-structured interviews,
and then we do follow-up sessions with particular people
who tend to be very reflexive on their data.
And then we move on to people
who are actually working with the data day to day,
the researchers managing the data.
That's often post-docs or research scientists,
graduate students.
They understand a lot more about the stages,
the versioning, the standards, the tools that are at play.
And so then what we do, you know,
we write our own papers and do our deep comparative analysis.
But in the meantime,
we want to push as much as possible
back to the infrastructure team at Johns Hopkins
where they're building infrastructure
and interfaces and all of that.
So we set up a rapid reporting framework
for doing that where we would analyze data very quickly
and push these key observations back
that had implications for metadata, policy,
other things that were going on.
Now, as you can imagine, you know,
we are challenged by the fact that, you know,
we hear a lot of things that we have to interpret.
So we hear scientists say all the time,
of course I'm willing to share my data publicly.
And, in fact, sometimes they do,
but they have one bad experience, and then they stop.
They say things like, there are no standards in my field.
No data standards, no metadata standards.
Well, we know that's not true,
but from their perspective, it's true.
Or, my data will never be of use to anyone else.
We hear this all the time,
but yet we can go somewhere else across campus even
and see an opportunity where data could play very nicely
for answering the questions there.
So one of the early observations we've made
is most of the forms of data
that scientists are willing to share
are not the ones that are most valued for reuse.
So a lot of what we're doing now is
how do we get scientists working in a way,
get their workflows in a way
that they can be producing reusable data.
So we want data that can be used for raw materials of research,
that's of application to other fields.
At the same time, we have all these stories about,
and more worries than we have for real cases,
but worries and concerns about misinterpretation,
misappropriation.
We do have cases of, you know,
good faith efforts not being followed
when data aren't shared.
So there are real concerns,
a lot of legitimate ones,
but a lot are worries that we're not so sure
are going to come to be true.
So again, we were talking, as I mentioned earlier,
we're interested in both producer and consumer sides,
so we've been looking a lot in early years
of the data conservancy on utility for producers
and looking at small science where it's very complex.
This often means bringing together
lots of different kinds of data into some kind of unit
that they feel is essential for understanding the science
and for reuse of the data.
So in the serious geobiology that we've been studying,
we call these units site-specific time series,
and they tend to include things like spreadsheets,
which themselves are aggregates of many different kinds of data,
but also microscopy images,
lots of photos, lots of annotations on the photos,
microbial genomic data, et cetera,
that's just a subset.
In volcanology, another area we've looked very deeply in,
we call their units rock profiles
because they start out with the physical rocks
they get from the field,
they do sampling and sectioning of that,
they then digitize those,
but they also have chemical analyses,
photographs, field notes, et cetera.
And then just to note on the bottom,
there's no sharing conventions for any of these
in these particular communities.
It's often by request,
usually after they've already made their publications
or made their stamp on it,
and no real repositories where they feel their data can go,
except for the genomic data.
So again, for us, as trying to educate and do research
about what the actual curatorial tasks will be,
we map out what parts of those data need to be put together,
in what ways, how they need to be preserved,
how they need to be related, what the provenance is.
These are the kinds of curation tests that you can't read,
probably, but the identifiers, the linking,
how we provide access, how we manage provenance, et cetera.
But in fact, the other thing we've really looked at is
where do we see the utility for the consumers here?
Because it's a great, huge investment
to curate those very complex sets
from the perspective of the producers.
At the same time, we're getting evidence
that parts of those data are very valuable
to other communities.
So utility for consumers is a very important issue.
So what's the most reusable component?
Well, right now, it looks to be those chemical analyses
in the spreadsheets, for instance.
So we see lots of evidence of the isotopes,
the iron, the titanium oxides.
We don't care about them, but we know others that would.
And that, in fact, the geochemical data does not,
necessarily, the same kind of reliance on the field context
that we see in geobiology and local knowledge, for instance.
So the real challenge here, of course,
and that we're trying to get a handle on
is anticipating what those other communities might be
because you have to curate it
in a way that it's actually usable for them.
So tracing this, you know, we know that
with the Geobiology Time Series,
we know who the designated communities are
in the OAIS sets,
the original communities.
And then you come to learn more
as we learn more about the use, et cetera,
about the potential communities.
Now, once we have big repositories with lots of use,
we can track these things easily.
But right now, we have means of sort of tracking
the potential users.
So chemistry, evolutionary biology,
bio-prospecting is very big with some of this data.
The pharmaceutical companies, in particular.
The U.S. Park Service has a great interest
in a lot of the geobiology data.
This is a real case where microbial data
was, it wasn't the original intent of the data,
but was able to use to find the presence
and extent of disease in certain animals
and the risk associated with that.
Same with volcanology.
We know the original designated communities,
but then glaciology was able to take field photographs
and assess basiotemporal glacier change over time.
Lots of interesting results on climate change there.
So, emerging questions for us are,
what do we know, what data do we invest in,
and can we anticipate value over time?
And so, thinking about that glaciology example,
our colleagues at National Snow and Ice Data Center
say this is a classic example
where the glacier photo collection,
which 10 years ago, nobody knew about,
and no one thought it was worth investing in digitizing it,
is now their second most popular data set.
And they have some very popular data sets.
All the AP journalists go there.
But they also have noted, it's more anecdotal at this point,
that there's these 30-year peaks of use.
And so, they've been sort of tracking, in a way,
we have some data that we can analyze
and hope that data will go in,
and it's used a lot when it's first released,
but then about 30 years later,
it becomes high interest again,
Other really big questions,
so our colleague at the National Research Council
wrote a paper that we like a lot,
but he makes this claim,
the value of data increases with their use.
Well, we like that idea,
but we also wonder, is this true?
If it's true, how do data actually gain value through use?
Can we document it?
So we sort of tried to go after that
with some of the data we'd collected
from these particular communities.
We were able to go back to the climate of soil,
volcanology, sensors, stratigraphy,
and sort of get a sense,
try to get a sense of what are these real indicators
of value based on,
again, this is from the producer perspective.
So the first three, not so surprising, I guess,
reputation of data collectors,
spatial coverage, longitudinal coverage.
But the next set is really interesting to us.
Unique conditions,
areas that are rarely studied,
politically volatile areas, or data,
and data that requires permitting
in order to get access to it.
So very high value, a little more scarce.
But the next sets are really important as well.
Multiple sources for triangulation and context,
and then data with documentation of workflows.
So these are things that we know are...
And then we went to try to really see,
think about this idea of gaining value.
So can we see when and how data gain value through reuse?
And this is just three little scenarios
that we were able to track.
So the first case is where we saw richness
in verification through reuse,
re-users getting complimentary evidence to bring to bear.
So ocean modelers working with field campaign data,
going out and trying to find additional data
on the weather during the particular flight patterns
of the time, actual satellite serial numbers,
looking for irregularities in open sea mooring.
And as they do that, they're again building these,
refining the data, but also bringing other data sets
that have important complimentary.
You see other cases of refinement and fit.
This is a group of sensor engineers
that transformed a set of data several times
for different audiences.
So reworking water measurements to share
with search and rescue teams, triathlon organizers,
fishermen, and even the ship industry.
So lots of reworking of the data for different audiences
and versions that are of interest.
And the last one we don't understand quite as well yet.
This is sort of a case of recalibration and feedback.
Some rainforest researchers had gotten some block sensor data
and wanted to know more about the original measurements,
went back, got the instrument level calibrations,
and then sort of packaged that up again
and sent it back to the original providers,
which is kind of a nice feedback there.
So for us at the Data Conservancy,
there's some lessons here, as we know more about the value
and what happens with reuse.
So what are the implications for the data services
they're trying to produce?
Well, you might want to actually recruit data
with multiple indicators.
At minimum, you can sort of assess what's coming at you
and set priorities based on that.
You might want to anticipate and track
long-term reuse cycles, because it really takes 30 years
to get the improvement in the data.
It might be worth waiting for and catching it when it happens.
Promote reuse to increase the value,
and then you have to find ways to capture
and represent the reuse as it improves.
Okay.
So I wanted to take a few minutes
to talk about the next project we're on.
I just got back from the first workshop
we held with the stakeholders in this project.
It's called Site-Based Data Curation
in Yellowstone National Park.
It actually grew out of the data conservancy work
with the geobiologists.
Co-PI is Bruce Fouk on this campus
in geology and microbiology.
We also have a co-PI at Yellowstone National Park,
Ann Rodman, and Saeed Chowdhury of the data conservancy.
And here we're just really focusing
on developing a framework of policies and processes
for curation of research data
generated at scientifically significant sites,
and Yellowstone's our model site for that.
And then within Yellowstone,
systems geobiology is the exemplar case,
in particular beginning by starting on the microbial data
because more than 50% of the data,
the permitted data,
that comes out of Yellowstone happens in microbial.
So this is important to us.
It's a really nice intersection of earth and life sciences.
Yellowstone's a mecca for geobiology data collection.
The people doing research there
are studying everything from the origins of life on earth
to life on other planets.
And what we realized later, after the fact,
is that there are the multiple data value indicators
going on with these data.
Longitudinal coverage, very unique conditions,
politically volatile, and permitting requirements.
Not everybody gets to go.
Scientists want to go year after year after year.
And lots of multiple sources for triangulation and context.
So very important for both Yellowstone
in terms of their resource management.
So they want to be investing in strategic science,
but they don't have good ways of tracking what happens
over many, many years.
And the researchers themselves know
that many people collect data at that particular hot spring
or that geyser or that particular event at that hot spring,
and they want access to it.
It's been going on for decades and decades,
and they don't know where it is,
but they know what they can do if they can find it.
So we're there to help them get there.
So to get a sense of what we're talking about,
this is just a description of one scientist's
hot spring data parameterization.
So rock deposits, water, microbes, climate.
All of them, interestingly enough, include.
And after this workshop, we've really come to realize
where we thought photography was probably low priority,
we now think it's really high priority
in terms of if we're worried about context,
if we're worried about bringing data together
to help people understand context,
it's going to be high priority.
And at the bottom, it all includes modeling,
and it's then everything in between to get to the modeling.
Am I running short on time?
Well, I cut out a few slides
because I thought we were running short,
but I know I was also asked to maybe talk
a little bit about our education program,
so I'll do that.
The research that we're doing in these two and other projects
is feeding directly into our specialization in data curation
in our master's in library and information science.
We started that with a grant in 2006 in the sciences,
and then got a parallel effort in humanities.
And so we've been graduating students since about 2008
in specializations in data curation.
They're going to data centers, research centers.
They're also going to research libraries, large museums.
They're going to the private sector.
They're going to lots of the public sector.
People are going to the FBI, to city governments,
public broadcasting, those sorts of things.
So everybody needs to take care of their data
and make it work for them, right?
There's a big demand here.
So we got in the business at the right time.
But I want to talk in particular about the most recent effort
in what we call the data curation education and research centers.
This is a collaboration with the National Center
for Atmospheric Research,
who is a partner on data conservancy.
And we're also partnering with another iSchool,
which is at the University of Tennessee.
And we're taking our core curriculum
in the specialization of data curation.
It may be available across the schools.
It's a distance program, so it's easy to do that.
But the biggest component, the thing we're most excited about,
is the field experiences for the students.
So we're putting master's students there
for seven weeks for summer internships.
We're putting doctoral students there for two semesters
to really look at the research problems coming out
of these big, big state-of-the-art data management centers.
And the best thing is, we were able to pair students up
with two kinds of mentors.
A data mentor, someone who's involved
in the day-to-day data management,
these very large, you know, things that everybody...
we're talking about today,
even some of the big climate models and all.
But in addition to the data mentors,
we're also pairing them with the science mentors.
So someone who's there actually conducting science on those data.
And again, this is how we've constructed our program.
It's very much about how you are partners in research
with scientists, you know, working alongside them
in order to get up to their...
you know, help them achieve their scientific goals.
So NCAR is helping us do that with the internships as well.
There's a video you can watch
that explains a little more about this.
So internships are very important.
They're difficult to arrange,
but we've been doing a pretty good job.
These are the internship sites,
the practicum sites we've had for students over the years.
Not every student gets this opportunity,
but we have many more.
We have students willing to go.
Most students, you know, these graduate students,
don't necessarily want to pick up and move somewhere away from here.
But as often as we can,
we put them in scientific centers
or, you know, big libraries or big research centers,
or even small ones, when it makes sense.
So these are the kinds of positions that students are getting.
As you can see, they haven't come together
in any kind of real uniform way.
As was mentioned, I'm on the National Academy of Science study committee
that's trying to understand the career trajectory
for these kinds of roles.
There'll be a report coming out in September, we hope,
that will try to explain, you know,
what's happening in science,
what's happening in digital and data curation,
you know, where the convergence needs to happen
in informatics and elsewhere,
so that we can get a little more coherence
and better, broader education.
Okay, so I'll finish there,
and I'm happy to take any questions you might have.
So, yeah, so the levels...
Oh, I took that slide out.
Yeah, so in fact, the workshop we were just in
was helping the scientists understand...
So NASA has levels, you know,
levels of processing.
So it starts from the raw data all the way on to the models,
and there's four levels in between.
We try to get scientists to understand.
You know, they just think of it as, you know,
I transform my data, or I process my data,
and which one do we keep, which one do we preserve?
You know, we try to get them to think about
what's the most shareable, usable one.
And so part of this workshop
was exactly trying to get them to stop thinking about,
I need to produce this table for my publication,
to I need to submit a data set that's coherent
and well-described that can be used by someone else.
And so in oceanography and other areas,
we've seen the communities develop their own levels
of understanding how to take those levels
and which of those are the most shareable.
Okay, any other questions?
So we have, you know, one last thing.
So before we do that, I need to step out.
I need to find out who won, because I don't know.
Right, so give me a second.