Paper/Presentation Description: This paper examines the challenges and opportunities that "big data" poses to scholars advancing research frontiers in the social sciences. It examines the strengths and weaknesses of machine-based and human-centric approaches to information extraction and argues for use of a hybrid approach, one that employs tools developed by data scientists to leverage the relative strengths of both machines and humans. The notion of a progressive, supervised-learning approach is developed and illustrated using the Social, Political and Economic Event Database (SPEED) project's Societal Stability Protocol (SSP). The SSP generates rich event data on civil strife and illustrates the advantages of employing a supervised-learning approach in contrast to conventional approaches for generating civil strife data. We show that conventional event-count approaches miss a great deal of within-category variance (e.g., number of demonstrators, types of weapons used, number of people killed or injured). We also show that conventional efforts to categorize longer periods of civil war or societal instability have been systematically mis-specified. To demonstrate the capacity of rich event data to open new research frontiers, SSP data on event intensities and origins are used to trace the changing role of political, class-based and socio-cultural factors in generating civil strife over the post WWII era.
Speaker Bio: After completing his doctoral work in political science at Northwestern University, Professor Althaus joined the University of Illinois faculty in 1996 with a joint appointment in the Political Science and Communication departments. He is currently a professor in both departments, and also associate director of UIUC's Cline Center for Democracy, where he has been a faculty affiliate since 2004. Professor Althaus's research and teaching interests center on the communication processes by which ordinary citizens become (in theory, at least) empowered to exercise popular sovereignty in democratic societies, as well as on the communication processes by which the opinions of these citizens are conveyed to government officials, who (in theory, at least) must transform the will of the people into political action. His work therefore focuses on three areas of inquiry: (1) the processes and constraints that shape the journalistic construction of news about public affairs, (2) the processes and constraints that influence how news audiences receive and utilize public affairs information, and (3) the channels of communication that allow individual members of a polity to speak in a collective voice as a public. He has particular interests in the quantitative analysis of political discourse, opinion surveys as channels for mass communication and political representation, the impact of strategic communication activities on news coverage and public opinion, the psychology of information processing, and communication concepts in democratic theory.
Professor Althaus serves on the editorial boards of Critical Review, Human Communication Research, Journal of Communication, Political Communication, and Public Opinion Quarterly. His research has appeared in the American Political Science Review, the American Journal of Political Science, Communication Research, Journalism and Mass Communication Quarterly, Journal of Broadcasting & Electronic Media, Journal of Conflict Resolution, Journal of Politics, Public Opinion Quarterly, and Political Communication. His book on the political uses of opinion surveys in democratic societies, Collective Preferences in Democratic Politics: Opinion Surveys and the Will of the People (Cambridge University Press, 2003) , was awarded a 2004 Goldsmith Book Prize by the Joan Shorenstein Center on the Press, Politics and Public Policy at Harvard University, and a 2004 David Easton Book Prize by the Foundations of Political Theory section of the American Political Science Association. He was named a Merriam Professorial Scholar by the UIUC Department of Political Science and the Cline Center for Democracy (2012-4, 2010-2), a 2004-5 Beckman Associate by the UIUC Center for Advanced Studies, and a 2003-4 Helen Corley Petit Scholar by the UIUC College of Liberal Arts and Sciences.
I want to talk about some of the challenges we've been having in trying to overcome some
significant data problems for understanding the world around us.
The problems begin by recognizing that there are two types of conflict that affect the
course of nations.
If you're thinking about the rise or fall of nations, conflicts between them, conflicts
within.
On the left side is wars between nations, right, big armies, navies, and so on.
On the right side is conflict that happens in a much smaller scale within nations.
This is a picture from 1963 of a Buddhist monk self-immolating to protest the Republic
of South Vietnam's policies towards Buddhists.
These are two forms of conflict at very different scales, but they can have equally important
and substantively important results.
So if we look over the course of the last half century, from World War II on up, and
we look at the number of major militarized conflicts all around the globe, in every country
in the world, we see that the level of conflict overall, or the number of conflicts, has been
rising over time.
What very few people realize is that these bottom categories represent across-country
conflicts.
The larger gray scale represents within-country conflicts, civil wars, and other forms of
lower-level conflict that are very destructive.
So most of the kinds of human conflict in the recent past have been gravitating towards
within-country conflicts.
We want to understand that because these kinds of conflicts are very lethal.
So these are estimates of the global casualty rate, deaths caused by these militarized conflicts,
over time, from 1946 to the late 2000s, and there have been a lot of people killed in
these conflicts over time.
If we disaggregate by the two types of conflicts, it's a complex pattern, but you'll see that
more people are getting killed today in these within-country conflicts than in the between-country
conflicts.
They're becoming rarer, the within-country conflicts are becoming more lethal.
So back to our two types of conflicts, well, it turns out there's a data problem that's
wrapped up in trying to understand these conflicts, and that is that the across-country conflicts
are very loud.
They make a lot of noise in the world system, governments talk to one another, the international
news system courses with information about them, it's easy to document that they are
starting, they have ended, and so on.
The other kinds of conflict are very quiet, and it's often very difficult to even know
that a protest has happened if you're not located in the local area where it has occurred,
where there's some informational coverage of that kind of conflict.
So for instance, when Condoleezza Rice, in the opening days of the Bush administration's
rollout of its plans to invade Iraq in 2002, when Condoleezza Rice gets on TV and says
what she needs to say, there's a great deal of information about exactly what that means,
the consequences for other countries in the world.
But when this lady gets up and wants to have her say on things as well, we're much less
likely to hear about her, or why she's doing what she's doing, or what cause she's advocating.
We just don't know.
She's angry, we know that, but we wouldn't even know that if we weren't located in some
level of physical proximity to where she was most of the time.
So how do we find out about these various kinds of conflict?
Well, through the news.
But the news presents particular problems for trying to trace the ebb and flow of these
conflicts, particularly the lower resolution within-country civil unrest kinds of conflicts,
because of the problems that Dan introduced in his presentation.
This is unstructured text, there is a great volume of it, and it's very difficult to pull
out just the kind of information that we want to pull out about these small-scale civil
unrest events in order to generate a larger picture of what's going on in the world, how
they're happening.
So the Kline Center for Democracy has, over a period of several years, invested a great
deal of time and money to pull together a unique archive of global news coverage, representing
every country in the world.
The top two sources are open-source intelligence operations that have been run by the American
and British governments since World War II, and we are looking at the most recent components
of these two.
These are setting up listening stations in every country in the world, translating news
broadcasts from those countries at a very local level into English, using native speakers
of whatever languages are in these countries, who are culturally resonant with the country,
translating that into English, bringing it back to Washington, D.C., it's used for short-term
intelligence work and then probably largely forgotten.
So we've been pulling this together as a data science problem to try to extract the information
that we'd like to know about these civil unrest events out of this huge volume of news data,
and not just textual data, but we're also dealing with the historical records of these
open-source intelligence operations that are on microphone and microfiche images, and we've
got segmentation problems and a lot of other challenges to overcome.
I'm going to be talking today out of a data set, a demonstration data set that we pulled
together out of the entire population of news stories in the New York Times from 1945 to
the mid-2000s.
And so when you've got volume of data on the scale that we're talking about, the next question
typically, if you've decided to analyze it, is do you use machines or do you use humans
to do it?
And the social sciences have been wrestling with these problems for quite a long time.
In fact, the first efforts to use machines to work with unstructured textual data in
the social sciences were back in the 1960s, and so there have been lots of efforts to
try to see whether machines or humans can do a better job on problems like this.
We aren't, at the Klein Center, coming out on one side or the other.
We see machines and humans as having relative advantages.
And so the machine-based systems, the kinds of systems that Dan was just talking about
for us, are very good at processing a large volume of data quickly and accurately, but
processing what social scientists would call manifest content in that data.
So manifest content would be an example of a, does the word demonstration or protest
or strike appear in a news article?
It does or it doesn't.
A machine is going to process that question using a rule set or any other way that you
want to choose much faster and more reliably than a human would do.
The humans would be good at the beginning, but they're going to get tired and need a
coffee break and so on, and they're going to lose accuracy.
So machines are very good at processing manifest content.
Humans tend to have a relative advantage in processing latent content, where the meaning
is not just right there in an obvious yes or no checkbox fashion, but along the lines
that Dan Roth was discussing, there's ambiguity, there's variability in the reference that
could be used.
But what's funny is, even Inspector Clouseau can figure out these ambiguity problems that
are taking Dan Roth's group into the upper stratosphere of research on trying to run
this through machine-based systems.
We're used to dealing with that kind of ambiguity all the time.
The problem is we can't take that knowledge to scale that the machine-based systems could
take.
But the relative advantages of humans over machines, I think, can be summarized pretty
succinctly in just observing that humans get the joke, okay?
So here's the joke, you ready?
One person says, my dog has no nose, and the second person says, how does he smell?
Awful, says the third person, okay?
And with all due respect to Monty Python, not only do humans get that that was a joke,
but we also get that it wasn't a very good joke, all right?
It was a clever wordplay on awful, we get that.
Machines would have a very difficult time.
So we want to leverage, at the Klein Center, the ability of humans to disambiguate context
very easily, because the context that humans bring to them, and not dependent on just what's
in the text, but to merge that with the scalability and reliability of machine-based systems.
So we have been working on what we're calling a progressive supervised learning system.
So you've probably heard about supervised learning systems, and the standard way of
implementing them is you get humans to come up with a gold standard set of classifications,
and you feed those gold standards to a machine system, you fine-tune it, you optimize it,
and you come out with a classifier.
So the human input is at the beginning, and there's an optimization process, and then
you run the machine system from there.
The system that we're employing is a hybrid system that brings humans and computers together
interactively over time, and so this is a complicated diagram, but our process starts
with having some repository of documents, of these articles, which get fed into a classifier.
And in this diagram, the circles represent just repositories or databases, the squares
are automated machine-based processes, and the triangles are human-based annotation processes.
So we send these things into a classifier that makes a decision for us whether this
particular news article is likely to contain information about a civil unrest event or not.
Is it relevant? Is it irrelevant?
With the New York Times data, 85% of the articles are kicked out immediately, not relevant at all.
15% move on through our system.
We then send that 15% through a series of natural language processing modules that extract
entities, that identify dates, that come out to geocode any location references, and so on.
That then, metadata, along with the original article, gets sent to a repository, a document store,
out of which some subset is selected for further analysis and sent up to human coders who are
equipped with machine systems in their interfaces.
So this is through a web interface, and machine systems will, for instance, present a list
of known entities and organizations and ask, is this particular event related to this entity
or that entity, both of which show up or are named in the article.
So the humans help to correct any errors that might be part of the machine-based process,
but having the machine capabilities may maximize the efficiency of the human coder's time
with the article.
Human coders ultimately discard some of these articles, know they didn't contain any civil unrest
event data, or keeps them, they did, and there's additional coding and annotation of those done
by the humans that are not currently within reach of the machine systems to do.
But then this is what makes this a progressive supervised learning system.
We then take the output data that the humans have produced, and we send them back to produce
new test training data sets to refine our classifier, and this can be done in pretty much
real time as well, so our classifier gets better and better as we move on.
But we also go back and correct out the natural language processing part of our component.
Maybe an entity was given wrong or a date was extracted incorrectly, and we can modify
those algorithms also in real time to be able to refine that.
So the idea behind a progressive supervised learning system is that machines do what machines
can do best and do well.
Humans do the subset of tasks that need to be done, but the machine capabilities are not yet
at the point where they can substitute for humans, but the humans are trying to work themselves
out of a job, producing more and more test training data to get the classifiers to operate
more and more efficiently.
Eventually we move the humans off to the next hard task that the computers can't do,
and that is a progressive supervised learning system.
So let me take you through some of the benefits that we've seen at the Klein Center in trying
to come up with this new generation of civil unrest data for which there is no central repository.
There's no structured data set out there that we could turn to, so we have to get it out of the news.
So one of the benefits that we get by bringing machines and humans together in these processes
is that we're able to measure for the first time the intensity of these civil unrest events.
So there have been, for many years, several applications that are automated entirely that will
count up whether some kind of civil unrest event has happened or not.
But obviously there are differences between events, and what we find in our data is that
the median number of participants in small scales giving a speech against...