Project Description: Recent studies have shown that over 85% of the information people and organizations deal with is unstructured -- the vast majority of which is text in different forms. A multitude of techniques has to be used in order to enable intelligent access to this information and support transforming it to forms that allow sensible use of the information. The fundamental issue that all these techniques have to address is that of semantics -- there is a need to move toward understanding the text at an appropriate level, beyond the word level, in order to support access, knowledge extraction and synthesis. We will discuss two key issues: (1) From Data to Meaning: There are several dimensions of text understanding that can facilitate access to information and the extraction of knowledge from unstructured text, and (2) Trustworthiness of Information: While we can locate and extract information quite reliably, we lack ready means to determine whether we should actually believe them.
Speaker Bio: Dan Roth is a Professor in the Department of Computer Science and the Beckman Institute at the University of Illinois at Urbana-Champaign and a University of Illinois Scholar. He is the director of a DHS Center for Multimodal Information Access & Synthesis (MIAS) and also has faculty positions in Statistics, Linguistics and at the School of Library and Information Sciences. Roth is a Fellow of the ACM, AAAI and the ACL, for his contributions to machine learning and natural language processing. He has published broadly in machine learning, natural language processing, knowledge representation and reasoning, and learning theory, and has developed advanced machine learning based tools for natural language applications that are being used widely by the research community. Roth is the Associate Editor-in-Chief of the Journal of Artificial Intelligence Research (JAIR) and will serve as Editor-in-Chief for a two-year term beginning in 2015. Prof. Roth has given keynote talks in major conferences, including AAAI, EMNLP, and ECML and presented several tutorials in universities and conferences including at ACL, NAACL and EACL. Roth was the program chair of AAAI'11, ACL'03 and CoNLL'02, is or has been on the editorial board of several journals in his research areas and has won several teaching and paper awards. Prof. Roth received his B.A Summa cum laude in Mathematics from the Technion, Israel, and his Ph.D. in Computer Science from Harvard University in 1995.
I'm going to talk about making sense of data.
I actually added trusting, which wasn't in the original title,
because I think it's very important also
to trust the data we get.
And I want to start first by thanking a lot of people
in my group that are working on these aspects of the research
that I'm going to talk about here and funding agencies.
And let me start by saying something
that we all know about.
We have a lot of data.
And people talk about 25% of the data we have
is, in fact, textual data.
Books, newspapers, web, internet activity
of different units, and so on.
So really a lot of data is textual data.
And until, in the old times, maybe 10 years ago,
we used to be able to be satisfied with databases.
And if the data is in databases,
we can actually do whatever we want with it.
We know how to create a database.
We know how to join.
We know how to do a lot of things.
When the data isn't in a database, it's unstructured,
what we would like to do is actually
to pretend that it is in a database,
essentially to think about or to try
to handle it as if it is in a database with a known schema.
And this raises a lot of problems,
from the simple problem of locating the relevant data
to organizing it, accessing it, analyzing it,
and synthesizing unstructured data.
And really what you want to do when
you deal with this kind of data, you
want to deal both with the content and with the network.
So who connects to whom, who talks to whom,
who authors the data, and so on.
So the research program that I'm going to hint about
really aims at developing theories, algorithms,
and tools that will allow us to transform raw data
into useful and understandable information.
And of course, as important is to integrate it
with existing resources.
So I want to talk very briefly about two aspects of this work.
So the one that we talk about most of the time
is this transformation from data to meaning,
basically trying to figure out in automatic ways,
what does the document say.
And I want to, at the end, hint a little bit
about the research program that we've
developed over the last year or two.
The three aims are talking about, can we believe the data?
What should we trust?
OK, so let me start with this kind of anecdote
that all of us are familiar with.
When you try to download the software package,
or when you try to log in in a hotel or any other place,
you typically get this contract.
Along with the contract comes a question, do you accept?
You never read the contract.
You just say, yes?
Yes.
But this really doesn't make us happy.
But really what you would like to do is you would like to say,
you know what?
I want to make sure that this contract guarantees
a few things that I care about.
Don't give my email away.
Don't do this.
So you'd be very satisfied if you
could set, in actual language, two or three conditions,
and someone, some broker, would read the contract
and tell you, yes, they guarantee that.
Or no, they don't.
We don't do it.
We don't have these programs yet.
And it's difficult.
In fact, we don't have these programs
only with a single contract.
We actually have a lot of documents, a lot of text
around that actually requires some level of understanding.
So it's not only a deep level of understanding,
something that requires to know whether this contract is going
to give our email away or not, but also it's
impressive because there's a lot of this.
Why is this so difficult?
So just to illustrate, so really we're
talking about natural language.
And when we talk about natural language,
you can think about this as the bottom layer here.
Above it is the level of meaning.
And one of the two key difficulties
are that, first of all, there's ambiguity.
Whatever you say in natural language
could have multiple meanings, depending
on the context in which it's being said.
And there's a second problem, which I call variability.
When you want to express a meaning,
there are many, many different ways you can express it.
So these two issues together make the problem very difficult.
Let me illustrate it with a couple of examples.
So here's a very simple type of example.
I call it context-sensitive paraphrasing.
Three sentences, they use the word title.
Here is a list of synonyms taken from a dictionary
for the word title.
And I color-coded to make the specific interpretation
of the word title in these examples.
And you can see that they vary a lot.
In fact, you could even see antonyms in this list.
Because depending on the context, it could vary a lot.
And this is a problem from ambiguity
that we have to resolve.
In fact, we resolve it unconsciously.
We don't even think about it.
But if you want to develop automatic tool,
you have to address this.
Variability is another problem.
Say I want to know whether in a given document
or a given document collection, it
says that Jim Carpenter works for the government.
There are many, many ways for us to say that.
And they range from saying simply,
Jim Carpenter works for the US government,
to saying something like, the Russian interior minister met
yesterday with his US counterpart, Jim Carpenter.
This also says that he works for the US government.
So in order to address this variability,
we can't just deal with this straightforward method
of Google-like keyword search.
This method cannot deal with variability
of expressive meaning, nor with ambiguity.
We really need to dive deeper, understand relations
expressed in text, understand entities,
understand semantic classes.
We need to bring knowledge from external resources.
Because we use everyday knowledge
whenever we read text and understand what it says.
We should be able to integrate information
across large collections of texts.
We should be able to identify what's
being said about the entity, about events, and so on.
So there are many tasks, all that
have to do with essentially resolving
ambiguity and variability.
What will we get if we can do this?
I'm sure you can come up with your own examples.
Of course, scientifically, you care about this
because it's going to allow us to move forward
toward natural language understanding.
But there are many, many interesting applications.
Here are two simple applications.
A law office is interested in getting
a list of all the people they corresponded with
in the last few days, years, or so.
Even that is a difficult task because we don't
know the names of the people.
We can't just search.
We just want to know, give me a list of all the people.
And in fact, I want a little bit more.
I want to be able to classify whether these people were
mentioned in a good or a bad way in these documents.
Simple task, not so easy to automate.
Think about a training facility in a large lab
or a corporation.
They want to be able to provide new employees easy access
to relevant information.
So when they read material, they want
to know what are the key concepts, what
are the key entities, who's working on what,
what applications, what methodologies were developed.
And I want this to be mapped to the relevant projects
and to some regular information.
Again, this requires a really good level
of processing in the document.
And here are two other examples.
Scott is going to tell us a little bit later
about the perspective of political scientists on this.
But you can think about the stuff
that Don was talking about before.
Let's assume that we want to study climate change
and its effect on societal stability or instability.
It may require us to identify all events related
to demonstration, protests, parades,
analyze them at the level of who is participating,
when does it happen, where does it happen, why does it happen,
generate some kind of a timeline of events,
and a causality check.
And very relevant today, the notion
of electronic health records.
There's a large amount of data in the medical domain
that essentially today, that records a lot of information.
Most of it is really narratives with a little bit
of database elements inside of it.
And most of it is a write-only document.
Physicians write a lot of information.
No one can use it.
But you could imagine there's a lot of applications,
from medication selection and tracking
to scientific gain that we could get
if we could read this information
and understand what it says.
So as I said, it's a difficult problem,
mostly because it has to address these ambiguity
and variability problems.
And conventional programs are not going
to do the work in this case.
So it's even difficult to program a simple predicate
like, in a given sentence, is this word available enough?
Think about it.
Rather easy for any fourth grader.
But it's not easy to write a program,
because almost all words in English
can take multiple power switch states.
And definitely, this has become harder
as you want to move up this chain.
So all the work in this domain really
is based on statistical machine learning and inference,
with kind of two research focus.
So one has to do with modeling and developing
learning algorithms for different phenomena.
And I'm splitting it here to things
that I call classification models,
just determining a property of the text, which
is something that I would say is well understood,
to structure models, where you can think about it
as multivariate models, where we want
to assign value to multiple variables at the same time.
And these variables are interdependent.
And to the issue of supervision.
So one important secret, perhaps, in this domain
is, yes, we have a lot of data.
All of you can see that there's huge amounts of text out there.
What we're lacking is supervision.
In order to learn models, you really
need someone to supervise the model
and tell you what's the ground truth.
What does it mean here?
What does it mean there?
And that's very expensive to get.
So a lot of the work in this domain, in fact,
struggles with how to develop learning protocols
and that we deal with very weak indirect supervision.
So that's one aspect of the work.
And the second aspect is what I call influence,
which is essentially decision making.
Once I've learned models, we want
to be able to incorporate into it some background knowledge,
domain information, and together with these two,
make some decisions.
So really, the way you can think about this,
you want to learn models, you want
to acquire knowledge of constraints on these models,
and together make decisions.
This is the type of work that we do.
So most of the rest of my talk is going to be examples.
And I'm not going to give all the examples
that I'd like to give.
I want to give you some pointers.
So definitely, if you go to my books
right there, you will see a lot of demos.
Basically, demos with a lot of work
on different aspects of the problem
or different tasks in this problem,
ranging from variations of semantic parsings
to looking at temporal extraction,
how to identify temporal expression,
normalize them, build timelines, looking
at linking text to external resources like Wikipedia,
coreference, and so on.
So there are many options.
And I want to give a few quick examples.
And the first one is going to be the simplest example we can
give, which is dealing with entities.
By entities, I mean, here's a piece of text.
I want to identify, these are the people.
These are the locations.
These are the organizations that I mentioned.
Even this problem that seems kind of simple
is actually very difficult.
Because entities are inherently ambiguous.
JFK would be both a location or a person.
And you want to be able to identify this.
And the only way you have is using the context.
So building long lists could help in some cases,
but really isn't the solution.
Because if you look at any reasonable size
collection of documents, you will
see that as the number of entities you see grows,
the number of new entities that you see also grows.
There are always new things, new name, new organization,
and so on.
So you really need to use machine learning approaches.
And many difficulties occur there,
especially as you try to move from one domain to another.
If you train more than a newswire data,
it's not going to behave well when you move up.
Here is one simple example.
You can see here the word Wednesday.
Wednesday used to be a day in the week.
It's also a name of a certain week.
We want to be able to identify it as an organization.
What helps us here is the fact that somewhere else
in the document we saw that there
is an organization called Sheffield Wednesday.
And probably they mean the same.
But again, this requires learning and a level of reasoning
in order to be able to do this.
Things become even more complicated
as you move to other domains.
And I want to give you an example from analyzing text
in the medical domain.
So these are discharge reports on hospitals that we look at.
And in this case, it's the same problem.
You want to identify concepts and entities.
But the type of concepts and entities
that you want to identify is completely different.
So here, for example, I want to identify mentions in the text.
And then classify them to problems, treatments, tests,
and people that are mentioned.
Conceptually, the same problem that I mentioned before,
but very different.
And in fact, requires a lot of background knowledge
in order to solve it.
Actually, we want more than that.
We want to be able not only to identify mention,
but also to understand it at some level.
I want to know, for example, that what's written here,
stabbing left side chest pain, actually
refers to the same thing that is written here as neuropathic.
If I do not know that these are the same thing,
these refer to the same thing, I will not
understand what's happening.
And won't be able to support decisions the doctors make.
So the decisions are actually hard.
This is called the co-reference resolution problem.
One of the most difficult problems to address.
And of course, there are many other needs.
I want to be able to do temporal recognition reasoning.
I want to deal with relations between problems, treatments,
side effects, and so on.
I want to deal with quantities and reasons about them,
and so on.
And you can imagine a lot of applications
that this can have.
In order to analyze text at this level,
you really have to understand something
about the semantic level of the test.
And I'm going to give an example of something
we call semantic role labeling, which is really
a task of understanding this at the level of who does what
to whom, when, and where.
Here's an example that was generated, of course,
for a DOB agency.
A sentence about the car bomb that
exploded outside the US military base
and killed 11 Iraqi citizens.
We want to understand it at the level of, OK,
there are two predicates here.
There's an explosion, someone got killed.
We want to know who are the arguers.
So the car bomb, the location in which it happened,
who's the killer, who got killed, and so on.
So this is the level of analysis that we want to be able to do.
And this is a demo, again, for my group's web page
that shows what can be done.
In fact, this isn't sufficient, because relations in text
are expressed not only using verbs.
A lot of the relations are expressed
using prepositions, nouns, and so on.
And these are even more ambiguous than verbs.
So for example, look at this simple sentence.
And look at the preposition.
So someone died of pneumonia.
So of represents a cause.
But you can represent a cause also with from,
suffer, from, through.
But from can represent other things.
In this case, from represents a location argument.
And it will also represent a start state.
So again, it shows example for ambiguity and variability
of expressing relations in text.
And you want to be able to do this.
And the same kind of model that I talked about before
actually do it well here.
Some of the difficulties, again, have to do with data.
We don't have large amounts of data
where people sat down and annotated.
The of here represents a cause.
The of here represents something else.
The from here represents this.
So we have to develop techniques that
try to do these things in a semi-supervised way.
And also, to look at multiple phenomena in the text
and kind of constrain them against each other.
OK.
I was going to talk a little bit about the math here.
But I'm going to skip that.
And I want to jump to two other examples.
So one of the examples that we deal with
is identifying events.
Here are two very simple descriptions of events.
I have an arrest event here.
And I have a kid event here.
Again, each one of these events could
be expressed also in a whole paragraph
where it's a lot more difficult to identify.
But we want to be able to identify events.
We want to be able to identify causality between events.
This happens because of that.
We want to identify temporal arguments.
When did it happen?
Two days after Christmas is OK.
But when is two days after Christmas?
In order to be able to reason about it,
I should be able to identify this temporal argument,
but also to convert it to December 27 relative,
and maybe the year, relative to some other place that
mentions the year in this document.
Otherwise, I won't be able to reason about this.
Why do we want to do it?
So this is a transition to Scott's lecture
in a few minutes.
We want to be able to support identifying
events that happen.
This is a collaboration that we have
with the Klein Center for Democracy,
where they're looking at really quantitative political science.
He's going to tell you a lot more about this.
So this is just an example of how
you can use this kind of natural language understanding
in order to support other domains.
So the final example that I want to give
is something that I call the reference problem,
that really you can think about in many respects
as a way to extract or to acquire knowledge so as
to support language understanding.
So when you read documents, you can
see that this is a document that mentions some Kennedys.
Here's another document that mentions Kennedys.
Here's yet another document that mentions Kennedys.
You typically don't even remember
how the name of the person was mentioned in the text
because you know that the first and second document probably
talk about this guy.
And the last document talks about this guy.
You actually ground it in some knowledge that you have.
And this supports your disambiguation
and consequently also acquiring knowledge about this.
How do we do this in text?
You read about Chicago.
Chicago means a lot of things.
Chicago could mean font.
It could mean a band.
It could mean an album.
And it could mean many, many other things that we all know.
Again, in order to understand what it means,
we need to be able to ground it in something.
We ground it in something that we know in the world.
And in order to kind of simulate this and acquire knowledge,
we actually have an effort that looks at text,
tries to ground it also.
And we're using Wikipedia for that.
Wikipedia, as the world, allows us to ground things into it.
Not so easy to do, but very powerful
because if you are able to do it
and generate some kind of an information network
between things that you read in text and Wikipedia,
it gives you a lot of power.
In fact, we can learn a very nice global model
that identifies concepts, disambiguates them
and grounds them into Wikipedia
and allows us to learn a lot of things
about things that are mentioned in text.
And as important is,
we can train this model in a completely unsupervised way
because we rely on links that people put into Wikipedia.
Here are a couple of examples.
So here is a blog that talks a little bit
about the Israeli-Palestinian conflict.
And you can see two names mentioned here,
Mahmoud Abbas and Abu Mazen.
Some of you that maybe know something about this conflict
would know that this is the same person.
But the writer of the blog used these words
and didn't reveal that this is the same person.
So he knew and he assumed that the readers do,
but if you don't know,
the context team will not let you do it.
So our model, though,
you can see points Mahmoud Abbas to Mahmoud Abbas webpage
and Abu Mazen also to Mahmoud Abbas.
So basically, our model figured out
that it has to be the same person,
given the context,
given the context of other pages
that are pointed to from this page.
So it not only identified conflicts that are important,
but also disambiguates and makes them correct.
Here's an application of this in the medical domain.
So again, when I started to work
on these electronic health records,
I can tell you even now, in fact,
I don't understand a word of what they're saying.
All this terminology is too complicated for me.
However, using the wiki file,
I can actually look at a prototype here
for Bournemouth's process,
and I can run the wiki file,
and it turns out that it points this to Born-Tuner.
So the surface representation doesn't matter so much.
It figured out that the corresponding page
is actually Born-Tuner.
If you go and read it, you'll see that this is right.
So really, the significance of this
is not only that we can identify
important concepts in tests, important entities,
and disambiguate them and learn something about it.
I think that this is actually something
that can be useful as a tool
for scientific literature
and even for educational application.
Okay, so that basically ends what I wanted to say,
and I want to spend two minutes
on another aspect of this work
that I think is really important
and really has some societal significance.
So, you know, we already know and say
that there's a lot of information out there,
but it's of uncertain quality.
And the quality deteriorates
as the amount of information out there
and the competition on being first
to reveal something grows.
And we all know, you know,
the Boston events just recently,
and just with this event,
I can fill up a lot of slides
with examples of what has happened
and how come we cannot count on anything that happened.
But basically, a lot of stories went out,
and the relation to the facts were,
you know, very, very weak.
So the question is,
that of distributed trust.
There's a lot of data sources
that give us information about things.
And sometimes by mistake,
and sometimes for a reason,
they don't give us the right information.
One of my favorite examples here
is the story of Senator Kyl,
who two years ago stood on the Senate floor
and said something like,
well over 90% of what Planned Parenthood does
is supporting abortion.
The truth is 3%.
But, you know, it's a reliable resource, kind of,
and it's already there in the transcript of the Senate,
so is that the truth?
So we really need to be able to develop
some kind of a computational trust system
that will be our proxy.
And maybe it will be a proxy
between text and some other systems,
question answering systems,
navigation systems,
news aggregation systems,
warning systems, and so on.
Something that will take a lot of information,
figure out what's true and what isn't,
and tell us something about it.
And, you know, another favorite example of mine
is the medical group.
There are thousands of support groups,
media forums, and so on,
and really hundreds of thousands of people
get the medical information from the internet.
They search for the best treatment for it,
side effects, and so on.
But some of the users on these news groups
actually have an agenda
from certain companies, for example.
So the question is,
to what extent can we believe what we read there?
So the goal of this research,
and by the way,
this problem isn't easy even for people.
And again, I have a lot of examples,
and I just talked on this one,
from the recent Supreme Court decision
on approving Obamacare.
If you remember,
in the first few minutes,
or more than minutes after this came out,
people misinterpreted it.
So it's not so easy.
So the research problem that I'm talking about
is a research problem that attempts to look at
sources and claims they're making,
and try to be able to score
trustworthiness of claimed sources.
So think about it as developing the ability
to generate a probability distribution
over trustworthiness of sources,
along with a distribution over the credibility
of the claims they're making.
And we are dealing with,
basically developing the algorithmic tools
that will allow us to do something with this.
So again,
I don't have time to talk about
the content of this research direction,
but I think it's clear that we need something with this.
It's clear that it's very difficult,
and we've been dealing over the last couple of years
with trying to develop some algorithmic tools
to support that.
And with this, I'm going to end.
Hopefully I gave you some idea
of what's happening in this area,
what are the difficulties,
and some of the progress that we've started to make.
Thank you.
Inspiration of that.
Just a curiosity.
For example,
in your JFA,
it would be a person or a location.
So could be Dallas, for example.
We use Dallas just to make my point.
For example,
today, if the word Dallas appears,
almost independently of a context,
it's very much likely to be a location
because of the inauguration,
the library inauguration in Dallas, Texas.
Today.
Today, today.
And that's the question I wanted to ask you.
In the present context,
does it come into the picture of what you are...
Yeah, yeah.
Very good question.
Of course.
So you could say,
even generalizing what you say,
if you start reading the document
and you see the word Dallas,
and in the beginning you think that it's a location,
it's very likely that later on in this document,
other occurrences of Dallas will also be locations.
Right?
And you are generalizing it, you know.
Today, Dallas is more likely to be a location.
Of course, we are using this.
So this is what I call structured prediction problems
where you have to make decisions with respect to many things
and they are not independent.
If I've seen Dallas,
and it's today,
and I know that there is an inauguration of something in Dallas,
I have to take this into account
and this is going to change something about
the probability my model should assign to occurrences of Dallas.
And this happens all the time.
You know, the sense of the word.
So the word table could mean many things.
It could be also a noun, it could be a verb.
But in a given context,
if I see table and I think,
well, this is probably a noun here,
it's very likely that in the same context
or in the same document,
it's also going to be noun and mean the same thing.
So we are using these things, of course.
But try to learn this automatically from Dallas.
Thank you.
What's a comfortable data size
that you'd like to use to have a nice prediction?
Okay, it's...
That's a very important question
and it varies over time.
So, for example, in order to train entities,
an identifier,
this is kind of one of the easier problems today.
So with a couple hundred documents,
a couple hundred examples,
you can already develop a solid model.
But if you want to do semantic parsing
that involves multiple decisions with respect to each sentence
and a lot of interaction between them,
you probably want to have thousands of examples.
So it really varies on the complexity of the problem.
Okay, so that's, again, a very, very important question
that intensifies the issue of the variability of data.
So most of these tools have been developed in English.
And the reason is we have a lot of data.
So people collected COCOA and annotated COCOA.
In many other languages, we don't have a lot of COCOA.
So there's a lot of effort, first,
to annotate data in other languages
and also to develop techniques that allow us to use
what we call sort of weekly parallel COCOA
in order to transfer abilities of analyzing English
to something else.
So as an example, I can take the BBC,
and BBC has news in Russian, in English,
in, you know, many other languages.
And I could assume that if it's today,
about the same international news are going to be covered
in the Russian BBC and in the English BBC.
So this will allow me to use models
that were trained for English
and make the predictions to the Russian text
and try to improve models for Russian, for example.
And, I mean, there's a lot of research
on trying to do these things,
basically to bridge over the fact
that in most languages, we have no resources.
Yeah, very important direction.