Modeling and Decoding the Humain Brain
1 Video View·Sep 5, 2023
Modeling and Decoding the Human Brain
Jack Gallant, UC Berkeley
at the University of California, where he's a professor of psychology,
but befitting this program and his work.
He's also affiliated with a number of other programs,
including programs in bioengineering, bioethics,
I have to read because it's a long list,
neuroscience and statistics, maybe also, no.
Yeah, but I didn't catch one on there.
All right, there's none on there, but I think we know about that.
Dr. Callan's research is primarily focused on generating computational models
to describe how the brain represents information during natural tasks,
including how people process, for example, natural food.
And actually, more recently, the ultimate natural scene,
which is a trailer for Hollywood movies,
which maybe the topic is not so natural,
but certainly the type of visual scene is very natural,
very similar to what we experience in real life.
And this work has shown in very exquisite detail
that the visual information that a person is perceiving
can be, in fact, decoded from the brain patterns,
blindly, basically, of what the person's doing.
And so you can pretty much read what the brain does.
And this has not only the great advantage of showing you how the system works,
but also has potential really interesting clinical and non-clinical applications.
And ultimately, Dr. Callan's research program is really a fantastic example
of the benefits that can be gained by cross-fertilizing research in neuroscience,
computer science, and engineering,
which is really the focus of the IGER program here.
And without further ado, please let's welcome Professor Callan,
who will speak to us about modeling and decoding the human brain.
Thank you.
Thanks.
Thanks very much for that kind introduction.
The one fact you left out is actually,
when I was at Caltech in Wash U, I was David Van Essen's post-doc.
So everything I know about the brain, I learned from David.
And I love coming to engineering conferences,
especially student engineering conferences,
because of my particular bias in neuroscience,
which is that understanding the brain is basically just a big,
ugly, complicated reverse engineering problem.
So here's our problem.
We want to understand this computer over here.
And we don't really know a heck of a lot about it.
I mean, we've got hundreds of years of stuff, information and data,
but there's still a lot we don't know.
But for comparison, we can think of this computer over here,
which is a standard I486 CPU, a few years old now.
So there are several systematic differences between these.
This system's gone through about eight generations of design,
intentional design by humans.
This has gone through about 25,000 generations of evolutionary design,
at least since the human lineage broke off.
The parts for this system are known and clear,
because it was designed this way.
So we know there are some graphics registers and some memory registers
and all kinds of specific kinds of subsystems in the CPU chip.
And we know where they are.
And somebody put them there, and they do a specific job.
And here, there's a bunch of parts.
We don't really know what they are.
As David pointed out, Brodman kind of divided it up into parts
based on coarse anatomy.
And you can use other methods for dividing it into parts.
But what the parts are is kind of complicated and mysterious.
We know how information flows through this network,
because it was designed with a certain language and a certain word
order and clear bits and essentially pathways of information
flow defined by information theory.
And over here, we don't know what the language is at all.
I'm a neurophysiologist by training.
So I assume that everything's basically encoded in spikes.
But who really knows?
It could be some other mysterious thing going on in there.
And basically, this problem, if you wanted to reverse engineer it,
would be at least tractable.
It would be really hard.
But you can imagine this is a fairly simple design system that
is designed by very specific structured rules.
And this system is designed by evolution.
And there may be a lot of dead ends and sort of extra cruft in there
that's not even all that useful.
It's just how the system ended up working, because it's an evolved system.
It's not going to be optimized.
Generally, it's going to be optimized down the sort of branch of the tree
that it's on.
And here, we could sort of analyze this system for as long as we want.
This system, usually, we only have for a very finite amount of time.
We can't get that much data from it.
So this is a really hard problem.
Essentially, this is a reverse engineering a nonlinear dynamical system.
And for those of you in engineering, you know that if you have a linear system,
the world is wonderful, easy to solve.
Nonlinear system, well, kind of a pain, but you can solve it.
Nonlinear dynamical system, well, everybody just throws up their hands
and gives up, right?
People can't even get the heating, ventilation, air conditioning right in
this room.
And you can imagine what it's like trying to reverse engineer this brain.
I mean, we don't even have math to describe what this thing does.
So it's a hard problem.
Now, there's really two aspects of this problem because the brain is a very
special kind of computer.
This is something that was drilled into me by David and Charlie Anderson.
The brain is not like a regular computer.
A regular computer, you have hardware, and you run software on it, and you run
whatever software you need on your computer platform.
But the brain is a hardware computer.
In the short term, the wires are fixed.
You can't change them.
It takes about a half an hour to grow a new synapse.
So you can only work with the hardware you have.
And therefore, the computations that the brain is performing have to be
determined ultimately by the connections.
And hence, David's fascination and interest with the connections of the
brain.
But the problem is, as David so eloquently pointed out, is that we don't
have very good ways to measure the connections of the brain.
And we're not going to get them anytime soon.
So our connectional information is going to be coarse.
And we're never going to know, say, you know, exactly what all the synapses are
in the human brain anytime soon or exactly what the biophysical mechanisms
are for different types of synapses.
And all those factors matter for function.
So my contention would be, rather than just looking at connection, we need to
combine connection information with functional information.
And of course, measuring function in the human brain is also hard because we
can't just take humans apart and do anything we want with them like we can
with, say, a simpler model system.
So right now, the best method by far we have of measuring activity, functional
activity in the human brain, is magnetic resonance imaging that David talked
about.
Now, I know many of you do MRI, but I'll just give the two-sentence version.
Basically, this is a big, gigantic stationary magnet.
You slide somebody in a stationary magnet.
It aligns some vanishingly small fraction of the protons in their brains.
And then you apply a radio frequency pulse, knock the protons off axis.
And as they precess back down, they give off the radio frequency energy.
And you can measure that.
And that's coded digitally.
So you can essentially get a digital information about the location of these
protons relaxing in the brain.
And it turns out by some biological quirk, oxygenated and deoxygenated blood
have slightly different paramagnetic signals.
And so we can sort of get some proxy for brain activity due to blood oxygenation
and deoxygenation with MRI.
But these signals are pretty complicated and indirect, right?
Because really, we're looking at the plumbing.
We're looking at the plumbing of the system, not neurons.
In fact, there's nothing even close to neurons in functional MRI signals that we
measure.
First of all, they're very indirect.
Here's a neuron.
There's a presynaptic membrane and a postsynaptic membrane.
This neuron sends some neurotransmitter over to the postsynaptic cell.
Now, you can't leave the extra neurotransmitter here in the synaptic
cleft.
Otherwise, you'd become paralyzed.
So you've got this vacuum cleaner system that's astrocytes that surround the
cleft.
They take up the neurotransmitter.
They repackage it and deliver it back to the presynaptic cell.
And as a side effect, they have these little filopodia on the capillaries.
And the astrocyte absorbs neurotransmitter and says, oh,
somebody around here fired, so I better open my filopodia because they're going
to need oxygen and sugar from the bloodstream.
And then the capillaries get larger.
And then blood flow, blood volume increase in this capillary.
And the oxygen extraction changes over time in the venous system.
So we're not measuring the brain.
We're measuring this bold signal, blood oxygen level dependent signal.
And we're measuring blood vessels.
In general, we measure blood vessels in MRI.
The blood vessels that have the highest signal to noise are the blood vessels
that are about the same size as a voxel.
So in a typical functional MRI experiment,
we're talking about voxels two by two by two millimeters.
So most of our signal is going to come from blood vessels about two
millimeters across.
Now, two millimeters is kind of a big volume.
And you might realize, if you're a neurobiologist,
that that's actually a hypercolumn.
This is the ice cube model of V1.
We've got ocular dominance columns, orientation columns.
And these are arranged in a grid, in a retinotopic map.
And a collection of orientation columns and hypercolumns together,
orientation columns and ocular dominance columns, is called a hypercolumn.
And the resolution of MRI is even coarser than a hypercolumn.
So we're not going to be able to get fine scale information.
We're going to be able to get information that's pooled over a couple million
neurons and indirectly coupled to those neurons through this hemodynamic response.
And this hemodynamic response is fairly slow.
If we show a checkerboard in a little small regions, wedges here, and
we look at a few checkerboards, these colored ones here.
We look at the time course of the response in voxels sensitive to those checks.
You can see that some of the checks elicit a positive bold response.
Some of them elicit a negative bold response.
But there's about a four second delay between when the check is flashed and
the bold response.
And that's because plumbing is really, really slow.
So this is a very slow, very indirect method of measuring the brain.
The way, the analogy I often give is, imagine you have this weird job where
somebody tells you to go in and figure out what's going on in an office building.
And the natural way to do this would be you go in and
you talk to everybody in the office building and say, what do you do?
What's your job?
But instead of doing that, you decide no, you go into the kitchen and
you figure out how much water they're using.
Because you figure if people are working hard,
they're probably going to be making more coffee and
they're going to be using more water.
And so if the water's flowing more,
that floor of the office building is probably doing more work.
That's essentially the thing we're doing in MRI.
And if you do some back of the envelope calculations
about how much information you get out of MRI.
In the best case scenario, you're getting about one one-millionth
of the information that's available in the brain at any given point in time.
But that sounds bad, but remember there's a lot of neurons in the brain.
So even one one-millionth is a huge amount of information.
And the amazing thing about MRI is that this technique works way better than any
other method we have for measuring brain activity in that you can get out a huge
amount of information even though this is very indirect.
So let's look at the kind of information we get out of the brain.
Here's a movie on the left and
a brain activity on the map, kind of like on the right, kind of like David showed.
Now the brain is crunched up inside the skull, which is very inconvenient.
And one of the first things David taught me is,
don't look at it crunched up inside the skull, flatten it out.
So here we have a flat map of the brain.
On here, red means bore, bold.
Blue means less bold.
Bold is always a relative signal, not an absolute signal.
And in the back here, this is the visual system.
This is the front of the brain.
This is the top and the bottom.
So you can see that we are getting systematic changes in brain activity
while people watch this movie.
All they're asked to do in this task is watch the movie and
fixate on a small green spot, which is not drawn on here except with my laser pointer.
So they're passively watching this movie.
And as the movie plays, you see changes in bold signal over time,
mostly in the back of the brain here in the visual system,
which is kind of what you'd expect.
This is, after all, a movie.
But you also see changes in bold up here in the front,
even though the person's not actually doing a task.
And up here in sort of the motor somatosensory strip,
very subtle changes, a lot up here in parietal,
and some changes down here in auditory cortex.
So you can think of the problem of trying to reverse engineer the brain at a functional level
as trying to find a systematic mapping or relationship
between these movies and these brain activity signals.
That's all you're trying to do.
It's basically a regression problem.
Now, on here, we can measure about 30,000 individual points using a typical 3T bold.
So we're going to build 30,000 potentially models.
Each model is going to tell how an individual point in the brain maps onto this movie.
In other words, what features in this movie drive responses at each individual point?
And then, of course, these individual voxels and volumetric pixels are aggregated into areas,
like V1's going to have maybe 3,000 or 5,000 of these points.
So all of the neurons in V1 are probably coding in some common space,
and all of the voxels in V1 should code in some common space.
So you're going to need a common model for V1,
and you'll have an individual model for each voxel.
And if you pick another area, like V2, you'll have a common model for V2
and an individual model for each voxel.
Now, what do those models look like?
Well, in general, all you really know is that they're nonlinear,
because there's no point in having a completely linear brain.
If you have a linear brain, then you might as well just look at the pixels,
because you're not going to get any more information after all linear transformation,
just a rotation of the coordinate system.
So what the brain has to do is apply nonlinearities to the input
so that it can extract information that's implicit in the input
and make it explicit so that it can be read out.
And of course, also the brain has to combine current information
with prior information it has about the world
in order to optimize learning and optimize behavior.
So if you're measuring one point in the brain, a bold response,
you start with a stimulus, you want to find the relationship
between the stimulus and the bold response.
One way you can think of this is what I call linearizing the problem.
You can find a nonlinear mapping from the stimulus into some feature space
such that the relationship between the feature space and the bold response is linear.
And this is actually a convenient way to think about the problem,
because then you can essentially come up with a theory about the nonlinear mapping,
implement that theory by deterministically moving the stimulus
from the stimulus space into the feature space,
and then using regression to map the feature space representation of the stimulus
onto the bold responses.
And the nice thing about doing regression in this particular case
is it's going to be linear regression.
And of course, linear regression is one of those things we can do really well.
Now, once you have a model that you're happy with by criteria that I'll explain in a second,
if you're interested in it, you can do decoding.
And in decoding, you take all the bold responses across a bunch of voxels.
These all have models, so you can now linearly map those models back into the feature space.
And then if you so desire, you can nonlinearly map back into the stimulus space
and reconstruct the stimulus.
And these are symmetric problems.
The encoding and the decoding problems are related by Bayes' theorem.
Essentially, the encoding model gives you the likelihood
the probability of response given a stimulus.
You multiply it by the times of prior on the stimulus,
and you get a posterior over the stimuli,
the probability of each stimulus eliciting a response.
And so now you have encoding and decoding in the same framework.
Now, if somebody wants to know later, there are specific reasons
it's better to build an encoding model than a decoding model,
but for this talk, it's not really important.
So, the problem here is this is a large-scale data analysis problem
because we have 30,000 voxels.
And we don't know what model's appropriate for each voxel.
So, we can take a data set.
Here's a stimulus and here's a response.
We take a bunch of responses, and we can break these responses into two parts,
an estimation data set that we're going to use to train the model,
and a validation data set that we're going to use to test the model.
And then for each individual voxel, we can start fitting models.
So, we might have a hypothesis.
Say, I think this voxel is sensitive to edges.
So, I'm going to transform my stimuli into an edge map,
and I'm going to find a set of weights that map this voxel's responses
onto the edge map.
Essentially, that will tell me what edges this voxel cares about.
And then I can try a different model.
Well, maybe this voxel cares about semantics.
So, I'm going to take semantic information that occurs in this picture.
These words actually don't occur in this picture because I changed the picture.
Forgot to change the words.
But anyway, now I'm going to have a set of weights that map these words
into this voxel.
And so, I can fit multiple models to each voxel.
Now, I have to test which model is right for each voxel.
So, to do that, I go over to my validation dataset,
and I get predictions from my model, say, the edge model,
and I just correlate them with the observed responses in that voxel.
And I can compare the predictions from my edge model to my semantic model
to my attention model, say, or however many models I want for each voxel.
And I can do this as many times as I want because I'm not overfitting.
I'm saving data out to do this.
This is a large-scale data-fitting problem.
We're going to fit dozens of models to 30,000 individual voxels.
And, of course, the thing to remember is, if there's, say, as David implied,
but didn't state something like 300 distinct human brain areas,
then we're going to need 300 model types distributed over 30,000 individual voxels.
So, there's going to be a lot of modeling going on.
All right, I'll skip this.
Okay, to make a long story short, let's just think about vision for the time being.
And I'm going to just demonstrate a few classes of models
that will give you an idea of how this framework works.
We're going to focus on vision because it's the easy part of the brain.
Vision is hard, but everything else is much harder than vision.
Vision is the dumbest part of the brain.
So, it turns out that people have done a lot of other MRI experiments
that have identified certain regions of interest or areas.
These are usually done by subtraction.
For example, you might show somebody some faces and then show somebody some places
and subtract the bold activity and locate the places on the cortical surface
that actually have more of a response to faces and places.
And if you do that, you get these white lines.
You can see there's a whole bunch of white lines.
These white lines are retinotopic areas.
This is the perihippocampal place area that likes places.
This is the fusiform face area that likes faces.
This is the exostriate body area.
There's some auditory areas, some frontal eye field areas.
There's a bunch of areas.
People have parceled out the brain using these functional localizers
to a reasonable, modest extent.
There's a whole bunch of parts of the brain.
Of course, we don't know what they do.
There are no functional localizers that really give you anything clear at all.
And we don't know that these functional localizers are actually correct.
I mean, if you use a functional localizer where you show faces and places
and your area that you're recording from is actually the kumquat area
and really it prefers kumquats,
well, maybe faces just look more like kumquats,
so it looks like a face area to you, but really it's a kumquat area.
It's a problem with localizers.
So it turns out there's two broad families of models that you can fit to the brain.
One is structural models and one is semantic models.
And structural models are models that essentially go from the stimulus space
into some feature space using a mathematical algorithm that you can understand.
For example, an edge map or a spatial frequency map.
Semantic models go from the feature space into a semantic space,
usually through some transformation that you don't really understand.
And I'll explain why that is later.
And you can see that these are distributed differently over the brain.
The structural models model well early visual areas like V1, V2, V3.
And the semantic models model well these higher-order visual areas
like PPA and FFA and so on.
And so I'm going to show you each of these models and how they all kind of fit to the data.
So this is a structural motion energy model.
This turns out to be the best model for early human vision that we have right now,
that anybody has.
And the model works thusly.
We start with our input, our movies, and we have some bold responses that we're trying to predict.
So for each and every voxel, we fit a model.
And the model actually has two stages.
The first stage is essentially a latent variable neural model
that's meant to model an underlying population of neurons
that we don't actually have access to.
And the second stage is hemodynamic coupling terms
that describe the relationship between these virtual neurons that we didn't measure
and the bold responses that we actually measured.
Now, there's a bank of about 2,000 to 8,000 of these motion energy filters.
They vary in their spatial frequency, their orientation, their position,
their phase, and their direction.
And each motion energy filter is actually a cascade.
We start by stripping off color.
We take a spatiotemporal Gabor wavelet.
We multiply it times the input.
We take two of these in quadrature phase
and combine them to get a complex cell, model complex cell.
We stick in a compressive nonlinearity because it helps predictions.
And then we temporally downsample from the rate of the movies,
which in this experiment was 30 hertz,
to the rate of the bold, which in this experiment was actually only a half hertz.
So now, this, remember, is deterministic.
This has nothing to do with our data.
This is essentially a theory about an underlying neural population.
So it's completely deterministic from the input, but it is a nonlinear transformation.
Now, we have a bank of 2,000 to 8,000 filters.
And for each individual filter, for each feature channel,
we fit a six-tap finite impulse response hemodynamic filter.
So we're fitting essentially 2,000 to 8,000 of these six-tap filters.
And we take the sum of those to predict the bold responses.
Now, this is a really large-scale fitting algorithm
because you have to do this for each one of the 30,000 voxels that you measured.
But this is kind of what our lab's bread and butter is.
So we have a lot of software for solving this kind of problem.
Now, once you fit this model, your first question should be,
well, how well does it predict?
And so you can just take a separate data set that you saved and look at predictions.
This shows predictions of the motion energy model.
You notice it predicts well here in the back of the brain.
And if we compare that to, say, a local change model that doesn't have direction or motion,
that model doesn't work so well.
A static model that doesn't have motion works even worse.
And this is a face model that's just pathetic.
It's not even worth thinking about.
So this motion energy model works really well for early vision,
which was exactly what it was designed to do.
Now, it's always entertaining to think, well, what do these voxel models look like?
After all, you're measuring from a million neurons and you've got 8,000 features.
Does it make any sense at all?
Well, you can treat a voxel like a neuron and look at its voxel-wise receptive field.
So there's a receptive field for one voxel.
It's located here in the spatial display.
And now we can marginalize out the excitatory channels and the suppressive.
Well, these aren't really excitatory and suppressive.
These are more like positive bold and negative bold.
And we can look at orientation channels and phase channels and aggregate frequencies together.
And you can see that there's a lot of substantial weights in this matrix.
There's 8,000 basis functions here and a lot of them has positive weights.
And you can't really tell what the tuning is from this.
And you kind of expect that.
After all, we are aggregating information over millions of neurons.
So to make this a little clearer, you can marginalize out, say,
spatial frequency and temporal frequency.
So on the left here is a receptive field for one neuron.
You can see that this has high spatial frequency and low temporal frequency tuning.
Here's a different neuron with high spatial frequency and medium temporal frequency tuning.
And you notice that we can recover temporal frequency tuning at our temporal frequency rate
far higher than the sampling rate of the bold response.
And that's because we have this underlying neural model.
So to make a long story short, basically you can do this one experiment
and recover pretty much everything anybody's ever reported in any MRI experiment in human V1.
Just by using natural movies in this model.
This model basically accounts for every previous report.
How well does this model work?
Well, one thing you can do is to look at predictions.
Another thing you can do is do movie identification.
So in this case, we have 400 movie clips.
These are one second long.
And we interrogate the model to figure out
if it can figure out which of these 400 movie clips that it actually saw.
And it does this by just looking at the bold response
and then running the model backwards
and trying to figure out which clip was most likely the one that elicited that bold response.
You can see that the model's doing almost perfect.
We're up about, I don't know, 90 something ridiculous percent correct.
And if we increase the number of candidate movies in a clip to say 10 to the 6th,
we're still well above 80%.
So with 10 to 6 movie clips, we can tell at 80% accuracy exactly which movie clip you saw
in a one second time window.
That's a pretty impressive result, I think.
Especially considering we're measuring the plumbing and not neurons.
So I told you about some structural models.
They fit well, or any structural model fits well in all these early areas.
And I'd like to move on to a semantic model that fits well in these higher areas.
Now, there's a problem with semantics.
And that's not the fault of neuroscience.
That's actually the fault of computer vision.
So computer vision algorithms are pretty good
at finding the local structural features in an image.
But there's no computer vision system out there
that could say, look out at this room and just label all the things in this image.
Or look up at me and label all the things.
You know, there's a slide projector over here.
There's a whiteboard.
There's a picture behind me.
There's me.
There's a microphone.
No computer vision system can do that.
So we don't really have a way to transform these images directly into semantic information.
We have to instead essentially do this by hand, which I consider cheating.
But there's really nothing to, no other way around this right now.
So we need a semantic space that we want to use for this.
The semantic space we decided to use was something called WordNet.
WordNet is a semantic tree of about 80,000 nouns and verbs in the English language.
And it essentially recapitulates the is-a relationships between nouns and verbs.
So at the root of WordNet, this actually just shows the 1,700 most common nouns and verbs.
There's a main tree here that has three branches and some subtrees for the verbs.
And these are is-a relationships.
So the base of all the nouns is thing.
And then you can see that all these things are artifacts.
And here's furniture.
Here's cars.
Here's indoor scenes, outdoor scenes.
Here's kind of things you can pick up.
Here's people.
Here's animals, and so on.
So this gives us kind of a hierarchical semantic tree that we can work with that's a convenient space
and has been validated with English language and perceptual psychology.
So how do we fit this data to our stimulus?
Well, we have to do this by hand.
So you take a one-second movie clip, and you find all the nouns and verbs in that movie clip.
So you say, oh, I think there's a dog in the movie clip.
And then because there's a dog, I know there's a canine, which is a superordinate category,
and I know there's an animal.
And I think there's a woman in this clip, and I know there's an adult, and I know there's a person.
And then I'll go to a different clip where there's a jaguar, which means there's a feline,
an animal, an organism, and an object, and so on.
You can do this for all the movie clips in the stimulus,
which if you want to see a graduate student become very unhappy, this is a quick way to do it.
But he designed this so he has only himself to blame.
This is Alex Hoof.
So we now have two hours of natural movies,
and we've essentially nominally transformed these movies into an indicator matrix
where we've got a row for each of the 1,700 nouns and verbs,
and a column that indicates whether that noun or verb is present in that one-second cube of space-time.
So now we can do regression between every single voxel and this matrix.
We take our voxel's bold response.
We shove it through the same regression procedure we used earlier.
In general, these are MRI data.
They've all been sphered.
They're Gaussian.
So usually ridge regression is the thing you use for this.
And then you're going to get a finite impulse response that indicates the hemodynamic coupling
for each individual noun and verb that happens in the movies
and what the hemodynamic response is of that voxel to those nouns and verbs.
Now you can interrogate a single voxel.
So these are two different voxels from two different brain areas,
the right precuneus and the left parahippocampal place area.
Let's look at this first.
Red here means a more bold response than average,
and blue means a lower bold response than average.
And the size of the ball just indicates the magnitude of the bold response.
So I've labeled some subset of these so you can kind of get an idea for what's going on.
But if you can't read them, I'll tell you.
This particular voxel gives a positive bold response whenever you show any kind of vehicle
or any kind of artifact that you can pick up or any indoor scenes or outdoor scenes
or people or canines.
And it tends to be suppressed by other kinds of animals that aren't canines,
by atmospheric phenomena, and by plants.
Okay, now here's a separate voxel in the right precuneus.
This voxel gives a positive bold response when you show people or communication verbs
or carnivores or vehicles or things you can pick up.
And it gives a negative bold response as it were suppressed by a lot of other kinds
of artifacts, outdoor scenes, indoor scenes, plants, atmospheric phenomena,
traveling verbs, ungulates, which it doesn't like, and so on.
See, you can see these data are pretty damn complicated, right?
Really, we've done a whole thousands of functional localizer scans in one experiment here.
And we can interpret this data by going through all the individual voxels
because we've got 30,000 voxels.
That would be absolutely crazy.
Really what we have is 30,000 voxels, a matrix of 30,000 voxels by 1,700 nouns and verbs.
We've got a weight matrix.
And we have to interpret that.
How the heck do we do that?
Well, as engineers always know, given a big matrix that you need to interpret,
the very first thing you do is PCA.
So let's do PCA on this matrix and see what happens.
Okay, so we did PCA on this matrix.
And we're going to get out, of course, 1,700 principal components.
Let's just look at the first few.
They're probably going to be the only ones that matter.
And let's look at it for several subjects.
So here's we've got subject SN, AH, and AV.
This is three subjects.
We've got five subjects who've done this experiment.
They all show the same result.
These are not the PCs.
These are correlations between the PCs.
You can see that all these subjects show very strong correlations between the first four
principal components of this semantic space.
So that's good.
That means we're no longer in a random world where everybody's got some hopelessly complicated
semantic thing we'll never understand.
Now we see all subjects share four principal components.
And we have some hope of actually interpreting that.
So the first thing you want to do, though, is make sure this isn't a stimulus artifact.
So we need to compare the stimulus to correlate the principal components with the stimulus PCs.
You can see that only one of the principal components correlates with the stimulus PCs.
And so let's not worry about that.
It looks like it's not much of a problem.
So now that we know that all subjects share the same semantic space, we can actually aggregate
all the data together, do PCA on the data aggregated across subjects, extract one principal
component space for all the subjects, and then project each individual person's data
into that principal component space.
And that's what we did here.
You can see it works pretty well.
Now we've done dimensionality reduction.
The next stage is to visualize the data.
Now remember the matrix, this weight matrix, maps between the feature space and the brain
space.
So to interpret the weight matrix, we can either look at the feature space or the brain
space.
Let's project into the feature space first.
So this is the feature space.
This is a four-dimensional space.
Of course, we can't plot four dimensions on a two-dimensional surface.
So let's pull out the first PC because it's probably boring.
And in fact, the first PC is boring.
The first PC is always boring in any problem.
It's energy, bright, fast-moving things versus slow, dark-moving things.
Of course, bright, fast-moving things make more neurons fire, and that makes more bold
signal.
So you'd expect that that would be the first PC.
Let's ignore that.
Now we'll take the other three PCs and we'll color code them.
So we can take each individual feature and color code it in this three-PC space and map
the individual features back on the WordNet tree.
Now if you don't really understand where the heck this came from at this point, and I wouldn't
blame you if that was true, really all you need to know is similar colors on this map
are semantic concepts that are represented similarly in all human brains.
So all these yellow things here are animals, and they're all represented similarly.
People are all represented similarly.
These are all indoor and outdoor scenes.
Well, indoor scenes.
They're all represented similarly.
These are all vehicles.
They're all represented similarly.
These are communication verbs.
They're all represented similarly.
There's a lot of data in here.
And of course, this is a three-dimensional structure, so it might be a little easier
to look at a movie out of it.
Now remember, there's 1,700 colors here.
I've just included a little cheat sheet of 16 colors here so you can kind of see what's
going on.
I'll walk you through this.
This flat yellow thing here is animals.
It's kind of on a branch off by itself.
This green thing here is people.
This stick that's sticking out here, completely different from everything else, this turns
out to be atmospheric phenomena.
And this little ball that's sticking out all by itself turns out to be text.
They're different from everything else.
This purple stuff down here is vehicles.
These blue things are indoor scenes.
Now there's some weird things in here.
You notice there's this little red ball here that kind of seems off by itself.
It's hanging off of humans, but it's kind of out here, and it looks like it's reaching
toward this little ball here.
This little red ball turns out to be athletes.
Athletes are kind of represented differently from humans because these were a long view
of athletes, so they're in a kind of athlete context, which is different than the normal
human context.
And then this little ball here, you wonder what this is.
It's kind of reaching down towards athletes.
This turns out to be talking animals because in these movies, we had cartoons, and even
though there was no sound, the animals were making talking motions.
So when your brain sees a talking animal, it says, well, what the heck is this?
Is that an animal?
Is it an athlete?
I'm not really sure.
So it kind of...
Anyway, all right.
So there's a lot of data in here, okay?
Now you can also take this matrix and you project it onto the flat maps.
And this tells you where semantic concepts are reflected on this or organized on the
surface of the brain.
So I'm...
For some reason, I showed a left hemisphere here.
Here's the visual system at the back.
Here are the structural areas are here on the far right.
And these are the semantic areas.
And this is somatosensory and motor.
This is speech down here.
You can see that these semantic selectivity is widely distributed across the entire surface
of the cortex here.
Now, remember, these are all vectors I'm plotting.
So the vectors have two components.
They have a length and a direction.
And here I'm only plotting the direction.
So I'm not showing you how much of the variance in the voxel's responses these semantic selectivity
vectors account for.
I'm just showing you what the tuning is toward.
And the main thing to take away from this is it's not like there's individual islands
of semantic selectivity that are organized for certain specific semantic categories like
faces and places.
Instead, there are these broad gradients of semantic selectivity that spread across cortex.
This is humans and animals.
This is indoor scenes, motion, vehicles are here back in the structural areas.
But I think that's because although we tried to remove all the motion energy from this
model before we projected onto the surface of cortex, the vehicles still have some residual
motion energy we couldn't completely remove.
So this is a fairly complicated map.
And now the problem is, as David pointed out, although subjects in some sense have a similar
semantic space, it turns out if you look across subjects, the details of every single subject's
map are different.
And aggregating those subjects into a common map space turns out to be an interesting challenging
problem.
Now, one thing that people oftentimes ask me is, well, okay, this is there.
People aren't doing anything in this test.
They're just staring at a dot and they're ignoring the movies.
What happens if they pay attention to the movies?
That's a legitimate question.
After all, if you record in neurophysiology, even as far back in V4, when an animal searches
for pattern A versus pattern B, you'll see changes in the gain of the neurons, you'll
see changes in the baseline response, you'll see changes in the tuning of the neurons back
in V4.
They're not huge changes in tuning, they're about 10%, but they are there.
If you go to the frontal cortex, Earl Miller's lab has shown that neurons can completely
change their tuning depending on what the animal's looking for.
So can we see that kind of effect in humans?
To do this, we did a simple task.
Instead of having people just stare at the movies, we had them stare at the dot while
searching for something.
In task one, search for humans.
Whenever you see a human, hold down the button.
When there's no human, release it.
In task two, search for vehicles.
Whenever you see a vehicle, hold down the button, and when you don't, release it.
Simple task.
Now we fit two models, one under searching for humans, and one under searching for vehicles.
So we can look at the exact same voxel under these two situations.
Here's a voxel in the fusiform face area.
You can see when you search for humans, this voxel's mostly tuned for humans, and also
animals, they have faces too.
But really, the person was searching for humans here, and this voxel mostly responds to humans.
But when you ask the person to search for vehicles, although the voxel retains its human
tuning, it now all of a sudden sort of lifts up its leg and expands its tuning curve to
also include vehicles and objects in a similar sort of semantic space from vehicles.
So this voxel is shifting its tuning curve depending on what you look for.
You can average across all the voxels in the brain while searching for humans versus searching
for vehicles, and then you can just subtract the models for all the voxels, and you can
see when you're searching for humans, essentially tuning for humans increases, and when you're
searching for vehicles, tuning for vehicles increases.
And if you want to know where this happens in the brain, you can just plot this black
on the flat maps of the brain.
So this is a really complicated color map that I hate, but I've never figured out anything
better.
The way to think about it is basically red means the voxel shifted towards humans, and
green means the voxel shifted towards vehicles, and I'm just showing you the situation when
you're searching for humans and searching for vehicles, and what you can see is the
shifts are enormously, sorry, the shifts are distributed all over the cortex, right?
They're everywhere, and in fact, structural part of the cortex, early retotopic cortex
does not shift all that much.
Most of the shifts occur in these semantic areas, higher order visual system, but also
up here in frontal cortex.
You see a lot of substantial shifts up here.
So this is a very widespread phenomenon, and the way I like to think about it is think
of your brain like a computer farm, except it's not like Amazon Cloud where any computer
can do anything.
Instead, there are sort of sub farms in this computer farm.
There's a bank of computers that's used for vision, there's a bank of computers that's
used for audition, and there's a bank of computers that's used for somatosensation, and those
are kind of fixed, but then you've got this more general like Amazon Cloud that's your
general purpose computer farm, and that's going to do whatever you need to do at any
point in time.
So if searching for humans is what your job is, then your whole brain, most of the front
of it becomes a giant human detector.
Now one thing you might ask is, well, wait a minute, maybe this is just an artifact.
After all, the BOLD response is a blood flow response, and blood flow is related to blood
pressure, and blood pressure is related to arousal.
So maybe what happens is when people detect humans in the displays, they get excited.
They say, I've detected the human.
They hit the button, their blood pressure goes up, BOLD goes up, and then when we do
regression, it looks like tuning got shifted to humans, even though it was just blood pressure,
which would be the most boring thing possible.
So how do we get around that?
Well, it's really easy in this task.
We actually just go through the movies, and we remove all the cases where humans or vehicles
ever appeared.
So now there's no humans or vehicles.
They never make a positive response, but we can look at the tuning of all the voxels for
anything that's not humans or vehicles.
And because we have this semantic space, we can look to see if the tuning of all the non-humans
and vehicles voxels shifts towards the location in semantic space where humans and vehicles
are located, even though there are no humans and vehicles in the task.
And that's what this shows in a kind of arcane way.
Basically, you get shifts predominantly over toward the direction of the attended target.
So this does not need the targets present.
In fact, I think this is the only MRI experiment he's ever done where you could do this, where
you could show an attention effect without the targets being present.
There was probably not related to blood pressure.
So if we just collapse over humans and vehicles, you see that red here means the tuning shifted
toward the target, and blue means the tuning shifted away from the target.
You notice in most parts of the brain, it looks like tuning shifted toward the target.
It's kind of like your brain is acting like a matched filter.
If you're searching for dogs, your brain says, well, okay, I'm gonna try to basically
increase the representational space for dogs at the cost of representing other things because
they're not important.
I'm gonna be able to represent dogs with more resolution and more fidelity, and it's got
more hardware, and so everything's gonna get better, and you're gonna do better at your
dog detection task.
There are a few places, however, that don't actually shift toward the attended target.
They shift away.
The TPJ, the cerebral frontal cortex, and this thing up in the precuneus.
And those are three very interesting areas because those turn out to be part of the default
network, and they're a part of supposedly self-awareness part of the brain.
So our current theory about this is that essentially when you're doing an external detection task,
a search task, it's very important for you not to get confused between events happening
in the external world and your fantasies of what you'd like to see in the external world.
So actually, the parts of the brain that monitor your internal state actually shift away so
that they're less likely to interfere with the targets.
So in the last few minutes, yeah, this is all gonna be online.
If you're curious about this, there's a lot of data in here.
So we're building a little tool that'll enable you to basically go in and just click any
point in the brain and pull up its semantic tuning and its structural tuning and actually
other kinds of tuning that we have.
And then you can sort of just steer the brain around.
So here I'm looking at the front, and this front part of the brain, it's non-visual.
It's still tuned for people, it looks like.
Here's a location, FFA.
It's tuned for people.
And if I go to a different place in FFA, it turns out it's not really tuned for people,
it's tuned for text.
And I go over to PPA here, and this is tuned for vehicles and objects and so on.
So this little tool will be online.
Anybody can just go in and run it.
It's a pretty neat tool to play with.
Now in the last few minutes, last five minutes, I'd like to talk about decoding, which is
probably what a lot of you are here for.
You may wonder why.
I thought he was a decoder.
Why did he spend the whole day talking about this weird neuroscience encoding stuff?
And the reason is, I think if you want to do decoding, the best thing to do is build
an encoding model rather than just try to decode blindly.
If you try to decode blindly, you're basically in the machine learning world where it's all,
the quality of your decoder is going to depend on how much data you have.
In this business, you're never going to get enough data.
But if you build an encoding model and then turn it into a decoder, then you can leverage
all the information we have about neuroscience and about the way the brain works to build
your encoding model, and then you get decoding out of it for free.
So obviously, if you could build a decoder, you could do a lot of things with it.
You could control wheelchairs or pinball machines, which would be awesome, I think.
Or you could build a speller.
All these systems use NERS or EEG.
NERS or EEG are, well, actually, I think this is also an EEG system.
These are all EEG systems.
EEG is not a very good way to measure brain activity, and you're only going to get a couple
bits out of EEG.
If you want to build a good brain decoder, you're going to have to use MRI or the portable
version of MRI, which is going to be NERSDOT or something like it.
This is a refresher for what I talked about before.
We've got a stimulus and some brain activity.
We build an encoding model, which gives us the probability of the response given some
function of the stimulus.
Now, to do decoding, we take this likelihood.
We multiply it times a prior, and we get a posterior.
No problem.
It's simple, trivial math, except for one thing.
We don't know this, P of S. We don't have a prior on natural movies.
This is a problem of computer vision again.
Computer vision people have not solved the prior on natural movies, so we don't really
have a P of S to use.
So what do we use?
Well, in our lab, we basically use what's known as an empirical prior.
We just sample a large number of images or movies.
So if we're decoding images, we have 50 million images we downloaded from Flickr, and we use
those.
If we're decoding movies, we have 5,000 hours of video we downloaded at random from YouTube.
Google hates us for this because it took us six months, and they do not like you doing
it.
But we have this 5,000 hours of video that we can use for a decoder and so on.
So basically, we're going to decode in terms of these libraries, and our quality of decoding
is going to be limited by the size of our libraries.
So this is the first paper we published back in 2008.
On the top are three images we showed.
On the bottom are the three reconstructions from our Flickr database.
Remember, these images were not in the database, so we could not actually find the actual image
you saw.
So here you notice there's a harbor scene.
Here's a harbor scene.
Here's a bunch of people in a room.
Here's a bunch of people in a room with their hands up.
Here's two buildings in the front, one in the back.
Well, this is a coliseum.
It's kind of a building.
You notice the contrast is lost here because, of course, we're measuring MRI, and we lose
contrast information because the visual system codes luminance as above the mean and luminance
as below the mean in two different channels.
And those get aggregated together in a bold signal, so we don't have contrast.
But these reconstructions are quite good.
And if we had, say, 50 billion images instead of 50 million images, we stand a really good
chance of finding an image that was very close to the exact one that you saw.
This is a different...
You might think, okay, well, you can decode actual images that you showed, so maybe you
could decode something like visual imagery.
Is that possible?
Well, Thomas Nasalaris in my lab did an experiment where, for a week, he just stared at these
four images.
He'd just sit at his desk and stare at the Mona Lisa for hours, trying to make the perfect
internal image of the Mona Lisa.
And then he went into the magnet, and we just put up a little word, Mona Lisa.
And he had to, like, make a mental image of the Mona Lisa.
And now we pretend that this is the same data we had before, and we shove it through the
regular model, and we see what we get.
And that's here.
So the Mona Lisa becomes Salma Hayek, and the soldier becomes a horse, and the kitty
becomes a dog, I think, and the vegetables are vegetables.
So this actually works pretty well.
And the really cool thing about this is, remember, this is the wrong model.
You know this is the wrong model, because your brain must do something different when
you're looking at an actual image versus when you're doing visual imagery.
Because if your brain did exactly the same thing in those two cases, you wouldn't be
able to tell the difference, and you'd probably walk in front of a car or something.
So it's the wrong model, but it still kind of works.
OK, I'm just going to take you through a few other models.
This is a model that leverages some intermediate visual areas that have both structural and
semantic information.
So we set this decoder up to be, you query it.
You can basically ask it four questions.
Where are the animals?
Where are the man-made objects?
Where are the people?
Where are the plants?
And it just returns these pink squares.
It doesn't know anything about the image.
And then you overlay the image to see if you got it right.
So here it finds the swan.
Here it finds these buildings.
Here it finds the people.
And here it finds the office plant.
So that kind of works.
That's nice.
Oh, this is an interesting decoder that uses latent Dirichlet allocation.
This is a machine, a decoder where the underlying feature space was learned using machine learning.
And we never implemented it directly by hand, but then we fit it to people's brains.
And to take a long story short, this decoder essentially returns a probability that a scene
belongs to certain scene categories, and it returns a probability of objects.
So you see there's this land.
It thinks this is a guy with a spider on his face.
It thinks, I think this is land animals and few people and maybe food.
I'm not sure.
And there's a man and maybe a woman and a person and a head, and it's an animal.
And here it gets a sports scene.
It says there's a sporting event and some large crowds and few people, and I think there's
an athlete and some people and a man.
You can see in all these cases, it's doing really, really well.
So many of you have probably seen this.
This was our structural decoder using the motion energy model.
This was on YouTube last fall and was very popular.
On the upper left here, we have the movie that we actually showed someone.
Now, when we decode, remember, we're not just decoding into one image.
We're getting a posterior distribution over all 5,000 hours of video.
So each one second of these 18 million seconds of video has a certain probability.
Here I'm showing the 15 most likely clips in the posterior.
And when the image has a face, then all of the clips in the posterior look kind of face-like.
So when the image is something really abstract and there's nothing like it in YouTube, then
all the clips that we have over here in the posterior are really bad.
So in that case, we just average over the top 100 and we show this here.
So this is called the maximum a posterior solution up here, and this is the average
high posterior solution down here.
And these are both valid decoding results.
You can use whichever one you want.
Now I'm kind of bored with this.
It's like a three-year-old result now.
I don't really want to look at it anymore.
So the thing that's really cool that I like is when we use the semantic model to decode
the semantic concepts from these exact same data.
And that's shown here.
So we're decoding 1,700 nouns and verbs simultaneously from people's brains.
And I've just, I'm outlining, I've arrayed them on a two-dimensional surface here just
so they don't all overlap each other.
The size here refers to the probability that that thing was happening in the movie at any
given time.
So you can see that here it gets hand, man, woman, talk, dog.
You may notice it came up a little early, but there was the dog.
And the timing's a little bit off right now.
It's pulling up things a bit early.
This is kind of a really short, mixed-up scene, but then it kind of settles down.
Clouds, body of water, mountain.
Here's some buildings, sky, vegetation.
You see people here, road, walk.
Then in just a second it will change.
You see here's some women.
So then we change.
We get text for the text here.
Then talk, woman, man.
The man will come up in a second.
There he is.
Room.
Your brain isn't actually predicting.
You just haven't adjusted the time right.
You can't see into the future with this device.
Flower.
And then we switch to underwater here.
Body of water, fish.
Water, swim.
It's not perfect.
You know, we don't have a manatee, so it thinks it's a whale.
Well, OK, that's an honest mistake.
In general, this thing works amazingly well.
It's really quite impressive because, again, we're not really optimizing these decoding
models.
We optimize our encoding model.
Our decoder, we're just kind of getting just by implementing Bayes' theorem.
It's not really any magic at all.
Oh, in the interest of time, I'm going to skip over this.
This is decoding navigation.
So where are we now?
Well, I like to think of this as the beginning of photography.
So this is the very first photograph that was ever taken by Joseph Neepche in the south
of France in like 1825 or 1882.
I can't really remember.
A long time ago.
Almost 200 years ago.
And it's a lousy photograph, but you can still tell it's not random.
It's not white noise.
It looks like maybe there's a building.
Maybe this is a roof.
This is maybe the side of a barn.
This is maybe another building here.
This may be a building here.
Maybe it looks like there's a horizon here.
This is probably the sky.
You can kind of tell what this is.
And, you know, between this very first photograph and today, now we have so much resolution
and so many pixels and such high quality video that people actually don't even want to pay
for it.
It's not even worth it anymore.
So video is kind of a solved problem.
In this case, brain reading and our ability to decode brain activity, it's all contingent
on the quality of brain activity measurements that we can make.
And right now, there's a little bit of by MRI.
The quality of models that we have, and those are very, very coarse.
And the quality of computers, which is still fairly slow, although, of course, that doubles
every few years.
So you can bet that brain decoding technology is going to advance really, really rapidly,
especially once we have better measures for measuring brain activity.
And thank you for your time.
I appreciate it.
I think we have very brief time for questions.
So I guess one of your grad student questions.
I'm just wondering, like, the biases that he has when he's making the model for, like,
grouping things together, like, across cultures.
Does that change at all?
Because if you introduce someone from, say, like, Africa or something, and they've never
seen some of those things in the images before, how does your model cover that?
Okay.
So the word net was not done by my graduate student.
That was actually done by this group of linguists and psychologists at Princeton.
It's been worked on for 20 years.
And it's a very, very, very, very, very, very, very, very, very, very, very, very, very, very,
very, very, very, very, very, very, very, very, very, very, very, very, very, very, very, very,
The bias part came in when my graduate student decided to code things in the movies.
And in that case, you know, nobody wants to go through and code every single thing, so
he picked the salient things, and there could have been some bias that was introduced at
that stage.
But that's going to be a bias of omission, meaning there was just a bunch of things he
didn't code at all, so we couldn't represent them in the encoding model.
We couldn't account for the part of the variance in the brain that relates to the things that
weren't encoded, and we can't decode them.
As far as the cultural issue, I feel pretty confident that at least for people who come
from an industrial society, it doesn't matter, because one of our subjects was born in Japan
and just came here a couple years ago, one of our subjects was born in Sweden and moved
here as a child, one of our subjects was born in Korea, and one of our subjects was born
in Russia, and one of our subjects was born in America.
So we've got pretty much the whole spectrum of industrialized countries in the lab, and
those are the, our people are the actual subjects here.
So I would assume that this will generalize to any group of smart Berkeley graduate students.
Yes?
You are examining here only things that are actually seen, but that way, things that can
be inferred from the pictures.
And I assume that this will be a very interesting talk for you, whether you can determine whether
people have inferred something.
Yeah, I like that, because that's not just imagery, that's like making inferences about
the world, sort of having a narrative, constructing a narrative of what's going on.
So have you tried anything with this sort?
We haven't for vision.
Movies are really a giant pain in the rear end to work with, because you have to code
everything by hand.
So to get at this issue, it's actually easier to use speech.
So we have a whole other part of the brain lab that I haven't talked about, because nobody's
ever presented at a meeting yet, on language decoding, where we fit various sorts of language
models to just corpuses of normal speech, and these issues then come out in that kind
of data.
I think, I think your intuition is correct.
You can essentially, this is a completely open modeling frame, where you can model anything
you want.
If you decide to make a set of regressors that have to do with inferences about the
scenes rather than the scenes, you're perfectly entitled to do that.
And that's an avenue we're pursuing in various ways.
Yes?
How big does the sample view have?
So how many subjects did you cast?
So, in my, since my background is both in human psychophysics and single cell physiology
in animals, I tend to be a low end kind of person.
I want to get as much data as I can from a few people, rather than getting a lot of data,
sorry, a little tiny data from a lot of people.
So all the experiments in my lab are generally small, and experiments, this experiment with
movies, I think we've got a total of seven subjects now that have, that have watched
these movies and for which we've built models.
And each individual subject is modeled individually.
There's no aggregation that ever goes on.
So these 30,000 models you have to build, have to build, have to be built for each individual
person.
Now, the idea of aggregating across humans and pooling the data together is really, you
know, really important for understanding things.
And I kind of showed you one way to do that, which is to just aggregate the weights and
take the principal components and see if there's anything in common.
But that's not really satisfactory because ultimately what you're going to want is to
take this functional data and link it to David's connectivity data.
And so that means you have to have a way to basically map the individual people, either
get the connectivity for the individual people, or map the individual people back into the
canonical space.
And that's, you know, as David pointed out, that's a challenge.
And the government doesn't pay me very much money to solve that challenge, so I'm hoping
someone else will do it first.
All right.
I think we need to get ready for the tours, but let's thank Professor Young again.
Thank you.

