Medical Imaging System Design
Sep 5, 2023
Nov. 8, 2012. BioEngineering Seminar Series. University of Illinois Urbana-Champaign
"Advances in the science of medical imaging system design"
Michael F. Insana, Professor and Department Head, Bioengineering, UIUC
Nghia Q. Nquyen,
Sara Bahramian,
Craig K Abbey,
Yeah, yeah, I'll let you know. I'll let you know.
I just wanted to see on her picture.
I gave him a candy.
You know what we're trying to get?
No, it wasn't like I just like punched it like that.
No, what happened was, okay, so I was throwing a bar at him.
I just threw it at him.
No worries.
I stopped making him mad and threw it back at him.
And so I just catched it in the air and threw it back at him.
Well, that worked fine until he started to get smart.
I didn't throw it when he thought I was looking.
I kept catching it, but there was this one time when he threw it super close to the wall.
It just hit the wall.
And I wasn't looking.
So out of my peripheral, I reached for it.
But of course, the wall.
Yeah, so I just like reached for it and I like basically punched the wall.
And I was like, ow.
All right.
That was hot as hell.
Amazing.
We're going to get started.
Usually I'm the guy introducing people.
And since I'm the guy, I'm not going to introduce myself.
But I'll warn you, there's some autobiographical stuff in here, right?
And I always think, especially since many of them are students here, I know when I was a student,
I was always interested in somebody's trajectory and how it's very rarely ever a linear trajectory.
Most of the time, you bounce off walls like a pinball machine.
But in my case, I was a little bit directed.
So I'll tell you a little bit about that.
But first, I'll point out that this is me.
And these are two of my students from the electrical engineering department.
Nia just graduated last year.
He's at Los Alamos doing imagery construction for breast CT systems.
And a lot of what I'm going to talk about today was part of his Ph.D. thesis.
Sarah's off to the FDA.
I'm going to talk about the FDA today because that's where I kind of grew up.
I did my post-doc there.
And it was an important part of that.
And so she's actually working in the same lab that I did when I did my post-doc a few years ago.
And then Craig Abbey is a professor at the University of California, Santa Barbara.
He's been a collaborator of mine for more than a decade.
His area, he's a Berkeley math guy, but his area is in visual perception.
So it's a really nice sort of combination of what we did.
So what I did was almost exactly 29 years ago, this week, on November 11th at 11 o'clock,
the 11th hour, the 11th day of the 11th month, I defended my Ph.D. thesis at the University of Wisconsin.
And I was successful.
And a few months later, I worked really hard after doing a post-doc where I was to finish up writing some papers.
I worked really hard to get into the FDA because of what was happening there at the time.
So I'm going to tell you something about that story.
It was a very exciting time.
I was a little bit late.
They had just sort of finished things.
We were trying to do some new stuff in ultrasound, which was my background.
So what I'll do is I'll start out by talking about the 1970s, and there was this big crisis in medical imaging
and how that would motivate new thinking about what image quality really meant.
And actually, that would begin the leverage of trying to quantify this thing.
So we started leveraging things like signal communications theory, visual perception ideas,
and a whole bunch of the work that was just done 20 years earlier in television.
We talked about concepts like what is information quantitatively and what is performance.
How do we rate performance?
We define them rigorously.
And out of that grew this thing called medical image science,
and it really started to grow up very, very quickly in the 80s and 90s, mostly for photon-based imaging systems.
I'll talk a little bit more about where that is if you're interested in that.
And then what I'm going to spend most of the time on is our recent breakthrough in this context of doing this for ultrasound
and where there was a conceptual problem that Nia was a big part in helping us figure out.
So what's this crisis I'm talking about?
In 1977, the United Nations Scientific Committee on the Effects of Atomic Radiation put out a report that said,
mammographic mass screening with high-dose techniques carried out on young women can increase the occurrence of breast cancer.
You can imagine what people thought of that.
It made it on 2020 with Barbara Walters sticking her microphone under people's nose and saying,
tell me why we're killing young women.
So this was a big crisis because now people are losing confidence.
The AMA had just been trying to convince people to get breast mammograms, and now it turns out that the quality was not there.
In fact, they found that radiation dose in breast per mammograph radiation is in the range of 10 to 35 rads,
which in today's SI units is centigrade.
Now, you may not know what that is, but that's a very large dose.
So at the time, these two film techniques that were coming out, where you basically do two orthogonal views,
generally could be undertaken for a breast dose less than about one hundredth of the average that was being applied.
And if you look today at 2012 digital mammographic systems, we're about that same range right now.
So obviously there was something going on here.
We weren't using the information that we wanted to.
So let me give you a little bit of history from this citation right here.
So Johnny on the Spot was the FDA team.
They were created in 1971 for the responsibility of consumer protection against unnecessary exposure to radiation-emitted devices in home and occupational use.
And the branch at the time was called the Bureau of Radiological Health.
And there was a bunch of group, a group of people.
Almost all physicists at the time were working there to try to figure out some of these things.
So this is, let's see, I started graduate school in 1980 and I finished in 83, at the end of 83.
So I was there in 84 through 87.
So it was a little bit after the beginning of this sort of thing.
But in 1972, this team was assembled.
This guy right here is my mentor, Bob Wagner, a famous guy in imaging who died a few years ago.
And I think of them as Dave and the three Bobs.
These are the guys that I worked for when I first went there.
Dave was the boss.
And Bob Gagney, Bob Wagner, and Bob Jennings were physicists that were dealing with what is radiation, how do we minimize the dose, maximize the information.
So today it's the Food and Drug Administration looks like this.
And in fact, BRH has morphed into CDRH for Center for Devices in Radiological Health.
They're responsible for all kinds of devices, radiating or not.
And they just moved a couple years ago to Silver Spring away from Rockville, Maryland.
So here's the kinds of questions that we would ask.
These are three images that might be taken from a patient.
These are mammograms, breast imaging.
These are not ultrasound images.
These are x-ray images from a digital system.
And the questions that they were asking at the time in the 70s to try to address this is what information is there available from these images and how do we quantify it?
What are the relevant tasks?
By tasks, this is the reason we're taking these images.
So it's related information but not necessarily.
How should I measure diagnostic performance when I have a bunch of people looking at images?
How do we know what's good and what's not good?
And then once I have the benefit, how do I evaluate that versus risk?
So here are the pioneering people that Bob turned to at the time.
Albert Rose, a very famous guy, came up with a vision model.
Again, a physicist from Cornell.
He was the head of RCA Labs.
And he was put together to pioneer photography and television, essentially.
So during this period of time, he was the head of the lab.
And he worked with a guy named Otto Shade, who was another one of these double E's at RCA.
And he was the first one to develop a unified theory of image analysis.
And he did this right after the war.
I mean, it was like immediately once a bunch of declassified information became available.
So he studied their work quite a bit.
And then he also looked at the work by a guy named Horace Barlow, who I think is still around.
He's a UC Berkeley MD.
He was the first to postulate a computational objective of the human visual system.
He actually mapped this out computationally.
It was a very exciting time to give this quantitative understanding of what was going on from our eyes into our brains.
His claim to fame is he's also the great-grandson of Charles Darwin.
I met this guy once.
He was a really interesting guy.
Rodney Shaw, another Englishman from Cambridge, spent most of his career in industry.
Xerox and Kodak, big imaging companies.
He was then recruited away to begin the first image science Ph.D. program at the Rochester Institute of Technology in Rochester, New York, where he did this.
And it exists today.
He then left to join Hewlett-Packard Labs, where he's still working.
And he was the first one to model the imaging system as an information channel.
And he championed a concept known as NEQ.
Those of us who were trained in X-ray imaging know noise-equivalent quanta as a quantitative information-theoretic measure for how much information is flowing that can also be related to all the instrumentation property we can measure in the lab.
And then along came Bob Wagner, who synthesized all of this.
He was there from 1972, almost the beginning, until he died in 2008.
I'll tell you more about Bob.
He's the guy I work with.
And then the ideas that Bob came up with were actually synthesized and put into a mathematical framework over 20 years that became this book right here, The Foundations of Image Science.
It's written by Harry Barrett at the University of Arizona.
And Kyle Myers, one of his students, as is Craig Abbey, who I work with.
And Kyle took my job when I left the FDA, and she's now the boss.
He's in charge of lots of things there.
So these guys created this book, and it's really a 1,600-plus page tome of not light reading.
Nevertheless, it's all there.
It's one of those kind of books.
Rodney Shaw, with another British engineer, Chris Dainty, generated this book, Image Science.
It's the first one that I know of.
I used it when I was a Ph.D. student at Wisconsin.
It's a little dated now, but there's been quite a gap between the two.
So Bob says there are rules for image quality.
He says, first of all, quality of the medical image depends on the task, the reason for obtaining the image, and the observer.
So he says it's a package deal.
It's not just the instrument.
It's what you're looking at and who's making decisions.
Image quality is very statistical.
Not only the noise that gets generated, but patients have variabilities.
They have to be averaged over, and readers have variability that need to be averaged over.
Therefore, he turned to statistical decision theory, where we look at human observer analysis,
and there's a science called psychophysics, where a lot of this evaluation was put together back, again.
It began in World War II when people were looking at radar screens, looking, is that an airplane, or is that a flock of birds?
They had to try to make that decision when they were working on these things.
And out of that came what's today the gold standard for evaluation of these systems,
called receiver operating characteristic analysis, or ROC curve analysis.
To combine that with the human observer, in statistical decision theory, there's a concept known as the ideal observer.
And this is someone able to use all the information that's available in the data, but only that which is available,
and it's hindered by the noise that's in there.
So it's a mathematical construct, in fact.
And what we would say is this is a mathematical observer limited by task performance,
only by the information that's contained in the data.
So let me start by showing an x-ray system.
Here's a current mammographic system that what we start out with is the patient.
I'm going to use this.
Those of you taking my class know that I take high-dimensional objects and I string them into long vectors, right?
So vector F is my object.
This is the task, so I'm going to see this lesion in this compressed breast right here that's placed in this area.
And what comes out of it from the primary detector are data that I'm going to call a shorter vector called G,
which is my primary detector data.
And then what happens is all of this stuff from the x-ray source, filtration, column meter, compression paddle,
anti-scattered grid, the detector physics itself is all the system that maps objects into data.
And what I've done with that, I display the data.
In the old days, I would develop the film.
Currently, I would process the electronic data in a form, and I produce the mammogram like this.
So what we would say today is that we have patients, given by F, that enter into an acquisition process.
So we have a linear mapping here between objects and data through this linear operator that maps the two.
We add some noise.
That's our acquisition stage.
And then we have a display stage, right, where we take the data and we do some processing to it,
and we get some representation of the original patient data, and then at the last step is an observer.
So this is what Bob had come up with.
He says there's patients, there's acquisitions, there's displays, and there's observers, and there's mathematical formulisms for this.
So one of the things we can do then is we can take the output of this, measure the human observer's response,
and stop at this acquisition point, measure the ideal observer's response, and then compare the two.
And the comparison gives me an efficiency.
And it says if I'm 1% efficient, I have 99% of the information I'm missing.
So there's reasons to go on.
This is a motivation for things like signal and image processing, right, to try to mine more of the information that comes out.
And it could be through the display or just the limitations of the human eye-brain system that provides a lot of these things.
Okay, so here's a typical experiment.
It's actually not a typical experiment.
This is just because I'm using the same images again.
But I would give two images, and I'm going to do something called a two-alternative forced-choice experiment.
So one of my images has some pathology I would like somebody to detect that's quite subtle usually, and the other one does not have it.
And so you show people the pathology, what it might be.
I've included this little round structure in this one just to point to something and say, tell me which of the two that are in there.
And when you have that, what you do is you say this one is the benign case.
What you do is you look at the probability distribution of the responses of the observer to the data, G.
When it's a benign condition, you get a distribution like this.
The probability of the responses to the data when there's a malignant condition, you get a response like that.
And if they're separated a little bit, you have some diagnostic capability.
If they're widely separated, that's very good.
If they're not so wide, if they completely overlap, that's very bad.
So what you can do is basically draw lines like this threshold right here and start integrating probability distributions to get these cumulative distributions on either side.
And what you can do is start to plot the false alarm rate to the probability of detection, and you generate a curve called the ROC curve from this kind of a data.
And again, when they completely overlap, these two, the line is a 45, the area of the curve is 0.5.
When they're completely separated, you have a perfect imaging system.
Then the area is 1 because the curve looks like this.
And so the whole universe exists between these two.
And again, this is the gold standard a lot of times for image analysis.
So what Bob did is he came up with this classifier for the ideal observer.
I'm using Harry Barrett's notation for this.
And what he says is the test statistics, the scalar quantity derived from the data that you look at, take a whole bunch of data and get one number from it.
And what that is, it's basically given by fairly standard stuff.
It's in Van Tree's book on signal detection.
It's the log likelihood ratio.
So this is the probability density of the data under condition 1, which may be the malignant case, versus the probability of the data under condition 0, which is the benign case.
You take the ratio, and I take the log, and I get this test statistic, and it's guaranteed to be the best performing given as long as my probability distributions are valid.
And if I put in for these Poisson cases that I have generating this data, generally what we find is something that Harry calls the Hotelling observer.
It's a machine observer that takes data.
Here's the data right here as a vector.
And it applies this filter to it.
And that filter gives me a scalar that I compare to a threshold.
And that's how I make decisions using an ideal observer, unlike a human, which would just look at it and tell you which one is there.
So the important thing here is this test statistic is linear in the data.
That's a very important thing.
And it's also proportional to the difference in the means between the two data sets, the mean difference and linear in the data.
And I'll come back to that when I do ultrasound in a minute.
So now what you can do is take moments of these distributions, right, the means, difference of the means squared, the difference between this mean and that mean,
divide it by, square it, divide it by the average variance between the two, create a signal-to-noise ratio for the ideal observer.
And then what Bob's contribution was was to separate this in an integral expression in terms of spatial frequencies into the task.
This is the Fourier transform of the contrast that's inside the body.
Fourier transform of that squared, magnitude squared, kind of proportional to a power spectrum, times this NEQ quantity, noise equivalent quanta.
And so this was really interesting.
It says the performance of an imaging system depends on the task.
Not only that, it depends on the response of the system in the same spatial frequencies the task exists.
Integrate that across all spatial frequencies, and I can predict the best possible performance of a system.
So this NEQ, which was really a good insight, can be broken down into image contrast, modulation transfer function squared, which is spatial resolution, and noise power spectrum, which is noise, right?
So when contrast is high, image contrast is high, that's good.
When MTF is high, that's good.
When noise is high, that's bad.
So we want to put that in the denominator, right?
So he was able to map all this out.
And I'll show you a difference of that for ultrasound in a minute.
So the signal to noise ratio that we get here for the ideal observer, it can be mapped into a detectability index that can be related to ROC analysis.
So now we can compare theory based on all the instrumentation properties, predictions with actual measurement, and then the ratio of those detectability indices squared is efficiency.
So that's sort of a summary of image analysis today in a lot of ways.
I'll show you one example quickly that I sort of glommed together from these NEQ measurements from a second generation CT system.
So this is back in the 80s, right?
This is kind of old stuff.
So NEQ, so large NEQ is good.
That means many of the photons are carrying information when the numbers are high.
This is spatial frequency, right?
So you can see this object right here, which you can also see here.
And this is being represented through two imaging systems, A and B.
And A has a higher NEQ, looks a little nicer than B does.
And so what's happened is if I look here at this one, that task is way down here in the low spatial frequencies.
So when I were to multiply that task by this NEQ and add them up as I've suggested here, my ability to see those objects is approximately equivalent as you might be able to see between the two.
But as I start to rotate around here counterclockwise, right, you see there's four objects here and then smaller, four more objects, and then smaller, four more objects, and it just keeps going.
And what happens is I can clearly see these, but these are really pretty mottled.
And so what happens is you go from this system to this system and you lower the NEQ for the same task, the system transfers less information.
So that was the sort of interpretation on this and which Bob used.
And, in fact, this is a standard in a lot of X-ray imaging techniques today.
FDA requires you to measure things like NEQ and other quantities.
So what is information?
I'll mention something very quickly.
You know, Shannon in 1948 defined information pretty simply.
He says if you have some message, right, we'll call that M, send it through an encoder, transmit the message,
encode it, and you look at what you receive, then the information contained in that message is equal to the log of 1 over the probability that that message is going to happen.
So you can imagine if the message has a probability of 1, then 1 over 1 times, and you take a log of that, you get 0.
That means there's no information transferred, right, because you already know what the message is before it got transferred with probability 1.
But if you receive a message and the message is a very low probability message, then there's a lot of information in that particular one, right?
So what we can do is then take that quantity, which I'm just re-representing here, over all possible messages and average it by multiplying by the probability distribution and summing it up,
and you get the average information that people use now, right?
And this looks like entropy.
So what we can do in imaging is we can take the same thing.
We can look at information and diagnostic performance.
We can look at the message being sent as the object.
The encoder is the imaging system.
We get some data, which gets transmitted to the display, which gets decoded, and then we end up with some data here that a decision maker looks at.
So we get the same sort of information measure.
In fact, these guys Felgett and Linfoot back in 55 actually used this sort of analysis.
And then what Rodney Shaw did that I showed you earlier in the 70s is he derived NEQ, and he related it back to the information metrics.
He related it to the Shannon information channels using Fourier methods, provided you had a linear shift invariant system and stationary processes,
which, of course, never occurs, but it's a very nice way to begin that process.
So what he said was then the detective quantum efficiency, that I throw photons at somebody,
what fraction of them are providing information to its efficiency measure depends on the NEQ in those spatial frequency channels divided by the total number of photons.
Therefore, we think of NEQ as the number of photons that I throw at patients that carry information that aren't absorbed or whatever else that can happen to them, right?
So Wagner-Brown in 85 applied this same analysis that came together through the ideal observer to relate information to performance.
So here we have performance that we can measure ROC-wise to a signal-to-noise ratio from the ideal observer that can be broken down into the task and the instruments.
So that's a complete sort of analysis for all that.
All right, now I'm going to switch over to the kinds of stuff that we do in sonography.
So we tried the same analysis in ultrasound, and we ran into some problems.
Here's an ultrasound system scanning breast tissue, and, of course, we have an object function as well.
Instead of linear attenuation coefficients for X-ray, it's going to be an impedance discontinuity, and that's what gives us our contrast.
So we have that object function mapping.
We acquire the data.
We add noise, and we get RF data, G, and we can choose to display it in a B-mode display, which is our little F hat for ultrasound, and then let human observers look at it and make decisions, right?
Or we can send this off to a computational observer, like some of the ones I showed you, that hotelling observer that's quite often used if you want to see what the best linear unbiased observer would do to the data.
Of course, we don't know if the human visual system is not linear, so we don't know if it mimics the humans in many ways, but it still can be used, and we use a different version of that.
And then we can also tap off the RF, compute the ideal observer, and start to compare various versions of this to try to follow the information flow as it goes through the system in some way.
And we end up with the same equations I showed you originally, just with new physics attached to them.
So here's how ultrasound works a little bit.
We electronically scan the linear array.
We have a 45-millimeter aperture with, you know, very finely cut elements that are generally about a tenth of a millimeter in size with a really, you know, between 0.1 and, I'm sorry, yeah, a tenth of a millimeter, which is 100 microns.
So that's our pitch between elements.
And what we do is we scan this across while beamforming, and every time we send out one pulse or we receive it, we get an RF signal.
This is what I was calling G.
Then we map this into by taking the envelope and maybe log compressing it, and that becomes one line of our beam mode image.
So we just keep electronically scanning, and we can produce one of these frames easily within a 60th of a second in a progressive mode, so we can do real-time imaging, no problem, in ultrasound, one of its main advantages.
So beamformers are the way you make your money in this business, right?
So you think about how you're going to transmit, how you're going to encode information on the transmission aperture, and then when you receive it, how you're going to put them together.
And typically what we do now is a simple display and sum beamformer, and I'll talk a little bit more about that later.
But basically you get this sort of flow of information as beams are focused.
So you start off by exciting a bunch of elements that kind of flow down this channel into this little pulse here, and as they go through, they start to send back echoes, right, that are received by the same transducer that are listening.
And then these get summed in some way, however we want to beamform this signal, to increase the signal-to-noise and give us some focusing properties.
But the reason I want to show you this is what's intrinsically different about ultrasound in pulse echo mode is that the information is not encoded in the mean of the tissue like it is in x-ray.
It's actually encoded in the variance of the tissue.
And where that comes from, it takes a while to explain, but I'm not going to do this justice, but basically what's going to happen is if I look at the mean value of the acoustic impedance, I get no signal back if it's homogeneous.
In fact, if I put a ramp on it, a constant first derivative, I still get nothing.
I have to have a second derivative.
The second derivative is curvature, it's surface.
So I need an impedance surface change before sound will reflect off of it.
And I can show if that's the case that that's a second-order measure, right, or basically it's encoded in the variance.
All right, so that's what I have here, right?
Object contrast is encoded in the variance of these stochastic objects.
Okay, so now let's go ahead and do the same two AFC experiments with a malignant benign case.
We get these distributions from the human observer case.
We generate ROC curves just like we did in mammography.
Now let's look at what the ideal observer shows.
Here's what we had put together.
The log likelihood ratio, as I showed you before, we no longer have a Poisson process.
In fact, we have information encoded in the covariance matrix.
So what happens is I now have the ideal observer's test statistic dependent on the square of the data.
It's quadratic in the data.
It's not linear anymore.
And here's where the contrast comes in, is in the inverse covariance matrices under the two different conditions.
So it's embedded in the variance.
So it's a very different sort of mathematical formulation that comes out of the ideal observer.
So now I have to use a different information matrix.
I'll talk about the Kolbeck-Liebler divergence that we use.
It's basically going to be given by the difference in the mean values of the test statistic.
And what we came up with, this is Nia's thesis topic, is basically we've been able to separate, again,
the contrast from the object from something to do with the imaging system.
But it's no longer NEQ.
We call it the acquisition information spectrum.
And this is no longer a difference in the means.
It's a difference in covariance matrices in some weird way.
So what we found when we did this, it takes a long time to explain and I can't,
but we get these NEQ kind of quantities, you know, how the system is going to be sensitive to information.
But we have to autocorrelate it before we can figure out what the information spectrum is.
And I'll show you what that means in a minute.
This was, we think, sort of the missing link that prevented us from making this connection.
And now we have this connection now between information, between performance,
and then we can relate that to the area of the ROC curve that we can measure for human observer performance.
So now information in this case, because we're doing a two alternative forced choice,
we have two images that we're deciding between.
So what Kovac did back in the 50s is he created this quantity here, J,
which is the average information in data needed to discriminate in favor of one class over another.
So you got to assume you have two classes, benign and malignant, all right?
And how do you do that?
Well, first of all, you substitute probability density functions in for probabilities.
And you get J is, if you look here, there's a summation, right, this integral over probability density.
So this is the mean value of a log of the ratio of these two things.
And so instead of log of P0, as you would have for a Shannon information metric, it's P1 over P0.
And similarly, we can do the average for over P1, right?
So it's P0 over P1.
And you add these together, and this gives us what Kovac-Liebler has used as the divergence
for measuring the information difference between these two sets, these two classes of data.
So it took us a long time, and I can't tell you about all the things,
but the information contained in the data is not our objective.
It took me a while to understand that.
We don't want to always see all the information.
We just want to have maximum performance.
In other words, if there's a discrimination between malignant and benign, if that's my task,
I want to maximize that performance, not necessarily the information I can withdraw,
because if I withdraw information I can't access with my eye-brain system, it's of no use to me.
So it turns out for a lot of ultrasound tasks,
the information and performance are going to be equivalent to each other.
So here's how we got going when I worked with Craig on this.
We took the class of data.
This is a malignant lesion.
This is a benign breast lesion.
And we basically did sort of a basis decomposition of breast lesions, okay?
So we're applying a lot of sort of analysis to breast lesions.
We did this with physicians.
Physicians helped us figure out how to do this so that we can write, you know,
likelihood functions based on diagnostic features.
And one feature is basically is there a lesion there or not.
If I take the difference between these two, I get this one, and I call that my task.
Another one is if you look here, this malignant feature tends to be more vertically oriented
than a benign feature.
That's just typically the case.
And so what we did was we used eccentricity.
And you can see by taking the difference between these two what the difference would look like.
These malignant lesions tend to have fuzzy edges, so a fuzzy edge versus a sharp edge.
Speculations, you see the little outgrowths here in various ways versus smooth edge.
So we have speculations.
And then finally the fact that they're both sort of low contrast lesions, but this one's anechoic.
There's nothing in it.
And this one's hypoechoic.
So we then took a difference between the two just like this, but we made them both high contrast
and then took a subtle difference between the two.
Now why did we do that?
Well, because then what we could do is take one task at a time, create images, image pairs,
and do two AFC studies with humans as well as analysis on that
and discover how much information was being transferred for various kinds of imaging technologies.
And what I'll point out is these three tasks right here are very edgy tasks,
whereas these here on the ends are sort of like large area tasks.
And it turned out to be an important difference between how we did this.
So here's how we synthesize images.
We take a Gaussian white random field, we multiply it by a task spectrum,
and that modulates the variance essentially in our white Gaussian field.
So we have low variance in here, high variance out here.
We then image that, add some noise, we create RF data.
Here's why humans don't look at RF data.
It's hard to see stuff, right?
Even though there's more information here than there is over here,
we still have to convert it so our eye-brain systems can appreciate it.
So we run it through a display operator.
And here's the B-mode image for a fairly subtle lesion.
And one of the things that the ideal observer told us to do is after we beam form,
but before we envelope detect for our display,
we run it through a various form of a Wiener filter, which decorrelates the data,
and then we display it.
And you can see that we get a little bit nicer data form when we do that.
So that's what we'll look at.
I'll look at B-mode images and then some Wiener filtered B-mode images later on.
And we put this out about six years ago now.
So here's some of the data that we got.
We start out with patient information, run it through the acquisition,
we study the ideal observer, we display it, and look at the human observer.
So for case A, which is this data right here, the gray line is the B-mode images,
and the black line is the Wiener filtered B-mode images with human observers.
There's 20 human observers looking at thousands of images that give us this data.
And what you can see immediately is that we're only between 0.1
and 10 percent efficient in extracting information from B-mode images
for all the five tasks that we look at.
The three edgy tasks were very, very challenging compared to the large area tasks.
If we Wiener filter, things get a little bit better, in fact, about an order of magnitude better,
which was interesting.
So we use that as motivation for doing some post-B formation processing,
and we're working with a company on doing this.
Now if you look here, what we can do the next phase is, if you watch up here,
I'm going to change this diagram, and I switch it from human observers now to the Smith-Wagner observer.
So I get rid of the vagaries of the human eye-brain system, and I put an algorithm at the end.
And when I do that, I get this data right here, which says I get things kind of boosted up a little bit,
but I get a huge separation still between these two responses for the B-mode and the Wiener filtered B-mode
with more than an order of magnitude difference between the two.
And finally what I'm going to do is look at human observers versus my machine observer
while looking through the display, and here I even get an inversion between the two.
Now it's a little bit more efficient for humans to look at the original B-mode image than it is here.
So what this tells me is the efficiency of the human eye-brain system relative to a machine observer,
and that's not very limiting.
I mean, still, when you're down around 10%, you would think it's pretty limiting,
compared to over here where we're seeing most of our loss is passing it through the display of the system.
So to display things, what we do is we take real signals to make analytic signals out of them,
which gives me a real and imaginary part, and then we do some processing.
When we're done, we collapse this back into a real part and throw away the phase information,
and that will come back a little bit later.
So the takeaway from this is that we're very inefficient.
Depending on the task, our efficiency varies wildly, and the processing also.
Look, you can see this processing doesn't do much, but here it works tremendously.
So as instruments now become more and more task-oriented, with that information,
we can go ahead and reprogram our tasks based on the kind of patients that we're looking at.
So let me do a little bit of kind of hardcore stuff here.
I apologize for this, but I'll see if I can have this make some sense.
In ultrasound, this K is a spatial frequency property.
This is like any Q.
So it basically says this is the magnitude squared of my system response function.
This one right here is sort of the speckle noise properties,
and then this is my additive noise properties here.
So what I want us to see is high spatial resolution.
I want to see small amounts of speckle, and I want to see small amount of noise.
And what I can do is I can do this case where I have very little noise in there.
So when that drops out, I can cancel these two terms.
In this case of S, over a bandwidth of just one over the object, and that's what I plotted here.
So I can show that at the center frequency of my signal over some bandwidth,
I end up with this sort of sensitivity of my system to certain scattering data from my patient
is given by this information right here.
This is what I'm sensitive to things.
Now, if I want to ask the question, where's the information that I'm sensitive to in the image?
You know, I look here.
I'm not sensitive to anything with very large, low contrast values.
So what I have to do is autocorrelate this function here,
and then what I do is I find this AIS term which maps into the contrast spectrum
and tells me what my performance would be.
And when I autocorrelate that, I get this function.
Now, off what you can see, here's the data right here.
If I have a large, low contrast lesion, that typically means I have an object spectrum that looks like this,
someplace in the low frequencies.
So when I integrate this and add it up, I can see this data quite well.
If I get outside the higher frequencies in my envelope bandwidth,
that is I get structures smaller than the speckle size that I'm showing here, I get nothing, right?
And that's kind of what you expect.
The thing we didn't expect is this thing over here, right, this high-frequency probe.
I mean, it makes sense to us physically because, you know, in optics,
you can typically see half a wavelength kind of is the theoretical resolution,
and this is at twice the center frequency.
So it's physically possible.
But I'll show you an experiment where we actually probe this a little bit,
and we could see that we can be sensitive to information.
And even though the information relative to the object is way up here at this high frequency,
it all has to be filtered through the communication channel,
which is where the system's interacting with the tissues.
So it's a little bit more complicated system for ultrasound than it is for other things.
And we've just published one part of this this year, and Nia's still got two papers out coming in.
So here's kind of a cool experiment that we did.
And we said, can we put in a task spectrum, an object contrast spectrum,
that kind of looked like a delta function, a very narrow band task spectrum.
And when we did that, we used this little Gabor pulse here,
which is an ellipse with a sine wave modulating it.
If we image it, it looks like this.
And if we meter filter, it looks like this.
So you can see something's lost in the translation.
But this pattern right here, if we look along that axis, is at that frequency right here.
It's a fairly narrow band, not this narrow band, a fairly narrow band.
And then what we can do is, if I multiply this, run it through the imaging system,
that's like convolving a delta function with AIS,
and what I get is basically the value of AIS at that location.
And then I can change this frequency and do it again,
and I can sweep through the entire bandwidth,
and I can experimentally validate that this curve is correct.
Now, unfortunately, NIA shows it here in a log scale and down here in a linear scale,
but this red curve right here, which is what we predict, is the same as this blue curve right here.
It's kind of hidden over the side, okay?
And then what we did was we predicted, basically this red curve is our prediction,
which is the blue curve here, and then what we did is we made measurements, right?
We actually did this experiment, and we ran it through a computational observer,
and when we did this, we get this black curve right here, which was fairly close,
saying our prediction was close, and, in fact, we are sensitive to this high-frequency information.
We did a couple other things.
We then took and we looked at the B-mode image, okay,
and when you do that, you lose the phase information by that,
and here's what we got when we did that by looking at these images.
Our information spectrum kind of dropped down to here, and right when we get to the minimum,
it just stayed at zero, so we lost information, right, because of the speckle that's here.
It's correlated data, and we lost all the phase information,
and if we ran it through a linear filter, this image right here,
we could recover part of the information that comes from the amplitude of the signal
but nothing from the phase.
So I think we're starting to learn how these things function and work.
Okay, where does this pay off?
As I said before, if you want to make money in ultrasound, you figure out how transducers work, right?
You make very highly efficient devices, and you create really interesting beamformers
that adapt to all kinds of patient situations.
It's a real hot topic in our area.
So if you take the, here's a transducer, here's a simulated object right here.
If we look at the output channels from each of the received channels from each of this,
here's what the impulse response looks like from each channel.
So you can see they're slightly diverted a little bit because you're pitched at slightly different angles.
You can add noise to each of those.
We can create RF images from each individual channel, run it through a beamformer to create the beamform data,
and then display it into this kind of an object here.
And when you focus the beam, here you can barely see here is a highly focused beam.
Here's the envelope.
You can see it looks like a big spike, right?
And as we know, spiky kind of impulse response functions are really good for imaging.
That means we're not blurring much.
And as we go to the far field, you can see how it's sort of spread out.
If you look at the envelope, you can see there's a very broad, blurry sort of response down here.
So what Nia did for his thesis is he looked at all these different tasks.
But here's for task four with speculations versus a smooth case, the difference between the two.
So here these are all lesions with speculations on them.
These are the smooth cases.
And you have to look at them.
And maybe here you can start to see that there's some edgy anomalies.
And here it's kind of smooth.
But if you look at a delay-in-sum beamformer, a typical system here,
we get this sort of efficiency given by the green curve.
It says you get highly task-dependent efficiencies for the observers.
And it's not too good, right?
We're below about 10 percent in most of the cases.
You can use a match filter to line up beams.
Instead of using a delay circuitry, you can just match filter your data,
and it lines everything up, which is great.
The downside is it blurs everything.
And so here's the price you pay.
You get a little bit lower sort of performance.
In fact, this is what the image looks like when you use a match filter technique.
If we use our Wiener filter technique versus a minimum variance beamformer,
which people are developing now in the industry quite a bit,
you can see that we can pick up our performance quite a bit.
So now we think that this sort of machinery is there
to evaluate different beamforming systems under different conditions.
And we think that's going to be a big help
by quantifying the performance under these different situations.
And the final thing I want to show you, here's a picture of a fetal picture,
a little baby down in here at 8 weeks.
One of the things that ultrasound has done is back in 1976 they decided,
I don't want to worry about bioeffects,
so what I'm going to do is sort of cap the bioeffects at the current standard.
And what that standard did, one way to measure that is a mechanical index,
which is 1.91 and draws it right here.
So that says right now if you want to get FDA approval for any of your devices,
you cannot go past a mechanical index of 1.91 without a lot of explanation
and basically go through a PMA process, which is very painful and expensive.
So I was talking to Bill O'Brien, who's spent his career looking at bioeffects,
which is really important for things like babies.
And so the question that we ask is, why did they pick 1.91?
It seems kind of arbitrary.
There's no real physiological reasons for picking that.
So we said, now that we have something that we can compare,
let me look at my three tasks for lesion detection
and study how the efficiency increases with mechanical index.
So if I go above 1.91, do I get any net benefit?
And it turns out I do.
In fact, it's not as much as maybe this change as I'm getting here.
So now we're going to start to ask the question, because I can rate performance,
maybe it's time to revisit sort of these artificial bounds on how ultrasound is being
and what outputs you can put out there, how much power you can put into a patient.
So we're thinking of working with Bill to try to look at the bioeffects
and then couple that with performance to perhaps give some new concepts
and ideas of where the ending should be.
So here's my summary.
What I want to leave you with is this whole idea of image science is very cool stuff.
And I went to the FDA because I had this sense that cool things were happening there
in the development of image science at the time, and I wasn't disappointed.
I was there for a big part of many of the things that were going on,
the tail end of sort of screen film technologies
and the beginning of digital technologies and x-ray,
but I spent a lot of time with MR and ultrasonic systems.
Image science is cool because it blends medicine and biology with instrumentation,
statistics, physics, and communications theory.
There's a whole bunch of really broad-spectrum things that kind of have to come together for this.
And one of the things that we found in this particular study that we did
is the physics of acoustic scattering present a very different relationship
between the instrument and tissue properties compared with other modalities.
So the way we do this for x-ray isn't going to work for ultrasound,
and it's because information is encoded in variance, not in mean.
So we get these quadratic test statistics, which have big impacts on how performance is rated.
Beam-mode imaging is only 1 to 10 percent efficient, sometimes less than that.
So the largest loss is from this demodulation step that happens in image formation.
So we think there's a really important role for signal processing.
First of all, how do you combine the elementary beams?
How do you weight them, filter them individually, then combine them in various patterns,
and then maybe filter them afterwards?
And all of that could have an enormous effect.
And not only that, but beam-forming strategies should vary with clinical exams.
So as you walk up to an ultrasound scanner today, you generally push a button and say,
I'm going to do breast imaging, or I'm going to do abdominal imaging, or kidney imaging.
It only changes various things, but we can get really specific.
If you want to say, I'm going to look for a lesion, or if I want to look at lesion boundaries or something,
you can actually optimize that system in ways.
It doesn't cost anything.
It's all software reformatting of the system.
So we think that's really exciting.
Performance is highly task-dependent.
So if I look at one thing versus another, the only way I can rate the quality of an imaging system
is in the context of what I'm looking at.
And that suggests that beam-formation strategies are going to vary with clinical exams.
As I said, we have tools to re-examine output guidelines.
And I just want to finish by saying medical image science has developed to the point where it now guides,
it does in x-ray, guides the development and evaluation of most imaging modalities.
And we've tried to make our impact in ultrasound for that.
So since I have a couple minutes left, I wanted to point out one funny thing that Bob was always told jokes.
He was a very serious guy in a lot of ways, right, in science.
Here he is standing on the beach in California with a drink in his hand,
and we're discussing important things, right.
So he was always having fun.
So here's this paper that I was referring to in 1985.
It was published in Physics and Medicine Biology.
And the eventual title was Unified SNR Analysis of Medical Imaging Systems.
Very noble.
What he proposed, what he actually sent to the editor at first,
was Grand Unified Analysis of Medical Imaging Systems.
And I said, Grand Unified.
Well, he's a physicist, so everything's a grand unified theory, right, in that.
And he goes, No, no, no.
Here's why I did it.
If you take these acronyms and you get guano medical imaging systems, right.
So if you don't know what that is, it's bat excrement, right.
So he was trying to make fun of himself, a little self-deprecating humor,
and he didn't want to tell anybody.
He just wanted to put this out there, and then he could tell that story many years later.
But the editor would have none of it.
So he trimmed his title back, and that's what it was.
Okay.
Well, thank you very much for your attention.
Hi, Ron.
Hi.
So great talk.
I really enjoyed it.
Thank you.
It seems like you didn't give me really a limiting time here.
The thing that I didn't bring up is the aberration the body provides, right.
So just like in telescope imaging, you want to see point stars, right,
but you've got to go through the atmosphere.
The same way the body tends to give a sound speed like a random screen, right.
So as you're passing this phase front through this random phase screen,
it sort of messes up your focusing ability.
So you can only focus so big, and so you're going to be limited in your total aperture
because the apertization effects or the aberration effects get worse as your aperture grows.
I don't know if it's due to the aperture bump.
If you're focusing on a certain camera, we'll see the aperture bump as well.
There are, and I think the main problem right now is until we get 2D arrays that are functioning at real-time rates,
which we can't do.
There's many efforts to do this, but it's not fully functioning without doing a sparse array kind of analysis.
It's the second, so when you only do one-dimensional, it's not quite enough, right.
You need the second dimension.
Plus, they're only doing phase correction, and I think they have to do some,
Bob Wagg at Rochester talked about doing some amplitude correction on each channel as well.
So I think the solution will happen, I think, quite naturally with 2D arrays
with both amplitude and phase correction adapted.
And if we can do that in real-time, we're done.
Right now, the problem is if you want to have, say, 100 by 200, maybe 1 or 200 by 1 or 200 element array.
So that's 10,000 transmit-receive channels, right, and you need a wire for each one of those.
You've got to put in a little tiny wire that you can put on a patient.
And then the heat, they haven't been able to figure out how to add that high-power sort of stuff
without dissipating a lot of heat, so you start burning your hand when you hold these devices.
I've been to Siemens and looked at some of these things, and they said,
here, grab onto this, and you start going, and you go, whoa, it's getting hot.
Yeah.
So part of it is going to be just technology.
Interconnect technology and sort of low-power electronics has been solved.
And sparse arrays.
Yeah, people really look at sparse arrays, and there's been no solutions that people have found
that have been satisfactory.
I don't really have an analysis of what's limiting there,
but the, you know, they just, without side-lobe cancellation that you get with sparse arrays,
you know, there's a problem.
Yeah.
Any other questions?
Sure.
You said you worked with physicians to actually get those typical tests, those five tests that you identified.
Yeah.
And this was based on discussion, or how did that happen?
Well, it turns out about 1995, there's a guy named Stavros that wrote in Radiology,
a widely-used clinical journal, what are the criteria for lesion detection?
And he basically listed a whole bunch of them.
So there was a lot of discussion amongst physicians.
So I was working with Karen Lindfors at UC Davis when I was there at the time,
and we asked her, what do you think of these things?
And she said, oh, this one's important, that one's not.
And I said, well, tell me why.
And she started showing us images and cases and things.
So it just came like that, basically, is she kind of put a rank order on what she uses in her practice
for how she makes the diagnostic.
It wasn't a detection problem in that case.
It was a diagnostic problem, trying to classify a known lesion as benign or malignant.
And we didn't use some of them, like shadowing effects.
There were another vast array of things that we couldn't sort of codify into likelihood functions.
But that was kind of fun to try to work with physicians
and then see if we could write this into these conditional probability distributions.
Yeah, it was fun.
Any other questions?
All right.
Well, thank you.
Thank you for coming.
Thank you.

