A Scalable Method for Cellular Blood Flow and Other N-body Systems
Sep 5, 2023
Natalie Beams is a graduate student being supervised by Dr. Luke Olson and Dr. Jonathan B. Freund. Her current research interests are in computational studies of blood cells flowing in small vessels. Her work has included both using simulations to try to understand specific aspects of blood flow and attempting to improve the computational methods behind the simulations.
Good.
So real quick, right before I get started,
I just want to acknowledge my two advisors on this project,
Professor Olson from Computer Science
and Professor Friend from Mechanical slash Aerospace.
So what was the original motivation behind this project?
As you may have guessed from my title,
it had to do with blood flow.
So Professor Friend and several previous and current members
of our research group have done numerous studies
about aspects of the mechanics and dynamics of blood flow,
red blood cells, transport of white blood cells.
Here's an example of when I was trying
to look at what are the conditions that
determine whether cells end up looking like this or like this,
the same size tube with the same flow rate.
But the problem was we were coming up
against some limitations in terms
of the scalability and feasibility
of certain types of simulations with our current code.
So trying to come up with new ideas in this
was the motivation behind it originally.
But there's other potential applications
of this electrostatics.
I bolded because I'm going to talk more about it, quite
a bit more about it, actually.
But also these other types of plasma dynamics,
gravitational dynamics.
Really the common thread is just applications
where you have dynamics that can be described
by Green's function solutions to the Poisson equation.
So that means combinations of the solutions
to the singular Poisson equation,
where the right-hand side here is the delta function that's
only non-zero at one point, so the singular force
on the right-hand side.
So more about what I'm talking about with the singular Poisson
problems here.
The model problem we've been using is from electrostatics.
And so we just have a collection of n-point charges
with certain strengths.
And the potential due to all these charges at a location
is a discrete sum of these singular Poisson problems
associated with the location and the strength
of the different charges in the collection.
And then over here with the blood flow,
we have Stokes flow, so where the Reynolds number goes to 0.
And we're satisfying these two equations here.
And here the delta functions, the singular forces
are on the surfaces of the cells,
so the effects of the cell surface on the fluids.
So the Green's function solutions for these
look like this.
Again, for electrostatic, we have just a discrete sum
of the strengths and the Green's function.
And this Green's function here is the 1 over r decay,
the familiar electrostatic potential from physics class.
For the cells, of course, it looks
more complicated with this boundary integral formulation,
but it's still just a combination
of these Green's functions, except now we
have an integral over the cell surfaces
instead of a discrete sum.
And it does get more complicated.
I see, oh, this is not the updated version of my slide.
But this is the same as electrostatics plus an extra 1
over r cubed term.
And then this other Green's function
is formed from two derivatives of this Gij.
And it has also r to the fifth in it.
So basically just a more complicated version of this.
But the two problems are definitely related.
And that's why we're using the electrostatic problem
as our model problem for this.
Maybe on the surface, they may not seem like they're related.
But that's why I'm going to be talking
more about electrostatics.
Green's functions have global support.
So the solution at one point in your domain,
you need information from all the other points
in the domain because of this very slow decay of the 1
over r type potential.
So a lot of the current codes in these areas
fall into two major groups to avoid
the n squared cost of a direct evaluation
of this global method.
And if you were here this morning for Professor
Rabiot's talk, you heard a lot about fast multiple methods.
Great talk.
But the idea behind that is that you
can, for groups of points that are further away
from the point you're looking at,
you can use this multiple expansion instead
of having to do every point individually.
Or another major category of these methods
is the so-called P3M methods, where
what the idea behind these is it takes that Green's function
that has slow decay and splits it into two parts.
One part is short range, so it does have quick decay.
So when you're evaluating it, you
can just keep a certain small subset of points
instead of the entire collection of points, which
cuts down on the cost.
And then the leftover, the smooth part here,
is solved for everything on a mesh.
And the proposed method I'm talking about
falls into this P3M framework, though I
do have a few things related to multipole in there.
So more about P3M then, since this
is the framework I'm talking about.
The most popular variant is the PME, the Ewald sum version
of it.
And the way that it accomplishes this splitting of the Green's
functions is by introducing Gaussian screens.
So here we have a singular force, a delta function.
Then if we add in the screen, and also subtract it back out
there, we still have our original force
we started with.
But if you take the potential of these two things together,
that's how you get this.
It's a short range thing that decays much more quickly
than the original one over our potential.
And because we form these screens out of Gaussians,
then you can solve this mesh problem really well
with the Fast Fourier Transform, or the FFT.
And this is great.
In a lot of ways, FFT has a lot of really great properties.
But unfortunately, scalable parallel implementation
is not really one of them, due to the communication
requirements of the FFT.
It becomes a real bottleneck in the code.
And this is the code I mentioned earlier
that I had the pictures from the simulations
on the second slide.
This was a PNE code.
This is the main problem for scalability of that code.
So the idea here is, can we make a new method, a new version
of P3M, which has the same idea, but we're creating it
such a way that all the components are going to be more
local in nature, so that we can have better parallel scaling.
The idea that Professor Ben came up with
was, what if instead of using Gaussian screens,
we build our screens out of piecewise polynomials
of order p?
And then for this mesh shell, the smooth part,
we can solve it with finite elements of p plus 2 order.
We add the 2, because we need two integrations
for the solution.
And the neat thing about that is,
because we're using this specific space, the p plus 2,
we're formulating it so this finite element solution
is exact.
So the potential we're looking for,
which is two orders higher than our screens,
it is in this finite element space
that we're looking for solutions in.
And then we can replace that FFT
with a multi-grid solver, which has
a lot better parallel scaling.
And it's really well-suited to the Poisson problem.
Just a little bit more about these screens.
So how do we actually choose screens that will work?
You set the coefficients of the piecewise polynomials
such that the integral of the screen
is equal to the strength associated with that point.
And then you cancel as you set to 0 as many moments
as you can about the screen around the point charge.
And what this does is, if you look
at terms from the multipole expansion,
it removes leading terms from the multipole expansion
for all the moments you can cancel.
And so then that increases the decay rate
of your short-range potential that's left over.
And the correlation here is, if your screen is order p,
you get short-range.
It's going to decay like 1 over r to the minus p plus 2.
So only your screen would be like 1 over r cubed instead
of 1 over r, et cetera.
And here are just some examples of slices
from increasing order of screens.
And you can see, as you increase the order of p,
you can kind of hone in on the location of the charge
there that you're wanting to screen.
So that's why you can get better and better decay,
more moments canceled.
So that's great, but does it actually work?
This is a really simple test case,
just to show the ideas here about this trade-off
between order and the two different parts of potential.
So we have 1,000 randomly distributed point charges
in a triply periodic box.
And I'm taking a look at the potential
along the diagonal here at 100 different points,
just sampled along this diagonal through the cube.
And here's what the potential looks like along that diagonal.
It's got order of magnitude of 10 to the 2.
And then for my initial calculation,
I used this method to calculate the potential.
And I used linear screen, so p equals 1,
and a relatively short cutoff distance.
So that's how many of the points do
I need to keep in that smaller subset of the short range
potential part.
And here, the magnitude of the error
is 10 to the minus 2 for this case here.
And if we want to decrease the error, what can we do?
We can either increase that cutoff distance
so we can keep more charges in the short range part
before we say, OK, I'm going to cut it off here and only use
this part.
Or to get a more drastic improvement in error,
we can increase the order of the screens to quadratic screens.
And of course, as you increase the order of the screens,
you also are incurring more work in the other things
you do in the solve.
So there's always this trade-off between what's
the optimum value of p for what you want for the problem.
So big picture, big things I was talking about,
you have best multiple methods for these problems.
You also have P3M methods, which includes the Ewald sum, which
is very popular now, and also what
we are proposing, a new version of P3M
designed to be parallel.
And takeaway points, so we're going
to replace the screens that are designed for an FFT solve
with screens that are designed for a finite element solve.
An exact finite element solve.
And the nice thing about this property in finite element
solve is that you can choose the mesh size based
on your architecture or maybe physical aspects
of the problem, something you want to model.
You don't have to worry as much about accuracy demands
for your mesh.
So that gives you a lot of flexibility
in tuning the method to what you're running on.
And finally, it's compatible with general boundary
conditions.
Basically, if you can impose the boundary conditions
on your finite element solve, then you
can put them in this method without too much
changing of the other parts of what's going on.
So thank you for your attention.
And if you have any questions, I will take them.
I didn't have any experiments to check
if your bounds are good bounds in terms of scalability
and scarcity in terms of problem size.
I haven't actually scaled it up into a lot yet.
That's what I'm working on right now.
So this is still just running on very simple problems.
So if not, that's exactly what's again.

