μ -BFBT Preconditioner for Stokes Flow Problems with Strongly Heterogeneous Viscosity (Johann Rudi)
6 Video Views·Sep 5, 2023
14th Copper Mountain Conference on Iterative Methods
μ -BFBT Preconditioner for Stokes Flow Problems with Strongly Heterogeneous Viscosity
Johann Rudi
3/24/2016
The paper is very well written. I enjoyed reading it. The overall look, figures and tables, is very polished.
The numerical results are very impressive.
It's an important application. It's well-motivated.
It's a pretty impressive paper for a student.
And lastly, a good extension of Stokes' solver to a very viscosity case and very large-scale parallel computation.
Thank you.
Looking forward to talking with you.
Thank you very much.
Okay. Yeah, that was a very intimidating introduction.
So, I'm working – I'm at Weisbaumer Gattas.
And Georg Stadler is a good colleague of mine, and I consider him almost as a mentor.
He's at NYU.
And so, yeah, you mentioned there's no theory behind it.
I'm working in a regime where it's very difficult to come to any theoretical conclusions.
Or maybe if you get to something, then the bounds are so large you cannot really use it.
And I would actually like to start with one of the main results in this presentation to get everybody kind of excited for the topic,
because you have been sitting here for a while.
So, I want to solve a Stokes system where the A block, the 1-1 block, depends on a viscosity called mu.
And I want to use a preconditioner, which is an upper triangular block preconditioner.
And in turn, you have to find approximations that always have a tilde in my presentation.
So, you have to find an approximation for the inverse A block and the inverse Schur complement.
It's defined in this way.
So, this is the nasty part of the Schur complement, the inverse of another big matrix.
And this first slide just compares one method that I call state-of-the-art for the Schur approximation.
And the newly proposed method here, I call it mu-BFPT.
And the key to it, this is the mu-BFPT.
It's in blue.
This is the state-of-the-art.
And the key component will be a certain scaling matrix, which is lumped mass of the velocity weighted with the square root of the viscosity.
And what we can see, if I set up a certain problem, and we'll show you the problem in a minute,
that as we increase problem difficulty this way, then the state-of-the-art preconditioner, it struggles a little bit, which is maybe okay, you might say.
But at some point, the number of iterations just go up dramatically.
And what I will be presenting is something that's very robust over a wide range of very difficult parameters in the viscosity in this block.
Okay.
Now, let's explain better what I do and motivate, and I'll tell you the challenges.
So, in computational science and engineering, commonly, you have to face, you want to solve creeping non-Newtonian fluid models.
So, typically, you take the incompressible Stokes equations, and you have a nonlinear rheology, and oftentimes it's a power law rheology.
So, if you linearize that with a Newton method, for example, then you get a spatially varying heterogeneous viscosity.
So, that's one component where the heterogeneity can come from.
Another one can be just in your model, you have a heterogeneity already built in, no matter if you have to linearize or not.
So, this is the PDE I want to solve, and this is already a linearized version.
So, this only depends on space.
The mu, the viscosity depends on space X.
I'm seeking the velocity called U and pressure called P.
It's incompressible, the right-hand side forcing, yeah, is determined by the problem.
So, then I apply, I want to apply an unstable finite element discretization, and that's why there's a zero in this block.
The viscous stress block becomes this matrix A dependent on mu.
I want to highlight the dependence on mu.
The gradient is B transpose.
The divergent becomes the B.
And now, this needs to be solved, or in large problem dimension, we want to solve it iteratively.
And we apply this Aperturing Block preconditioner, and you have to find approximations for A tilde, for the viscous block called A tilde, and this inverse for the shock complement.
And to be more precise, you have to find operators where you can find the action of a vector, or where you can compute the action of a vector onto this operator.
You don't need to actually form these operators.
Now, the problem I'm mainly working on and I'm mainly interested in is earth-mantle convection, and it is a pretty complex and challenging problem, and it has to run in parallel.
So, some of the challenges I want to just briefly tell you.
It's the rheology.
It's nonlinear, heterogeneous, and isotropic.
It has sharp gradients in the viscosity.
Look at this picture, for example.
From the blue to the red, this is a drop of six orders of magnitude, and that's only within just a couple of kilometers.
You have highly localized features and a wide range of spatial scales.
This is one of the main features and problems, because the plate that's kind of extending in this direction will be moving depending on what is happening here, and these are complicated nonlinear physics.
Local mass conservation is crucial.
Okay, these are a lot of challenges.
I'm only going to focus on a couple.
So the MOON-BFPT for the Schur complement approximation will be the main part.
I also will mention a hybrid spectral geometric algebraic multigrid method that will be used for the elliptic operators.
There will be three.
One is the A block, and two B cycles will be used for the MOON-BFPT.
Discretization will be unstable.
I would consider it quite advanced discretization.
It has a QK elements, or there's K for the velocity, and PK minus one with discontinuous molar pressure space for the pressure.
Okay, I will make some simplifications, but always keep the earth's mantle convection as the target application.
So all what I'm going to tell you should apply to earth's mantle convection, or it can be extended.
So no complicated rheology will be used, no spherical shell, and no adaptive meshes, just uniform meshes.
Now what's the benchmark problem that we want to test our preconditioners with?
It will be a cube domain.
It has a low viscosity medium, and inside this medium I will include so-called sinkers, the high viscosity in blue.
And I will vary the number of sinkers, and in addition they will be placed at random points.
Sorry.
And this makes for a more complicated flow, and it makes it more difficult to solve.
The viscosity, I've wrote down the equations, are fairly straightforward to implement.
There's a scaling of an indicator function.
The indicator function will tell you where the sinkers are.
There's a shift by minimum viscosity.
I want to point out that the minimum should be much lower than one.
The maximum should be much higher than one.
Every sinker is kind of a Gaussian function, and I put them together by multiplying them all together.
The right-hand side forcing is what makes the sinkers go down.
It's kind of a gravity pull.
So two parameters will be right.
Number of sinkers, dynamic ratio, it's a global property of maximum viscosity or minimum viscosity.
And for algorithmic scalability, I will increase finite element order K,
and I will increase mesh refinement level L.
Okay, what is MU-BFPT?
Recall this equation that I showed you two slides before.
This is what I want to solve, and this is the sure complement that I'm trying to find,
or rather a good approximation for it.
So the underlying principle of MU-BFPT is that you want to find a special matrix X
that if you formulate this viscous block times the gradient,
then you want to flip those and find the X that would satisfy this equation so that it almost vanishes.
Another way to put it is as a least squares minimization problem,
and this is done here, and there's a special norm that you have the flexibility to use different norms.
And also, I haven't been mentioning this, matrices C and D will be diagonal scaling matrices
because that's the only feasible option if you want to compute things.
So solving this minimization problem, you arrive at the BFPT approximation for the pressure.
It looks long, but this is a Poisson operator for the pressure.
These are straightforward matrix applications, and this is again the Poisson operator for the pressure.
It just has a different scaling.
Okay, now what will make this BFPT robust is actually the choices of the scaling matrices C and D.
And this is what I've been doing, finding them numerically,
and I will show you the results that actually show that it's a very robust choice.
And the choice will be the mass matrix of the velocity weighted with the square root of the viscosity,
and the lumping makes it diagonal.
Okay, here.
This is the important result in terms of robustness.
This is the classical approach, the mass matrix.
If you just take one sinker and you increase contrast or dynamic ratio,
everything seems to look fine, and actually most often you find these kind of tests in the literature.
But looking at it from another perspective and putting in more and more sinkers,
this number of iterations blows up, and the stopping criterion was six orders of magnitude reduction.
And on the other hand, for BFPT, it's a very nice behavior of, it's a very stable preconditioner,
no matter whether dynamic ratio or the number of sinkers is right.
And this turns out to be important for mental connection problems.
Okay, another view of it.
I showed you 3D examples before.
Now I switch to 2D so I can plot some eigenvalues.
And these are the eigenvalues of the exactly assembled Schur complement,
so they go throughout the whole range.
If you apply mass matrix preconditioning, you get a nice clustering over here,
but also kind of a tailing in the small eigenvalues,
and I think this is pretty interesting when I saw this,
that when I applied the MU-BFPT preconditioner,
there seems to be kind of a lower bound for the minimum eigenvalues.
And what I was trying to do with these circles and the colors,
I was trying to identify which are the eigenvalue, eigenvector pairs,
which are responsible for the trial of solver to take long.
So I was decomposing the residual into these eigenvector components
and trying to see which of these modes are taking longer to converge.
Okay, now there's a technical issue.
It's pertaining to the boundary, Dirichlet boundary conditions.
So if we have this commutator and open domain, no boundaries,
viscosity constant, then you can satisfy the commutator in infinite dimensions exactly.
So I use an operator notation for these infinite dimensions.
Now let's apply a bounded domain,
and let's attach boundary conditions to the velocity,
and then you cannot satisfy this anymore on the boundary.
And this turns out to pose a problem for algorithmic scalability.
One way, and experimentally we found this,
to get around this difficulty is to change the weighting
in C and in D.
So what I will be using, the weighting is,
there's a certain amplification on the boundary.
These are elements that are touching the boundary.
And when you invert this, this acts kind of as damping.
So you kind of reduce the weight of your boundary elements
in this commutator relationship.
And these are numerical experiments.
These are, again, number of iterations.
The colors show where the convergence is fast.
You can see making no change, setting both to 1.
This looks, 33 iterations is nice.
This is level 5.
Going to level 7, you get 44, 45, which is not very nice.
But if you either use, amplify the C or the D matrix,
then you get, again, a very nice convergence,
in just one iteration more than you would on the other hand.
And this is a similar behavior.
Instead of increasing the level, you increase the order.
So you see a similar kind of pattern of fast convergence.
And we will be using these kind of observations
from our extensive numerical testing
in order to assess the algorithmic scalability.
Now, our last component for algorithmic scalability
will be a multigrid method.
We want to use finite elements.
So we begin with the spectral coarsening,
down to finite element order 1.
Then the geometric coarsening, where we reduce elements.
And at the same time, we repartition
on fewer and fewer cores, reduce the MPI communicator.
And then, at some point, we turn over to AMG,
and it coarsens down to a direct solve.
And the rationale behind this is,
imagine you would have adaptively refined mesh
at the beginning.
And at some point, coarsening these adaptive meshes
gets inefficient, and the reduction rate for each level
is not desirable anymore.
But AMG can still figure out a way to coarsen fairly quickly.
But by going down to a certain level,
we avoid this high setup cost of AMG.
So this is done in parallel, parallel repartitioning,
on subset, of course.
What makes the algorithmic scalability work
is higher-order L2 projections.
If we go from one level to the other,
and combining those with Chebyshev-accelerated Jacobi
point smoothers, Chebyshev's from Patsy,
and the combination of these two make
the very nice algorithmic scalability behavior.
You can see it here.
I increased the level from 4 to 10.
I took one of the sinker problems as a model problem.
With that, I increased one of the boundary
amplifications.
And then I only solved the A block
so I could see how the V cycles of HMG
are performing as a preconditioner.
And this is perfect, 18 iterations
for six-orders reduction.
I was testing the same for the pressure Poisson, which
is here named Kd.
And this is also very nice.
And in order to see how everything works together
with mu BFPT, you can see the number of iterations.
It's very robust.
So this is a very nice algorithmic scalability.
I did the same thing for the order, K.
It goes from 2 to 8.
And here, you see a mild increase, which is actually,
in a way, you would expect some mild,
or at least from the literature, I
think you would expect some mild increase.
So just four iterations is, I think, very good.
Similar behavior for the other block.
And one thing maybe I forgot to mention
is that in this pressure space, which
is discontinuous and modal, there's
an additional smoothing step and the redistributization
step onto continuous nodal finite elements.
And actually, that's what makes the multivariate V cycle work,
because going through this discontinuous pressure
space and coarsening that one, we
had difficulties with that, difficulties with convergence.
And then finally, putting the V cycle together,
these and these, combining it with mu BFPT,
you see from 33 iterations to 50, order 7,
and then a slightly larger jump.
And I think I'm actually very happy with this result,
even though it increases mildly, because it's
the first time I achieved something
for this kind of very difficult to precondition problem.
And I haven't seen anything similar in the literature
before.
Now finally, the parallel scalability.
I was using a machine at the Texas Advanced Computing
Center.
It has been built or set up just in January.
It has up to 30,000 cores, which I almost use all of them.
It's an Intel Haswell architecture.
And what I show you here is the weak scalability
for the solve.
It's normalized for the varying degrees of freedom
or number of iterations it took.
And then also the setup time.
And typically, in a nonlinear solve,
the solve time would dominate.
So this is the important figure or number to look at.
And this is 90% weak scalability over a 600-fold increase,
of course.
And yeah, this is a very nice result.
And also for the setup, actually 71% weak scalability
is very nice.
If you're interested in more extreme scalability,
we did this just last year for the Earth's mental convection
with adaptive meshes.
And we used up to 1.6 million cores on the IBM BGQ.
And on this system, we achieved 79% weak efficiency.
And this was reported in the Golden Boyle prize-winning
paper.
So finally, an implicit solver has some strong scaling limit.
And we also see that here, that over an 80-fold increase,
things work out quite nicely.
I will explain the different lines in a minute.
But as you go to the 30,000 core range, things flatten out.
And the differences between the four runs
or the four experiments are that I
was replacing MPI ranks by OpenMP threads
and looking at the performance.
And I leave you with the references.
And thank you very much for your attention.
Thank you.
Questions?
So I see that in your application,
there's no inertia with variables.
Yeah, I'm not sure what you're after.
Well, I'm just saying, I'm asking you to crystal ball
gaze, I guess, in the sense that suppose
that one didn't have these very slow mental flows,
that had a flow of something, and viscosity of something,
and a different situation.
Do you think that your BFPT would,
if BFPT deals with inertia, enable you to solve such problems?
Yeah, I don't have any experience.
So I'm afraid.
Can you repeat the question?
So the question is simply that BFPT was originally
invented to solve an obvious type of problem.
So problems with inertia.
This is being used here for the Stokes algorithm.
Variable viscosity, Navier-Stokes problems
could be solved with this method.
So one idea I have is it might be
that the symmetry of the Stokes system
is playing a role, how this preconditioner performs
for these.
So since we have been developing these methods
with numerical experiments, and trying
to see which way we should go.
And now, at the moment, we're trying
to figure out a theoretical framework that would
support numerical experiments.
It is still premature to tell any results from that.
But yeah, one idea I have, because I've
been trying to figure out it theoretically,
it could be a property that needs symmetry.
So yeah, we will see in the future.
But it would be interesting to see how it carries over
to Navier-Stokes to this right.
More questions?
For Johan?
So in this drunk scaling, you have this tuning off.
Why?
Can you explain why that was happening from 4,000 to 30,000?
Yes.
So what happens is the discretization,
let's say it would be a Poisson problem.
It would actually look the same.
So you have the continuous discretization space.
That means you have to always communicate between neighbors.
So there's always somehow you partition your elements
on processors.
And there's always the boundary which
you have to communicate with.
So as you increase the number, of course,
this kind of boundary of communication
gets larger and larger.
And at some point, you have to communicate with many processors.
So communication is just taking over.
And yeah, you're barely doing any computation at some point.
And that's, I think it's an inherent problem
of implicit solvers.
So it's just something that might be theoretically,
a theoretical kind of bound.
And you have to see how far you can push it.
Yeah?
What's the percentage of unknown that's
done in the AND gene compared to the other two
person in the process?
You mean how much we hit the residual?
Yeah, roughly.
How much is AMG being responsible for
versus the other two of reduction?
Yeah, so it depends, of course, on what kind of problem
you're solving, what kind of viscosity you have.
There are tests I compared the HMG with just plain AMG.
So it turns out that by knowing these features
of the finite element discretization,
knowing the problem, and using these kind
of high order projections, I could always
have a convergence rate better than with just plain AMG.
So I would guess that AMG is very
good at this linear discretization.
But as you go to high order, you get in trouble
because it just cannot as effectively
reduce the residual as you go through the V cycle.
So you see, it depends on the discretization.
And yeah, there are these.
So you're saying it depends on what basis functions you want.
Yeah.
And I want high order, so it would be worse.
Yeah.
OK, any more questions?
If not, let's thank you all again.

