Hello, dear friends, welcome
to Ouyang's video channel again
. In this issue,
we continue
the advanced tutorials
of the ComfyUI series . After we have understood
the use and application of IPA for such a long time,
in this issue we finally It needs to
be combined with animation.
We also said before that there are too few methods
to generate animation
. Now
we have a very good method.
Not much to say.
Since it is video generation,
we will go directly to the server.
Open a server
. In this issue,
Ouyang mainly shares with you
how to combine IPA with AD,
which is equivalent to the basic application of IPA in animation.
Ouyang calls it static animation of IPA.
This workflow is quite familiar
to us.
The basic node built with this
impact and the reference used for IPA
were still the same image we had before.
The main model here
still has the same inscription as
Dreamshaper8.
Basically
, there is not much change from the inscriptions in previous issues
. What happens next? I hope that this character will be more model-oriented
, so here
I will reduce its weight to 0.6
and give it an ease in.
Such a weight will
make it more model-oriented.
It still feels a little weak.
Add a little more.
OK. Then it's okay. Not bad.
Then let's talk about
what effect it can have when IPA
is added to Animatediff.
First of all, it can have
the effect of stabilizing the character and the picture.
When we add IPA,
its changes and flickering will be reduced a lot.
Here we can For comparison,
first here
we find this animatediff
and then we find node one.
We first use this node one to load it.
At the same time, here we simply use a V2 model.
If we don't use IPA,
we can connect it and take a
look and directly use Ad to generate it. For such an effect,
here we give the frame number to 16 frames.
Here we also need to change this preview image and
change it to our previous
video . Drag out the
merge assistant to search for video
and merge the video
into h264 encoding.
Others are not saved by default. The video
is OK. Let's directly generate it now.
During the generation process,
let's talk about the problems that everyone has encountered before.
Some people may say that the screen will be blurred.
The biggest problem is this batch.
You can't give it too small
. Maybe I said If you want to try to
give batch 1
, it is definitely not enough
because it renders it to have
a length of a batch.
Secondly, batch 8 and batch 16 are very different.
Batch 8 may not be as good as 16
because This model
, our motion model
, is trained on the basis of 16 frames
, so the effect of 16 frames is the best.
This has been tested many times.
No matter which model it is,
which motion model
, the effect of 16 frames is the best. Okay,
now let's come back and take a look at
this character
and his rendering.
The number of steps for my rendering is 25,
which is not a big problem.
Then you will see that the effect is a bit blurry.
This is also the case
with our previous redrawing. It feels like
it will be too flat,
including its light, shadow and contrast, which are not that good.
Now, when we connect to this IPA,
it is also such an assassin's inscription.
And here,
it's just that we have a reference.
Of course, this sports model is also in the model. The pipeline
is specifically connected to the IPA process.
There is not much difference between the front and the back
because in the end their parameters will be merged together.
It’s just a matter of which one is merged first.
So the final parameters are all the same.
OK, now let’s generate
it again . You can see that after we use IPA,
the entire picture will be much more stable.
The stability is not as large as the previous one
, that is, the range of motion is not large.
The second thing is
that the performance of the entire picture will be much clearer
and much better.
Yes. You see, both the edges and the light and shadow
are very clear, right?
This is the most important role of IPA.
Without the intervention of IPA,
our animation may be very messy.
For example, if the range of motion is one now,
even if I increase it to 1.2,
we will come back. If you take a look at it,
you can see that it is still in good condition, right?
It’s just that the movement of the hair and
the background is more intense.
This is the core function of IPA to generate video.
Now all the video workflows that you see are
animatediff’s work . Streaming
is basically inseparable from IPA
. So when we understand that
IPA will be used to generate animation,
let's discuss static animation.
What is static animation?
To do different things on static reference images,
first let's talk about The simplest solution
is to mask, right?
If we want to make some articles on the background
, for example,
we still drag out a copy in the same way as before
, and then we edit the mask
. In the mask area,
we only frame the characters. Of
course, our reference image is not the same as our final image
here. Let's copy this image.
Let's re-edit
the instructions and give them to the characters, right?
This is one of our most basic methods of using IPA.
Then we Rename the background.
For example, this translation
is a night sky.
We want it to have the feeling of a shooting star.
Let's try it now.
Copy it to the line below and then
transfer the mask to our IPA.
Then we Go directly to the generation.
Now you can see that we have successfully changed the background.
So how to deal with static images,
we will deal with dynamic images.
The principles are similar.
You can see that there is no problem in the second generation.
It seems that there is no problem with
Meteor. Our movement is related
because the movement model
will also accept different movement effects during training
. For example, let's change it to V3 now
, and then here
we add a V3 Lora
search. There are too many.
Let's find V3.
OK, lower the weight of this Lora
to 0.6. Now let's generate
it again. You can see if the effect of V3 is different.
It may change in style.
Some friends also asked why
the animation produced by AD is different from the picture produced by my model. This is different
because our motion model
actually has this graphic element,
so when our original model information has been passed through AD,
its picture has actually been modified,
so it cannot be completely like our
previous main model. The image generated by the model
is OK. After understanding this,
let's change it to V2 now.
Because during the test process,
I still think the effect of V2 is more beautiful,
but the sports performance of V3
will be better. This motion effect should be more natural.
It depends on our needs.
Of course, in the advanced settings of AD,
we can also mix V2 and V3 and various motion models
. We will talk about this later
in the advanced settings of AD.
OK, now we put Delete this Lora.
Next,
let's talk about the second one
, which is the partition animation we mentioned before
. At this time,
we still delete this
as before.
I also created such a mask group.
This is We have used the case
preview before to preview the mask.
This is after inversion.
This side is not inverted.
Then we apply it to IPA. So
how to apply it?
Here, we have to add
the same For two IPA application nodes
at the same time, we give two different pictures.
Here, I prepared an image of water.
Here is a picture of fire. You can see
that these two
images are not very regular. Ah,
we all need to preprocess,
so at this time we do a little preprocessing on the image.
Clip directly searches for the preprocessed image.
Then we finally preview it.
OK, these two can see that this is like this.
The water may be cropped.
Let's Align it downwards
OK so there is no problem.
Then we pass these two images
one by one to
the water one. We pass it to the first
fire one. We pass it to the second one
. We sharpen it a little bit
like this. The whole effect
will be sharper
and clearer.
Then we turn it off.
Next, we pass the mask below to our IPA.
The first one on the left is water.
The second one is inverted on the right, which is fire.
Then we go here. First, set its
weights to linear
, and then
give the weights to
the IPA behind 0.8. We also connect the model in series
, and finally give the model to ad.
Next, let's change the prompt word.
In this place, we will simply use this
In order to make it more dynamic, I gave
a prompt word of water and fire
here, a flying water and fire
, and then
a
colorful motion background.
After that, we turned on the sampler
and directly rendered
the range of motion here. It's a bit big. 1.2
We should test it from the beginning
. That doesn't matter.
Now let it generate OK.
Now we can see that water is on the left
and fire is on the right.
The biggest problem here is that
there is an obvious boundary line, right?
That's the weight. The problem
is that the linearity we gave is relatively high
, so we can lower its weight and
give each reference to ease in.
In this way, it may reflect
the colorful background behind us.
With such an inscription
, we can clearly see
the water and The two backgrounds of fire are blended together,
but the animation effect is not particularly good.
Let's slightly adjust its movement intensity
and then change our random seed.
This will form
a picture of water and fire coexisting.
So here we are Simply use a mask
to distinguish the two objects
. Of course,
in addition to this method,
we can also make the two objects blend better
. As for the fusion method,
there are two types.
So here?
We don't need to use a mask to distinguish.
First, we cancel the mask.
The first way is to use weights
. For example, our first one is in
, and the second one we can out.
That is to say, it is the image of water at the beginning
, and then when When it slowly exits,
if
our fire enters again,
the two will alternate.
Of course, the fire may be weaker.
Now,
let's generate it again and take a look.
Now we can see another scene,
which is water and The fire is fused together.
The second way
we can solve this problem is easier.
First, change it to linear.
We can use the start and end time, right? We
have tried this before.
What about animation?
Let's try it too. For example
, if we let the water go from 0 to 0.5, it will go
to the front half
, and the fire will go to the back half
from 0.5 to 1.
This is half of half.
Let's generate it again. In
this case,
there will be more water
and fire will enter. Slower,
so if its effect
is not so obvious
, if I change these two,
fire will be stronger,
water will be weaker,
for example, 0 to 0.5
, then water will enter
0.5-1 from the back
, so this At that time, we need to
adjust which one of the two sides is merged first.
As for the two best fusion methods
, I believe everyone can see
that the fusion we used before using this weighting method
will be better.
In terms of percentage,
we need Finely adjust
their order.
Of course,
we can also superimpose it
, for example, from 0 to 0.7
, and then from 0.5 to 1 here,
so it can be superimposed by 20%, right?
Then,
let's give it a widescreen 768
Then we change the weight to
ease in
. Then
we can also overlay it a little bit.
For example, here is 0.4,
and here is 0.6,
with a 20% overlap in the middle.
In this way, we get a blend of water and fire. Picture
: What if I want to use it as a background and
I also want to put a character in front of it?
Let's do the third fusion.
Now we still use ipidopter.
Same thing. Let's copy an ipa
and give it a reference picture
here. Use a
reference picture like this of a cultivating figure,
and then we pass the image over
and at the same time we connect the model to
this IPA. We can put it anywhere,
at the back or
in the front.
Then here it is,
I think. Its weight may be higher in the front.
Let's put it in the front
and string it to the front.
Then we drag this model
over.
Now it's a plus model.
Then let's keep the weight here unchanged first.
Or this ease in. What
's next?
Let's add another inscription to the final effect
. For example, here we copy an
inscription like this
, which is a handsome man sitting cross-legged and meditating
, with long hair
behind it. I added some styles,
such as anime
illustrations and Chinese style,
and then generate it.
OK, so we A
There is an effect of characters and a moving background.
We can continue to adjust this effect in depth.
You can see that there is a moon in
the background.
The moon is moving a bit, right?
And in our reference picture, you can see that
there is something behind it.
How to block the background of the moon
? Do we also use a mask here, right
?
Let’s edit its mask here as well.
We only frame the characters
and not the background
, which means iPA only recognizes this. After the character
, we pass the mask to this IPA
and change the random seed.
Let's generate it again.
You can see that our background
is now separated from the original background effect, right?
But our character may be slightly biased
because of our mask. The photo is not drawn with this resolution.
In fact, it should be drawn with this resolution.
Here we copy the original image
and then draw its mask on this image,
making it slightly larger. It doesn't matter.
Then we transfer this mask
to the IPA of the character above us.
Now we generate OK.
In this way, we are out of the background of the original character reference image.
So we can adjust this effect
as we like. You can see
it. The effect of water is a little bit weak now, right?
Then we can enhance it through here.
Slightly adjust the start time of water. Adjust
the end of water to 0.8
and start to 0.2.
In this way, there may be more water.
Of course, we can also adjust it
more. Let's adjust it roughly.
For example, 0 to 0.5
here is 0.
Then
let's let him start from 0.4 to 1
, and then we will generate it again.
After we know this area control,
these weights are all ours. It can be realized by adjusting it
casually.
Now we see another scene.
So this weight adjustment is really interesting.
Now
we can see that we have no background,
so we can do some articles on the background
, for example.
The background I used here is
the cosmic starry sky and golden light
with a little weight,
because you can see that our side is all ease in,
so we are qualified to reflect the background inscription.
You can see that the background has been reflected smoothly, right?
But this golden light may be melted together with our water.
Water and fire are melted together.
They are related.
So this is how we
use weights and our masks to divide
our different objects from static
. Finally, we have to perfect this. As for the animation,
in fact, sometimes after the AD is generated, we will feel that it is blurry
, but if we enter the IPA,
it can actually greatly alleviate the blur. Just like
when we did not enter the IP
when we first generated
it , you can see it. It's very blurry, right?
If we want to improve the quality and details of this animation,
it's actually very simple.
We only need to perform a secondary redraw.
Here, we
use the simplest way
to directly redraw the sampler. Let's make a copy.
We directly pass this latent
and this PIP.
Then we don't change anything about the rendering parameters here. We
slightly change the redraw amplitude to 0.5
0.6. 0.5 is a bit low.
In fact, the parameters are all the same.
And here
we Add a slightly later scaling to it.
For example, let's give it 1.5.
In this case, our resolution is
more than 1,100 of 768
, which is equivalent to 1280.
Then pass this latent over and
put the
video below. Let's merge another video.
Just like here,
we still need to decode
and re-merge one video.
Let's just redraw it a second time.
OK. Now let's compare
the two videos
and perform a second redrawing with 1.5 times magnification.
You can see that the details on the right side are much more. Right,
it includes water flow,
fire, and the details of the character's clothing.
So if we want to improve the quality of our animation, it
's very simple to do two-step rendering.
Of course, after that,
we can also perform a post-stage amplification
and post-fill frame
. It will make our video more detailed
, including you can see
the golden silk thread of the water flow
below .
OK, so these are some methods about the static use of IPA. So
what is static?
Let’s re-understand
the so-called static, which means we use
a Use a reference picture
to generate the model data of the reference picture
. This model data is actually a solid state,
which means that it will not move.
As for the model data output by IPA,
it is not dynamic.
It is solid state. It
only provides a reference data .
For reference data,
we use different mask areas,
weights,
inscriptions, and
combinations
to generate the video content we want to generate.
What is dynamics?
When talking about dynamics,
we have to mention another application
in our IPA.
About an application of Batch, right?
We can find IPA.
Here we can see advanced and batch. What
we want to talk about is batch.
The so-called batch.
We also said before that
it will send out many pictures at one time
. Each picture
corresponds to each frame
, so this is a dynamic transformation.
We can let different frames
use different reference pictures.
Is it dynamic transformation
during the generation process,
unlike ours which only starts from the beginning? In the end
, for example, let's say we have 16 frames.
The reference content and reference information given by
him from frame 0 to frame 16
are all the same
, but we have passed in three
different references.
Now,
let's delete all of them.
Then here As for the batch,
we only need to keep one
and the others can be deleted
because we can
transfer various images into this batch, right?
So here
we transfer two images,
the first image is
still the one we had before What about the second image
of such a female assassin
? Let’s pass in
the same female assassin.
You can see that I redrawn it
and closed her eyes.
In this case,
we can pass in the image we composed. A batch .
We can calculate
this batch frame by frame.
For example, what we generate here is 16 frames.
Then we let it repeat twice.
Each time is one second,
that is, let it repeat for two seconds.
So what about the beginning?
I hope it closes its eyes
. Then
I hope it opens its eyes.
So at this time,
we can directly copy a batch.
We directly search for batch to copy the image batch.
So for this one,
I asked him to do two frames.
For the next one,
I asked him to do 6 frames.
In this way, combine the two in a batch,
which is also a batch
. OK. Finally, let’s preview.
So combined together like this,
shall we? There is
a one -second batch, right? The first two are closed eyes,
and the second one
is open eyes.
Generally, when a person blinks, it takes two or three frames.
So we basically give him 2 to 3 frames. Then what happens
after that
? Let this second repeat twice.
Here is a copy of 2
and then send it over.
Let's take a
look . In this case, have we combined
the 16-frame image
? You can see that
the first 8 frames are closed first and then opened.
The same 8 frames are repeated below, so
what happens is that
our two-second animation will blink twice.
This is the role of our batch.
After that, we directly pass this batch to our
application of this batch,
batch to batch. Now
, let's open our sampler
and adjust the resolution back to 512.
Open the merged video and let
's render directly
. Hey, I forgot something here.
I was a little careless.
We haven't changed the prompt word.
So you can watch it now. As for our assassin
, he blinked a little at the beginning, right?
His eyes went from closed to open.
The second time, there was no
blinking . In fact, we can actually emphasize it in terms of prompts.
For example, if we blink,
we can see it now.
Let's fix the animation of the eyes, right?
Because the weight we used before may be a little low,
so I raised it a little bit.
This is a change in batch dynamics.
Speaking of this, some friends may have thought of
this. Can I not give
the same character to the batch?
Can I change it?
For example, I change the character slightly
and then adjust the above image back to our previous reference image.
Finally, we slightly change the batch.
It's very simple to let the top occupy 8 frames per second and
the bottom occupies 8 frames per second.
After combining the two,
directly pass the combined image
to our batch here
and then we directly generate
it. Then we can see like this This effect
may not be what we want.
The effect we expected in our hearts should be that
the two characters changed smoothly and connected in the past.
But now
it is a jump
directly to another character.
So this is When it comes to a transformation that is often used
in our video animation production,
we call it a transition.
How do we use it in this workflow?
This brings us to advanced dynamic masks.
We leave these contents to In the next issue,
because the dynamic mask needs to be expanded, we will talk about a lot of content
. It is estimated that it will take a long time to combine in this issue.
Finally,
Ouyang will share with you a little animation production trick.
Here we will only give We will give such a reference picture
to 16.
The last tip I want to talk about is about
the AD process.
When Ouyang shared it with you last time, he also said that
he had made a major update.
So here we will briefly talk about it
. Let's simply build
an advanced usage method .
Now we are using a simple construction method.
So advanced?
We can find
this series of Gen2 in AD.
In fact, here it is divided into simple
loaders. In the three parts,
we can see that there are dynamic application parts,
dynamic loading parts and sampling parts.
Now
let’s load a dynamic model first
. So this node is
actually equivalent to ours. This part is the model name
and some of the model settings above.
We are You can look at them separately here
. For example, does this dynamic model Lora
exist in this place?
With dynamic scaling
, you can see that there is a scaling multi-value here,
which is the dynamic scaling on our side.
It separates this place.
Then Let's talk about the big part here
. First, we have to load a dynamic model, right?
The direct dynamic model loader is exactly the same as ours. We
also used V2's model here.
You can see that it is a dynamic group at the back. Then there is also a dynamic group
here. It is not a model pipeline
, so after the model group here comes out
, we need to find the advanced group in our AD,
which is the advanced sampler in
the group of Gen2.
In the advanced sampling step, we can connect this group over and
connect it over. After it is responsible for the input and output of the model pipeline
, we can connect this model pipeline
and output it to our sampler at the same time.
In this way, basically these three nodes
synthesize the simplest and original node above.
Then For the following nodes, you
can set up the relevant steps in more detail.
The main point we emphasize here is
a special algorithm in sampling. We can see
that there is a sampling setting here,
but the most basic place also has sampling settings.
When it comes to this sampling setting,
we can also drag out the sampling setter.
Of course, we can also connect it to the sampling setting below,
because we may use the advanced process later.
Here, we will briefly talk about
the construction of the advanced process
and basically complete it.
After that, we delete the above two nodes.
We talk about this advanced process
, and then in the sampling settings,
we can see that in the noise type,
we can set different noise types.
That is, after the major update of AD A
month later,
this free noise was launched.
It will freely generate some noise
during the animation generation process
, which is the process of our animation model operation.
It is equivalent to adding noise on our IPA side.
If we add noise on our IPA side, Adding a little negative noise
will make the picture smoother, right?
But
it can't be too much
, or it will be a bit greasy, right?
The same is true here in animation.
When we add this free Nois In the future,
many details
will be expressed better, softer and silkier.
You can understand it this way
. So this is a reference.
Now let's take a look.
This is an animation we generated before.
So for this animation,
let's move it out first. After making a backup
, we use this
option to add free noise.
What will the animation look like?
Let's generate it.
Here we can see that the left side has noise added,
the right side does not,
and the left side can see it more delicately.
The animation is more delicate.
The water on the right can be seen slightly less.
It means that its stability will be higher.
So after using this method,
we can increase the range of motion
and zoom in
. In this advanced node,
we can directly find
the scaling multi-value.
It is actually a scaling multiplication.
A scaling value of a multiplication
finds a multi-valued dynamic
and then we bring this floating point number up. Before we mentioned it,
it was 1.
Now when we mention 1.2
, it is equivalent to the dynamic scaling of our original node.
Now, let's
generate
that It can be seen that in this case, after we add noise
and enhance its motion value,
we can see that the motion of the background
is more natural and delicate
than the original effect on the right.
If we add noise on the right based on the original After 1.2,
it may fly this animation
, so this free noise is
an important point to tell everyone.
Then the second point
is that there is a little trick
about our model pipeline.
When we talked about IPA before,
we gave a model pipeline Model scaling
can adjust our diversity
, right? So if we can use model reduction in animation,
there is actually a node
in our original SVD.
We can find this sample.
In the sample, we can see that there is The video model
is a linear or triangular
one in SVD.
We can try both.
In this case, our default is to use a linear one.
We can reduce the minimum value of the linear one to 0.7. After that,
we will pass the model again
. Let's take a look at the changes
to our sampler .
OK, now you can see that its CFG is relatively low at the beginning
and slowly increases to 1.
So here we can adjust
the CFG. Let's adjust
the curves of our screen. You can see that the higher the CFG, the stronger the dynamic range.
So this is an effect of 1.5.
The entire saturation is slowly changing.
This is one of the functions of our CFG guidance.
So now If we use the previous static
CFG zoom, we can see
that it will give us more diversity in the picture
, so we can try various CFGs
in animation generation, which
are also worth exploring.
We can search for CFG to see There are a lot of
double-line guides,
scaling
, linear guides,
customization
, and this triangular guide.
Finally , here
we come to a complete set
, that is, we perform a secondary upgrade and increase.
At the same time, we perform a post-stage enlargement of the height image
through the model.
Then in the model, we load a us model
. For the outgoing image,
we give a supplementary frame to manually create a
model of the 49 we used before.
Of course, 47 can also be used.
The default 47 can also be
used. Then we give the multiple to three times.
Finally, the frame rate of the video we merged is 24 frames.
In addition, the magnification factor here is slightly larger,
and the coefficient is reduced to 0.5
, which means it is enlarged twice.
After it is enlarged, we will give it a sharpening
essentials node. We find
an image to automatically sharpen
itself. The better way to adapt to sharpening
is to separate the foreground and background.
Here we set a default value and give it a lower value. Let's
give it to 0.5. Finally, give the image to us.
Open the sampler
and then we generate it
. After this series of operations,
you can see Here is one of our finished products.
Let’s move over and take a look.
The CFG we gave here is a bit low,
so its color saturation is not very high.
In the later stage, we can slightly adjust its color
to see the smoothness of
our video. The level has reached a level, right?
Then take a look at our resolution.
Download it to the desktop and let's take a look. Its resolution is 2304. 1536
has reached a resolution of two k,
so this is what this section will do. I want to share with you
some of the most basic ways of combining IPA with AD.
I hope it will be helpful to you.
As for dynamic changes,
don’t worry.
The next section will come,
because for videos,
the entire debugging process takes a lot of time. It takes a lot of effort.
So finally,
if you think the content of this issue is quite useful
, don’t forget to
click a series of buttons to support Ouyang.
If you are interested in the AI field
, don’t forget to follow Ouyang’s video channel
in the future. Let's learn
and communicate together.
See you next time
.