Stable diffusion thread

[doogyhatts]

SparseCtrl is now available on ComfyUI Advanced Controlnet.
https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/issues/245

Ok I have sufficiently tested SparseCtrl using the RGB image encoder.
Currently, I am using only one input image into ComfyUI, and generate a series of 17 keyframes (discard the first one).

Sometimes the output may be distorted, so just need to keep generating until a stable one appears.
When comparing against SVD, AnimateDiff has much better arms and hands movement.
However, FreeInit has not been added into ComfyUI AnimateDiff-Evolved yet.

 

[doogyhatts]

Found a scribble workflow for AnimateDiff SparseCtrl.

 

[doogyhatts]

Quick tutorial on SparseCtrl using ComfyUI.

 

[doogyhatts]

DreaMoving demo has been released on HF!
https://huggingface.co/spaces/jiayong/Dreamoving

Their output results look really good!

 

[doogyhatts]

SVD is very good at fluid dynamics (water, smoke).

 

[doogyhatts]

Very useful LoRA that can fix generation of images above 512x768 size.

 

[zheng]

camera/motion controls

 

[doogyhatts]

New technique to fix hands using control-net!

 

[doogyhatts]

I managed to improve my understanding of ComfyUI due to this guy's tutorials.


Test results:
https://forums.hardwarezone.com.sg/...y-dall-e-3-and-is-free.6954638/post-150828081

 

[zheng]

I managed to improve my understanding of ComfyUI due to this guy's tutorials.


Test results:
https://forums.hardwarezone.com.sg/...y-dall-e-3-and-is-free.6954638/post-150828081

actually find it almost pointless for img2video now... very limited motion...
if really want to do motion need to go with controlnet
or direct text2video

 

[doogyhatts]

actually find it almost pointless for img2video now... very limited motion...
if really want to do motion need to go with controlnet
or direct text2video

Actually, I am just looking to do similarly to what Pika.Art has, but with higher success rate in generation.
So no need for unlimited motion, until DreaMoving and Animate Anyone are released.

 

[doogyhatts]

This one uses StableZero123.
Multiple images to video technique.

 

[doogyhatts]

New SVD model to be released soon. The new model is accelerated by TensorRT.
https://developer.nvidia.com/blog/new-stable-diffusion-models-accelerated-with-nvidia-tensorrt/

 

[x1243x]

New SVD model to be released soon. The new model is accelerated by TensorRT.
https://developer.nvidia.com/blog/new-stable-diffusion-models-accelerated-with-nvidia-tensorrt/

can't even get the tensorrt to work with sdxl.. sd1. 5 no issue.. but sdxl has issues

 

[doogyhatts]

With FreeInit, it is now possible to get a higher success rate with text-to-video generation.
No more third arms?


I will try doing one for the next video.

 

[doogyhatts]

Ok I have tested FreeInit and I can confirm it is worth it, even though the two additional iterations would lengthen the generation time from 6 min to 18 min for each AnimateDiff video.

The weights for batch prompt scheduling also needs to be increased as FreeInit seems to dampen them.

 

[doogyhatts]

Vividtalker is much better than Sadtalker.
https://humanaigc.github.io/vivid-talk/

Not released yet.

 

[doogyhatts]

He combined the usage of FreeInit with the mm_v14 motion module to get hair animation using text-to-video.

 

[doogyhatts]

He combined the usage of FreeInit with the mm_v14 motion module to get hair animation using text-to-video.

Ok I have tried to do the same.
But the success rate of getting a stable hair animation through AnimateDiff is very low.

So I opted for a different technique.
Select a subset of animation frames from the output of AnimateDiff.
Send them into SVD to generate the hair animations.

 

[doogyhatts]

FaceChain is Alibaba's consistent face generation solution.
Needs 19gb vram.