Stable diffusion thread

[doogyhatts]

Discovered new Image-to-Video solution!
Demo showcase: The Wondering Earth 3

Check out the huggingface demo sites.
Colab also available by Camenduru.

Spoiler

 

[doogyhatts]

The academic team at Alibaba group is preparing the release of their AnimateAnyone project.
We will then be able to do AI-assisted tiktok dance videos with accurate motions and possibly little interpolation errors.

 

[doogyhatts]

Cfg and motion bucket adjustments for camera motion.
Eye-blinking tutorial using SVD.

 

[doogyhatts]

New paper on animation by NUS & ByteDance!
Colab by Camenduru.

​

 

[zheng]

New paper on animation by NUS & ByteDance!
Colab by Camenduru.

​

nus but all tiong student

 

[doogyhatts]

Just managed to generate one short dance animation using MagicAnimate.
The face and hands are a mess, but the animation looks stable.

So have to use ADetailer to fix the face and hands.
If fixing hands still fail, have to use coloured hand-maps to paint over the terrible hands.

Custom motion sequence is not supported at the moment, not sure if it will be.
The demo has limited motions.

When custom motion feature is available, it will be useful for style transfer and changing clothing.
Then extract the control-net data for grid generation to achieve better overall quality.

 

[doogyhatts]

New paper from NUS again!
DeepCache.
Speeds up SD 1.5 by >100%.
Wait for A1111 or ComfyUI to integrate it.

 

[doogyhatts]

SVD AnimateDiff Refiner.
It works on 8gb vram GPU.

 

[doogyhatts]

SVD AnimateDiff Refiner.
It works on 8gb vram GPU.

This method requires 32gb sys ram.
I encountered an error so far during my test, and it is slow.

So a better choice (for me) is to tile-resample and upscale the original output from SVD.
And try to get more stable movements with the camera motion by controlling the Cfg and motion bucket variables.

 

[doogyhatts]

 

[doogyhatts]

New paper from NTU solved AnimateDiff's problems!
https://tianxingwu.github.io/pages/FreeInit/

 

[zheng]

New paper from NTU solved AnimateDiff's problems!
https://tianxingwu.github.io/pages/FreeInit/

very good improvement
2 more paper it will be better

 

[doogyhatts]

AnimateDiff v3 has been released with Sparse Controls.
Basically, it can now do similarly as what SVD is doing but with different types of input (Sketch Encoder, Depth Encoder, RGB Image Encoder).

https://github.com/guoyww/animatediff
https://guoyww.github.io/projects/SparseCtrl/

Spoiler

 

[doogyhatts]

AnimateDiff v3 motion module has much better facial animation.

 

[doogyhatts]

New paper on dancing animation by Alibaba AI group!
DreaMoving

Results look very good!
This is in addition to their Animate Anyone paper.

 

[doogyhatts]

Stability.ai announced Stable Zero123 for image-to-3d object several days ago.
https://stability.ai/news/stable-zero123-3d-generation

 

[doogyhatts]

I got access into Pika.art.
The output animation is better than SVD but gets slightly blurry.

Output is a 3 second video. I added the reverse manually.
The output can be appended with a 4 seconds extension.

 

[zheng]

New paper on dancing animation by Alibaba AI group!
DreaMoving

Results look very good!
This is in addition to their Animate Anyone paper.

now they have to add physic to clothing and floating hair

 

[doogyhatts]

SparseCtrl is now available on ComfyUI Advanced Controlnet.
https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/issues/245

 

[doogyhatts]

64 frames model.