Thread by @AIWarper

Viggle AI Experiment 💙

This is an incredible tool for style transfer.

Details and link to Discord below 👇

As always, if you enjoy this content please kindly like and share!

Firstly, I rotoscoped the subject using After effects. This isn’t necessary but I needed a mask for a later step anyways so - why not right?

I then fed the video with this image into ViggleAI

It gave me this video.

The next steps involved compositing back into the source footage.

This was the step that I need to improve on and have since come up with a better method for doing it involving in painting and gen fill

I then took the composited video and ran it through AnimateDiff using the LCM motion model, photon checkpoint, and Differential diffusion with a combined mask of the joker + the mask I made in the earlier step. I also used a Joker LoRa.

No controlnets.. very simple workflow.

This ended up cleaning it to about 70% of the quality but I had some issues with the jacket morphing into the wrong perspective.

Easy fix though I just cropped a good frame of the jacket, tracked it loosely to the footage, then reran again through ADiff.

The biggest take away is I need to clean the initial composite a bit better. You can see some artifacts of the original actor that I composited over. I’ve since come up with a new method to improve on this that’ll I’ll discuss in the future.

You can use Viggle for FREE in their discord. The crazy thing is this is a 27s video and it legit took less than 5 mins to get this result 🤯 @ViggleAI

discord.gg/viggle

@lilyachty