Thread by @CoffeeVectors

Testing Multi-ControlNet on a scene with extended dialog, a range of facial expressions, and head turning. No EBSynth used. #AImusic from aiva.ai. Breakdown🧵1/15 #aicinema #controlnet #stablediffusion #aiia #aiartcommunity #aiva #machinelearning #deeplearning #ai

Overall flow: pre-process video > img seq > play with prompts > initial controlnet settings > control net batch render > upscale > clean up in post > key out background > deflicker > video post-production 2/15

The approach I used here was figuring out an initial workflow. There’s definitely a lot to play with and improve on. The orig vid is low-res and a little blurry so I started with a pre-process pass by bringing out the edges/brightness & upscaling. 3/15

Then I took the first frame and played with prompts in SD. After finding an initial prompt, I’d then test it on more extreme frames to see how it held up. I’d then tweak the prompt/settings until I got a look that worked across a range of frames. 4/15

In multi-controlnet, after trying lots of settings, I ended up on a mix of HED, normal, and depth. (Again, you want to test with several frames through out the video.) I also kept the denoising low at 0.3 to keep better color consistency and follow the lip sync closely. 5/15

Since I was going for a more cartoonish look, I knew the softer edges of HED with lower guidance/weight would be good for following the input img but allow the features to get exaggerated. (Canny tends to be more exact in following edges.) 6/15

Normal and depth just seemed to help get the image where I wanted over a range of the more problematic frames, but that was just from tinkering. I didn’t go in knowing what precise effect the mix or weight/guidance would have. 7/15

After the batch img2img render, I upscaled the video to 4K using @topazlabs video enhance AI. Then downscaled back to 1080 HD. This brought in more contrast and sharpness. 8/15

I had issues with flickering spots so I had to go in and manually erase them which took forever. Also highlights in the lips would be confused for teeth. I did a few things to improve the situation but a better solution is needed. 9/15

With the increased clarity I went in on removing the background. I tried several approaches with depth maps and color keying, seeing how it held up around the hair across the whole video. The depth map approach didn’t really work the best in this case, 10/15

so I ended up using different versions of color keying and layered them together. The result isn’t perfect (neither was the starting input), but end of the day the intent was to surface a workflow and learn from it, so I think it’s good enough for that. 11/15

To decrease the boiling/flickering I used several rounds of the deflicker effect in #davinciresolve (from @CorridorDigital’s awesome Anime video breakdown!) I’ve tried deflickering approaches before but 12/15

they don’t always help depending on which program/plugin you’re using and settings you go in with, but the deflicker in Resolve is awesome! (Only available in the paid version.) If stacking several deflickers doesnt do enough, try diff settings and several repeat rounds. 13/15

I used an image I generated in #midjourney. Blurred and animated in post. And I added a little grain and sharpened everything (deflickering can soften details a bit.) Animated some slow zooming and dropped in some music from aiva.ai. 14/15

Hope some of you found this thread useful! If so please share it with others! And I’m excited to see if any of you run similar experiments and make awesome stuff! 15/15