Post

How to get URL link on X (Twitter) App

On the Twitter thread, click on or icon on the bottom
Click again on or Share Via icon
Click on Copy Link to Tweet
Paste it above and click "Unroll Thread"!
More info at Twitter Help

CoffeeVectors

@CoffeeVectors

Sep 12, 2022 • 9 tweets • 9 min read • Read on X

Scrolly

Took a face made in #stablediffusion driven by a video of a #metahuman in #UnrealEngine5 and animated it using Thin-Plate Spline Motion Model & GFPGAN for face fix/upscale. Breakdown follows:1/9 #aiart #ai #aiArtist #MachineLearning #deeplearning #aiartcommunity #aivideo #aifilm

@daz3d

First, here's the original video in #UE5. The base model was actually from @daz3d #daz and I used Unreal's #meshtometahuman tool to make a #metahuman version. 2/9 #aiartprocess

Then I took a single still frame from that video and ran it through #img2img in a local instance of #stablediffusion WebUI. After generating a few options I ended up with this image. 3/9 #aiartprocess

Then I used the Thin-Plate Spline Motion Model to animate the img2img result from #stablediffusion with the driving video from #UnrealEngine5. 4/9 #aiartprocess

github.com/yoyo-nb/Thin-P…

The result is 256 x 256. 5/9 #aiartprocess

Here's a split screen highlighting the consistency and coherence between the driving video and the results. I upscaled the video a bit using #TopazVideoEnhance. Notice the quality of the blinking and the coherence of the "birthmark" under her left eye. 6/9 #aiartprocess

Good as this was, there were some artifacts in the eyes I wanted to reduce. Made the video an img sequence. Used a local instance of GFPGAN to batch process all the frames. This increased the quality of the eyes and sharpened the image by enlarging 4x.(1st video)7/9 #aiartprocess

GFPGAN changed the alignment of some things slightly for some reason and shifted the color. It also faded the birthmark a bit. Not sure how to control the strength of the effect locally just yet. GFPGAN link below. 8/9 #aiartprocess

github.com/TencentARC/GFP…

There's still some artifacts here and there, but pretty good considering the look is from a single keyframe. I'll continue experimenting and see how to improve on this initial workflow. Hope this thread gives you some ideas! 9/9 #aiartprocess

• • •

Missing some Tweet in this thread? You can try to force a refresh

This Thread may be Removed Anytime!

Twitter may remove this content at anytime! Save it as PDF for later use!

More from @CoffeeVectors

CoffeeVectors

@CoffeeVectors

Oct 12, 2024

Made this video with iPhone photos I took of my friend Stephanie that I used as keyframes in @LumaLabsAI! With the camera controls I can gen transitions between shots. I also built a custom web app in Next.js to help me speedramp and edit all the clips! Breakdown 🧵(1/18)

Basically if you have some photos, throw in a start and end frame, and start your prompt with the camera move, and then stuff like “smooth camera, steadicam” I find minimal prompts work best. And don’t enhance the prompt (that tends to add hand held). (2/18)

Sometimes I’ll add ‘motion blur’, ‘drone racing’ or ‘music video’ to see how it changes the results. “Perfect face” can help reduce cross-eyes, etc. You’ll still need to experiment, but long prompts or prompts describing the scene don’t usually help with this effect. (3/18)

Read 18 tweets

CoffeeVectors

@CoffeeVectors

Jul 20, 2024

Testing how LivePortrait works lip syncing 24fps lyrics on top of slow motion footage. Was curious to see if it might help with music videos. Quick explanation below! 🧵

Started with a clip from an Eminem song and passed it through Adobe Podcast to get the acapella. Passed that through @hedra_labs with a Midjourney portrait for the face animation. Used that as input into LivePortrait using ComfyUI and a slowmo clip from Die Hard.

I find it helps for the Live Portrait input to have a plain background. Otherwise you might get extra warping in the background behind the head.

Read 7 tweets

CoffeeVectors

@CoffeeVectors

Dec 24, 2023

Made this video (🎶) with a Midjourney v6 image! Started by upscaling/refining with @Magnific_AI, pulled a Marigold Depth Map from that in ComfyUI, then used as a displacement map in Blender where I animated this camera pass with some relighting and narrow depth of field.🧵1/12

Here's the base image and the before/after in @Magnific_AI. Even though MJv6 has an upscaler, Magnific gave me better eyelid and skin details for this case. (Fun fact, this image was from a v4 prompt from summer last year, when MJ had just released a new beta upscaler.) 2/12

Next step was using the new Marigold Depth Estimation node in ComfyUI to get an extremely detailed depth map. Note that I'm saving the result as an EXR file (important for adjusting levels later), and that the remap and colorizing nodes are just for visualization. 3/12

Read 12 tweets

CoffeeVectors

@CoffeeVectors

Nov 15, 2023

Testing LCM LORAs in an AnimateDiff & multi-controlnet workflow in ComfyUI. I was able to process this entire Black Pink music video as a single .mp4 input. The LCM lets me render at 6 steps (vs 20+) on my 4090 and uses up only 10.5 GB of VRAM. Here's a breakdown 🧵[1/11]

Entire thing took 81 minutes to render 2,467 frames, so about 2 seconds per frame. This isn't including the time to extract the img sequence from video and gen the ControlNet maps. Used Zoe Depth and Canny ControlNets in SD 1.5 at 910 x 512. [2/11]

Improving the output to give it a stronger style, more details & feel less rotoscope-ish, will require adjusting individual shots. But doing the entire video in one go lays down a rough draft for you to iterate on—build on fun surprises, troubleshoot problem areas. [3/11]

Read 11 tweets

CoffeeVectors

@CoffeeVectors

May 25, 2023

@MatthieuGB

Timelapse of using #photoshop’s new generative fill feature to connect two images and build a scene around them using blank prompts. Was inspired by @MatthieuGB’s post doing something similar! Notice how I’m not adding any descriptions, but letting gen fill present options for… twitter.com/i/web/status/1…

Here’s the final image! 2/4

And here are the original images made in #midjourneyv51 3/4

Read 4 tweets

CoffeeVectors

@CoffeeVectors

Feb 28, 2023

Testing Multi-ControlNet on a scene with extended dialog, a range of facial expressions, and head turning. No EBSynth used. #AImusic from aiva.ai. Breakdown🧵1/15 #aicinema #controlnet #stablediffusion #aiia #aiartcommunity #aiva #machinelearning #deeplearning #ai

Overall flow: pre-process video > img seq > play with prompts > initial controlnet settings > control net batch render > upscale > clean up in post > key out background > deflicker > video post-production 2/15

The approach I used here was figuring out an initial workflow. There’s definitely a lot to play with and improve on. The orig vid is low-res and a little blurry so I started with a pre-process pass by bringing out the edges/brightness & upscaling. 3/15