CoffeeVectors Profile picture
Oct 14, 2022 8 tweets 8 min read Read on X
Started with a landscape in #midjourney. Used a depth map in @sidefx #houdini to create a scene mesh. Imported to #UnrealEngine5 @UnrealEngine to add grass and animate. Added music. Breakdown thread 1/8 #aiartprocess #MachineLearning #deeplearning #aiartcommunity #genart #aiia
I could have created a similar scene in just Unreal Engine and Quixel but I wanted to see what I could do with this landscape image I generated in #midjouney 2/8 #aiartprocess
I'm also trying to do more collaborations with other AI Artists so I used this as an excuse to research depth maps further and see how I could push them. I generated this LeRes depth map using "Boosting Monocular Depth Estimation to High Resolution" on Github. 3/8 #aiartprocess
I applied the depth map as a displacement map on a high-poly plane in @sidefx #houdini. I like Houdini because of the way it helps me deconstruct process with nodes and code and I can experiment with workflows easily. Rendered this natively in Mantra. 4/8 #aiartprocess
I then took the scene mesh and imported it as a .obj into #UE5. I wanted to apply grass with the footage tool directly onto the mesh, but that didn't work. So I sculpted a plane in the shape of the foreground terrain using Unreal's native modelling tools. 5/8 #aiartprocess
I then used the foliage brush to add grass to the plane. There's lots of YouTube tutorials like this one that explain how. 6/8 #aiartprocess
I applied a Third-Person game template in #UnrealEngine5 and did a screen recording of a character I controlled running around the scene in real-time. I also animated a camera and rendered a cinematic with Movie Render Queue. (1st post.) 7/8 #aiartprocess
There's clearly some rough edges to this project, but it's basically an exploration of different ways a 2D image might be coverted into 3D with depth maps. I have some other ideas I'm going try soon. If you have a cool process involving that please share! 8/8 #aiartprocess

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with CoffeeVectors

CoffeeVectors Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @CoffeeVectors

Dec 24, 2023
Made this video (🎶) with a Midjourney v6 image! Started by upscaling/refining with @Magnific_AI, pulled a Marigold Depth Map from that in ComfyUI, then used as a displacement map in Blender where I animated this camera pass with some relighting and narrow depth of field.🧵1/12
Here's the base image and the before/after in @Magnific_AI. Even though MJv6 has an upscaler, Magnific gave me better eyelid and skin details for this case. (Fun fact, this image was from a v4 prompt from summer last year, when MJ had just released a new beta upscaler.) 2/12
Image
Next step was using the new Marigold Depth Estimation node in ComfyUI to get an extremely detailed depth map. Note that I'm saving the result as an EXR file (important for adjusting levels later), and that the remap and colorizing nodes are just for visualization. 3/12 Image
Read 12 tweets
Nov 15, 2023
Testing LCM LORAs in an AnimateDiff & multi-controlnet workflow in ComfyUI. I was able to process this entire Black Pink music video as a single .mp4 input. The LCM lets me render at 6 steps (vs 20+) on my 4090 and uses up only 10.5 GB of VRAM. Here's a breakdown 🧵[1/11]
Entire thing took 81 minutes to render 2,467 frames, so about 2 seconds per frame. This isn't including the time to extract the img sequence from video and gen the ControlNet maps. Used Zoe Depth and Canny ControlNets in SD 1.5 at 910 x 512. [2/11]
Improving the output to give it a stronger style, more details & feel less rotoscope-ish, will require adjusting individual shots. But doing the entire video in one go lays down a rough draft for you to iterate on—build on fun surprises, troubleshoot problem areas. [3/11]
Read 11 tweets
May 25, 2023
Timelapse of using #photoshop’s new generative fill feature to connect two images and build a scene around them using blank prompts. Was inspired by @MatthieuGB’s post doing something similar! Notice how I’m not adding any descriptions, but letting gen fill present options for… twitter.com/i/web/status/1…
Here’s the final image! 2/4 Image
And here are the original images made in #midjourneyv51 3/4 ImageImage
Read 4 tweets
Feb 28, 2023
Testing Multi-ControlNet on a scene with extended dialog, a range of facial expressions, and head turning. No EBSynth used. #AImusic from aiva.ai. Breakdown🧵1/15 #aicinema #controlnet #stablediffusion #aiia #aiartcommunity #aiva #machinelearning #deeplearning #ai
Overall flow: pre-process video > img seq > play with prompts > initial controlnet settings > control net batch render > upscale > clean up in post > key out background > deflicker > video post-production 2/15
The approach I used here was figuring out an initial workflow. There’s definitely a lot to play with and improve on. The orig vid is low-res and a little blurry so I started with a pre-process pass by bringing out the edges/brightness & upscaling. 3/15
Read 15 tweets
Feb 1, 2023
Really impressed by the acting you can generate with @elevenlabsio #texttospeech! These are #AIvoices generated from text—dozens of "takes" stitched together. Breakdown thread: 1/8 #syntheticvoices #ainarration #autonarrated #aicinema #aiartcommunity #aiia #ai #MachineLearning
I started by having #chatGPT write a few rough drafts of a scene involving a panicked character calling her friend for help from a spaceship. I was going for something that would involve heightened emotions but not be too serious. 2/8
Then I wrote a short script using some of those ideas plus my own and put the whole thing into @elevenlabsio. I generated a few takes using low Stability (1-2%) and high Clarity (90-99%). Each take usually had parts I liked, or at least gave me ideas for direction. 3/8
Read 8 tweets
Jan 30, 2023
I wonder what the future of UX design (maybe apps in general) may be like if AI really allows us to customize our experience. Not to mention blend programs together through a 3rd party/custom UI if an AI can understand onscreen what's being displayed by the app's GUI. 1/5
Combined with no code platforms of the future and advanced templates, you could probably do weird stuff like Premiere Pro x Unreal Engine x a fighting game template x an anime you like and custom gameify your interface. 2/5
Or maybe you could just submit to a chat AI to combine several apps/aesthetics together and present different connections and gameification strategies based on knowledge of UI/UX design. 3/5
Read 5 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

:(